Sr Hadoop Developer Resume
Ann Harbor, MI
PROFESSIONAL SUMMARY:
- 8+ years of experience with Big Data tools and technologies including working in a production environment of a Hadoop project.
- Well versed in Installation, Configuration, Supporting and Managing of Big Data and Underlying infrastructure of Hadoop Cluster.
- Excellent understanding / knowledge of Hadoop architecture and various components such as HDFS, JobTracker, TaskTracker, NameNode, DataNode and MapReduce programming paradigm.
- Strong experience with big data processing using Hadoop technologies MapReduce, Apache Spark, Hive, Pig and Yarn.
- Developed Spark/Scala code to ingest data leveraging memory and optimizing performance
- Experience in analyzing data using Spark SQL, HIVEQL, PIG Latin, Spark/Scala and custom MapReduce programs in Java.
- Worked with Spark engine to process large scale data and experience to create Spark RDD.
- Exploring with the Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark - SQL, Data Frame, Pair RDD's, Spark YARN.
- Experience in designing and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.
- Strong Knowledge on Architecture of Distributed systems and Parallel processing, In-depth understanding of MapReduce programing paradigm and Spark execution framework.
- Parallelized and translated our logistic regression model into Spark ML and successfully deployed an ETL pipeline into production to predict new responses daily using PySpark.
- Exposure on usage of Apache Kafka develop data pipeline of logs as a stream of messages using producers and consumers.
- Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time
- Strong experience working with real time streaming applications and batch style large scale distributed computing applications using tools like Kafka.
- Implemented Kafka Storm topologies, which are capable of handling and channelizing high stream of data and integrating the storm topologies to filter and process that data across multiple clusters for complex event processing.
- Implemented Data ingestion framework using Kafka and Fume
- Good experience with NoSQL databases HBase, MongoDB and good understanding of Cassandra.
- Hands on experience in using Apache SOLR/Lucene.
- Good knowledge in streaming applications using Apache Kafka.
- Responsible for importing log files from various sources into HDFS using Kafka.
- Hands on experience with data acquisition into Hadoop cluster using Sqoop and Flume.
- Proficiency in Hadoop data formats like AVRO & Parquet.
- Practical knowledge of functionalities of MapR/Hadoop daemons, the interaction between them, resource utilization and dynamic tuning to make the cluster available.
- Experience in capacity planning, architecting, and designing MapR cluster from scratch.
- Experience in managing MapR/Hadoop cluster using MapR Control System (MCS). Imported Avro files using Apache Kafka and did analytics using Spark in Scala did troubleshooting using Scala problems to produce illustrative reports and dashboards along for ad-hoc analysis.
- Hands on experience on working with Amazon EMR framework transferring data to EC2 server.
- Used Cassandra to store the analyzed and processed data for scalability.
- Experience in working on CQL (Cassandra Query Language), for retrieving the data present in Cassandra cluster by running queries in CQL.
- Used Mongo DB as backend analytical database for VIN numbers hierarchy
- Written multiple MapReduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.
- Developed custom MapReduce(YARN) for cleaning, accessing and validating the data.
- Efficient in Dimensional Data Modeling, Process Modeling for Data Warehouse.
- Used Apache Flume to collect the logs and error messages across the cluster and loading log data into HDFS.
- Good understanding of cloud configuration in Amazon web services (AWS) and real time AWS command line interface, and AWS data pipeline, EMR, S3.
- Worked in different strategies as in Agile, Scrum and Waterfall.
- Solid knowledge in Maven for project build/test/deployment.
- Used JUnit for unit, integration testing and Log4j utility for error logging.
- Good communication skills, work ethics and the ability to work in a team efficiently with good leadership skills
TECHNICAL SKILLS:
Big Data Technologies: Apache Hadoop, Apache Spark, Apache Kafka, Apache Sqoop, Apache Hive, MapReduce, Apache Pig, Apache Flume and Oozie.
Languages: C, C++, J2SE 1.6, 1.7, 1.8, JEE, Scala, JUnit & Shell Scripting
Frameworks: Spring, Spring Boot.
Web Services: IBM Data Studio, Toad for DB2, SQL Advantage, SQL Developer, IntelliJ, Eclipse, JBoss developer, Netbeans, SOAP UI, MS PowerPoint, Excel Dashboards
Data Formats: JSON, AVRO, ORC, CSV, XML, Parquet and Proto Buffer.
Web Services: RESTFUL.
Deployments: Pivotal Cloudy Foundry and Chef.
Integration Tools: Jenkins and Team City.
NoSQL Database: HBase, Cassandra, MapR: DB and MongoDB.
Database: JDBC, MySQL, SQL Server, Oracle 10g and Apache Hive.
Development Tools: Eclipse Juno and IntelliJ IDEA.
Packages & Tools: MS Office Suite (Word, Excel, PowerPoint, SharePoint, Outlook, Project).
Operating Systems: Mac OS, Windows 10, Cent OS 7 and Ubuntu.
BI Tools: SAP Business Objects 4.1, Information Design Tool and Web Intelligence.
UML Modeling Tools: Visual Paradigm for UML 10.1 and Visio.
EXPERIENCE:
Confidential, Ann Harbor, MI
Sr Hadoop Developer
Responsibilities:
- Involved in understanding business requirements and prepare a design documents, coding, testing and go on live installation environment.
- Extracted data using Sqoop Import query from multiple databases and ingest into Hive tables.
- Involved in developing the Spark Framework to provide structure to data on the fly and process the data using Spark core API's, Data Frame, Spark-SQL and Scala Evaluated and improved application performance with Spark.
- Worked with analyst to determine and understand business requirements.
- Implementing new dimensions into Spark application upon on the business requirements.
- Worked on streaming pipeline that uses Spark to read data from Kafka, transform it and write it to HDFS.
- Involved in working with Spark on top of YARN for interactive and batch analysis.
- Implemented POC on writing programs in Scala using Spark and worked on migrating MapReduce programs into Spark using Scala.
- Implemented Spark SQL to update queries based on the business requirements.
- Convert MR algorithms into Spark transformations and actions by creating RDDs, pair RDDs.
- Used cloud computing on the multi-node cluster and deployed Hadoop application on cloud S3 and used Cloudera to run a Map-reduce.
- Hands on expertise in running the Spark & Spark SQL on Cloudera.
- Initiated spark context and processed live streaming information with the help of RDD using Kafka and Kafka brokers.
- Developed data pipelines using Kafka, Spark and Hive to ingest, transform and analyze logs.
- Auto Populate MongoDB tables with data coming from Kafka sink.
- Using Spark SQL allowing to the data from different formats it's currently in (like JSON, Parquet, a Database), transform it, and expose it for ad-hoc/interactive querying.
- Used Spark API over Cloudera Hadoop YARN to perform analytics on data in HDFS.
- Proficiency in Spark for loading data from the local file system, HDFS, Relational databases and using Spark SQL, import data into RDD and Ingesting data from a range of sources using Spark Streaming with JSON output and loading them to HDFS, text file.
- Worked with Amazon Web Services (AWS) using EC2 for hosting and Cloudera for data processing with S3 as storage mechanism.
- Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD's.
- Responsible for continuous monitoring and managing Cloudera cluster through AWS console.
- Integrated various business validation & prep flag rules from campaign managers and business analyst on the data.
- Experienced in Analyzing Cassandra database and compare it with other open-source NoSQL databases to find which one of them better suites the current requirements ingested data from RDBMS and performed data transformations, and then export the transformed data to Cassandra as per the business requirement.
- Involved in converting Cassandra/Hive/SQL queries into Spark transformations using SparkRDDs in Scala.
- Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Spark SQL and Scala extracted large datasets from Cassandra and Oracle servers into HDFS and vice versa using Sqoop.
- Developed a frame work to handle loading and transform large sets of unstructured data from UNIX system to HIVE tables.
- Deployed Spark application and Java web services in pivotal cloud foundry.
- Used AUTOMIC job scheduler for scheduling multiple applications in Production.
- Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scale.
- Handled Prod Deployments and provided production support for fixing the defects.
- Strong communication and analytical skills and a demonstrated ability to handle multiple tasks as well as work independently or in a team.
Environment: Apache Hadoop, Apache Spark, Apache Kafka, Apache Hive, Sqoop, Cassandra, AWS, Spring Boot, Pivotal Cloud Foundry, Cloudera, JUnit, Angular JS, IntelliJ, Maven, Team City, Automic and Git Hub.
Confidential, Eastpointe, MI
Big Data Engineer
Responsibilities:
- Worked with teams to run Hadoop, Spark on MapR converged platform by interacting with HDFS, HBase and JSON API.
- Created a connector to migrate data from HBase/MapR-DB binary to MapR-DB document database in Spark using Scala.
- Migrated data from SQL server to MapR using Hbase Java API.
- Worked on creating MapReduce programs to parse the data for claim report generation and running the jars in Hadoop. Co-ordinated with Java team in creating MapReduce programs.
- Created tables in MapR-DB using MapR control System maprcli and the HBase shell.
- Enabled StreamSets Data Collector to work with MapR-DB documents to read and write JSON into MapR-DB.
- Gathering business requirements from the Business Partners and Subject Matter Experts.
- Imported data from RDBMS to HDFS and Hive using Sqoop on regular basis.
- Created Hive tables and worked on them using Hive QL, which will automatically invoke and run MapReduce, jobs in the backend.
- Experience in managing and reviewing Hadoop Log files.
- Experienced in loading and transforming large sets of structured, semi-structured and unstructured data.
- Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
- Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.
- Worked with Spark to create structured data from the pool of unstructured data received.
- Configuring and performance tuning the Sqoop jobs for importing the input (raw) data from the data warehouse.
- Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as Map-Reduce, Pig, Hive, and Sqoop) as well as system specific jobs (such as Java programs and shell scripts). Deployment of Hortonworks stack and Spark.
- Implemented the Oozie Workflow Scheduler and coordination services to automate the workflow and schedule MapReduce jobs that extract the data to perform data mining.
- Used Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.
- Involved in scheduling and coordinating Oozie workflow manager jobs to run multiple Hive, MapReduce and pig jobs in batch processing mode.
- Experience on deployment of Apache Tez on top of YARN.
- Implemented Batch jobs using MR2, PIG, Hive, and Tez.
- Experience in using Sqoop to migrate data to and from HDFS and MySQL.
- Written Spark Application to implement Slowly changing dimensions (SCD Type I).
- Written PIG script to load processed data from HDFS into MongoDB.
- Involved in creating Hive tables loading with data and writing hive queries which will run internally in map reduce way.
- Used spark to parse XML files and extract values from tags and load it into multiple hive tables using map classes, developed code to pre-process large sets of various types of file formats such as Text, Avro, SequenceFile, XML, JSON, and Parquet.
- Developed Hive queries to process the data and generate the data cubes for visualizing.
- Used MongoDB to store processed products and commodities data, which can be further down streamed into web application (Green Box/ Zoltar).
- Deployed Spark application and java web services in pivotal cloud foundry.
- Involved in daily SCRUM meetings to discuss the development/progress of Sprints and was active in making scrum meetings more productive.
Environment: Apache Hadoop, Apache Spark, Apache Pig, Oozie, MapR/ MapR-DB, TEZ, Spring Boot, Pivotal Cloud Foundry, JUnit, IntelliJ, Maven, Team City, and Git Hub.
Confidential, Salt Lake City, UT
Hadoop Developer
Responsibilities:
- Developed Crawlers java ETL framework to extract data from Cerner client’s database and Ingest into HDFS & HBase for Long Term Storage.
- Written data processing pipelines in Apache Crunch to standardize and normalize the data and store the normalized data in HBase.
- Create ETL Pipelines using Apache Crunch to read data from HBase and calculate the KPI metrics and store the data in HDFS, and Bulk load data into HBase.
- Experience in ETL pipeline from HDFS to Vertica.
- Working on migrating all the Batch ETL Crunch pipelines to in-memory Spark Pipelines for quicker run times of the jobs.
- Created Oozie workflows to manage the execution of the crunch jobs and vertical pipelines.
- Cluster tuning to improve performance.
- Create Integration tests to check the validity of the data being crawled.
- Implementing new drill to detail dimensions into the data pipeline upon on the business requirements.
- Worked with multiple teams in resolving production issues.
- Deployment automation via Chef.
- Handled Prod Deployments and provided production support for fixing the defects.
- Responsible for creating business layer and its underlying data foundation and connection layer using Information Design Tool.
- Analyze the reporting requirements with solution designers and come up with the technical specifications for the Business Objects reports.
- Created reports based on business requirements using SAP BO Web Intelligence and implemented sorting, filtering, ordering and labeling of reports.
Environment: Apache Hadoop, Apache Crunch, HBase, Apache Hive, Oozie, Chops, HDFS, Apache SOLR, Splunk, SAP Business Objects 4.1, Chef, JUnit, Ruby, Eclipse, Maven, Jenkins, and Git Hub.
Confidential, Pittsburgh, PA
Jr Hadoop Developer
Responsibilities:
- Extensively involved in Design phase and delivered Design documents. Experience in Hadoop eco system with HDFS, HIVE, PIG, SQOOP and SPARK with SCALA.
- Create Hive based reports to support the application metrics (These will be used by UI teams for reports).
- Extending HIVE and PIG core functionality by using custom UDF’s.
- Agile methodology including, test-driven and pair-programming concept.
- Worked with other teams to determine and understand business requirement.
- Created partitioned tables in Hive.
- Developed the Pig UDF'S to pre-process the data for analysis.
- Worked with business teams and created Hive queries for ad hoc access.
- Extensive experience in writing Pig scripts to transform raw data from several data sources into forming baseline data.
- Handled importing the data from various data sources and performed transformation using Java, Hive and Pig.
- Used Pig as ETL tool to do Transformations with joins and pre-aggregations before storing the data onto HDFS. Responsible to develop data pipelines from different sources.
- Strong communication and analytical skills and a demonstrated ability to handle multiple tasks as well as work independently or in a team.
Environment: Apache Hadoop, HBase, Apache Hive, Apache PIG, Chops, HDFS, Apache SOLR, Java, Splunk, SAP Business Objects 4.1, Chef, JUnit, Ruby, Eclipse, Maven, Jenkins, and Git Hub.
Confidential
Java Developer
Responsibilities:
- Extensively worked in acquiring the requirements from the business analysts and involved in all requirement clarification calls.
- Understanding the design documents.
- Developed JMS API using J2EE package.
- Involved in Detail level design and coding activities at offshore.
- Involved in Code review.
- Writing and testing the JUNIT test classes.
- Provide support to client applications in production and other environments.
- Working on tickets raised by the real-time users and continuous interaction with end users.
- Prepared the Technical Design Document, understanding document and test cases (UTCs and ITCs).
- Provided Technical & Functional support to the end users during UAT & Production.
- Continuous monitoring of application for 100% availability.
Environment: Java, J2EE, Spring MVC, MIMA ORM, CVS, AQT, WebSphere, Oracle 10g and HPSM Ticketing tool.
Confidential
Java Developer
Responsibilities:
- Developed front-end screens using JSP, HTML and CSS.
- Developed server-side code using Struts and Servlets.
- Developed core java classes for exceptions, utility classes, business delegate, and test cases.
- Developed SQL queries using MySQL and established connectivity.
- Worked with Eclipse using Maven plugin for Eclipse IDE.
- The application was developed in Eclipse IDE and was deployed on Tomcat server.
- Involved in scrum methodology.
- Supported for bug fixes and functionality change.
Environment: Java, Struts 1.1, Servlets, JSP, HTML, CSS, JavaScript, Eclipse 3.2, Tomcat, Maven 2.x, MySQL, Windows, Linux.
