Big Data Engineer Resume
Syosset, NY
SUMMARY
- Extensive IT experience of over 8+ years with multinational clients which includes 5 years of Big data related architecture experience developing Spark / Hadoop applications.
- Hands on experience with the Hadoop stack (MapReduce, Pig, Hive, Sqoop, HBase, Flume, Oozie,Corn,ESP).
- Proven Expertise in performing analytics on Big Data using Map Reduce, Hive and Pig.
- Implemented POC to migrate map reduce jobs into Spark RDD transformations using Scala.
- Developed Apache Spark jobs using Scala in test environment for faster data processing and used Spark SQL for querying.
- Strong experience working with both batch and real - time processing using Spark frameworks. Proficient knowledge on Apache Spark and programming Scala to analyses large datasets using Spark and Storm to process real time data.
- Design and develop reusable components and frameworks for ingestion, cleansing, and data quality using technologies such as Apache Spark, Python, Scala(Databricks, Delta lake,Intelij, Data Grip Click house, Snowflake, AWS Redshift,EMR, S3, Glue.
- Implement Data Lake in Azure Blob Storage, Azure Data Lake, Azure Analytics, Databricks Data load to Azure SQL Data warehouse using Polybase, Azure Data Factory.
- Implement Data Lake in Google Big Query, GCP, SQL Scripts to load data to Big Query, Composer (Confidential) for running the Talendand Query Scripts.
- Experienced in Spark Core, Spark RDD, Pair RDD, Spark Deployment Architectures.
- Build and run data pipelines using an orchestration framework such as Apache Airflow.
- Experienced with performing real time analytics on NoSQL data bases like HBase and good knowledge in working with Impala, Storm and Kafka.
- Worked with Oozie work flow engine to schedule time based jobs to perform multiple actions.
- Have experience on Spark Core, Spark Streaming, Hive Context, Spark Sql and MLlib for analyzing streaming data.
- Experienced in importing and exportingdatafrom RDBMSinto HDFS using Sqoop.
- Analyzed large amounts of data sets using Pig scripts and Hive scripts.
- Good working knowledge on processing Batch applications.
- Experienced in writing MapReduce programs and UDFs for both Hive and Pig in Java.
- Used Flume to channel data from different sources to HDFS.
- Experienced with Maven, Jenkins and GIT.
- Experience in importing and exporting data from RDBMS to HDFS, Hive tables and HBase by using Sqoop.
- Experience in importing streaming data into HDFS using flume sources, and flume sinks and transformingthe data using flume interceptors.
- Exposure on usage of Apache Kafka to develop data pipeline of logs as a stream of messages usingproducers and consumers.
- Experience in integrating Apache Kafka with Apache Storm and created Storm data pipelines for real timeprocessing.
- Experience in using NiFi processor groups, processors and concepts on process flow management.
- Experience in Apache NiFi which is a Hadoop technology and also Integrating Apache NIFI and Apache Kafka.
- Good experience in Hive partitioning, bucketing and perform different types of joins on Hive tables and implementing Hive SerDe like JSON and Avro.
- Experience in converting Hive/SQL queries into Spark transformations using Spark SQL, Python and Scala.
- Worked on Data Serialization formats for converting Complex objects into sequence bits by using AVRO, PARQUET, JSON, CSV formats.
- Experience using file transfer protocols, sftp, scp, MFT/CFI.
- Experience in Performance Tuning, Optimization and Customization.
- Experience with Unit Testing Map Reduce programs using MRUnit, JUnit.
- Experience in Active Development as well as onsite coordination activities in web based, client/server and distributed architecture using Java, J2EE which includes Web services, Spring, Struts, Hibernate and JSP/Servlets along with incorporating MVC architecture.
- Worked with leading clients, which includes understanding client Requirements, Estimations, Analysis of Functional specifications, Technical Design Specification, Review of Technical Design, Development, Testing and Implementation activities.
- During this period I have also acquired strong knowledge of Software Quality Processes and SDLC (Software Development Life Cycle).
- Experience in working with ETL tools like IDQ, informatica.
- Creating dashboards using tableau with different KPI for monitoring upstream application.
- Build and run data pipelines using an orchestration framework such as Apache Airflow
- Extensively worked on Java development tools, which includes Eclipse Galileo 3.5, Eclipse Helios 3.6, Eclipse Mars 4.5, WSAD 5.1.2.
- Ability to work in teams as well as an individual, quick learner and able to meet deadlines.
TECHNICAL SKILLS
Hadoop Ecosystems: Hadoop, MapReduce, YARN, Spark, Sqoop, Hive, Oozie, Pyspark, PIG, HDFS, Flume, Impala, Storm, Apace Kafka, Solar, Hbase, Azure.
Java Technologies: Java 5, Java 6, JAXP, AJAX, I18N, JFC Swing, Log4j, Java Help API.
J2EE Technologies: JSP 2.1 Servlets 2.3, JDBC 2.0, JNDI, XML, JAXP, Java Beans.
Frameworks: MVC, Struts, Hibernate.
Application Server: Apache Tomcat 5.x 6.0, Jboss 4.0,Airflow
SDLC Methodologies: Waterfall, Agile/Scrum.
MS Office Suite: MS Word, MS Excel, MS PowerPoint, MS Access, MS Project, MS Visio.
Database Management: SPSS, MS Excel, MS Access, Oracle 10g, DB2, Mongo DB.
Operating Systems: Windows 95/98/2000/XP/Vista/7/8/10, Linux, Macintosh.
Programming Languages: C, Java 6.0, Scala 2.11 SQL, PL/SQL, PIG Latin,Python,HiveQL, Unix shell scripting
Reporting Tools: Qlikview, Tableau.
PROFESSIONAL EXPERIENCE
Big Data Engineer
Confidential, Syosset, NY
Responsibilities:
- Develop and analyze large scale, high speed and low latency data solutions using Big Data, Apache Spark,Python, pyspark and Capable to adapt to new technologies and develop the data driven solutions using the new technologies.
- Experience architecting data warehouses and/or data lakes for large amounts of structured and unstructured data.
- Experience with data modeling, data warehousing, and building ETL pipelines.
- Experience working in data pipelines and batch processing using Apache Spark, Pysparkand Python.
- Exploring with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's.
- Load the data into Spark RDD and do in memory data Computation to generate the Output response.
- Design and develop reusable components and frameworks for ingestion, cleansing, and data quality using technologies such as Apache Spark, Python, Scala(Databricks, Delta lake,Intelij, Data Grip Click house,Snowflake,AWS Redshift, EMR, S3,Glue).
- Worked with structured and unstructureddata at scale and comfort with a variety of different stores (key-value, document, columnar, etc.) as well as traditional RDBMSes and data warehouses.
- Developed Spark scripts by using Scala shell commands as per the requirement.
- Implemented ELK (Elastic Search, Log stash) stack to collect and analyze the logs produced by the Spark Cluster.
- Extensively worked on the core and Spark SQL modules of Spark.
- Involved in converting Hive/SQL queries into Spark transformations using Spark SQL, Python and Scala.
- Build and run data pipelines using an orchestration framework such as Apache Airflow.
- Creating dashboards using tableau with different KPI for monitoring upstream application.
- Comfort with version control systems (e.g. Git, Git lab).
- Lead by example, role-modeling best practices for unit testing, CI/CD, performance testing, capacity planning, documentation, monitoring, alerting and incident response.
- Stored the output files for export onto HDFS and later these files are picked up by downstream systems.
- Writing unit test cases (Test Driven Development) is a mandatory and expected task once code development is complete.
- Writing automated jobs in Spark and Scala to validate the data migrated from existing Hadoop systems to the PostgreSQL DB.
Environment: Hadoop, Java, HDFS, MapReduce, Spark, Python, Scala, Pig, Hive, Impala, Sqoop, Flume, HBase, Oozie, Java, SQL scripting, Linux shell scripting, Databricks, Delta lake, Intelij,tableau, Data Grip Click house, Snowflake, AWS Redshift,EMR, S3, Glue, Eclipse and Cloudera.
Spark/ Hadoop Developer
Confidential, Franklin lakes, NJ
Responsibilities:
- Responsible for building scalable distributed data solutions using Hadoop.
- Loading JSON from upstream systems using Spark streaming and load them to elastic search.
- Written various key queries in elastic search for retrieval of data effectively.
- Used Spark-Streaming APIs to perform necessary transformations.
- Configured deployed and maintained multi-node Dev and Test Kafka Clusters.
- Worked on Apache spark writing Python applications to convert txt, xls files and parse.
- Involved in converting Hive/SQL queries into Spark transformations using Spark SQL, Python andpyspark.
- Spark Streaming collects this data from Kafka in near-real-time and performs necessary transformations and aggregation on the fly to build the common learner data model and persists the data in NoSQL store (Hbase).
- Hands-on experience to write Java based Kafka and Spark streaming programs
- Written trivial Spark and Kafka jobs with Java to read data using Spark-SQL.
- Developed Spark scripts by using Scala shell commands as per the requirement.
- Used Spark API over Hadoop YARN to perform analytics on data in Hive.
- Implemented ELK (Elastic Search, Log stash, Kibana) stack to collect and analyze the logs produced by the Spark Cluster.
- Used Reporting tools like Kibana to connect with Hive for generating daily reports of data.
- Involved in development of Storm topology for ingestion of data through XML payload and then load them to various distributed stores.
- Extensively worked on the core and Spark SQL modules of Spark.
- Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
- Developed REST services which processed several requests triggered from UI.
- Integrated the Hive warehouse with HBase.
- Stored the output files for export onto HDFS and later these files are picked up by downstream systems.
- Load the data into Spark RDD and do in memory data Computation to generate the Output response.
- Exploring with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
- Used RESTFUL web services in JSON format to develop server applications.
- Experience using file transfer protocols, sftp, scp, MFT/CFI.
- Experienced working with HDP2.4.
Environment: Hadoop, Java, HDFS, MapReduce, Spark, Pig, Hive, Impala, Sqoop, Flume, Kafka,HBase, Oozie, Java, SQL scripting, Linux shell scripting, Eclipse and Cloudera.
Hadoop Developer
Confidential, Atlanta, GA
Responsibilities:
- Involved in converting Hive/SQL queries into Spark transformations using Spark SQL, Python and Scala.
- Visualize the HDFS data to customer using BI tool with the help of Hive ODBC Driver.
- Implement Data Lake in Google Big Query, GCP, SQL Scripts to load data to Big Query, Composer (Confidential) for running the Talendand Query Scripts.
- Work on Python / Bash / SQL Scripts to Load data from Data Lake to Data warehouse.
- Familiarity with a NoSQL database such as MongoDb.
- Used Impala for data analysis.
- Developed Spark scripts by using Scala shell commands as per the requirement.
- Extensively worked on the core and Spark SQL modules of Spark.
- Used Spark API over Hadoop YARN to perform analytics on data in Hive
- Implemented Optimized join base by joining different data sets to get top claims based on state usingMap Reduce.
- Created HBasetables to store various data formats of data coming from different sources.
- Responsible for importing log files from various sources into HDFS using Flume.
- Worked on analyzingHadoop stack and different big data analytic tools including Pig, Hive, HBase database and Sqoop.
- Done Proof of Concept in Apache Nifi workflow in place of Oozie,Cronto automate the tasks of loading the data into HDFS and pre-processing with Pig.
- Designed Nifito pull data from various sources and push it in HDFS and Cassandra.
- Worked extensively with the No SQL databases like MongoDB and Cassandra.
- Integrating bulk data into Cassandra file system using MapReduce programs.
- Worked with Nififor managing the flow of data from source to HDFS
- Created customized BI tool for manager team that perform Query analytics using HiveQL.
- Used Hive and Pig to generate BI reports.
- Created Partitions, Buckets based on State to further process using Bucket based Hive joins.
- Moved Relational Data base data using Sqoop into Hive Dynamic partition tables using staging tables.
- Worked on custom Pig Loaders and storage classes to work with variety of data formats such as JSONand XML file formats.
- Used Oozie workflow engine to manage interdependent Hadoop jobs and to automate several types ofHadoop jobs such as Java map-reduce Hive, Pig, and Sqoop.
- Used Oozie Operational Services for batch processing and scheduling workflows dynamically.
- Experienced in Monitoring Cluster using Cloudera manager.
Environment: Hadoop, HDFS, HBase, MongoDb, MapReduce, Java, Hive, Pig, Sqoop, Flume, Spark, Oozie,Corn,ESP,Hue, SQL, ETL, Cloudera Manager, MySQL.
Hadoop Developer
Confidential, Camarillo, CA
Responsibilities:
- Worked with team members for upgrading, configuration and maintenance of various Hadoop infrastructures like Pig, Hive, and HBase.
- Written elastic search template for the index patterns.
- Implemented and extracted the data from Hive using Spark.
- Involved in converting Cassandra/Hive/SQL queries into Spark transformations using Spark RDDs, and Scala/Python (PySpark).
- Extracted files from NoSQL database (MongoDB) and processed them with Spark using mongo Spark connector.
- Extensive knowledge in NoSQL databases like HBase.
- Involved in creating Hive tables, and loading and analysing data using hive queries.
- Responsible for analysing and cleaning raw data by performing Hive/Impala queries and running Pig scripts on data.
- Written Hive queries on the analysed data for aggregation and reporting.
- Developed automated processes for flattening the upstream data from Cassandra which in JSON format. Used Hive UDFs to flatten the JSON Data.
- Involved in creating Hive tables and loading with data.
- Imported and exported data from different databases into HDFS and Hive using Sqoop.
- Worked on loading log data into HDFS through Flume.
- Used the Apache Nififor data ingestion from TCP traffic
- Used Sqoop for loading existing metadata in Oracle to HDFS.
- Inserted Overwriting the HIVE data with Hbase data daily to get fresh data every day.
- Developed Python scripts, UDF's using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop.
- Used BI Tool Tableau for the generating of dashboard reports and visualization of data.
- Developed graphs by using Tableau ETL tool.
- Implemented Map Reduce jobs using Java API and PIG Latin.
- Participated in the setup and deployment of Hadoop cluster.
- Involved in loading and transforming large sets of structured, semi structured and unstructured data from relational databases into HDFS using Sqoop imports.
- Developed Sqoop scripts to import export data from relational sources and handled incremental loading on the customer, transaction data by date. Used Apache Kafka for tracking data ingestion to Hadoop cluster.
- Hands on design and development of an application using Hive (UDF).
- Developed the Pig UDF'S to pre-process the data for analysis.
- Experience using file transfer protocols, sftp, scp, MFT/CFI.
- Worked in AWS environment for development and deployment of Custom Hadoop Applications.
- Install Hadoop, Map Reduce, HDFS, and AWS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
Environment: MapReduce, HDFS, Hive, Pig, Spark, Apache Kafka, Flume, Zookeeper, Yarn, Linux, Sqoop, Java, Scala, Tableau, Python, CDH4, CDH5, AWS,EMR,S3, Eclipse, Oracle, Git, Shell Scripting and Cassandra.
Hadoop developer
Confidential, Austin, TX
Responsibilities:
- Responsible for building scalable distributed data solutions using Hadoop.
- Worked on importing data from various sources and performed transformations using MapReduce, HIVE to load data into HDFS.
- Configured Sqoop jobs to import data from RDBMS into HDFS using Oozieand Corn workflows.
- Worked on setting up Pig, Hive and HBase on multiple nodes and developed using Pig, Hive, HBase and MapReduce.
- Worked on loading the data from MySQL to HBase where necessary using Sqoop
- Solved small file problem using Sequence files processing in Map Reduce.
- Installation and Configuration of other Open Source Software like Pig, Hive, HBASE, Flume and Sqoop
- Created HBase tables to store variable data formats coming from different portfolios.
- Performed real time analytics on HBase using Java API and Rest API.
- Implemented HBase Co-processors to notify Support team when inserting data into HBase Tables.
- Worked on compression mechanisms to optimize MapReduce Jobs.
- Analyzed the customer behavior by performing click stream analysis and to ingest the data used Flume.
- Experienced with working on Avro Data files using Avro Serialization system.
- Implemented business logic by writing UDF's in Java and used various UDF's from Piggybanks and other sources.
- Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
Environment: Horton works, Map Reduce, HBase, HDFS, Hive, Pig, Java, SQL, Cloudera Manager, Sqoop, Flume, Oozie, Corn,ESP,Java, Eclipse.
Hadoop/Java Developer
Confidential
Responsibilities:
- Worked with Cloudera’s CDH3 distribution.
- Performed analytics on structured data using Hive.
- Created and maintained technical documentation for executing Hive queries and Pig Scripts.
- Loading the data from the different Data sources like (Teradata and DB2) into HDFS using Sqoop and load into Hive Tables, which are partitioned.
- Developed the Apache Pig scripts to process the HDFS data.
- Developed the Sqoop scripts in order to make the interaction between Pig (ETL) and MySQL Database.
- Worked on Oozie for loading data into HDFS and for job work flow scheduling.
- Designed, developed and validated the front-end using HTML, CSS, AJAX, JavaScript and JSP, JSF Tag libraries for reusable web interface components including the open source DisplayTag and AjaxParts Tag libraries.
- Developed Business Logic and Interfacing components using Business Objects, XML, and JDBC.
- Used JAXB2 for marshalling XML to java objects and the inverse.
- Used Spring 2.5 Framework for Dependency injection and integrated with Hibernate and Struts frameworks.
- Participated in weekly design reviews and walkthroughs with project manager and development teams.
- Provide technical guidance to business analysts, gather the requirements and convert them into technical specifications/artifacts for developers to start.
Environment: Apache Hadoop (Cloudera), Hive, Pig, Oozie, Corn,Sqoop, Java/J2EE, Struts, Hibernate, Java Script, AJAX, JSON, JSTL, XML, HTML, Log4J, Shell Scripting, Tomcat, MySQL, LINUX, UNIX.
