SUMMARY

• Experienced Data Engineer with 7+ Years of progressive experience on Analysis, Design, Development, and Implementation of end - to-end production data pipelines.
• Experience in software analysis, datasets, design, development, testing, implementation of Cloud, Big Data, Big Query, Spark, Scala, Hadoop and in maintaining data pipeline.
• Expertise in designing, creating, testing, and maintaining the complete data management from Data Ingestion, Data Curation, Data processing/Transformation and Data Provision with in-depth knowledge on Spark APIs like Spark-SQL, Streaming, by working on different file formats like parquet, JSON, and performance tuning of spark applications from various aspects.
• Expertise in writing end to end Data processing Jobs to analyze data using Spark, and Hive.
• Experience with Apache Spark ecosystem using Spark-Core, SQL, Data Frames, RDD's and knowledge on Spark MLLib.
• Extensive Knowledge on developing Spark jobs by developing RDD’s, Dataframes (Resilient Distributed Datasets) using Scala, PySpark and Spark-Shell.
• Experience in developing enterprise level solutions using batch processing (using Spark and Hive) and streaming frameworks (Spark Streaming & Kafka).
• Proficient in data processing with both MapReduce & Apache Spark.
• Performed complexdata analysisand provided critical reports to support various departments.
• Extensive experience in understanding security requirements of Hadoop and data governance.
• Proficient in Oracle Database, SQL, PostgreSQL, Python programming and DBMS concepts.
• Strong programming skills in Python and Scala to build efficient and robust data pipelines.
• Experience in using and tuning relational databases (e.g., Microsoft SQL Server, Oracle, MySQL) and columnar databases (e.g., Amazon Redshift, Snowflake).
• Expertise in end-to-end Data Processing jobs to analyze data using MapReduce, Spark, and Hive.
• Strong Experience in working with Linux/Unix environments, writing Shell Scripts.
• Extensive experience with Apache Airflow, Bash/Python scripting for scheduling tasks and process automation.
• Worked with Jenkins for CI/CD pipelines as part of automated build and deployment of artifacts.
• Designed, configured, and deployed Amazon Web Services (AWS) for a multitude of applications utilizing the AWS stack (Including EC2, Glue, Lambda, SNS, S3, RDS, Cloud Watch, SQS, IAM), focusing on high-availability, fault tolerance, and auto-scaling.
• Experience working with Google Cloud Platform (GCP) and utilizing GCP services like GCS, Dataproc, Cloud Sql, Composer and BigQuery.

TECHNICAL SKILLS

Programming: Python, Java, Scala

Big Data: Spark, Hadoop, HDFS, MapReduce, Hive, Oozie, HBase, Impala, Hue

Big Data Platforms: Cloudera, Hortonworks, AWS EMR, Dataproc

Database Technologies: Teradata, HBase, Redshift, Athena, Snowflake.

Cloud Services: S3, EMR, Redshift, Lambda Functions, Step Functions, BigQuery, Composer, GCS

Scheduling tools: Apache Airflow, Oozie, AWS Step Functions

Version Control Tools: GitHub, BitBucket

Tools: JIRA, Rally, Confluence, Jenkins, Jupyter Lab, IntelliJ

PROFESSIONAL EXPERIENCE

Confidential, Reston, VA

Data Engineer

Responsibilities:

• Developed Spark Applications to implement various data cleansing/validation and processing activity of enterprise datasets ingested from traditional data warehouse systems.
• Worked both with batch and real time streaming data sources.
• Developed custom Kafka producers to write the streaming messages from external Rest applications to Kafka topics.
• Developed spark streaming applications to consume the streaming json messages from Kafka topics.
• Developed data transformations job using Spark Data frames to flatten JSON documents.
• Worked with the Spark for improving performance and optimization of the existing transformations.
• Used Spark Streaming APIs to perform transformations and actions on the fly for building common learner data model which gets the data -from Kafka in Near real time and persist it to HBase.
• Worked and learned a great deal from AWS Cloud services like EMR, S3, RDS, Redshift, Athena, Glue.
• Migrated an existing on-premises data pipelines to AWS.
• Worked on automating provisioning of AWS EMR clusters.
• Used Hive QL to analyze the partitioned and bucketed data, executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business specification logic.
• Experience in using Avro, Parquet, ORC file and JSON file formats, developed UDFs in Hive.
• Worked with Log4j framework for logging debug, info & error data.
• Used Jenkins for Continuous integration.
• Generated various kinds of reports using Tableau based on client specification.
• Used Jira for bug tracking and Git to check-in and checkout code changes.
• Responsible for generating actionable insights from complex data to drive real business results for various application teams and worked in Agile Methodology projects extensively.
• Worked with Scrum team in delivering agreed user stories on time for every Confidential .

Environment: AWS, S3, EMR, Spark, Kafka, Hive, Athena, Glue, Redshift, Teradata, Tableau

Confidential, Seattle, WA

Data Engineer

Responsibilities:

• Ingested gigabytes of click stream data from external servers such as FTP server and S3 buckets on daily basis using customized home-grown Input Adapters.
• Created Spark JDBC ingestion jobs to import/export data from RDBMS to S3 data store.
• Developed various spark applications using Scala to perform cleansing, transformation and enrichment of these click stream data.
• Involved in data cleansing, event enrichment, data aggregation, de-normalization and data preparation needed for machine learning and reporting.
• Troubleshooting Spark applications for improved error tolerance and reliability.
• Fine-tuning spark applications/jobs to improve the efficiency and overall processing time for the pipelines.
• Created Kafka producer API to send live-stream json data into various Kafka topics.
• Developed Spark-Streaming applications to consume the data from Kafka topics and to insert the processed streams to Snowflake.
• Utilized Spark in Memory capabilities, to handle large datasets.
• Used Broadcast variables in Spark, effective & efficient Joins, transformations and other capabilities for data processing.
• Experienced in working with EMR cluster and S3 in AWS cloud.
• Creating Hive tables, loading and analyzing data using hive scripts. Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Involved in continuous Integration of application using Jenkins.
• Interacted with the infrastructure, network, database, application and BA teams to ensure data quality and availability
• Followed Agile Methodologies while working on the project.

Environment: AWS EMR, Spark, Snowflake, Hive, HDFS, Sqoop, Kafka, Scala, Java, S3, Cloudwatch, Aws simple workflow

Confidential, Minneapolis, MN

Data Engineer

Responsibilities:

• Involved in writing Spark applications using Scala to perform various data cleansing, validation, transformation, and summarization activities according to the requirement.
• Involved in creating data lake in Google Cloud Platform (GCP) for allowing business teams to perform data analysis in Big Query.
• Responsible for developing data pipelines involving ingesting raw json files, transactional and user profile information from on prem data warehouses and processing them using spark and finally loading the processed data to Big Query.
• Automated launch of Dataproc clusters and autoscaling the clusters and submitted spark jobs to Dataproc clusters.
• Utilized cloud Sql as external hive Metastore for Dataproc clusters so that metadata is persisted across multiple Dataproc clusters.
• Utilized Spark-Big query Connector for writing the processed data from spark to big query directly.
• Utilized Google Cloud Storage as data lake and ensured all the processed data is written to GCS directly from spark and hive jobs.
• Written Kafka producers for streaming real time json messages to Kafka topics and processed them using spark streaming and performed streaming inserts to big query.
• Worked extensively on performance tuning of Spark application to improve job execution times and troubleshooting failures.
• Worked on different file formats like Text, Avro, Parquet, JSON, and Flat files using Spark.
• Developed daily process to do incremental import of data from Teradata into Hive tables using Sqoop.
• Work with cross functional consulting teams within the data science and analytics team to design, develop and execute solutions to derive business insights and solve client’s operational and strategic problems.
• Extensively worked with Partitions, Dynamic Partitioning, bucketing tables in Hive, designed both Managed and External tables, also worked on optimization of Hive queries.

Environment: GCP, Google Cloud Storage, Dataproc, Big query, Spark, Scala, Java, Kafka, Spark Streaming, Hive, Teradata, Bigtable, Cloud Sql

Confidential

Java Developer

Responsibilities:

• Involved in Requirement Analysis, Design, Development and Testing of the risk workflow system.
• Involved in the implementation of design using vital phases of the Software development life cycle (SDLC) that includes Development, Testing, Implementation and Maintenance Support.
• Applied OOAD principle for the analysis and design of the system.
• Implemented XML Schema as part of XQuery query language
• Applied J2EE design patterns like Singleton, Business Delegate, Service Locator, Data Transfer Object (DTO), Data Access Objects (DAO) and Adapter during the development of components.
• Used RAD for the Development, Testing and Debugging of the application.
• Used WebSphere Application Server to deploy the build.
• Developed front-end screens using Struts, JSP, HTML, AJAX, JQuery, Java script, JSON and CSS.
• Used J2EE for the development of business layer services.
• Developed Struts Action Forms, Action classes and performed action mapping using Struts.
• Performed data validation in Struts Form beans and Action Classes.
• Developed POJO based programming model using spring framework.
• Used IOC (Inversion of Control) Pattern and Dependency Injection of Spring framework for wiring and managing business objects.
• Used Web Services to connect to mainframe for the validation of the data.
• SOAP has been used as a protocol to send request and response in the form of XML messages.
• JDBC framework has been used to connect the application with the Database.
• Used Eclipse for the Development, Testing and Debugging of the application.
• Log4j framework has been used for logging debug, info & error data.
• Used Hibernate framework for Entity Relational Mapping.
• Used Oracle 10g database for data persistence and SQL Developer was used as a database client.
• Extensively worked on Windows and UNIX operating systems.
• Used SecureCRT to transfer file from local system to UNIX system.
• Performed Test Driven Development (TDD) using JUnit.
• Used Ant script for build automation.
• PVCS version control system has been used to check-in and checkout the developed artifacts. The version control system has been integrated with Eclipse IDE.
• Used Rational Clear quest for defect logging and issue tracking.

Environment: Windows XP, Unix, RAD7.0, Core Java, J2EE, Struts, Spring, Hibernate, Web Services, Design Patterns, WebSphere, Ant, (Servlet, JSP), HTML, AJAX, JavaScript, CSS, jQuery, JSON, SOAP, WSDL, XML, Eclipse, Agile, Jira, Oracle 10g, Win SCP, Log4J, JUnit.