SUMMARY

• Extensive 8 +years of IT experience as anHadoopDeveloper and Data Engineer like Hive, Sqoop, Airflow, Impala, PySpark, AWS lambda, NIFI,Python, Snowflake, Atana and AWS services.
• Manage, Design and develop big data applications using PySpark, Snowflake, Airflow, Hive, S3 and other AWS services
• Developed PySpark scripts to migrate data ingestion from Teradata and landing into S3
• Performance tuning for already existing spark prod jobs
• Convert long running hive queries to PySpark application for more performance
• Developed a POC using AWS Lambda and secret Manager.
• Created Snowflake Consumption views for BI Reporting.
• An excellent team player and self - starter wif good communication skills and proven abilities to finish tasks before target deadlines.
• Hands on Experience using CI/CD Jenkins pipelines
• Experience in writing Hive UDF based on requirements
• Experience in importing and exporting data from different data sources using Sqoop.
• Expertise on job/workflow scheduling using Airflow,Oozie and Autosys.
• Experience in installation, configuration, supporting and managing Hadoop clusters.
• Expert in working wif Hive tables, data distribution by implementing partitioning and bucketing, writing and optimizing the HiveQL queries.
• Strong experience in writing Map Reduce programs for Data Analysis. Hands on experience in writing custom partitioners for Map Reduce
• Clear understanding on Hadoop MRV1 architectural components viz. HDFS, Job Tracker, Task Tracker, Name Node, Data Node, Secondary Name Node and YARN architectural components viz. Resource Manager, Node Manager and Application Master.
• Implemented in setting up standards and processes for Hadoop based application design and implementation.
• Worked on environment setup activities for isolated business testing.
• Worked on POCs and Bench Marking the performance of multiple Hadoop eco system technologies (Hive/Impala/Avro/Parquet) in contrast wif our business use cases to help leadership determine the right tools to use.
• Experience in managing and reviewing Hadoop log files.
• Worked in Waterfall and Agile methodologies (Scrum).
• Experience in Software Development Life Cycle (Requirements Analysis, Design, Development, Testing, Deployment and Support).

TECHNICAL SKILLS

Bigdata Technologies: Hadoop,AWS,S3,Hive, Airflow, Snowflake,atana, PySpark,DynamoDB, Lambda, EMR

Programming Languages: Python

Databases: RDBMS, MySQL, Oracle, Teradata, DB2

IDE and Tools: pycharm

Operating System: Windows, Linux/Unix. Mac

PROFESSIONAL EXPERIENCE

Confidential, Beaverton OR

Sr Data Engineer

Responsibilities:

• Manage, Design and develop big data applications using PySpark, Snowflake, Airflow, Hive, S3 and other AWS services
• Worked on Analysis of the Sources and Design of the pipeline for Validator Guide.
• Develop PySpark business rules integrate wif AWS services.
• Develop automating scripts to move data from Teradata,DB2 to AWS s3 bucket.
• Develop Airflow scripts to integrate Pyspark code and schedule jobs.
• Create Snowflake Tables, views for users to access history data.
• Grant snowflake roles and groups to team.
• Develop common framework using spark.
• Convert long running hive queries to pyspark application for more performance.
• Performance tuning for already existing spark prod jobs.
• Hands on Experience using CI/CD Jenkins pipelines.

Environment: PySpark, Snowflake, Airflow, Hive, S3, Teradata,DB2, Spark SQL, Git, Bitbucket, EMR,Python and Agile Methodologies

Confidential, Beaverton OR

Sr Data Engineer

Responsibilities:

• Developed ETL for Responsive Business dashboard.
• Developing Airflow DAG for building and scheduling jobs to run on daily, weekly, monthly basis.
• Developed scripts to migrate data ingestion from Teradata and landing into S3
• Developed scripts to query snowflake.
• Convert long running hive queries to pyspark application for more performance.
• Worked on connecting snowflake to SSAS cubes POC.
• Developed and Maintained the SSAS cubes for planners.

Environment: PySpark, Snowflake, Airflow, Hive, S3, Teradata,DB2, Spark SQL, Git, Bitbucket, EMR,Python and Agile Methodologies

Confidential, WI

Hadoop Developer

Responsibilities:

• Developed data processing frame work through which client can run/call any queries/ETL based on their requirement.
• Developed impala views on tables and handled multi-tenancy.
• Developing Oozie workflows and coordinators for building pipelines and scheduling pipelines to run on daily basis.
• Created Impala views for users to access history data.
• Scheduling allHadoop/Hive/Sqoop/Spark jobs using Oozie.
• Used Autosys scheduler to automate the jobs.
• Experience in productionizing various applications.
• Designed the plan for migrating the application to a centralized multi tenant cluster
• Developed automation scripts to migrate data from one cluster to another cluster to migrate data.
• Developed scripts to migrate metadata from one cluster to another cluster wif changes
• Experience in developing optimized spark logic.
• Experience in performance tuning and faster processing using Spark.
• Developed Python scripts to generate the dynamic hive queries based on the reference data.
• Developed a framework to scrub the history data based on business requirements.
• Big data analysis using Pig, Hive and User defined functions (UDF).
• Performed joins, group by and other operations in MapReduce using Java or PIG Latin.
• Worked on cluster migration activities like data copy & code deployment from old to new cluster.
• Used setter and getter methods of Java in the reducer to set/get values to and from the java jar
• Processed the output from PIG, Hive and formatted it before sending to the Hadoop output file.

Environment: Spark SQL, HDFS, Hive, Pig, Sqoop, Scala, Shell scripting, Linux, MySQL Eclipse, Git, Oozie, MySQL and Agile Methodologies.

Confidential, Charlotte, NC

Hadoop Developer

Responsibilities:

• Enhanced the retail simplification project to include wholesale and enterprise data.
• Designed and developed data lake for retail, wholesale and enterprise related data using Big data technologies.
• Designed the plan for migrating the application to a centralized multi tenant cluster.
• Developed automation scripts to migrate data from one cluster to another cluster to migrate data.
• Developed scripts to migrate metadata from one cluster to another cluster wif changes. experience in setting the standards and developing code according to them.
• Modified existing code and developed code as per the new standards .
• Designed and developed data pipelines to process data from various data sources including big data technologies.
• Designed and developed common utilities using Spark.
• Developed a utility to compare two different datasets using Scala spark.
• Developed a common framework to perform data integrity checks across various platforms.
• Experience in developing spark logic using scala and java for various use cases.
• Experience in developing optimized spark logic.
• Experience in performance tuning and faster processing using Spark.
• Created Impala views for users to access history data.
• Scheduling allHadoop/Hive/Sqoop/Spark jobs using Oozie.
• Used Autosys scheduler to automate the jobs.
• Experience in productionizing various applications.

Environment: Eclipse, Oracle 10g, Sub Version, Hadoop, Hive, Linux, MapReduce, HDFS, Hive, Java (JDK 1.6),HadoopDistribution of Cloudera, MapReduce.

Confidential, Blue Bell, PA

Hadoop Developer

Responsibilities:

• Migrated the existing data to EC2 from RDBMS using Sqoop for processing the data.
• Developed ETL pipeline to efficiently process data from MySql to CDH on EC2.
• Developed logic to process the data from mysql in EMR cluster using spark.
• Designed solution to move data between S3 and EC2 for the application.
• Designed workflows to process data from EMR cluster to S3 and S3 to CDH.
• Used Hive data warehouse tool to analyze the data in HDFS and developed Hive queries.
• Implemented Optimized Hive queries.
• Worked on implementing cost and time efficient solutions in AWS for the end clients.
• Developed solutions to reduce SLA from days to hours to provide data efficiently to the users.
• Writing Hive/Pig scripts to convert existing Value added processing running on Mainframe/Teradata environment to Hadoop.
• Designed Standard Oozie templates (Sub-Workflows) for data processing which is common across all LOBs.
• Sqooping History data from Teradata and Netezza and created Parquet tables on history data.
• Implemented Partitioning, Dynamic Partitions, Buckets in Hive.
• Developed custom UDF’s to handle exponential values and to convert different date formats to desired format.

Environment: Hadoop, HDFS, Spark, MapReduce, Pig, Hive, Sqoop, Oozie, Python, Java, SQL Scripting and Linux Shell Scripting, Cloudera, Cloudera Manager, EC2, EMR, S3, AWS

Confidential

Java Developer

Responsibilities:

• Implemented the application using Agile methodology. Involved in daily scrum and sprint planning meetings.
• Actively involved in analysis, detail design, development, bug fixing and enhancement.
• Driving the technical design of the application by collecting requirements from the Functional Unit in the design phase of SDLC.
• Developed Micro services using RESTful services to provide all the CRUD capabilities.
• Creating requirement documents and design the requirement using UML diagrams, Class diagrams, Use Case diagrams for new enhancements.
• Developed the Application Module using several design patterns like Singleton, DAO, DTO, and MVC.
• Involved in writing JSPs, Java Script and Servlets to generate dynamic web pages and web content.
• Used JBoss application server deployment of applications.
• Developed communication among SOA services.
• Involved in creation of both service and client code for JAX-WS and used SOAPUI to generate proxy code from the WSDL to consume the remote service.
• Designed the user interface of the application using HTML5, CSS3, JavaScript, Angular JS, JQuery and AJAX.
• Designed Node.js application components through Express.
• Implemented AJAX functionality to speed up web application.

Environment: Java, J2EE, Java Swing, HTML, Java Script, Angular JS, Node.JS, JDBC, JSP, Servlet, UML, Hibernate, XML, JBoss, SDLC methodologies, Log4j, GitHub, Restful, JAX-RS, JAX-WS, Eclipse IDE.