SUMMARY

• Senior Data Engineer having 12+ years of experience with a demonstrated history of working in various industries.
• Skilled in Databases, Azure, Shell Scripting, Hadoop, and Data Mining.
• Strong Information Technology professional with a master’s degree focused in Computer Science from Confidential at Charlotte.

TECHNICAL SKILLS

Hadoop Ecosystem: MapReduce, Hive, Pig, Spark, Sqoop, Oozie, Kafka, Azure

Databases: Oracle 9i/10g/11g, SQL Server 2005/2008.

Languages: Java, C.

No SQL databases: Cassandra.

Reporting Tools: Cognos 8.X, Cognos 10.x Report Studio, Query Studio, Metric Studio, Analysis Studio, Event Studio, Framework manager

ETL Tools: IBM DataStage 7.5/8.1/8.5 Designer client, Director Client, administrator Client.

Other Tools: SQL Management Studio, Eclipse, Toad 9/10, Quality Center, Serena Version Control Tool, MATLAB.

Cloud: Azure, AWS .

PROFESSIONAL EXPERIENCE

Confidential, Boston, MA

Sr Data Engineer

Responsibilities:

• Created all the pipelines required to build the on - prem datalake.
• Worked with data scientists to build the pipelines required for Customer Churn Model.
• Reduced the latency of spark jobs by tweaking the spark configurations and following other performance and Optimization techniques.
• Worked extensively with Spark dataframes.
• Involved in converting various Teradata scripts to Spark code for the SVC project.
• Worked with Spark SQL in transforming various business requirements.
• Experience in Azure Cloud, Azure HDInsight Big Data Technologies (ApacheHadoop and ApacheSpark) and Data bricks.
• Design and implement end-to-end data solutions (storage, integration, processing, visualization) in Azure and Databricks.
• Experience managing Azure Data Lakes (ADLS) and Data Lake Analytics and an understanding of how to integrate with other Azure Services.
• Migrate data from traditional database systems to Azure bigdata platforms.
• Worked on Gen1 to Gen2 migration in Azure.
• Developed the ETL workflows, scheduling the jobs and enabling triggers using Azure Data Factory.
• Created numerous pipelines in Azure using ADFv2 to get the data from different source databases by using different Azure Activities like Move,Transform, Copy, filter, for each, Databricks etc.
• Built all the end-to-end data pipelines for the Data science project.
• Worked in Teradata to Snowflake migration project.
• Translated the Teradata bteq scripts to the Snow sql.

Environment: Hadoop, HDFS, Hive, Sqoop, Unix, Azure, Tidal, Spark,Snowflake.

Confidential, Chicago IL

Data Engineer

Responsibilities:

• Designed and implemented ETL pipeline involving Apache Kafka and Apache Spark - collect the data from various sources; transform it and load the data.
• Reduced the latency of spark jobs by tweaking the spark configurations and following other performance and Optimization techniques.
• Worked with various RDD’s, pair RDD’s and Data Frames.
• Worked with Sqoop to transfer data between RDBMS and HDFS.
• Worked with different file formats like Avro, Sequence files and Parquet.
• Worked extensively on EMR and S3 in AWS.
• Migrated and deployed the applications using AWS cloud.
• Creating/Managing buckets on S3 and store the database logs backup.
• Used compression codecs to compress data in HDFS.
• Create / Modify shell scripts for scheduling various data cleansing scripts and ETL loading process.
• Developed DQ framework to validate data between RDBMS and Hadoop using Spark and Python.
• Implemented a python model to calculate the deltaV for the APD data.
• Worked with Amazon S3 and Redshift.

Environment: Hadoop, HDFS, Hive, Sqoop, Unix, Oozie,python, AWSRedshift.

Confidential, Charlotte NC

Hadoop Engineer

Responsibilities:

• Worked with Mainframe fixed width and delimited files.
• Designed and Implemented Sqoop incremental imports, delta imports on tables without primary keys and dates from Teradata and appends directly into Hive Warehouse.
• Worked on data migration from existing data stores to Hadoop File System.
• Worked with Avro and Parquet file formats.
• Used Compression Techniques (snappy) with file formats to leverage the storage in HDFS.
• Worked on data migration from existing data stores to Hadoop File System.
• Provided ad-hoc queries and data metrics to the Business Users using Hive.
• Developed customized Hive UDF’s.
• Worked with Partiti/oning, bucketing and other optimizations in hive.
• Used Impala for querying the HDFS data.
• Developed and implemented core API services using Scala and Spark.
• Optimized the performance while dealing with large datasets using Partitions, spark in memory capabilities, effective and efficient joins,transformations and other heavy lifting during the ingestion process itself.
• Used Spark API to perform analytics on data in Hive.
• Transforming the existing hive queries and pig scripts into multiple Spark jobs.
• Worked with Pig to flatten the multiple datasets.
• Written shell scripts for DI and other error handling and mailing systems (DUSTc).
• Designed & developed ETL workflow using Oozie for business requirements that includes automating the extraction of data from Oracle/ Netezza /Teradata database into HDFS using Sqoop scripts and loading into the hive tables using hive scripts.
• Conducting root cause analysis and resolve production problems and data issues.
• Worked with Autosys scheduler to automate the jobs.

Environment: CDH, Hadoop, HDFS, Map Reduce, Hive, Pig, Sqoop, Unix, Oozie, Autosys.

Confidential, CA

Hadoop Developer

Responsibilities:

• Designed and implemented ETL pipeline involving Apache Kafka and Apache Spark - collect the data from various sources; transform it and load the data.
• Worked with various RDD’s, pair RDD’s and Data Frames.
• Worked with Sqoop to transfer data between RDBMS and HDFS.
• Worked with different file formats like Avro, Sequence files and Parquet.
• Worked extensively on EMR and S3 in AWS.
• Migrated and deployed the applications using AWS cloud.
• Creating/Managing buckets on S3 and store the database logs backup.
• Design highly available, cost effective and fault tolerant systems using multiple EC2 instances, Auto Scaling, Elastic Load Balancer.
• Used compression codecs to compress data in HDFS.
• Developed map reduce programs to structure the data.
• Developed custom MapReduce programs and custom User Defined Functions (UDFs) in Hive to transform the large volumes of data with respect to business requirement.
• Created internal and external Hive tables and defined static and dynamic partitions as per requirement for optimized performance.
• Running Hive queries for data analytics purpose.
• Implemented Partitioning - Static and dynamic Partitionsand Bucketing in Hive.
• Wrote Pig Latin scripts for advanced analytics on data for recommendations.
• Configured daily workflow for extraction, processing and analysis of data using Oozie.
• Create / Modify shell scripts for scheduling various data cleansing scripts and ETL loading process.

Environment: AWS, MapReduce, Pig 0.11, Hive 0.10, Sqoop, Unix, Spark

Confidential, Charlotte

Senior Developer

Responsibilities:

• Develop and validate the Reports and Dashboards.
• Performed Report and Model Migration.
• Scheduling Reports and grant permissions to the reports only to the appropriate users.
• Deployed Reports from Dev to UAT and UAT to Production.
• Support for User Acceptance testing and fixing defects.

Environment: Cognos 8.4.1, Service Center, TOAD, Unix.

Confidential

ETL Developer

Responsibilities:

• Translating the business requirements into Datastage jobs.
• Performed thorough testing in all the environments and move to Production environment.
• Tuned the long running jobs and reduced the running time of the Datastage jobs to great extent.
• Worked with XML transformer and other complex transformations.
• Improving the query performance using the parallel hints wherever possible.
• Writing shell scripts to trigger the Data stage jobs.
• Scheduling the Datastage jobs using the ESP Scheduler.
• Logging the defects in Quality Center.

Environment: Datastage 8.5, 7.5, Oracle 10G, Quality Center 9, TOAD 9/10, Mainframe, UNIX, Windows XP, Serena Version Control tool.

Confidential

Reports Developer

Responsibilities:

• Data loading using the Microsoft Management Studio.
• Created several crosstab and list reports.
• Developed Reports which mainly deals with customer Churn analytics
• Developed Dashboards that would be helpful for the executives.
• Validating both reports and dashboard and involved in creating test records and performing Unit and Performance testing.
• Performance improvements of the reports by optimizing report query, indexes in DB2.

Environment: IBM Cognos 10, SQL Server 2008.

Confidential

Reports Developer

Responsibilities:

• Data loading using the Microsoft Management Studio.
• Analyze the data and come up with new and innovative reports.
• Developed Reports which mainly deals with Sales and Profits.
• Developed Dashboards that would be helpful for the executives.
• Validating both reports and dashboard.

Environment: IBM Cognos 10, SQL Server 2008.