SUMMARY

• Above 17 years of professional experience in Information Technology, wif extensive experience as Big data Engineer and Hadoop/Spark Developer, AWS, ETL, Data Quality, Reporting, Data warehouse and Data/system analysis.
• Expert at full life cycle implementation using Spark Ecosystem.
• Experienced wif design, development and implementation of data pipeline using Spark/Python Applications
• Good Knowledge and experience on Integration tools and deployment tool like Jenkins in setup of CI/CD pipeline.
• Experienced wif AWS cloud platform to perform Bigdata Analytics work using EMR cluster.
• Experienced wif development and implementation of various AWS services including Aethna, Lambda, EC2, Elastic Search, Dynamo DB, Arora, Redshift, cloudwatch, S3 and Glue.
• Developed Python Scripts to handle the complex data structures for analytics and for data ingestion into Enterprise cluster.
• Experienced in developing and deploying enterprise - based applications using major Hadoop ecosystem components like Job Tracker, Task Tracker, Name Node, Data Node, Map Reduce concepts and YARN architecture which includes Node manager, Resource manager, Hive, Pig, HBase, Flume, Sqoop, Spark Streaming, Spark SQL, Storm, Kafka, Oozie, Zookeeper and Cassandra.
• Experience in using different file formats like CSV, Sequence, AVRO, ORC, JSON and PARQUET files and different compression Techniques like LZO, Gzip, Bzip2 and Snappy. Write abstract loader and reading and handling metadata
• Experienced to implement ad-hoc queries using Hive Query Language, Partitioning, bucketing and Hive-Custom UDF's.
• UsedSpark StreamingAPIs to perform transformations and actions on the fly for building common learner data model which gets the data fromKafkain near real time and persist it toCassandra.
• Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Experience in transferring Streaming data, data from different data sources into HDFS, NoSQL databases using Apache Flume and Apache Kafka.
• Experience in creating complex data warehouse and application roadmaps and BI implementation specializing in Teradata platforms and design, development and maintenance of ETL code.
• Analyzed large amounts of data sets and migrate ETL operations using RDD, Dataframes, Datasets and Spark SQL.
• Experience writing Shell scripts in Linux OS and integrating them wif other solutions.
• Hands-on programming experience in various technologies like Python, Scala.
• Strong Experience in developing ETL applications using different ETL tools like IBM Information Server, Informatica and Ab initio.
• Extensively development experience in different IDE like Eclipse, Anaconda.
• Strong experience in core SQL and Pl SQL.
• Very good experience in Scrum, Agile and Waterfall models.
• Good experience in Qlikview for Data Visualization and analysis on large data sets, drawing various conclusions.
• Strong team player wif good communication, analytical, presentation and inter-personal skills.

TECHNICAL SKILLS

Big Data Technologies: MapReduce, Hive, Pig, Impala, Hue, Sqoop, Kafka, Storm, Oozie, Flume

Spark components: RDD, Spark SQL (Data Frames and Dataset), Spark Streaming.

Programming Languages: Python 3, SQL, Scala 2.11, Core Java

Databases: Oracle, Teradata, DB2

NoSQL Databases: DynamoDB, Hbase, Aurora

Data Warehouse tool: Redshift

Serverless Application: AWS Lambda, glue

Scripting and Query Languages: Shell scripting, SQL and PL/SQL

Operating Systems: Windows, UNIX/Linux

Build Management Tools: Jenkins, Maven

IDE’S & Command line tools: Eclipse, Anaconda3, Toad and WinSCP

ETL Tools: Datastage, Informatica, Ab initto, ODI

Data Quality Tools: OEDQ, IBM Quality Stage

BI Tools: Qlikview, Cognos, Power BI

PROFESSIONAL EXPERIENCE

Confidential - CA

Sr. Spark Developer

Responsibilities:

• Acted as Lead Data Platform Engineer and worked in data management, data integration, data analysis
• Preparing HLD and LLD document wif ETL methodology transformation rules for data pipelines.
• Migrated existing application from PySpark to Spark SQL to halp increase Developer productivity.
• Designed highly performant migration pipelines to transform and migrate data from Teradata to Delta-lake, Tuned the jobs, provided recommendation and findings along wif best practices for future pipeline development.
• Developed and automated large-scale, high-performance frameworks and visualization to ensure reliability and meet critical business requirements.
• Analyzed the business needs, profile large data sets, design, develop and tune data products on Databricks platform.
• Provisioned a secure infrastructure to Ingest, enrich and combine data from multiple sources to present curated data products. Provided architecture patterns, development best practices and built a secure framework to manage data security and governance.
• UsingSpark Streamingto bring all real time data into the Databricks environment.
• Design, implement, and maintain modern data pipelines to deliver optimal solutions utilizing appropriate cloud technologies and integrations in collaboration wif data scientists and ML engineers.
• Collaborated wif product owners and business SMEs to analyze customer requirements and provide a supportable and sustainable-engineered solution.
• Shared noledge by clearly articulating results and ideas to key stakeholders.
• Participate in requirements gathering and understand the scope of the project and business requirements.
• Experienced in read and ingesting data in to one or more AWS services like S3, Redshift, Glue and data processing using Databricks.
• Leaded multi-functional engineering projects and implementation. Including but not limited to data warehouse/data lake architecture, data governance, data quality, and data privacy.
• Migrated data from traditional database systems to Databricks Delta Lake.
• Shared project solutions and outcomes wif Databricks colleagues to improve delivery on future projects.
• Recreated existing application logic and functionality in the AWS using Delta Lake and Databricks
• Providing support to customers about issues relating to the storage, handling, and access of data.
• Involved in change management, source control, and continuous deployment strategies.
• Worked wifJenkinsto build and deploy Python code fromGitHubto the production servers usingContinuous Integration and Continuous Delivery (CI/CD) pipeline.
• Collaborated wif analysts, data stewards, developers, and system owners in the testing of newly incorporated databases and integration measures.

Environment: Databricks, UNIX Shell Scripting, PySpark, Spark Streaming, Jenkins, AWS S3, Azure Blob Storage, RDS, Glue, EMR, Elastic, Cloudwatch, Lambda, EC2

Confidential

Sr. Spark Developer

Responsibilities:

• Acted as Lead Data Platform Engineer and worked in data management, data integration, data analysis
• Participate in requirements gathering and understand the scope of the project and business requirements.
• Experience ingesting data in to one or more AWS services like S3, Redshift, Glue and data processing using Databricks.
• Migrated data from traditional database systems to AWS S3
• Designed and implement streaming solutions using Kafka or Spark Stream Analytics
• Recreated existing application logic and functionality in the AWS using Data Lake, Glue, redshift and Databricks. Experience in DWH/BI project implementation using AWS Databricks.
• Designed and Developed ETL jobs to extract data from PostgreSQL and load it in data mart in Redshift.
• Involved in sourcing, processing, and transforming events data into tables from Kafka, One Lake (AWS-S3), Dev Exchange (API) to create consumable views of the data.
• Designing and building scalable DataStage solutions.
• Configuring clustered and distributed scalable parallel environments.
• Updating data wifin repositories and data warehouses.
• Monitoring jobs and identifying bottlenecks in the data processing pipeline.
• Testing and troubleshooting problems in system designs and processes.
• Improving existing ETL processes for the business.
• Providing support to customers about issues relating to the storage, handling, and access of data.
• Performed review and analysis of the detailed system specifications related to the DataStage ETL and related applications to ensure they appropriately address the business requirements
• Evaluated impact of proposed changes on existing DataStage ETL applications, processes and configurations
• Designed and developed the generic ETL utility to load data from PostgreSQL database
• Responsible for moving code from one environment to another. DEV to TEST and tan to PROD
• Ensured the stable operation of ETL processes in the production environment to agreed service levels
• Involved in Modeling, Estimation, Requirement Analysis and Design of mapping document and planning using ETL, BI tools.
• Worked wifJenkinsto build and deploy Python code fromGitHubto the production servers usingContinuous Integration and Continuous Delivery (CI/CD) pipeline.
• To meet specific business requirements wrote UDF’s inPySpark.
• Driving initiative to automate the recurring manual activities for monitoring and operations using Unix Scripting.
• Working on Hive to transform the data in stages for business reporting needs.
• Development of Hadoop MapReduce programs and data migration from existing data source using Sqoop.
• Devised schemes to collect and stage large data in Hadoop HDFS and worked on compressing the data using various formats to achieve optimal storage capacity
• Configured Hadoop Big Data, Map Reduce, HDFS, and developed multiple Map Reduce jobs for data cleaning and preprocessing
• Analyzed system specifications, designed, and developed test plans, worked on ingestion process of the web log data into Hadoop Big Data platform, and worked in extensive data integration using Big Data.
• Analysis and development ofSpark Cassandraconnector to load data from flat file toCassandra.
• UsingSpark Streamingto bring all real time data in the Hadoop environment.
• Involved in discussion wif source systems for issues related to DQ in data.
• Development of MapReduce programs and data migration from existing data source using Sqoop.
• Devised schemes to collect and stage large data in HDFS and worked on compressing the data using various formats to achieve optimal storage capacity
• Built re-usable Hive UDF libraries which enabled various business analysts to use these UDF's in Hive querying.
• Extensively worked on Text, ORC and Parquet file formats and compression techniques like Gzip.
• Extensively worked on Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
• Developed job processing scripts using Oozie workflow.
• Worked wif Spark-SQL context to create data frames to filter input data for model execution.
• Used AWS Glue to load data into S3 buckets and build RDS on top it for analytics purposes
• Cloudwatch for monitoring the AWS jobs.

Environment: IBM Information Server 11.5, Hadoop, HDFS, HiveQL, Oozie, UNIX Shell Scripting, PySpark, Kafka, Jenkins, AWS S3, RDS, Glue, EMR, Elastic, Cloudwatch, EMR cluster, Redshift, Lambda, EC2

Confidential

Sr. Spark Developer

Responsibilities:

• Engineered Customer 360 view from analyzing different customer domain data using Hive, Spark and Oozie.
• Developed multiple POCs using PySpark and deployed machine learning models on the Yarn cluster.
• Design dimensional model, data lake architecture, data vault 2.0 on Snowflake and used Snowflake logical data warehouse for compute.
• Processing of incoming files using Spark native API.
• Used Spark API over Hortonworks Hadoop Cluster.
• Implemented static Partitioning, Dynamic partitioning and Bucketing in Hive using internal and external table.
• Used Hive to do transformations, joins, filter and some pre-aggregations after storing the data to HDFS.
• Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time and Persists into Cassandra.
• Usage of Spark Streaming and Spark SQL API to process the files.
• Developed Spark scripts by using Scala shell commands as per the requirement.
• Processing the schema oriented and non-schema-oriented data using Scala and Spark.
• Designed and developed Spark Scala code to fast processing of HiveQL queries.
• Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.
• Developed Scala scripts, UDFFs using both Data frames/SQL/Data sets and RDD/Map Reduce in Spark 1.6 for Data Aggregation, queries and writing data back into OLTP system through Sqoop.
• Handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, TEMPEffective & efficient Joins, Transformations and other during ingestion process itself.
• Working on Snowflake modeling and highly proficient in data warehousing techniques for data cleansing, Slowly Changing Dimension phenomenon, surrogate key assignment and change data capture.
• Designed, developed and did maintenance of data integration programs in a Hadoop and RDBMS environment wif both traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis.
• Created Map Reduce Jobs to convert the periodic of XML messages into a partition Avro Data.
• Used Sqoop widely in order to import data from various systems/sources (like MySQL) into HDFS.
• Created components like Hive UDFs for missing functionality in HIVE for analytics.
• Developing Scripts and Batch Job to schedule a bundle (group of coordinators) which consists of various.
• Used different file formats like Text files, Sequence Files, Avro.

Environment: Hadoop, HDFS, Hive, Java, Sqoop, Spark, VMG, Teradata, Oracle12c, Apache Oozie, SFTP

Confidential

Sr. Spark Developer

Responsibilities:

• Requirement gathering involves requisite technical data which includes source files, installation, uninstallation instruction and any configuration details
• Evaluate requests and participating in functional specification review to determine feasibility and estimate effort required.
• Developed Spark Programs for Batch processing and managing data coming from different sources.
• Developed Spark scripts by using python shell commands as per the requirement.
• Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
• Performance optimization when dealing wif large datasets using partitions, broadcasts inSpark, TEMPeffective and efficient joins, transformations during ingestion process.
• Used Spark Sql wif python for creating data frames and performed transformations on data frames.
• Implemented Spark SQL to access hive tables into spark for faster processing of data.
• Experienced in defining job flows managing and reviewing Hadoop log files
• Supported MapReduce Programs those are running on the cluster.
• Jobs management using Fair scheduler and Cluster coordination services through Zookeeper.
• Hands on Experience in Oozie Job Scheduling.
• Developed Simple to Complex Map Reduce Jobs using Hive and Pig.
• Implemented Partitioning, Dynamic Partitions and Bucketing in Hive for efficient data access.
• Involved in running Hadoop Jobs for processing millions of records of text data.
• Created connection through JDBC and used JDBC statements to call stored procedures.
• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.
• Design dimensional model, data lake architecture.
• Implemented multiple Map Reduce Jobs in java for data cleansing and pre-processing.
• Moved all RDBMS data into flat files generated from various channels to HDFS for further processing.
• Involved in creating Hive tables, loading data and writing hive queries.
• Imported and exported data into HDFS using Sqoop which includes incremental loading.

Environment: Hadoop, MapReduce, HDFS, Pig, Hive, Spark, Flat files, Oracle 11g, Sqoop

Confidential

Sr. Hadoop Developer

Responsibilities:

• Requirement gathering involves requisite technical data which includes source files, installation, uninstallation instruction and any configuration details
• Analyzed the data using Map Reduce, Pig, Hive and produce summary results from Hadoop to downstream systems.
• Used Pig as ETL tool to do transformations, event joins and pre-aggregations before storing the data onto HDFS.
• Developed data pipeline using flume, Sqoop and pig to extract the data from weblogs and store in HDFS.
• Used Sqoop to import and export data from HDFS to RDBMS and vice-versa.
• Created Hive tables and involved in data loading and writing Hive UDFs.
• Exported the analyzed data to the relational database using Sqoop for visualization and to generate reports.
• Created HBase tables to load large sets of structured data.
• Managed and reviewed Hadoop log files.
• Worked extensively wif HIVE DDLs and Hive Query language (HQLs).
• Developed UDF, UDAF, UDTF functions and implemented it in HIVE Queries.
• Implemented SQOOP for large dataset transfer between Hadoop and RDBMs.
• Created Map Reduce Jobs to convert the periodic of XML messages into a partition Avro Data.
• Used Sqoop widely in order to import data from various systems/sources (like MySQL) into HDFS.
• Created components like Hive UDFs for missing functionality in HIVE for analytics.
• Developing Scripts and Batch Job to schedule a bundle (group of coordinators) which consists of various.
• Used different file formats like Text files, Sequence Files, Avro.
• Assisted in creating and maintaining Technical documentation to launching HADOOP Clusters and even for executing Hive queries and Pig Scripts.
• Assisted in Cluster maintenance, cluster monitoring, adding and removing cluster nodes and
• Installed and configured Hadoop, Map Reduce, HDFS, developed multiple Map Reduce jobs in java for data cleaning and pre-processing

Environment: Hadoop, HDFS, Map Reduce, Hive, Pig, Sqoop, HBase, Shell Scripting, Oozie, Oracle 11g

Confidential

Technical Lead

Responsibilities:

• Understanding the Requirement Documents and documenting Interface Agreements.
• Responsible for preparing the Logical data model and physical data model.
• Helped in preparing the mapping document source to target and preparing technical Design Document.
• Documented High Level Design & Low Level Design Documents.
• Implement Parallelism using partition concept to improve performance of Datastage Jobs.
• Prepare and present POC on identifying Duplicate profiles by using EDQ’s Match and Merge functionality.
• Assess the quality of data by using EDQ’s Business rule functionality.
• Responsible for creating EDQ code for Profiling and validating the Results.
• Responsible for code review and performance tuning of Business Rules coding.
• Used DataStage as an ETL tool to extract data from sources systems, aggregate the data and load into the Oracle database.
• Understanding the Customer Requirements and the Technical Specifications Documents.
• Design, Develop and Implementation of Datastage Jobs and sequence jobs and changing existing parallel Jobs as per client need.
• Creation of Jobs using Different Transformations like Database Connectors, Joiner, Expression, Aggregator,, Sorter, Lookup and CDC Transformation.
• Creating the sequence jobs and Running them by using Director.
• Guided the development team in building DataStage jobs and Assisted QA Team.
• Performed DQ Analysis and documented the issues.
• Responsible for Preparation of Design Documents / Test Case Specifications.
• Identifying performance bottlenecks and fine-tuning ETL mappings and workflows to improve performance and troubleshooting defect.
• Involved in unit, performance, Regression and integration testing of data stage jobs and prepared Documentation.

Environment: IIS 8.1, Oracle 11g, UNIX, SVN, Source Forge

Confidential

Technical Lead

Responsibilities:

• Understanding the Requirement Documents and documenting Interface Agreements.
• Helped in preparing the mapping document from source to target and preparing technical Design Document.
• Documented High Level Design & Low Level Design Documents.
• Involved in the discussions wif SME’s to validate the reports generated by Investigate stage (Quality Stage) and preparing of ACL (Action Check List) document suggestions made by the Business.
• TEMPEffectively used standardized stage (Quality Stage) in standardizing the source data by using the existed rule sets like name, address etc. for multiple countries and generated the valid and invalid data reports.
• Designed the jobs by using different survive techniques available in Survive stage (Quality Stage) by passing the Match stage output as input to get the best breed of record to the target.
• Updated/Modifying the existed rule sets specifically the PAT file to handle the unhandled patterns according to the ACL doc wherever it is needed.
• Involved in Building of Customized Rulesets for the countries for which there are no default Rulesets.
• Involved in the design of Match Templates suggestions made by the Business and validated the results to identify the duplicates coming from the source depending on different Match types.
• Designed the jobs by calling the Match Templates prepared and generated the results.
• Used DataStage as an ETL tool to extract data from sources systems, aggregate the data and load into the Oracle database.
• Communicate wif customer in getting the requirements /Issues /Risks verified & Resolved.
• Guided the development team in building DataStage jobs and Assisted QA Team.
• Performed DQ Analysis and documented the issues.
• Perform Code Reviews and Involved in end to end Implementation of the Project
• Responsible for Preparation of Design Documents / Test Case Specifications / Performance
• Debugging the Jobs and preparing the analysis reports to client.
• Created Shell Scripts to perform file level validations.
• Fine-tuned Data Stage jobs and routines for optimal performance.

Environment: IIS 8.7, AIX, Oracle 10g, VSS

Confidential

Senior ETL Developer

Responsibilities:

• Communicate wif customer in getting the requirements /Issues /Risks verified & Resolved.
• Responsible for preparing ETL Specification for developing Jobs.
• Prepared the mapping documents from source to target and Detail Design Document.
• Developed various jobs, performed data loads and transformations using different stages and pre-built routines, functions.
• Used DataStage as an ETL tool to extract data from different sources systems, perform transformations on data and load into target systems using File and DB Stages.
• Used Director for monitoring Job logs to resolve issues and fix warnings.
• Responsible for Preparation of Design Documents / Test Case Specifications.
• Debugging the Jobs and preparing the analysis reports to client.
• Fine Tuning Data Stage jobs and routines for optimal performance.
• Responsible for constructing and testing the sequencer jobs.
• Involved in unit, performance, Regression and integration testing of datastage jobs and prepared Documentation.

Environment: IIS 8.5, Solaris 9, Oracle11g, Mainframes, IBM DB2 V9

Confidential

Senior ETL Developer

Responsibilities:

• Responsible for preparing ETL Specification for developing Jobs.
• Helped in preparing the mapping document source to target and Preparing Detail Design Document.
• Created Low-level design and High-level design documents.
• Developed various jobs to perform data loads and transformations using different stages, pre-built routines and functions.
• Developed Parallel jobs using Parallel stages like: Aggregator, Filter, Union, Joiner, Sorter, Source Qualifier, Input and Output Transformations.
• Excessive usage of Director for monitoring job logs to resolve issues.
• Performed DQ Analysis and documented the issues.
• Wrote Shell Scripts to run data stage jobs.
• Responsible for Preparation of Design Documents / Test Case Specifications.
• Debugging the jobs and preparing the analysis reports to client.
• Responsible for constructing and testing sequence jobs.
• Involved in unit, performance, Regression and integration testing of datastage jobs and prepared Documentation.

Environment: IIS 8.0, IBM DB2 V9, Oracle 9i, Unix

Confidential

ETL Developer

Responsibilities:

• Handle& Solve tickets taking priorities into consideration.
• Responsible Perform RCA and follow up through CR process (When Necessary).
• Involved in requirements gathering, source data analysis, and identified business rules for data migration and for new CSR's.
• Involved in Production support, performance and integration testing of Data stage jobs
• Handling all the Administrative related activities.
• Used Data Stage Director and its run-time engine to schedule, run, and test and debug its components, and monitor the resulting executable versions (on an ad hoc or scheduled basis).
• Debugging the Jobs and preparing the analysis reports to client.
• Exporting and Importing the Jobs from Production server to Development server and Vice- Versa.
• Developed Parallel jobs using Parallel stages Like Merge, Join, Lookup, Transformer (Parallel), Oracle Enterprise Stage.
• Interact wif source team for Source data changes.
• Fine-tuned Data Stage jobs and routines for optimal performance.
• Creating design documents for production support.

Environment: IIS 8.0, Teradata V2R6, Oracle 9i, Solaris 9

Confidential

ETL Developer

Responsibilities:

• Involved in Analyzing the Mapping Documents and maintaining the Issue Register to get coordinated wif Client
• Involved in preparing the design documents like LLD’s and HLD’s
• Had hands in running the Quality Stage Components
• Developed Reusable shared containers that can be used across the project.
• Had hands in altering the Configuration file based on the environmental settings
• Handling all types of Source files viz. Flat files wif De-limiters, Fixed width files and Files wif multiple delimiters
• Develop transformation for various complex logics taking performance and partitions in to account
• Involved in preparing the test scenarios for UNIT testing, Assembly Testing and Regression Testing
• Played vital role in Quality Processes like Defect Prevention while managing the development
• Prepare noledge sharing documents and share it across the team by conducting the meetings
• Created source to target mapping documents from staging area to Data Warehouse and from Data Warehouse to various Data Marts.
• Involved in low-level design and developed various jobs and performed data loads and transformations using different stages of DataStage and pre-built routines, functions and macros
• Used DataStage Designer for developing various jobs to extract, cleansing, transforming, integrating and loading data into Data Warehouse.
• Used DataStage Director to schedule running the server jobs, monitoring scheduling and validating its components
• Responsible for Preparation of Design Documents / Test Case Specifications

Environment: Datastage7.5.1a, IBM Db2, AIX

Confidential

ETL Developer

Responsibilities:

• Involved in developing the technical documents from functional specifications.
• Involved in preparing the design documents LLD’s and HLD’s.
• Extracting, Transforming and Loading the data from Source to Staging and Staging to Target according to the Business requirements.
• Developed Reusable components, which can be used across the project.
• Debugging and fixing the code bugs.
• Following up wif the Review teams (Migration team, DBA team and Production support Team) to sign off on the Releases.
• Responsible for performing testing in the application for correct loading of files.
• Performed bugs tracking and Risk management activities.

Environment: Informatica Power Center 7.1, Teradata, Oracle 9i, Unix