SUMMARY

• Above 17 years of professional experience in Information Technology, with extensive experience as Big data Engineer and Hadoop/Spark Developer, AWS, ETL, Data Quality, Reporting, Data warehouse and Data/system analysis.
• Expert at full life cycle implementation using Spark Ecosystem.
• Experienced with design, development and implementation of data pipeline using Spark/Python Applications
• Good Knowledge and experience on Integration tools and deployment tool like Jenkins in setup of CI/CD pipeline.
• Experienced with AWS cloud platform to perform Bigdata Analytics work using EMR cluster.
• Experienced with development and implementation of various AWS services including Aethna, Lambda, EC2, Elastic Search, Dynamo DB, Arora, Redshift, cloudwatch, S3 and Glue.
• Developed Python Scripts to handle teh complex data structures for analytics and for data ingestion into Enterprise cluster.
• Experienced in developing and deploying enterprise - based applications using major Hadoop ecosystem components like Job Tracker, Task Tracker, Name Node, Data Node, Map Reduce concepts and YARN architecture which includes Node manager, Resource manager, Hive, Pig, HBase, Flume, Sqoop, Spark Streaming, Spark SQL, Storm, Kafka, Oozie, Zookeeper and Cassandra.
• Experience in using different file formats like CSV, Sequence, AVRO, ORC, JSON and PARQUET files and different compression Techniques like LZO, Gzip, Bzip2 and Snappy. Write abstract loader and reading and handling metadata
• Experienced to implement ad-hoc queries using Hive Query Language, Partitioning, bucketing and Hive-Custom UDF's.
• UsedSpark StreamingAPIs to perform transformations and actions on teh fly for building common learner data model which gets teh data fromKafkain near real time and persist it toCassandra.
• Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming teh data to uncover insights into teh customer usage patterns.
• Experience in transferring Streaming data, data from different data sources into HDFS, NoSQL databases using Apache Flume and Apache Kafka.
• Experience in creating complex data warehouse and application roadmaps and BI implementation specializing in Teradata platforms and design, development and maintenance of ETL code.
• Analyzed large amounts of data sets and migrate ETL operations using RDD, Dataframes, Datasets and Spark SQL.
• Experience writing Shell scripts in Linux OS and integrating them with other solutions.
• Hands-on programming experience in various technologies like Python, Scala.
• Strong Experience in developing ETL applications using different ETL tools like IBM Information Server, Informatica and Ab initio.
• Extensively development experience in different IDE like Eclipse, Anaconda.
• Strong experience in core SQL and Pl SQL.
• Very good experience in Scrum, Agile and Waterfall models.
• Good experience in Qlikview for Data Visualization and analysis on large data sets, drawing various conclusions.
• Strong team player with good communication, analytical, presentation and inter-personal skills.

TECHNICAL SKILLS

Big Data Technologies: MapReduce, Hive, Pig, Impala, Hue, Sqoop, Kafka, Storm, Oozie, Flume

Spark components: RDD, Spark SQL (Data Frames and Dataset), Spark Streaming.

Programming Languages: Python 3, SQL, Scala 2.11, Core Java

Databases: Oracle, Teradata, DB2

NoSQL Databases: DynamoDB, Hbase, Aurora

Data Warehouse tool: Redshift

Serverless Application: AWS Lambda, glue

Scripting and Query Languages: Shell scripting, SQL and PL/SQL

Operating Systems: Windows, UNIX/Linux

Build Management Tools: Jenkins, Maven

IDE’S & Command line tools: Eclipse, Anaconda3, Toad and WinSCP

ETL Tools: Datastage, Informatica, Ab initto, ODI

Data Quality Tools: OEDQ, IBM Quality Stage

BI Tools: Qlikview, Cognos, Power BI

PROFESSIONAL EXPERIENCE

Confidential - CA

Sr. Spark Developer

Responsibilities:

• Acted as Lead Data Platform Engineer and worked in data management, data integration, data analysis
• Preparing HLD and LLD document with ETL methodology transformation rules for data pipelines.
• Migrated existing application from PySpark to Spark SQL to halp increase Developer productivity.
• Designed highly performant migration pipelines to transform and migrate data from Teradata to Delta-lake, Tuned teh jobs, provided recommendation and findings along with best practices for future pipeline development.
• Developed and automated large-scale, high-performance frameworks and visualization to ensure reliability and meet critical business requirements.
• Analyzed teh business needs, profile large data sets, design, develop and tune data products on Databricks platform.
• Provisioned a secure infrastructure to Ingest, enrich and combine data from multiple sources to present curated data products. Provided architecture patterns, development best practices and built a secure framework to manage data security and governance.
• UsingSpark Streamingto bring all real time data into teh Databricks environment.
• Design, implement, and maintain modern data pipelines to deliver optimal solutions utilizing appropriate cloud technologies and integrations in collaboration with data scientists and ML engineers.
• Collaborated with product owners and business SMEs to analyze customer requirements and provide a supportable and sustainable-engineered solution.
• Shared knowledge by clearly articulating results and ideas to key stakeholders.
• Participate in requirements gathering and understand teh scope of teh project and business requirements.
• Experienced in read and ingesting data in to one or more AWS services like S3, Redshift, Glue and data processing using Databricks.
• Leaded multi-functional engineering projects and implementation. Including but not limited to data warehouse/data lake architecture, data governance, data quality, and data privacy.
• Migrated data from traditional database systems to Databricks Delta Lake.
• Shared project solutions and outcomes with Databricks colleagues to improve delivery on future projects.
• Recreated existing application logic and functionality in teh AWS using Delta Lake and Databricks
• Providing support to customers about issues relating to teh storage, handling, and access of data.
• Involved in change management, source control, and continuous deployment strategies.
• Worked withJenkinsto build and deploy Python code fromGitHubto teh production servers usingContinuous Integration and Continuous Delivery (CI/CD) pipeline.
• Collaborated with analysts, data stewards, developers, and system owners in teh testing of newly incorporated databases and integration measures.

Environment: Databricks, UNIX Shell Scripting, PySpark, Spark Streaming, Jenkins, AWS S3, Azure Blob Storage, RDS, Glue, EMR, Elastic, Cloudwatch, Lambda, EC2

Confidential

Sr. Spark Developer

Responsibilities:

• Acted as Lead Data Platform Engineer and worked in data management, data integration, data analysis
• Participate in requirements gathering and understand teh scope of teh project and business requirements.
• Experience ingesting data in to one or more AWS services like S3, Redshift, Glue and data processing using Databricks.
• Migrated data from traditional database systems to AWS S3
• Designed and implement streaming solutions using Kafka or Spark Stream Analytics
• Recreated existing application logic and functionality in teh AWS using Data Lake, Glue, redshift and Databricks. Experience in DWH/BI project implementation using AWS Databricks.
• Designed and Developed ETL jobs to extract data from PostgreSQL and load it in data mart in Redshift.
• Involved in sourcing, processing, and transforming events data into tables from Kafka, One Lake (AWS-S3), Dev Exchange (API) to create consumable views of teh data.
• Designing and building scalable DataStage solutions.
• Configuring clustered and distributed scalable parallel environments.
• Updating data within repositories and data warehouses.
• Monitoring jobs and identifying bottlenecks in teh data processing pipeline.
• Testing and troubleshooting problems in system designs and processes.
• Improving existing ETL processes for teh business.
• Providing support to customers about issues relating to teh storage, handling, and access of data.
• Performed review and analysis of teh detailed system specifications related to teh DataStage ETL and related applications to ensure they appropriately address teh business requirements
• Evaluated impact of proposed changes on existing DataStage ETL applications, processes and configurations
• Designed and developed teh generic ETL utility to load data from PostgreSQL database
• Responsible for moving code from one environment to another. DEV to TEST and then to PROD
• Ensured teh stable operation of ETL processes in teh production environment to agreed service levels
• Involved in Modeling, Estimation, Requirement Analysis and Design of mapping document and planning using ETL, BI tools.
• Worked withJenkinsto build and deploy Python code fromGitHubto teh production servers usingContinuous Integration and Continuous Delivery (CI/CD) pipeline.
• To meet specific business requirements wrote UDF’s inPySpark.
• Driving initiative to automate teh recurring manual activities for monitoring and operations using Unix Scripting.
• Working on Hive to transform teh data in stages for business reporting needs.
• Development of Hadoop MapReduce programs and data migration from existing data source using Sqoop.
• Devised schemes to collect and stage large data in Hadoop HDFS and worked on compressing teh data using various formats to achieve optimal storage capacity
• Configured Hadoop Big Data, Map Reduce, HDFS, and developed multiple Map Reduce jobs for data cleaning and preprocessing
• Analyzed system specifications, designed, and developed test plans, worked on ingestion process of teh web log data into Hadoop Big Data platform, and worked in extensive data integration using Big Data.
• Analysis and development ofSpark Cassandraconnector to load data from flat file toCassandra.
• UsingSpark Streamingto bring all real time data in teh Hadoop environment.
• Involved in discussion with source systems for issues related to DQ in data.
• Development of MapReduce programs and data migration from existing data source using Sqoop.
• Devised schemes to collect and stage large data in HDFS and worked on compressing teh data using various formats to achieve optimal storage capacity
• Built re-usable Hive UDF libraries which enabled various business analysts to use these UDF's in Hive querying.
• Extensively worked on Text, ORC and Parquet file formats and compression techniques like Gzip.
• Extensively worked on Hive to analyze teh partitioned and bucketed data and compute various metrics for reporting.
• Developed job processing scripts using Oozie workflow.
• Worked with Spark-SQL context to create data frames to filter input data for model execution.
• Used AWS Glue to load data into S3 buckets and build RDS on top it for analytics purposes
• Cloudwatch for monitoring teh AWS jobs.

Environment: IBM Information Server 11.5, Hadoop, HDFS, HiveQL, Oozie, UNIX Shell Scripting, PySpark, Kafka, Jenkins, AWS S3, RDS, Glue, EMR, Elastic, Cloudwatch, EMR cluster, Redshift, Lambda, EC2

Confidential

Sr. Spark Developer

Responsibilities:

• Engineered Customer 360 view from analyzing different customer domain data using Hive, Spark and Oozie.
• Developed multiple POCs using PySpark and deployed machine learning models on teh Yarn cluster.
• Design dimensional model, data lake architecture, data vault 2.0 on Snowflake and used Snowflake logical data warehouse for compute.
• Processing of incoming files using Spark native API.
• Used Spark API over Hortonworks Hadoop Cluster.
• Implemented static Partitioning, Dynamic partitioning and Bucketing in Hive using internal and external table.
• Used Hive to do transformations, joins, filter and some pre-aggregations after storing teh data to HDFS.
• Used Spark-Streaming APIs to perform necessary transformations and actions on teh fly for building teh common learner data model which gets teh data from Kafka in near real time and Persists into Cassandra.
• Usage of Spark Streaming and Spark SQL API to process teh files.
• Developed Spark scripts by using Scala shell commands as per teh requirement.
• Processing teh schema oriented and non-schema-oriented data using Scala and Spark.
• Designed and developed Spark Scala code to fast processing of HiveQL queries.
• Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.
• Developed Scala scripts, UDFFs using both Data frames/SQL/Data sets and RDD/Map Reduce in Spark 1.6 for Data Aggregation, queries and writing data back into OLTP system through Sqoop.
• Handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, TEMPEffective & efficient Joins, Transformations and other during ingestion process itself.
• Working on Snowflake modeling and highly proficient in data warehousing techniques for data cleansing, Slowly Changing Dimension phenomenon, surrogate key assignment and change data capture.
• Designed, developed and did maintenance of data integration programs in a Hadoop and RDBMS environment with both traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis.
• Created Map Reduce Jobs to convert teh periodic of XML messages into a partition Avro Data.
• Used Sqoop widely in order to import data from various systems/sources (like MySQL) into HDFS.
• Created components like Hive UDFs for missing functionality in HIVE for analytics.
• Developing Scripts and Batch Job to schedule a bundle (group of coordinators) which consists of various.
• Used different file formats like Text files, Sequence Files, Avro.

Environment: Hadoop, HDFS, Hive, Java, Sqoop, Spark, VMG, Teradata, Oracle12c, Apache Oozie, SFTP

Confidential

Sr. Spark Developer

Responsibilities:

• Requirement gathering involves requisite technical data which includes source files, installation, uninstallation instruction and any configuration details
• Evaluate requests and participating in functional specification review to determine feasibility and estimate effort required.
• Developed Spark Programs for Batch processing and managing data coming from different sources.
• Developed Spark scripts by using python shell commands as per teh requirement.
• Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
• Performance optimization when dealing with large datasets using partitions, broadcasts inSpark, TEMPeffective and efficient joins, transformations during ingestion process.
• Used Spark Sql with python for creating data frames and performed transformations on data frames.
• Implemented Spark SQL to access hive tables into spark for faster processing of data.
• Experienced in defining job flows managing and reviewing Hadoop log files
• Supported MapReduce Programs those are running on teh cluster.
• Jobs management using Fair scheduler and Cluster coordination services through Zookeeper.
• Hands on Experience in Oozie Job Scheduling.
• Developed Simple to Complex Map Reduce Jobs using Hive and Pig.
• Implemented Partitioning, Dynamic Partitions and Bucketing in Hive for efficient data access.
• Involved in running Hadoop Jobs for processing millions of records of text data.
• Created connection through JDBC and used JDBC statements to call stored procedures.
• Developed Pig Latin scripts to extract teh data from teh web server output files to load into HDFS.
• Design dimensional model, data lake architecture.
• Implemented multiple Map Reduce Jobs in java for data cleansing and pre-processing.
• Moved all RDBMS data into flat files generated from various channels to HDFS for further processing.
• Involved in creating Hive tables, loading data and writing hive queries.
• Imported and exported data into HDFS using Sqoop which includes incremental loading.

Environment: Hadoop, MapReduce, HDFS, Pig, Hive, Spark, Flat files, Oracle 11g, Sqoop

Confidential

Sr. Hadoop Developer

Responsibilities:

• Requirement gathering involves requisite technical data which includes source files, installation, uninstallation instruction and any configuration details
• Analyzed teh data using Map Reduce, Pig, Hive and produce summary results from Hadoop to downstream systems.
• Used Pig as ETL tool to do transformations, event joins and pre-aggregations before storing teh data onto HDFS.
• Developed data pipeline using flume, Sqoop and pig to extract teh data from weblogs and store in HDFS.
• Used Sqoop to import and export data from HDFS to RDBMS and vice-versa.
• Created Hive tables and involved in data loading and writing Hive UDFs.
• Exported teh analyzed data to teh relational database using Sqoop for visualization and to generate reports.
• Created HBase tables to load large sets of structured data.
• Managed and reviewed Hadoop log files.
• Worked extensively with HIVE DDLs and Hive Query language (HQLs).
• Developed UDF, UDAF, UDTF functions and implemented it in HIVE Queries.
• Implemented SQOOP for large dataset transfer between Hadoop and RDBMs.
• Created Map Reduce Jobs to convert teh periodic of XML messages into a partition Avro Data.
• Used Sqoop widely in order to import data from various systems/sources (like MySQL) into HDFS.
• Created components like Hive UDFs for missing functionality in HIVE for analytics.
• Developing Scripts and Batch Job to schedule a bundle (group of coordinators) which consists of various.
• Used different file formats like Text files, Sequence Files, Avro.
• Assisted in creating and maintaining Technical documentation to launching HADOOP Clusters and even for executing Hive queries and Pig Scripts.
• Assisted in Cluster maintenance, cluster monitoring, adding and removing cluster nodes and
• Installed and configured Hadoop, Map Reduce, HDFS, developed multiple Map Reduce jobs in java for data cleaning and pre-processing

Environment: Hadoop, HDFS, Map Reduce, Hive, Pig, Sqoop, HBase, Shell Scripting, Oozie, Oracle 11g

Confidential

Technical Lead

Responsibilities:

• Understanding teh Requirement Documents and documenting Interface Agreements.
• Responsible for preparing teh Logical data model and physical data model.
• Helped in preparing teh mapping document source to target and preparing technical Design Document.
• Documented High Level Design & Low Level Design Documents.
• Implement Parallelism using partition concept to improve performance of Datastage Jobs.
• Prepare and present POC on identifying Duplicate profiles by using EDQ’s Match and Merge functionality.
• Assess teh quality of data by using EDQ’s Business rule functionality.
• Responsible for creating EDQ code for Profiling and validating teh Results.
• Responsible for code review and performance tuning of Business Rules coding.
• Used DataStage as an ETL tool to extract data from sources systems, aggregate teh data and load into teh Oracle database.
• Understanding teh Customer Requirements and teh Technical Specifications Documents.
• Design, Develop and Implementation of Datastage Jobs and sequence jobs and changing existing parallel Jobs as per client need.
• Creation of Jobs using Different Transformations like Database Connectors, Joiner, Expression, Aggregator,, Sorter, Lookup and CDC Transformation.
• Creating teh sequence jobs and Running them by using Director.
• Guided teh development team in building DataStage jobs and Assisted QA Team.
• Performed DQ Analysis and documented teh issues.
• Responsible for Preparation of Design Documents / Test Case Specifications.
• Identifying performance bottlenecks and fine-tuning ETL mappings and workflows to improve performance and troubleshooting defect.
• Involved in unit, performance, Regression and integration testing of data stage jobs and prepared Documentation.

Environment: IIS 8.1, Oracle 11g, UNIX, SVN, Source Forge

Confidential

Technical Lead

Responsibilities:

• Understanding teh Requirement Documents and documenting Interface Agreements.
• Helped in preparing teh mapping document from source to target and preparing technical Design Document.
• Documented High Level Design & Low Level Design Documents.
• Involved in teh discussions with SME’s to validate teh reports generated by Investigate stage (Quality Stage) and preparing of ACL (Action Check List) document suggestions made by teh Business.
• TEMPEffectively used standardized stage (Quality Stage) in standardizing teh source data by using teh existed rule sets like name, address etc. for multiple countries and generated teh valid and invalid data reports.
• Designed teh jobs by using different survive techniques available in Survive stage (Quality Stage) by passing teh Match stage output as input to get teh best breed of record to teh target.
• Updated/Modifying teh existed rule sets specifically teh PAT file to handle teh unhandled patterns according to teh ACL doc wherever it is needed.
• Involved in Building of Customized Rulesets for teh countries for which their are no default Rulesets.
• Involved in teh design of Match Templates suggestions made by teh Business and validated teh results to identify teh duplicates coming from teh source depending on different Match types.
• Designed teh jobs by calling teh Match Templates prepared and generated teh results.
• Used DataStage as an ETL tool to extract data from sources systems, aggregate teh data and load into teh Oracle database.
• Communicate with customer in getting teh requirements /Issues /Risks verified & Resolved.
• Guided teh development team in building DataStage jobs and Assisted QA Team.
• Performed DQ Analysis and documented teh issues.
• Perform Code Reviews and Involved in end to end Implementation of teh Project
• Responsible for Preparation of Design Documents / Test Case Specifications / Performance
• Debugging teh Jobs and preparing teh analysis reports to client.
• Created Shell Scripts to perform file level validations.
• Fine-tuned Data Stage jobs and routines for optimal performance.

Environment: IIS 8.7, AIX, Oracle 10g, VSS

Confidential

Senior ETL Developer

Responsibilities:

• Communicate with customer in getting teh requirements /Issues /Risks verified & Resolved.
• Responsible for preparing ETL Specification for developing Jobs.
• Prepared teh mapping documents from source to target and Detail Design Document.
• Developed various jobs, performed data loads and transformations using different stages and pre-built routines, functions.
• Used DataStage as an ETL tool to extract data from different sources systems, perform transformations on data and load into target systems using File and DB Stages.
• Used Director for monitoring Job logs to resolve issues and fix warnings.
• Responsible for Preparation of Design Documents / Test Case Specifications.
• Debugging teh Jobs and preparing teh analysis reports to client.
• Fine Tuning Data Stage jobs and routines for optimal performance.
• Responsible for constructing and testing teh sequencer jobs.
• Involved in unit, performance, Regression and integration testing of datastage jobs and prepared Documentation.

Environment: IIS 8.5, Solaris 9, Oracle11g, Mainframes, IBM DB2 V9

Confidential

Senior ETL Developer

Responsibilities:

• Responsible for preparing ETL Specification for developing Jobs.
• Helped in preparing teh mapping document source to target and Preparing Detail Design Document.
• Created Low-level design and High-level design documents.
• Developed various jobs to perform data loads and transformations using different stages, pre-built routines and functions.
• Developed Parallel jobs using Parallel stages like: Aggregator, Filter, Union, Joiner, Sorter, Source Qualifier, Input and Output Transformations.
• Excessive usage of Director for monitoring job logs to resolve issues.
• Performed DQ Analysis and documented teh issues.
• Wrote Shell Scripts to run data stage jobs.
• Responsible for Preparation of Design Documents / Test Case Specifications.
• Debugging teh jobs and preparing teh analysis reports to client.
• Responsible for constructing and testing sequence jobs.
• Involved in unit, performance, Regression and integration testing of datastage jobs and prepared Documentation.

Environment: IIS 8.0, IBM DB2 V9, Oracle 9i, Unix

Confidential

ETL Developer

Responsibilities:

• Handle& Solve tickets taking priorities into consideration.
• Responsible Perform RCA and follow up through CR process (When Necessary).
• Involved in requirements gathering, source data analysis, and identified business rules for data migration and for new CSR's.
• Involved in Production support, performance and integration testing of Data stage jobs
• Handling all teh Administrative related activities.
• Used Data Stage Director and its run-time engine to schedule, run, and test and debug its components, and monitor teh resulting executable versions (on an ad hoc or scheduled basis).
• Debugging teh Jobs and preparing teh analysis reports to client.
• Exporting and Importing teh Jobs from Production server to Development server and Vice- Versa.
• Developed Parallel jobs using Parallel stages Like Merge, Join, Lookup, Transformer (Parallel), Oracle Enterprise Stage.
• Interact with source team for Source data changes.
• Fine-tuned Data Stage jobs and routines for optimal performance.
• Creating design documents for production support.

Environment: IIS 8.0, Teradata V2R6, Oracle 9i, Solaris 9

Confidential

ETL Developer

Responsibilities:

• Involved in Analyzing teh Mapping Documents and maintaining teh Issue Register to get coordinated with Client
• Involved in preparing teh design documents like LLD’s and HLD’s
• Had hands in running teh Quality Stage Components
• Developed Reusable shared containers that can be used across teh project.
• Had hands in altering teh Configuration file based on teh environmental settings
• Handling all types of Source files viz. Flat files with De-limiters, Fixed width files and Files with multiple delimiters
• Develop transformation for various complex logics taking performance and partitions in to account
• Involved in preparing teh test scenarios for UNIT testing, Assembly Testing and Regression Testing
• Played vital role in Quality Processes like Defect Prevention while managing teh development
• Prepare knowledge sharing documents and share it across teh team by conducting teh meetings
• Created source to target mapping documents from staging area to Data Warehouse and from Data Warehouse to various Data Marts.
• Involved in low-level design and developed various jobs and performed data loads and transformations using different stages of DataStage and pre-built routines, functions and macros
• Used DataStage Designer for developing various jobs to extract, cleansing, transforming, integrating and loading data into Data Warehouse.
• Used DataStage Director to schedule running teh server jobs, monitoring scheduling and validating its components
• Responsible for Preparation of Design Documents / Test Case Specifications

Environment: Datastage7.5.1a, IBM Db2, AIX

Confidential

ETL Developer

Responsibilities:

• Involved in developing teh technical documents from functional specifications.
• Involved in preparing teh design documents LLD’s and HLD’s.
• Extracting, Transforming and Loading teh data from Source to Staging and Staging to Target according to teh Business requirements.
• Developed Reusable components, which can be used across teh project.
• Debugging and fixing teh code bugs.
• Following up with teh Review teams (Migration team, DBA team and Production support Team) to sign off on teh Releases.
• Responsible for performing testing in teh application for correct loading of files.
• Performed bugs tracking and Risk management activities.

Environment: Informatica Power Center 7.1, Teradata, Oracle 9i, Unix