SUMMARY OF SKILLS:

• Interacting with Client understanding the requirements and developing High Level design and Low Level Design document.
• Designing work flow design document and distributing task in team.
• Solving problem for data pipeline in Azure Databricks.
• Experienced in Apache Airflow, Data control API, Amazon S3 (Cloud Storage) and Pivoted cloud foundry (PCF), Azure Data factory, Datalake, Azure DataBricks, Apache Airflow, Data Warehousing, etc.
• Designing and implementing a scalable and durable data model in PySpark on Azure Cloud.
• Experienced in GIT for version control.
• Experienced in CI/CD integration with Jenkin, Ansible tower, Ansible Playbook, JIRA and GIT.
• Developing end - to-end automation for data pipeline, making dataset readily-consumable by visualization tool and notification system.
• Experienced in implementation of new data workflows, automation and API integrations
• Experience in implementing large-scale custom development and systems integration projects.
• Familiar with UNIX, Shell scripting, SQL.
• Developing HLD and LLD documentation.
• Provide guidance to development team working on PySpark as ETL platform
• Integrating Application with Reporting and Monitoring tool and fulfilling customer requirement accordingly.
• Interacting with business users to gather requirements, writing functional and technical specifications.
• Experienced in Reporting tool.
• Translating requirements into design specifications and experience in developing design documents.

PROFESSIONAL EXPERIENCE

Confidential

Data Integration and Migration

Roles and Responsibilities: -

• Developed spark applications in python(PySpark) on distributed environment (Azure Data Bricks) to load huge number of CSV files with different schema in to S3.
• Designing Data Pipeline job in Azure Data factory
• Using Azure Data lake for storage.
• Execute code in Stream mode and salable mode by azure Stream Analytics.
• Worked on reading and writing multiple data formats like JSON, Parquet,etc using PySpark.
• Designing Data Pipeline job to extract data through xml, Cloud S3, Flat File source.
• Using JIRA tools and Git for committing code.
• Implementing Test Driven Development.
• Creating and maintaining scalable data pipeline and build out new integration.
• Developing API and integrated with data streaming job, deploying them on Pivoted cloud foundry (PCF) on scaling mode.
• Executing job in scaling mode.
• Create and maintain optimal data pipeline architecture
• Optimization of PySpark code in Azure Data Bricks using best practices and right parameters.
• Created different Hive RAW and Standardize table for data validation and Analysis with Partition and bucket
• Helping team for solving critical issues, also optimisation of spark code.
• Responsible creating data streaming pipeline.
• Developing Proof of Concept and implementing in project to get desire result.
• Developing HLD and LLD document for the project.
• Writing Scripting for interacting Internal server with S3 bucket (Cloud Storage) for migrate the data.
• Interacting with client and understanding the requirement and developing the code.
• Creating Architecture flow diagram for Project.
• Responsible for delivery support and any code related issue.

Confidential

Data Migration

Responsibilities: -

• Developing Data pipeline in Azure Data factory.
• Interacting with customer understanding the requirement and Developing data pipeline job.
• Analysed the sql scripts and designed it by using PySpark SQL for faster performance in Azure Data Bricks.
• Implementing artificial intelligent to provide user request data.
• Developing big data pipeline for processing the data.
• Responsible for Data modelling of complex large data set.
• Implementing predictive analytics algorithms, methodologies to avoid the slowness while working on huge data.
• Create and manage project plans and roadmaps within an agile environment.
• Accepting challenges and managing risks TEMPeffectively and innovatively with own initiative.
• Understanding logical mapping and data models provided and design
• Troubleshooting and resolve data, system issues and performance issues.
• Developing and Executing tasks related to the production of the Test Plan, test cases, test scripts, and test data.
• Generating report by data visualization tool (Pentaho Reporting tool).

Confidential

Application Integration

Responsibilities: -

• Designed Data integration job for data transformation between multiple data base in Azure.
• Integrating API and PySpark code to interact with GUI.
• Wrote SQL scripting to developer Business Intelligence Report also shell script to filter ETL data.
• Configured and Scheduling PySpark job and bug fixing (Azure Data Bricks).
• Designed and building configuring ETL according to business requirement.
• Developed strategic testing plans to achieve performance quality by coordinating with the project manager.
• Developed testing documentation to ensure that it meets the customer requirement
• Evaluated defect metrics data and updating.
• Customized dash board and creating new according to requirement.
• Analyzed server logs and resolving issue.
• Interacted with customer and developing code according to requirement.
• Tested in code UAT and deploying in Production.
• Implemented productive algorithm for filtering data.
• Generating report by data visualization tool (UniBI Reporting tool)

Confidential

End to End Data Integration

Responsibilities: -

• Designed the Data Pipeline job for getting data from different applications.
• Designing and developing test Driven development.
• Implemented predictive algorithm to fill null value and get data transform.
• Developed and implemented test cases under the guidelines.
• Integrated multiple application and transforming data between multiple data bases by ETL job.
• Interacted with the project team to gain an understanding of the business requirements, technical context, and conformance criteria.
• Designed and scheduling ETL job.
• Developed reports by using Data integration.
• Performed testing functional applications and validated testing procedures.
• Supported business staff by providing technical information.
• Conducted research on new tool and applications to support business requirements.
• Prepared project deliverables documents by coordinating with team members.
• Wrote shell script for Data integration between multiple data base.
• Participated in designing the overall logical & physical Data warehouse/Data-mart data model and data.

TECHNICAL SKILLS

Database: Oracle, MySQL

IT Skills: Apache Spark, PySpark, Data Bricks, Apache Beam, Airflow, Amazon S3, Python, Shell scripts, Pivoted Cloud Foundry (PCF)

OS: Unix, Windows