SUMMARY

• A Senior developer having overall 9 years of experience in ETL and Hadoop technologies, seeking a challenging position in a dynamic organization where my experience and dedication can help the growth of organization and my career.
• Senior Bigdata Developer having experience in ETL (Informatica and Abinitio) & Hadoop technologies (HDFS, Spark using Python, HIVE, Sqoop and R Programming) with good understanding of Data warehousing and ETL concepts.
• Involved in direct discussion with Business and Data Analyst for the requirement gathering, translate the business requirement to technical specification, design the architecture and develop efficient ETL and Big data solution with reusable component following best practice and standards. Prepare effort and time estimation for project proposals.
• Worked with the Data modeler to set the structure of the Data flow in various layers for both ETL and Big data Projects
• Ingested the file into HDFS using Apache Sqoop and Abinitio graph
• Transformed the data from ingestion layer using Spark (written in both Python and Scala programming) and Abinitio graph
• Developed the mappings to Extract, Transform and Load (ETL) using Informatica and Abinitio
• Performed the data Load using Oozie scheduler for Big data HDFS and Control M, Informatica Workflow, Autosys for the ETL data
• Performed the Map reducer to load the Hive tables using Java
• Developed reusable Informatica jobs adhering to requirement and existing legacy logic. Design and implemented automated services as dynamic parameter generation, Error and Audit Balance Control and automated validation.
• Lead the team to develop the ETL jobs adhering to the requirement and following the organizations compliance, standards and framework, security policy for development, data/file transfer and consumption. Manage the team to complete the development and testing within the timeline to deliver the application. Document and maintain design, mapping specifications, nonfunctional requirement and knowledge transfer documents.
• Confer with the team to design the system and gather the statistics, progress of the development and maintenance to create weekly/monthly status report.
• Handle issues and deployment using incident management and change management respectively.
• Monitor and troubleshoot any production issue and provide an efficient solution within the SLA. Provide technical support to users in assisting any functional problems or modifications. Performance tuning of the application to improve the efficiency.
• Perform data analysis and profiling, by framing complex custom SQL queries based on customer need and providing report and statistics.
• Create, maintain, support and enhance Business Objects Universe, web and desktop intelligence reports. Administration and installation of Informatica, create and maintain users and repository services.

TECHNICAL SKILLS

ETL: Informatica PowerCenter and Abinitio

Reporting: SAP Business Objects

Hadoop Ecosystem: HDFS, HIVE, Pig, Sqoop and Oozie

Analytical Engine: Apache Spark, Spark SQL and Numpy

Database: Teradata, Oracle, Netezza, MS SQL server

Programming: Unix, PL SQL, Python, Scala, T - SQL,R Language and Java Map reduce

Others: Control M, Informatica Scheduler, JIRA, Bitbucket, Version one, HP ALM, Autosys

PROFESSIONAL EXPERIENCE

Senior Bigdata Developer

Confidential

Responsibilities:

• Understanding the business requirements and required Hadoop Ecosystem tools to process the various type of source data into Hadoop.
• Perform the data profiling on various source system and match against the requirement.
• Performed the analytics based on source data in the business data warehouse lab using R Language
• Create the Sqoop jobs in Hadoop Ecosystem to load or import the data from various RDBS such as Oracle, Teradata, MS SQL server into Hadoop as a Raw data.
• Transform the raw data using Pyspark and Apache Spark(Scala) as data frames and stored into the Hadoop as HDFS using different compression.
• Create the Hive tables over HDFS compressed file to manage and process the data for various business or testing purpose.
• Create the Oozie job scheduler to load the data from end to end which is from Source to Hadoop various layers such as Raw, Interim and Semantic layer.
• Perform unit testing and performance testing to verify the correctness of data load and performance of data transfer.
• Maintain the security and organize the data in HDFS by following the organization standards and frameworks.
• Adapt the inhouse framework built over the Hadoop ecosystem for any new process being implemented in the organization.
• Work with security team to maintain the secure access of the data in Hadoop environment.
• People management and technical leadership for solving complex programming tasks.
• Involve in creating the implementation plan, performance metrics to support any production release and co-originate every implementation plan and post production validation.
• Prepare knowledge sharing documents, seek and leverage information from other projects/module to integrate and improve the efficiency of the system.
• Check in the code developed or scripts in GIT Hub for future use.

Bigdata Developer

Confidential

Responsibilities:

• Develop Abinitio jobs with components such as Rollup, Scan, join, Partition by key, Partition by Round Robin, Gather, Merge, Lookup, etc.
• Create reusable Abinitio graph to handle the ingestion, change data capture and loading to Teradata and HDFS, handle different source and targets using the psets by calling the graph.
• Create dynamic transformation xfr based on the requirement. Manage data volume to Teradata by using proper load operators.
• Performed the Map Reducer to load the data into Hive tables using Java
• Maintain the security and organize the data in HDFS by following the organization standards and frameworks. Restrict access to the data in Teradata based on the need by executing proper grants.
• Create transformation rules using spark SQL and Data frames. Transform the data and store the data into HDFS using different file formats and compressions.
• Adapt the inhouse framework built over the Hadoop ecosystem for any new process being implemented in the organization.
• Support the business for yearend Loan Application Register report filing for HMDA Reporting.
• People management and technical leadership for solving complex programming tasks.
• Involve in creating the implementation plan, control M execution plan, performance metrics to support any production release and co-originate every implementation plan and post production validation.
• Prepare knowledge sharing documents, seek and leverage information from other projects/module to integrate and improve the efficiency of the system.
• Follow Test-Driven Development and other agile standards as part of the agile scrum model.

Senior ETL Developer

Confidential

Responsibilities:

• Develop reusable Informatica jobs adhering to requirement and existing legacy logic. Designed and implemented automated services as dynamic parameter generation, Error and Audit Balance Control and automated validation. Created UNIX shell scripts for emailing the reports and to execute the Informatica workflow in a recursive loop based on input from customer. Create labels and tags for code migration and schedule using the Informatica scheduler.
• Develop Abinitio jobs to read source file, ingest, transform and load into HDFS. Code migration using Express IT and scheduling using control M. Develop reusable Abinitio code, custom dynamic transformation, use proper Abinitio component, handle the Teradata load operator.
• Reverse engineering of legacy application to latest technology with no impact to business, implemented PL/SQL to Informatica and Informatica to Abinitio-Hadoop projects.
• Handled HDFS, Teradata, Oracle, SQL server as source and target with Abinitio and Informatica.
• Experience in creating and handling complex SQL queries in Teradata database, SQL performance tuning, relational model analysis.
• Performed the role of the Data Analyst for data profiling and prepared the mapping specifications for the ETL jobs.
• Design project experience, starting from the analysis, preparing the design document including the ETL architecture with Audit and Error handling. Data modelling, creation of the scripts, test cases and the mapping documents.
• Prepared automated regression testing for the jobs where the complete ETL logic is framed into SQL queries which are executed against the target for validation.
• Involved in release management to code deployed in production, tracking the release, publishing the release status/progress. Prepare and review the Implementation plans and supporting documents for every release.
• Document High and low-level design, mapping specification, implementation plan, control M execution, process flow of the batch.
• Co-ordination within the team for the project deliverables. Organize daily stand up calls with Onsite-Offshore to get the work completed before the scheduled time.