We provide IT Staff Augmentation Services!

Big Data Engineer Resume

SUMMARY:

  • 12 years of experience in software industry with good knowledge in Agile development environment. Highly experience in DWH projects and having good knowledge on Data Warehousing Architecture , Test - driven development ( TDD ), Guidewire ETL integration & Bigdata ETL integration.
  • An enthusiastic and project-oriented team player with sound leadership and problem solving skills to develop creative solution for challenging customer needs.
  • Designed and coded application components in an Agile environment utilizing a test driven development approach.
  • Hands on experience in end to end warehouse implementation which includes gathering client requirements, Analysis, Design, Coding and Testing.
  • Expert in Informatica tool like designer, workflow manger, workflow monitor, repository manger etc.
  • Integrated Hadoop with traditional ETL systems and performed extraction, transformation, and loading of massive structured and unstructured data.
  • Optimized existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames.
  • Exploring with the Spark improving the performance and optimization of the existing algorithms in Hadoop.
  • Converted text files into AVRO and loading into Hive tables.
  • Experience in Understanding project issues and ability to work independently or as a part of a team with excellent communication skills and analytical, presentation skills.
  • Provided the solution for the design changes, build the informatica data quality mapplets and powercenter mappings for the Customer.
  • Have hands on experience in Designing ETL & Data Integration Solutions, Development and Support using Informatica PowerCenter.
  • Worked on Data profiling for Data quality process.
  • Implemented best practices in DWH projects like naming standards, performance oriented and efficient mapping design.
  • Responsible for development and implementation of PL/SQL Procedures, Functions as per client requirement.
  • Have experience in adherence to coding standards & best practices using Informatica Power Center.
  • Imported and exported data into HDFS/HBase using scoop.
  • Responsible for managing tasks and deadlines for onsite and offshore ETL teams
  • Worked on Business Objects designer, Webi reports, report testing, report requirement analysis.
  • Experience in on impact analysis and map & gas analysis.
  • Worked in different role models like Software Quality Analyst, Team Lead and managerial roles.

TECHNICAL SKILLS:

Hadoop/Big Data: Spark1.6, HDFS, Mapreduce, HBase, Hive,Sqoop, Oozie .

ETL:: Informatica PowerCenter 9.x/8.x/7.x/6.x

RDBMS:: Oracle,Netezza,DB2

Data Quality Tool:: Informatica Developer,IDQ 9.6.1

Virtualization Tool:: Denodo platform 5.5

Languages/Utilities:: PL/SQL, SQL, PERL, Ruby, Python 2.5, Scala 2.11

Reporting Tool:: Business Objects XIR2

Other Software Tools:: TOAD, RubyMine, Harvest, GitHub

Industry Knowledge:: Finance, Aero, Telecommunications, Insurance, Pharma.

PROFESSIONAL EXPERIENCE:

Confidential

Big Data Engineer

  • Loaded files from LexisNexis to Hive tables using spark
  • Created Sqoop tasks to transfer data from DWH to Hive tables.
  • Scheduled Hive tasks using Ooze workflow tasks.
  • Designed Hive tables for the various subject areas like Customer driving habits, Driving time etc.
  • Importing and exporting data into HDFS/HBase using scoop.
  • Designed and created Spark data frames and RDD for the quick data analysis.
  • Worked on Hadoop POC in comparison with existing ETL jobs.
  • Flat legacy data in to Hive databases .
  • Responsible for loading data from UNIX file systems to HDFS. Installed and configured Hive and written Hive UDFs
  • Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames.
  • Worked with NoSQL databases like HBase in creating HBase tables to load large sets of semi structured data coming from various sources.
  • Scheduled Hive jobs and spark jobs using control M
  • Prepared technical specifications to develop Informatica ETL mappings to load data into various tables confirming to the business rules.

Confidential

ETL Architect and ETL Informatica Lead

  • Creation and maintenance of Logical and Physical Data Model diagrams.
  • Understand business and technical concepts to create optimal data design solutions
  • Working in designing, development and data modeling in business intelligence, analytics and data warehousing environment.
  • Extensively used Informatica Power Center to load data from Flat Files to DB2, Flat Files to SQL Server, DB2 to XML files, Flat Files to Oracle.
  • Involved and develop several complex mappings in Informatica a variety of PowerCenter transformations, Mapping Parameters, Mapping Variables, Mapplets & Parameter files in Mapping Designer using Informatica PowerCenter.
  • Provide the post production implementation support and work with Operations team to provide them the KT for continuous Application Support & Maintenance.
  • Facilitate any performance tuning activity for Informatica Workflows.
  • Work closely with cross-functional teams to design and build data stores, data flows, ETLs and related data management application/systems that serve both internal and external business.
  • Hands on interaction, technical evaluation, project support, and the establishment of patterns and standards with the various and multiple emerging data products and technologies.
  • Mentor junior associates.
  • Created Informatica mappings as per technical specifications.
  • Analyzing defects and incidents and document it.
  • Worked on database environment setting for System test and Performance test etc.
  • Responsible for on call and on call rotation.
  • Coordinating Production deployment.
  • Working with project teams to implement processes which result in a high degree of data quality and integrity, and assists in the resolution of data quality issues from source systems.

Hire Now