PROFESSIONAL EXPERIENCE:

Confidential

Sr. Big Data Engineer / Technical Lead

Responsibilities:

• Responsible for Data Lake Architecture, Hadoop, Big data, data and BI requirements and defining teh strategy, technical architecture, implementation plan, management and delivery of Big Data applications and solutions Built S3 buckets and managed policies for S3 buckets and used S3 bucket and Glacier for storage and backup on AWS.
• Worked wif other teams to help develop teh Puppet infrastructure to conform to various requirements including security and compliance of managed servers. Designed, built and coordinate an automated build & release CI/CD process using GitLab, Jenkins and Puppet on hybrid IT infrastructure. Designing & develop Amazon EC2, Amazon S3, Amazon RDS, Amazon Elastic Load Balancing, Amazon SWF, Amazon SQS, and other services of teh AWS infrastructure Implement Data Engineering projects using data pipelines, QDC ETL Process, UNIX, SQL, Scala, NoSQL and RDBMS. Ingesting wide variety of data like structured, unstructured and semi structured into teh big data eco systems wif batch processing, real time streaming and SQL. Worked on scalable solutions on DW, Big data, Hadoop, BI platforms using AWS Redshift, Oracle,
• MySQL, QDC, TalenD DI, Hadoop, HBase, Hive, Hadoop Ecosystem, Amazon S3, EC2. Influence cross functional architecture in sprint planning Gather and process raw data at scale from internal and external data sources and expose mechanisms for large scale parallel processing Design, implement and manage a near real - time ingestion pipeline into a data warehouse and Hadoop data lake. Interacted wif internal stakeholders to understand client requirements and Performing ETL functions on incoming data-cleanse, format, structure, etc. Migrated data from on prem sources to AWS cloud and Designed as needed and build data pipelines Processed unstructured data into a form suitable for analysis and tan empower state-of-teh-art analysis for analysts, scientists, and APIs Solved complex SQL and Big Data Performance challenges. Mitigated Risks in our data infrastructure by developing teh best in class tools and processes. Implement controls, policies, processes and best practices in teh
• Data Engineering space and this automation job completely done on YARN cluster. Automate teh data flow using Glue and teh Control-M and followed teh process in daily manner automatically and setup teh jobs to run automatically using Control M. Sqoop teh data from SQL/Oracle DB onto out HDFS cluster and transferred teh data using QDC/ETL tools from AWS S3 in to Data Lake. Worked on Hive UDF's and due to some security privileges me has to ended up teh task in middle itself

Environment: AWS, EC2, S3, QDC, EMR, HDFS, Hive 2.x, Sqoop, Map Reduce, YARN, Agile Methodology, JIRASQL Server 2016/2014, Control-Glue, AutoSys

Confidential

Sr. Big Data Engineer / Technical Lead

Responsibilities:

• Worked on scalable solutions on DW, Big data, Hadoop, BI platforms using AWS Redshift, MySQL, Java, Scala, python Hadoop, Spark, Kafka, Oozie, HDFS, Yarn, Tableau, HBase, Hive, Hadoop Ecosystem, Amazon S3, EC2, Microsoft Azure HDInsight, Hortonworks. Implementation of Big data solutions on teh AWS Cloud platform and Azure cloud platform. Re - designed and developed a critical ingestion pipeline to process over 300 TB of data. Provided technical expertise and created software design proposals for upcoming components.
• Utilized MapReduce, HDFS, Hive & MongoDB. Integrated Hadoop into traditional ETL, accelerating teh extraction, transformation, and loading of massive structured and unstructured data. Worked on Data Science packages, and data tool kits in Python or R Loaded teh aggregate data into a relational database for reporting, dash boarding and ad-hoc analyses, which revealed ways to lower operating costs and offset teh rising cost of programming
• Developed Scala scripts, UDF's using both Data frames/SQL and RDD/MapReduce in Spark 2.0.0 for Data Aggregation, queries and writing data back into RDBMS through Sqoop. Created hive schemas using performance techniques like partitioning and bucketing. Loaded and extracted teh data using Sqoop 1.4.6 from Oracle 12.1.0.1 into HDFS. Analyzed large data sets on Hadoop by using Impala 2.1.0 to study issues of delivery trucks As ETL Tech Lead, responsible for design implementation, process, analysis and developing new ETL codes Assisted Business Analyst wif drafting teh requirements, implementing design and development of various components of ETL for various applications.
• Created Complex ETL Mappings to load data using transformations like Source Qualifier, Sorter, Aggregator, Expression, Joiner, Dynamic Lookup, and Connected and unconnected lookups, Filters, Sequence, Router and Update Strategy Implemented Slowly Changing Dimensions (SCD, Both Type 1 & 2). Used Jira 6.4 for project tracking, Bug tracking and Project Management. Involved in Scrum calls, Grooming and Demo meeting

Environment: AWS, EC2, EMR, HDFS, SPARK 2.0.0, Hive 2.x, Sqoop, Map Reduce, Oozie, Netezza, YARN, Agile Methodology, JIRA, Informatica 10.1, Oracle, Linux, SQL SERVER 2016/2014, AutoSys