SUMMARY:

• 8 years of expertise as a Data Engineer with Hadoop Ecosystem products such as HDFS, Hive, Spark, and Python modules in diverse business areas.
• HDFS, MapReduce, Pig, Hive, Scoop, Spark, Yarn, and ZooKeeper are all Hadoop Eco System tools dat me am familiar with.
• Experience in exporting and importing data from HDFS to Relational Database Management Systems (RDBMS) and from RDBMS to HDFS using Sqoop.
• Extensive Big Data Analytics noledge, including hands - on experience creating MapReduce tasks on the Hadoop Ecosystem, including Hive and Pig.
• Expertise in building RDD transformations, Actions, Data Frames and case classes for the required input data, as well as converting RDD to Data frames using Spark Context.
• Hands on Bash scripting experience and building data pipelines on Unix/Linux systems.
• AVRO, Parquet, CSV, XML, and JSON are among the file formats with which me'm familiar.
• Hands-on expertise with Cloudera Hadoop distributions, including installation, configuration, support, and management.
• Used Spark Data Frames API over Cloudera platform to perform analytics on Hive data.
• On AWS, designed and built scalable, highly available, and fault-tolerant systems.
• Experienced in designing, built, and deploying and utilizing almost all the AWS stack (Including EC2, S3, EMR), focusing on high-availability, fault tolerance, and auto-scaling.
• Experience in Migrating SQL database to Azure Data Lake, Azure SQL Database, Databricks, and Azure SQL Data warehouse.
• Controlling and granting database access and migrating on-premises databases to Azure Data Lake store using Azure Data factory.
• Working noledge with Tableau, including the creation of reports with the software.
• Expertise in using Python to create Spark applications.
• Expertise in Cloud Migration of Existing Applications and Data Feeds.
• Knowledge in installation, configuration, supporting and managing Hadoop Clusters using Cloudera (CDH3, CDH4) distributions and on Amazo

PROFESSIONAL EXPERIENCE:

Confidential

Sr Big Data Engineer

Responsibilities:

• Developed Spark scripts in Python, Scala in accordance with the requirements. Created Spark apps with Python APIs to insert data into Hive. Developed Python scripts for data aggregation, Querying, and writing data in Spark 1.6 and Spark 2.1, using both RDD and Data Frames/ SQL Data sets Handling huge datasets with Partitions, Spark in Memory capabilities, Broadcasts in Spark, TEMPEffective and efficient joins, Transformations, and other techniques during the data ingestion process. In the pipelines,
• Python and Scala scripts were used for Spark transformations. Extract Transform and Load data from source systems to Azure Data Storage services using a combination of Azure Data Factory, Spark SQL, and Azure Data Lake Analytics. Created Azure Data Factory pipeline to insert the flat file, Orc file data into Azure SQL. Cloud based report generation, development and implementation using SCOPE constructs and power BI. Expert in U - SQL constructs for interacting multiple source streams within Azure Data Lake.
• Working experience on Azure Databricks cloud to organizing the data into notebooks and making it easy to visualize data using dashboards. Data Ingestion to one or more Azure Services like Azure Data Lake, Azure Storage, Azure SQL, Azure Data Warehouse, and processing the data in Azure
• Databricks. Created/Enhanced Teradata Stored Procedures to generate automated testing SQLs. Experience in creating configuration files to deploy the SSIS packages across all environments. Good Knowledge/Understanding of NoSQL databases and hands-on work experience in writing applications on NoSQL databases like Cassandra and MongoDB. Assisted with the creation of Hive tables as well as the loading and analysis of data using Hive queries. Using Spark -SQL to convert SQL codes to Spark codes for faster data processing and testing. Developed Informatica mappings, sessions, and workflows to load transformed data into EDW from various source systems such as SQL Server, Teradata, and Flat Files. Worked on poc configuring Spark streaming data to receive real-time data from Kafka and store it in HDFS. Interacted with client teams to gain a better understanding of their deployment needs. Used shell scripts to automate the SQL-Spark conversion. Worked closely with data scientists to understand the requirements and deliver the desired data.

ENVIRONMENT: Scala, Spark SQL, Hive, Data Warehouse, Informatica, NoSQL, Teradata, ETL, MapReduce, Microsoft Azure, Cassandra, MongoDB, Apache Hadoop, Git.

Confidential

Big Data Engineer

Responsibilities:

• Gathered requirements from Users to Develop data pipelines from different sources to Hadoop. Maintaining and Designing Data governance and security for data platforms on AWS Cloud. Was responsible for creating on - demand tables on S3 files using Lambda Functions and AWS Glue using Python and PySpark. Worked on
• Terraform to create resources in cloud. Built POC on AWS Glue. Implemented and integrated Hadoop based business intelligence and Data Warehouse system including implementations of searching, filtering, indexing, aggregation for reporting and report generation and general information retrieval. Experience in moving high and low volume data objects from Teradata and Hadoop to Snowflake. Developed reusable framework to be leveraged for future migration dat automates ETL from RDBMS systems to the Data Lake utilizing Spark Data Sources and Hive data objects. Performed day-to-day integration with the Database Administrators (DBA) DB2, SQL
• Server, Oracle, and AWS Cloud teams to ensure the insertion of database tables, columns and its metadata has been successfully implemented out to the DEV, QUAL and PROD region environments in AWS Cloud - Aurora and Snowflake. Performed ETL data translation using informatica of functional requirements to Source to Target
• Data Mapping documents to support large datasets (Big Data) out to the AWS Cloud databases, Snowflake and Aurora. Deployed Instances, provisioned EC2, S3 bucket, Configured Security groups and Hadoop eco system for Cloudera in AWS. Experience in using distributed computing architectures like AWS products (e.g., AWS EC2, AWS Redshift, and AWS EMR, AWS S3) and working on raw data migration to Amazon cloud into S3 and performed refined data processing. Worked on AWS Data
• Pipeline to configure data loads from S3 to into Redshift. Used SSIS, Python scripts, Spark Applications for ETL Operations to create data flow pipelines and was involved in transforming data from legacy tables to Hive, and S3 buckets for handoff to business and Data scientists to create analytics over the data. Expertise in migrating existing applications, Informatica Data feeds, and ETL Pipelines to the Hadoop, snowflake, and AWS. Created a process in Hadoop to load 1 billion records in span of one hour to match the latency and get the behind data for one of the important Dataset for Honeywell Created SharePoint Pages to store all the documentation for Project. Created Map Reduce programs to analyze data of customer and provide summary results from Hadoop and integrated big data technologies and analysis tools into the overall architecture Created Hive tables to store the processed results in a tabular format. Created Spark Applications using some transformations as Map, Flat Map, join Intersect, groupByKey, ReduceByKey, join, coalesce to the RDD and exported the items to RDD to Data frames and External storage using RDD Action items as count, collect, tak

Data Engineer

Responsibilities:

• Gathered requirements, code, functionalities, technical specs from Team Lead and Business analysts. Worked extensively on Hive, SQL, Scala, Spark, and Shell, Python Worked lot of time working with Sqoop to import and export data to HDFS. Created Complex tables using Hive, loaded them with data, and wrote Hive queries dat called and ran MapReduce processes in the backend. To meet the business needs, used Spark and Hive window functions to execute sophisticated data calculations. Multiple
• Hive UDFS has been written using OOPS concepts as well as spark functions in Python projects. To has the highest performance and querying speed, me used Hive to create partitioned tables. Created Map Reduce programs to analyze data of customer and provide summary results from Hadoop. Worked with scoop to import/export data from relational database to Hadoop and flume to collect data and populate in Hadoop. Performed data validation with Redshift and constructed pipelines designed over 100TB per day.
• Data processing experience with Apache Kafka (gathering, aggregating, and transferring data). Integrating Big data technologies and analysis tools into the overall architecture. Assisted senior - level data scientists in the design of ETL processes, including SSIS packages. Deployed the reports on to server and managed subscriptions to deliver the reports to end user based on Granularity. Developed scripts for end-to-end data management and synchronization across all clusters. Involved in Agile techniques, daily scrum meetings, and spring planning.

ENVIRONMENT: SQL, Hive, Scala, Python, PySpark, Kafka, Sqoop, Hadoop, Map Reduce, flume, Redshift, SSIS, ETL.

Confidential

Hadoop Developer

Responsibilities:

• Gathering requirements, Design, functional & technical specification, Coding, testing, and Document preparation. Involved in requirement analysis, design, development, testing, and documentation. Technical design documents were created to list the extraction, transformation, and loading processes as well as business rules. Worked with business analysts to convert new business needs into technical specifications. Sqoop was used to extract data files from MySQL and Oracle, which were tan placed in HDFS and processed
• Created Hive tables, loaded them with data, tan wrote Hive queries dat would run internally in a map reduce fashion. Wrote Hive queries for data analysis to meet the business requirements. Developed ETL Packages using SSIS to populate data from different sources, for example flat - files/SQL/Oracle Server to the Staging Database and tan into the Data Warehouse.
• Thepackages created included a variety of transformations, for example, Slowly Changing-Dimensions, Look up, Aggregate, Derived Column, Conditional Split, Fuzzy Lookup, Multicast, and Data Conversion. Data loading from UNIX file system to HDFS was one of my responsibilities. Responsibilities include Database Design and Creation of User Database.
• Worked on tuning SQL queries by adding the missing indexes. Evaluated suitability of Hadoop and its ecosystem to the above project and implementing / validating with various proof of concept POC applications to eventually adopt them to benefit from the Big Data Hadoop initiative. Using the Spark for Research Using Spark Context, Spark-SQL, Data Frame, Pair RDD's, and Spark YARN, we can improve the performance and optimization of existing Hadoop methods. Weekly meetings with technical collaborators, as well as active engagement in senior and junior developer code review sessions.

ENVIRONMENT: Hive, Spark, Unix, Oracle, HDFS, Spark-SQL, ETL, SSIS (SQL service Integration Services), MS Excel.