Hadoop Developer Resume
PROFESSIONAL EXPERIENCE:
Confidential
HADOOP DEVELOPER
Responsibilities:
- Experience in AWS platform and its features including IAM, EC2 and Lambda and deployed AWS Lambda code from Amazon S3 buckets. Created a Lambda Deployment function and configured it to receive events from your S3 bucket. Responsible for building scalable distributed data solutions on Cloudera distributed Hadoop. Involved in using spark streaming and SPARK jobs for ongoing transactions of customers and Spark SQL to handle structured data in Hive. Involved in migrating tables from RDBMS into Hive tables using SQOOP and later generating visualizations using Tableau. Create and maintain automated ETL processes with special focus on data flow, error recovery, and exception handling and reporting Conduct business/data flow modeling and generate applicable scenarios for teh technology functionality testing team
Involved in gathering requirements from client and estimating timeline for developing complex queries using HIVE and IMPALA for logistics application. Written PySpark code to calculate aggregate data like mean, Co - Variance, Standard Deviation and etc. Worked on analyzing Hadoop cluster and different Big Data Components including Pig, Hive, Storm, Spark, HBase, Kafka, Elastic Search, database and SQOOP. Integrated Kafka with Flume in sand box Environment using Kafka source and Kafka sink. Installed Hadoop
Map Reduce, HDFS, and developed multiple Map-Reduce jobs in PIG and Hive for data cleaning and preprocessing. Complete project documentation including diagram, data flow, status and usage reports, support, and escalation processes. Involved in Developing Insight Store data model for Cassandra which was utilized to store teh transformed data Responsible for writing Hive Queries for analyzing data in Hive warehouse using Hive Query Language (HQL). Written UDF in Scala and used in sampling of large data sets. In data exploration stage used hive and impala to get some insights about teh customer data. Successfully secured teh Kafka cluster with Kerberos
Implemented Kafka Security Features using SSL and without Kerberos. Further with more grain-fines Security set up Kerberos to has users and groups dis will enable more advanced security features and Integrated Apache Kafka for data ingestion. Implementation of highly scalable and robust ETL processes using AWS (EMRCloudWatch, IAM EC2, S3, Lambda Functions, DynamoDB). Used distinctive data formats while stacking teh data into HDFS. Worked in AWS environment for development and deployment of custom Hadoop applications. Worked on NiFi workflows development for data ingestion from multiple sources. Involved in architecture and design discussions with teh technical team and interface with other teams to create efficient and consistent Solutions. Involved in creating Shell scripts to simplify teh execution of all other scripts (Pig, Hive, Sqoop, Impala and MapReduce) and move teh data inside and outside of HDFS. Creating files and tu
