SUMMARY

• Confidential developer with good working experience in Designing and Building high performance and scalable systems using Big Data Ecosystem on windows and Linux environments.
• Experience in different Hadoop distributions like Confidential and Horton WorksDistributions ( Confidential ).
• Experience in installing Hadoop cluster using Apache Hadoop Hortonworks.
• Experience on Apache Hadoop technologies Hadoop distributed file system (HDFS), MapReduce framework, YARN, Pig, Hive, Sqoop, and Flume.
• Expertise in developing Map - Reduce programs to perform Data Transformation.
• Have hands on experience in writing Map Reduce jobs on Hadoop Ecosystem including Hive and Pig.
• Hands on experience in installing, configuring and using ecosystem components like Hadoop MapReduce, HDFS, Pig, Hive, and Sqoop.
• Good knowledge of Hadoop architecture and various Hadoop Stack elements.
• Familiarity on real time streaming data with Spark for fast large scale in memory MapReduce.
• Experience in developing pipelines and processing data from various sources and processing them with Hive and Pig.
• Good understanding of NoSQL databases and hands on work experience in writing applications on NoSQL databases like HBase.
• Experience in Hadoop Development in AWS Environment with EMR.
• Good knowledge of SQL server, SSIS and SSRS reports and Mysql.
• Hands on Experience in development and deployment of custom Hadoop application in both in house Clusters and Cloud based environment.
• Exploring with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
• Good understanding and ability to learn, apt new skills and good communicator with team spirit minded.

TECHNICAL SKILLS

Big Data: Apache Hadoop,HDFS,MapReduce,Hive,PIG,OOZIE,HUE,SQOOP, Spark, Confidential manager and Hortonworks

Database: MYSQL, Oracle, SQL Server, Hbase, Hive

IDEs: Eclipse

Languages: Java, PIG LATIN, Unix shell scripting, Python( Basics)

Reporting Tool: Talend 6.3-6.5.1, Tableau, Jasper

Web Technologies: HTML, XML, JavaScript, J query

WORK EXPERIENCE:

BigData Engineer

Confidential, Benotnville,AR

Responsibilities:

• Migrated Data from Existing Teradata to Hive and created a staging in Hive Data warehouse.
• Developed yaml files scripts to load the data into Hive Data lake.
• Created Talend workflows to load data from source Oracle OMS system to Teradata and from Teradata to external reporting systems.
• Used internal application to sqoop the data from teradata and automated thru one automic.
• Created Jobs, Workflows to schedule the loads using UC4 Automic client.
• Developed multiple POCs using PySpark and deployed on the Yarn cluster, compared the performance of Spark, with Hive and SQL/Teradata.
• Developed code in reading multiple data formats on HDFS using PySpark
• Implemented Spark using Scala and also used Pyspark using Python for faster testing and processing of data.
• Converted Hive SQL into Spark transformations using Scala and PySpark.

Environment: Sqoop, Hive, Teradata, Shell, Pyspark, Talend, Scala.

Hadoop Developer

Confidential, IL

Responsibilities:

• Designed, planned and delivered a proof of concept for business functions.
• Used Sqoop to migrate data from HDFS to My SQL and vice versa.
• Automated batch loading jobs using JAMS automation scheduler with shell script.
• Involved in loading and transforming large sets of structured and semi-structured data from relational databases into
• HDFS using Sqoop imports created Hive tables using HiveQL, ingesting data and automation for batch load with static and dynamic Partitions.
• Hands on working closely with business analysts and clients to meet their requirements day to day
• Used pig latin script for cleansing the data form ASCII format.
• Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
• Hands on EBCDIC data transformation and conversion to UTF format using Java.
• Wrote UDF on Hadoop for client custom requirements.
• Executed adhoc reports using talend and schedule thru to JAMS automation.
• Responsible for multiple projects for tableau dash board and adhoc reports and ingesting data into Hadoop.
• Designed a standard template for pre-job and post specifications on Talend 6.3.0
• Installed and configured Hadoop Horton works platform for UAT.
• Hands on Jasper as beginner and executed multiple pdf reports.
• Worked extensively on using processing components like, tAggregate, tSortRow, tFixed FlowInput, tMap, tJoin, tLogRow, tUnique in Talend.
• Sqooped the data from sql, cleansed the data using pig script in hdfs moved to Hive, applied business filters and sqooped to mysql database.
• Created dynamic partitioned tables in strict mode to break the existing sql pipeline and generated reports using Jasper.
• Developed PIG Latin scripts for handling business transformations
• Developed UDF functions and implemented it in HIVE Queries.
• Worked with NoSQL databases like HBase in creating HBase tables to load large sets of semi structured data coming from various sources.

Environment: MapReduce, HDFS, Hive, Pig, Bash shell, PowerShell, Sqoop, Java, TFS, MySQL, Sql, Eclipse, Jams Scheduler, Talend, Jasper.

Hadoop/Spark Developer

Confidential, Atlanta, GA

Responsibilities:

• Developed PIG scripts to transform the raw data into intelligent data as specified by business users.
• Worked in AWS environment for development and deployment of Custom Hadoop Applications.
• Worked closely with the data modelers to model the new incoming data sets.
• Involved in start to end process of Hadoop jobs that used various technologies such as Sqoop, PIG,
• Hive, MapReduce, Spark and Shell scripts(for scheduling of few jobs ) Extracted and loaded data into
• Data Lake environment (Amazon S3) by using Sqoop which was accessed by business users and data scientists.
• Expertise in designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase, Oozie, ZooKeeper, SQOOP, flume, Spark, Impala, Cassandra with Hortonwork Distribution.
• Installed Hadoop, Map Reduce, HDFS, AWS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
• Assisted in upgrading, configuration and maintenance of various Hadoop infrastructures like Pig, Hive, and Hbase.
• Used Spark API over Hortonworks Hadoop YARN to perform analytics on data in Hive.
• Exploring with the Spark improving the performance and optimization of the existing algorithms in
• Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
• Developed Spark code using scala and Spark-SQL/Streaming for faster testing and processing of data.
• Import the data from different sources like HDFS/Hbase into Spark RDD.
• POC on Single Member Debug on Hive/Hbase and Spark.
• Configured deployed and maintained multi-node Dev and Test Kafka Clusters.
• Performed transformations, cleaning and filtering on imported data using Hive, Map Reduce, and loaded final data into HDFS.
• Load the data into Spark RDD and do in memory data Computation to generate the Output response.
• Loading Data into Hbase using Bulk Load and Non-bulk load.

Environment: MapReduce, HDFS, Hive, Pig, Spark, Spark-Streaming, Spark SQL, Apache Kafka, Sqoop, Java, Scala, CDH4, CDH5, AWS, Eclipse, Oracle, Git, Shell Scripting and Cassandra.

Hadoop Developer

Confidential, IL

Responsibilities:

• Optimized Map/Reduce Jobs to use HDFS efficiently by using various compression mechanisms.
• Responsible for building scalable distributed data solutions using Hadoop.
• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS.
• Handled importing of data from various data sources, performed transformations using Hive,
• MapReduce, loaded data into HDFS and extracted the data from Oracle into HDFS using Sqoop.
• Analyzed the data by performing Hive queries and running Pig scripts to study customer behavior
• Installed and configured Confidential Manager for easy management of existing Hadoop cluster.
• Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.
• Responsible for managing and reviewing Hadoop log files. Designed and developed data management system using MySQL.
• Developed entire frontend and backend modules using Python on Django Web Framework.
• Wrote Python scripts to parse XML documents and load the data in database.
• Cluster maintenance as well as creation and removal of nodes using tools like Confidential Manager Enterprise, and other tools.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
• Worked on NoSQL databases including HBase and Elastic Search.
• Worked on importing and exporting data from Oracle and DB2 into HDFS and HIVE using Sqoop.
• Used Tableau as reporting tool as data visualization tool.
• Involved in Java, J2EE, Struts, Web Services and Hibernate in a fast-paced development environment.
• Followed Agile methodology, interacted directly with the client provide/take feedback on the features, suggest/implement optimal solutions, and tailor application to customer needs.
• Setting up proxy rules of applications in Apache server and Creating Spark SQL queries for faster requests.
• Designed and Developed database design document and database diagrams based on the requirements.
• Designed the XML Schema for data transmission using xml documents.
• Wrote Unit tests using JUnit.
• Responsible and involved in the start to end process of Hadoop cluster installation, configuration and monitoring.

Environment: HDFS, Hive, PIG, UNIX, SQL, Java, MapReduce, SPARK Hadoop Cluster, Hbase, Sqoop, OozieLinux, Data Pipeline, Confidential Hadoop Distribution, Python, MySql, Git. MapR-DB.