SUMMARY

• 6 years of professional IT experience, which includes hands on experience in Big Data Analytics and development.
• Expertise wif the tools in Hadoop Ecosystem including Pig, Hive, HDFS, MapReduce, Sqoop, Storm, Spark, Scala, Kafka, Yarn, Oozie, Zookeeper for complex business problems.
• Experience in using Scala and spark streaming for ongoing transactions for customers.
• Experience in developing, designing, testing and maintaining object - oriented application.
• Experience in writing complex MapReduce programs that work wif different file formats like Text, Sequence, Xml, parquet and Avro.
• Experience in migrating legacy Warehouse systems like Teradata, Netezza to Hadoop.
• Performed different Optimization techniques like distributed cache, map side joins, partitioning and bucketing in Hive. Leveraged Hive queries to create ORC tables.
• Executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business requirements. Strong knowledge of Pig and Hive's analytical functions, extending Hive and Pig core functionality by writing custom UDFs and macros.
• Worked on developing ETL processes to load data from multiple data sources to HDFS using Kafka and SQOOP, perform structural modifications using Map-Reduce, HIVE and analyze data using visualization/reporting tools.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and data processing.
• Loading data into Spark RDD and do in memory data Computation to generate the Output response.
• Expert in understanding the data and designing/Implementing the enterprise platforms like Hadoop Data lake and Huge Data warehouses. Experience in Oozie and workflow scheduler to manage Hadoop jobs by Direct Acyclic Graph (DAG) of actions wif control flows.
• Experience in building scalable and reliable data pipelines based on Big Data processing technologies like Hadoop, Map Reduce, Spark etc.
• Used various compression techniques like LZO, G-ZIP, and Snappy to save data and optimize data transfer.
• Experience in developing applications using all Java/J2EE technologies like Servlets, JSP, EJB, JDBC, SOAP, REST etc.
• Proficient in writing build scripts using Ant & Maven
• Experience in developing applications using HIBERNATE (Object/Relational mapping framework).
• Experience in all facets of Software Development Life Cycle (Analysis, Design, Development, Testing and Maintenance) using Waterfall and Agile methodologies.
• Experience working wif Apache SOLR for indexing and querying.
• Good understanding of Scrum methodologies, Test Driven Development and continuous integration.
• Ability to learn and adapt quickly and to correctly apply new tools and technology.
• Great verbal and composed relational abilities consolidated wif relational and conflict resolution skills and have solid analytical abilities.

TECHNICAL SKILLS

Big Data Ecosystem: HDFS, MapReduce, Yarn, Spark, Kafka, PIG, HIVE, Sqoop, Flume, Oozie, Impala, HBase, Hue,Storm,NIFI

Java& J2EE Technologies: Core Java, Servlets, JSP, JDBC, Java Beans

IDE’s: IntelliJ, Eclipse, Jupyter

OperatingSystems: Windows, Ubuntu

Programming languages: Python, Scala, Shell Scripting, SQL

Databases: Oracle 11g/10g/9i, MySQL, DB2, MS-SQL Server, HBASE

Visualization: Tableau, Qlikview

Version Control: SVN, GIT

Hadoop Distributions: Cloudera, HortonWorks, EMR

PROFESSIONAL EXPERIENCE

Confidential - MI

Big Data Engineer

Responsibilities:

• Worked on Horton works Data Platform Hadoop distribution.
• Involved in data ingestion into HDFS using Sqoop for full and incremental load from RDBMS.
• Experienced in implementing Spark RDD transformations and writing queries in Spark SQL using Scala and Python to implement business logic.
• Extensive experience in Spark through core Spark API running Scala & Python Scripts to transform raw data from several data sources into forming baseline data.
• Ingested real time/ near real time data using Kafka &Spark Streaming.
• Developed frameworks using Spark and Scala for Data Quality and Integrity Checks.
• Developed shell script to run business rules against incoming data which generate dynamic Hive Queries.
• Worked on Sequence files, RC files, Map Side Joins, Bucketing, Partitioning for Hive performance enhancement and storage improvement. Leveraged Hive queries to create ORC tables.
• Experience in integrating Spark wif Elastic Search.
• Load and transform large sets of structured, semi structured and unstructured data even joins and some pre-aggregations before storing data into HDFS.
• Created Hive and Tez views and providing permission to user and AD groups on Ambari server.
• Implemented POC to stream the data from log servers and other streaming sources using NIFI.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports by our BI team
• Developed and maintained the continuous integration and deployment systems using Jenkins, ANT and Maven.
• Developed UNIX shell script to run the spark submits and triggers it through Oozie flow.
• Developed a POC for creating complex workflow using Apache Airflow so as to decommission Oozie.
• Provided Level 3 Production Support.

Environment: Hadoop, Map Reduce, Spark, Spark SQL, Kafka, HDFS (Yarn), Hive, Pig, Sqoop, Oozie, Python, GIT, SQL, Shell script, Scala, Storm, Hbase, Elastic Search.

Confidential

Software Engineer-Big Data

Responsibilities:

• Involved in complete project life cycle starting from design discussion to production deployment.
• Worked closely wif the business team to gather their requirements and new support features.
• Developed Map Reduce Program for Generating Unique key for every incoming new record (Universal Key Generator).
• Worked on Data comparison tool (Recon Tool) which is used to compare two datasets.
• Worked on Generating Dynamic Case Statements based on Excel provided by Business using Python.
• Extensively worked on Hive, Pig and Sqoop for sourcing and transformations.
• Worked on Avro and Parquet File Formats wif snappy compression.
• Explored various file structures (Text, Avro and Parquet file formats) and compression techniques (Snappy and Gzip) for designing the underlying data as per business needs.
• Worked on loading and transforming of large sets of structured, semi structured and unstructured data.
• Experience in using Sqoop to import (IDL, Delta Loads) and export the data from Netezza and Oracle DB into HDFS and HIVE.
• Implemented calculated columns in Hive as per Business requirements.
• Developed and implemented hive and spark custom UDFs involving date functions.
• Created Partitioning, Bucketing, Map side Join, Parallel execution for optimizing the hive queries.
• Worked in transforming data from HBase to Hive as bulk operations.
• Worked wif Control-M/Oozie scheduler to automate the jobs.
• Performed POC on Spark SQL, Hive On Spark and Impala engine, finally Implemented the ETL on Spark SQL.
• POC on Migrating Map Reduce programs into Spark transformations using Spark SQL and Scala.
• Worked on Spark Streaming and Spark SQL to run sophisticated applications on Hadoop.
• Experience in working on Spark SQL queries, Data frames, import data from Data sources, perform transformations, perform read/write operations, save the results to output directory into HDFS.
• Hands-on experience wif AWS (Amazon Web Services), using Elastic MapReduce (EMR), creating buckets in S3 and storing data in them
• Developed UDF's in both Data frames/SQL / RDD/MapReduce and Scala scripts in Spark 1.1 for writing data back into OLTP system, Data Aggregation and queries.
• Worked wif data pipeline using Sqoop to import customer's data and historical data from data sources such as HDFS, MySQL, Oracle and Teradata and achieved good experience on Impala.
• Worked on Data Serialization and HIVE serialization formats, which involves converting Complex objects into sequence bits by using CSV, PARQUET, JSON and AVRO formats.
• Exported the analyzed data to Relational databases using Sqoop to generate reports and visualization for the BI team.
• Worked on Developing custom MapReduce programs and User Defined Functions (UDFs) in Hive to transform the large volumes of data wif respect to business requirement.
• Driving POC initiatives for finding the feasibilities of different traditional and Big data reporting tools wif the data lake Spotfire BO, Tableau etc.
• Implemented ELK (Elastic Search, Log stash, Kibana) stack to collect and analyze the logs produced by the Spark cluster
• Good understanding on DAG cycle for entire Spark application flow on Spark application Web UI.
• Involved in implementing test cases, testing map reduce programs using MR Unit and another mocking frame works.
• Involved in implementing Maven build scripts, worked on maven projects and integrated wif Jenkins.
• Worked in complete Software Development Life Cycle (analysis, design, development, testing, implementation and support).

Environment: Hadoop 2.3, AWS, MapReduce, Hive, Spark, Avro, Linux, Sqoop, Shell Scripting, Oozie, Git, XML, Scala, Java, Maven, Eclipse, Oracle.

Confidential

System Analyst

Responsibilities:

• Developed and deployed UI layer logics of sites using JSP.
• Struts (MVC) is used for implementation of business model logic.
• Worked wif Struts MVC objects like Action Servlet, Controllers, and validators, Web Application Context, Handler Mapping, Message Resource Bundles and JNDI for look-up for J2EE components.
• Messaging and interaction of Web Services is done using SOAP and REST Worked wif Restful web services to enable interoperability.
• Developed POJOs and Java beans to implement business logic.
• Used Hibernate, DAO, and JDBC for data retrieval and modifications from database.
• Developed JUnit Test cases for Unit Test cases and as well as System and User test scenarios.

Environment: Servlets, JSPs, Java Beans, POJO, HTML, CSS, JavaScript, SQL, JDBC, JSTL, Stored Procedures.