SUMMARY

• AWS Certified Developer Associate with over 7+ years of experience in the IT industry with Big Data implementing complete Hadoop solutions along with Scala and Python.
• Good working experience in using Apache Hadoop ecosystem components like MapReduce, HDFS, Yarn, Hive, Sqoop, Pig, Oozie, Flume, HBase and Zookeeper.
• Extensive experience working on spark in performing ETL using Spark Core, Spark - SQL and Real-time data processing using Spark Streaming.
• Expertise in working with RDD’s, Data Frames and DataSet API’s.
• Extensively worked with Kafka as middleware for real-time data pipelines.
• Strong knowledge on performance tuning Hive queries and troubleshooting various issues related to Joins, memory exceptions in Hive.
• Exceptionally good understanding of partitioning, bucketing concepts in Hive and designed both Managed and External tables in Hive.
• Strong experience using different columnar file formats like Avro, ORC and Parquet formats.
• Experience in Hadoop Distributions: Cloudera and Hortonworks,
• Experience in job workflow scheduling tools like Oozie and Airflow.
• Extensive knowledge in using SQL Queries for backend database analysis.
• Strong knowledge in NoSQL column oriented databases like HBase, Cassandra and its integration with Hadoop cluster.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems (RDBMS) and vice-versa.
• Led many Data Analysis & Integration efforts involving HADOOP along with ETL.
• Hands on experience on Enterprise Data Lake to provide support for various use cases including Analytics, processing, storing and Reporting of voluminous, rapidly changing, structured and unstructured data.
• Transferred bulk data from RDBMS systems like Teradata into HDFS using Sqoop.
• Strong experience working with AWS Cloud services like S3, EMR, Redshift, Athena, Glue metastore etc.,
• Strong experience in analyzing large amounts of data sets writingPySparkscripts and Hive queries.
• Worked in developing a Nifi flow prototype for data ingestion in HDFS.
• Well-versed in Agile, other SDLC methodologies and can coordinate with owners and SMEs.
• Hands on experience in IDEs Eclipse, IntelliJ, and version controller repositories like GitHub.
• Worked on different operating systems like UNIX, Linux, and Windows
• Diverse experience utilizing Java tools in business, Web, and client-server environments including Java Platform, Enterprise Edition (Java EE), Enterprise Java Bean (EJB), JavaServer Pages (JSP), Java Servlets (including JNDI), Struts, and Java database Connectivity (JDBC) technologies.
• Fluid understanding of multiple programming languages, including C#, C, C++, JavaScript, HTML, and XML.

TECHNICAL SKILLS

Hadoop Core Services: HDFS, Map Reduce, Spark, YARN

Hadoop Distribution: Horton works, Cloudera, Apache

NO SQL Databases: HBase, Cassandra

Hadoop Data Services: Hive, Pig, Sqoop, Flume

Hadoop Operational Services: Zookeeper, Oozie, Airflow

Monitoring Tools: Cloudera Manager

Cloud Computing Tools: Amazon AWS

Languages: C, Java/J2EE, Python, SQL, PL/SQLHive QL, Unix Shell Scripting, Scala

Java & J2EE Technologies: Core Java, Servlets, Hibernate, Spring, Struts

Application Servers: Web Logic, Web Sphere, JBoss, Tomcat

Databases: Oracle, MySQL, PostgreSQL, Teradata

Operating Systems: UNIX, Windows, LINUX

Build Tools: Jenkins, Maven, ANT

Development Tools: Microsoft SQL Studio, Toad, Eclipse, IntelliJ, NetBeans

Development methodologies: Agile/Scrum

Visualization and analytics: Tableau Software, QlikView, Qlik Sense

PROFESSIONAL EXPERIENCE

Confidential, Dallas, TX

Bigdata Developer

Responsibilities:

• Imported the data from various formats like JSON, Sequential, Text, CSV, AVRO and Parquet to HDFS cluster with compressed for optimization.
• Developed software code using C# programming language to meet design specifications, and user requirements as well as delivered high-quality applications within specified deadlines.
• Worked on ingesting data from RDBMS sources like - Oracle, SQL Server and Teradata into HDFS using Sqoop.
• Loaded all data-sets into Hive from Source CSV files using spark and Cassandra from Source CSV files using Spark
• Created environment to access Loaded Data via spark SQL, through JDBC&ODBC (via Spark Thrift Server). Developed real time data ingestion/ analysis using Kafka / Spark-streaming.
• Configured Hive and written Hive UDF's and UDAF's. Also, created Static and Dynamic with bucketing as required.
• Worked on writing Scala programs using Spark on Yarn for analyzing data.
• Managing and scheduling Jobs on a Hadoop cluster using Oozie.
• Written Hive jobs to parse the logs and structure them in tabular format to facilitate effective querying on the log data.
• Developed Oozie workflow for scheduling and orchestrating the ETL process and worked on Oozie workflow engine for job scheduling.
• Managed and reviewed the Hadoop log files using Shell scripts.
• Write Hive scripts to aggregate / transform core data and prepare for Solr indexing
• Migrated ETL jobs to Pig scripts to do Transformations, even joins and some pre-aggregations before storing the data into HDFS.
• Using Hive join queries to join multiple tables of a source system and load them to Elastic search tables.
• Real time streaming, performing transformations on the data using Kafka and Kafka Streams.
• Built NiFi dataflow to consume data from Kafka, make transformations on data, place in HDFS & exposed port to run Spark streaming job.
• Developed Spark Streaming Jobs in Scala to consume data from Kafka Topics, made transformations on data and inserted to HBase.
• Implemented Spark using Scala and SparkSQL for faster testing and processing of data.
• Experience in managing and reviewing huge Hadoop log files.
• Collected the logs data from web servers and integrated it into HDFS using Flume.
• Expertise in designing and creating various analytical reports and Automated Dashboards to help users to identify critical KPIs and facilitate strategic planning in the organization.
• Involved in Cluster maintenance, Cluster Monitoring and Troubleshooting
• Loaded all data-sets into Hive from Source CSV files using spark and Cassandra from Source CSV files using Spark/PySpark.
• Completed data extraction, aggregation and analysis in HDFS by usingPySparkand store the data needed to Hive.
• Developed Python code to gather the data from HBase (Cornerstone) and designs the solution to implement usingPySpark.
• Maintaining technical documentation for each and every step of the development environment and launching Hadoop clusters.
• Worked on different file formats like Parquet, Orc, Avro, Sequence files using MapReduce/Hive/Impala.
• Worked with Avro Data Serialization system to work with JSON data formats.
• Worked with the Data Science team to gather requirements for various data mining projects.
• Wrote shell scripts for rolling day-to-day processes and it is automated.
• Involved in building applications using Maven and integrated with Continuous Integration servers like Jenkins to build jobs.
• Worked on BI tools as Tableau to create dashboards like weekly, monthly, daily reports using tableau desktop and publish them to HDFS cluster.

Environment: Spark, Spark SQL, Spark Streaming, Scala, Kafka, Hadoop, HDFS, Hive, Oozie, Pig, Nifi, Sqoop, AWS (EC2, S3, EMR), Shell Scripting, HBase, Jenkins, Tableau, Oracle, MySQL, Teradata and AWS.

Confidential

Hadoop Developer

Responsibilities:

• Installed & configured multi-node Hadoop Cluster and performed troubleshooting and monitoring of Hadoop Cluster.
• Developed software applications using C++ and other object-oriented languages
• Installed and configured Oozie on the Hadoop cluster.
• Managing, defining and scheduling Jobs on a Hadoop cluster.
• Worked on installing cluster, commissioning & decommissioning of datanode, namenode recovery, capacity planning, and slots configuration.
• Resource management of Hadoop Cluster including adding/removing cluster nodes for maintenance and capacity needs.
• Wrote Hive queries for ad-hoc reporting and ETL.
• Worked with different file formats such as Text, Sequence files, Avro, ORC and Parquet.
• Pushing logs onto Hadoop cluster via SOAP UI calls.
• Involved in upgrading all the Hadoop components to the latest versions.
• Worked on the NOSQL database MongoDB.
• Involved in converting Map Reduce programs into Spark transformations using Spark RDD in Scala.
• Experience in managing and reviewing Hadoop log files.
• Optimizing Hadoop MapReduce code, Hive and Pig scripts for better scalability, reliability and performance.
• Developed the OOZIE workflows for the Application execution.
• Performing data migration from Legacy Databases RDBMS to HDFS using Sqoop.
• Worked on Creation of Optimized Environment for testing and demonstrating Spark, Cassandra & Kafka Connectivity with objective of minimizing Spark SQL query times, Over JDBC/ODBC for databases
• Writing Pig scripts for data processing.
• Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
• Implemented Hive tables and HQL Queries for the reports.
• Imported data from Cassandra into HDFS using Mongo export utility.
• Involved in developing shell scripts and automated data management from end to end integration work
• Experience in performing data validation using HIVE dynamic partitioning and bucketing.
• Written and used complex data types in storing and retrieving data using HQL in Hive.
• Developed Hive queries to analyze reducer output data.
• Implemented ETL code to load data from multiple sources into HDFS using pig scripts.
• Highly involved in designing the next generation data architecture for the Unstructured data.
• Developed PIG Latin scripts to extract data from the source system.
• Created and maintained technical documentation for executing Hive queries and Pig scripts.
• Involved in Extracting, loading Data from Hive to Load on RDBMS using Sqoop

Environment: Spark, Scala, HDFS, Map Reduce, MySQL, Cassandra, Hive, HBase, Oozie, PIG, ETL, Hortonworks (HDP 2.0), Shell Scripting, Linux, Sqoop and Oracle 11g.

Confidential 

Big Data Developer

Responsibilities:

• As a Big Data Developer, I worked on Hadoop eco-systems including Hive, HBase, Oozie, Pig, Zookeeper, Spark Streaming MCS (MapR Control System) and so on with MapR distribution.
• Installed and configured Hadoop MapReduce, HDFS, Developed multiple MapReduce jobs in Java for data cleaning and preprocessing.
• Built code for real time data ingestion using Java, MapR-Streams (Kafka) and STORM.
• Involved in various phases of development analyzed and developed the system going through Agile Scrum methodology.
• Worked on Apache Solr which is used as indexing and search engine.
• Involved in development of Hadoop System and improving multi-node Hadoop Cluster performance.
• Worked on analyzing Hadoop stack and different Big data tools including Pig and Hive, Hbase database and Sqoop.
• Developed data pipeline using Sqoop and pig to extract the data from weblogs and store in HDFS
• Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
• Worked with different data sources like Avro data files, XML files, JSON files, SQL server and Oracle to load data into Hive tables.
• Used J2EE design patterns like Factory pattern & Singleton Pattern.
• Used Spark to create the structured data from a large amount of unstructured data from various sources.
• Implemented usage of Amazon EMR for processing Big Data across Hadoop Cluster of virtual servers on Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3)
• Performed transformations, cleaning and filtering on imported data using Hive, MapReduce, Impala and loaded final data into HDFS.
• Developed Python scripts to find vulnerabilities with SQL Queries by doing SQL injection.
• Experienced in designing and developing POC’s in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.
• Responsible for coding MapReduce program, Hive queries, testing and debugging the MapReduce programs.
• Extracted Real time feed using Spark streaming and convert it to RDD and process data into Data Frame and load the data into Cassandra.
• Involved in the process of data acquisition, data pre-processing and data exploration of telecommunication projects in Scala.
• Implemented a distributed messaging queue to integrate with Cassandra using Apache Kafka and Zookeeper.
• Specified the cluster size, allocating Resource pool, Distribution of Hadoop by writing the specification texts in JSON File format.
• Exported event weblogs to HDFS by creating a HDFS sink which directly deposits the weblogs in HDFS.
• Used RESTful web services with MVC for parsing and processing XML data.
• Utilized XML and XSL Transformation for dynamic web-content and database connectivity.
• Involved in loading data from the Unix file system to HDFS. Involved in designing schema, writing CQL's and loading data using Cassandra.
• Independently analyzed and validated program specifications as well as designed and created reliable and reusable C# code that conformed to department standards and procedures.
• Built the automated build and deployment framework using Jenkins, Maven etc.

Environment: Hadoop, Hive, HBase, Oozie, Pig, Zookeeper, MapR, HDFS, MapReduce, Java, MS, Jenkins, Agile, Apache Solr, Amazon EMR, Spark, Scala, Cassandra, Apache Kafka, MVC