SUMMARY

• Around 8 years of overall IT development experience including 3+ years of experience exclusively on BIG DATA ECOSYSTEM using HADOOP framework and related technologies such as HDFS, MapReduce, Hive, Pig, Spark, Python, HBase, Flume, Oozie, Sqoop, Impala, Kafka, Splunk and Zookeeper. Excellent knowledge on distributed storages (HDFS) and distributed processing (MapReduce, Yarn) for real - time streaming and batch processing.
• Experience in developing Map-Reduce programs to perform Data Transformation in Java.
• Experience in writing Custom MapReduce programs in java and also extending Hive(hql) and Pig core functionality by writing custom UDFs.
• Extensive experience with big data query tools like Pig Latin and HiveQL.
• Experience in extracting the data from RDBMS into HDFS using Sqoop.
• Experience in collecting the logs from log collector into HDFS using Flume.
• Good understanding of NoSQL databases such as HBase, Cassandra and Mongo DB.
• Experience in analyzing data in HDFS through MapReduce, Hive and Pig.
• Experience in job workflow scheduling and monitoring tools like Oozie and Zookeeper.
• Knowledge on Hadoop administration activities such as installation, configuration and management of clusters using Cloudera Manager, Hortonworks and Apache Ambari.
• Solid knowledge of Hadoop architecture and core components Name node, Data nodes, Job trackers, Task Trackers, Oozie, Scribe, Hue, Flume, Kafka, HBase, etc.
• Extensively worked on development and optimization of Map reduce programs, PIG scripts and HIVE queries to create structured data for data mining.
• Worked with Scala framework for processing data pipelines through Spark.
• Very good experience of Partitions, bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.
• Experience in extending HIVE and Pig core functionality by using Custom User Defined functions.
• Working knowledge on Oozie, a workflow scheduler system to manage the jobs that run on PIG, HIVE and SQOOP.
• Knowledge and experience in Microsoft Technologies using C#, ASP.NET, ADO.NET, Web Forms, Win Forms, EF, LINQ, WCF, WPF, XML, XAML, Microsoft SQL Server and Oracle
• Experienced with RDBMS concepts which include database programming using SQL Server Components: Transact-SQL, Stored Procedures, functions.
• Experienced and worked with software development methodologies and process models like Agile/Scrum and waterfall.
• Experience and worked on SDLC tools like SPRITZ, Zephyr and JIRA.
• Highly motivated, result oriented professional with a short learning curve and strong communication skills.

TECHNICAL SKILLS

Hadoop/Big Data: HDFS, Map Reduce, Hive, Pig, Sqoop, Flume, Kafka, Zoo Keeper, Oozie, Spark (Scala and Python)

NoSQL Databases: MongoDB, HBase, Cassandra

.Net Technology: Net Framework 3.5/4.0/4.5, EF, LINQ, ASP.NET, MVC 4, ADO.NET

Internet Technologies: HTML, AJAX, XML, JavaScript, jQuery, CSS, AngularJS, Web API, SOAP, HTML5, XHTML, Bootstrap, XML, JASON

Languages: Python, Scala, Java, C#.Net, C++

Database: Microsoft SQL Server 2008/2012, Oracle

Development Tools: Eclipse, Visual Studio 2008, 2010, 2012, Azure

Operating Systems: Sun Solaris, Cent OS, UNIX, Red Hat Linux, Ubuntu Linux, Windows 10/8/7 and Windows Server 2012/2008

Web Servers: Web Logic, Web Sphere, Apache Tomcat, IIS

Version Controlling Tools: Team Foundation Server, Bit bucket

SDLC Tool: JIRA, SPRITZ, Zephyr and TFS

Bug Tracking Tools: IBM Clear Quest, TFS, JIRA

Methodology: Agile/Scrum, Waterfall

Tools: & Libraries: Log4Net, Beyond Compare, SOAP UI, FileZilla, Putty, Fiddler, EA, BuildFarm

PROFESSIONAL EXPERIENCE

Confidential, Georgetown, KY

Hadoop/Spark Developer

Responsibilities:

• Build distributed, reliable and scalable data pipelines to consume and process data.
• Used Sqoop to pull data from Teradata and different sources.
• Design and implement column family schemas of HBase within HDFS.
• Optimized Map Reduce Jobs to use HDFS efficiently by using various compression mechanisms.
• Responsible for data extraction and data ingestion from different data sources into Hadoop Data Lake by creating ETL pipelines using Pig, and Hive.
• Designed job using Talend Open Studio for Big Data to ingest into Data Lake and Data warehouse.
• Push data as delimited files into HDFS using TalendBig data studio.
• Involved in creating Hive tables, loading with data and writing hive queries which run internally in MapReduce. Loaded various formats of data like Avro, parquet into these tables and analyzed data using HQL.
• Used Oozie workflow engine to run multiple jobs which run independently.
• Worked on Kafka while dealing with raw data, by transforming into new Kafka topics for further consumption.
• Developed Kafka producer and consumers, HBase clients, Spark, and Hadoop MapReduce jobs along with components on HDFS, Hive.
• Developed Spark Programs for Batch and Real Time Processing.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
• Imported the data from different sources like HDFS/HBase into Spark RDD and implemented Spark RDD transformations, actions to implement business analysis.
• Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
• Used Spark SQL to create structured data by using data frames and querying from other data sources using jdbc and hive.
• In data exploration stage used hive and Impala to get some insights about thedata.Worked on large datasets to generate insights by using Tableau.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Scala.
• Involved in developing components on Reporting Dashboard with Spring MVC, Spring Framework and Hibernate.
• Worked hands on with ETL process. Handled importing data from various data sources,performed transformations.
• Worked on batch processing of data sources using Apache Spark, Elastic Search.
• Implemented indexing for logs to Elastic Search and analysis on integrating Kibana with Elastic Search.
• Written data pipelines in python to extract data from hive, MySql, Presto.
• Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
• Developed Spark scripts by using Python shell commands as per the requirement.
• Developed and implemented core API services using Python with spark.
• Involved in the design and development of Spark applications using Scala and utilizing Data frames and Spark SQL API that read the cleansed data from the TBDP Transformation layer (Hive).
• Develop efficient Pig and Hive scripts with joins on datasets using various techniques.
• Designed and developed the Data Models in HBase based on the data from existing subsystem data and create real time virtual line for calculating the parts required for manufacture of the vehicle.
• Developed Ingress Spark processes that load data from Staging area, Kafka Topics to TBDP Raw layer (Hive).
• Troubleshoot and debug any Hadoop ecosystem run time issues.
• Determine the operational impacts by reviewing the integration systems for better design of the system.
• Participate in the testing process through preparation of test cases, unit testing, system testing and supporting for User Acceptance Testing.

Environment: Hadoop, Spark, Python, MapReduce, HDFS, Hive, Pig, Java, Sqoop, Zookeeper, Impala, Scala, SBT, Maven, Team Foundation Server (TFS), Bit bucket, Zephyr, JIRA

Confidential, Georgetown, KY

Hadoop Developer

Responsibilities:

• Worked withHadoopEcosystem components like HDFS, Spark, Hive, Sqoop, Zoo Keeper, Pig with Cloudera distribution model.
• Involved in the design and development of Spark applications using Scala and utilizing Data frames and Spark SQL API that subscribe the xml files from the Kafka cluster and processes the.
• Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
• Written multiple Map Reduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats
• Worked on different Spark processes for business transactions.
• Developed Oozie workflows for spark jobs.
• Developed Map Reduce jobs to automate transfer of data from/to Hbase
• Created Sqoop scripts to export the data from external database to Hive external tables and developed Hive UDF's in java.
• Involved in the discussions with users on the posting definitions and configuring them.
• Developed Hive queries to do analysis of the data and to generate reports to be used by users.
• Used TFS as a code repository for managing agile project development process.
• Created SPF's (System proposal form) and user guides for each of the processes developed.
• Explored with Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frame, Spark YARN.
• Worked on Spark code using Scala, Java and Spark-SQL for faster testing and processing of data.
• Developed Spark scripts by using Python shell commands as per the requirement.
• Developed and implemented core API services using Python with spark.
• Used Kafka functionalities like distribution, partition, replicated commit log service for messaging systems by maintaining feeds.
• Participate in the testing process through preparation of test cases, unit testing, system testing and supporting for User Acceptance Testing.
• Support during the implementation and post- implementations of the programming applications.

Environment: Hadoop, Spark, Python, MapReduce, HDFS, Hive, Pig, Java, Sqoop, Zookeeper, Impala, Scala, Team Foundation Server (TFS)

Confidential, Lakeforest, CA

Sr Software Engineer

Responsibilities:

• Involved in software Development Life Cycle (SDLC) Process included Analysis, Design, Programming and Testing
• Using the Following Agile Development methodology and participating in regular meetings.
• Working on 3-tier architecture and involved in preparation of high-level design and detail design of the system.
• Worked with patterns like MVVM (Model View View Model) and also implemented the n-tier architecture with enhanced functionality, high performance.
• Working on 3-tier architecture and involved in preparation of high-level design and detail design of the system.
• Developed the Windows application in an Agile Methodology and always worked closely with the business team for effectively of the application.
• Developed High Performance applications using TPL in .NET Framework 4.0.
• Designed, modeled and architected multi-threaded, enterprise n-tier software agent using Windows Communication Foundation (WCF) web service hosted in a Windows service.
• Developed Windows clients in C# using Windows Presentation Foundation (WPF) and XAML.
• Used LINQ provider which will be exposed to the user and will be responsible for building and executing a query expression
• Used C# and ADO.NET to define and implement secure middle-tier components using both connected and disconnected environment strategies.
• Used Style cop and code analysis to improve the quality of code.
• Creating database tables, stored procedures, triggers, views and functions in SQL Server.
• Use Moles Framework to write automated unit tests.
• Involved in developing automated unit tests for the developed functionalities.
• Involve in Preparation of Test Cases and Unit testing for the developed functionalities.

Environment: Visual Studio 2012/2010, C#, VC++, WPF, XAML, MVVM, MVC, SQL Server 2008, IIS, JavaScript, AJAX, jQuery, Angular JS, React JS, SSRS, SSIS, Web API, SOAP, REST, Bootstrap, XML, JASON, WCF, WPF, EF, LINQ, Team Foundation Server (TFS), IBM Rational Clear case

Confidential

Software Engineer

Responsibilities:

• Involved in software Development Life Cycle (SDLC) Process included Analysis, Design, Programming and Testing.
• Working on 3-tier architecture and involved in preparation of high-level design and detail design of the system.
• Developed Windows clients and Windows services in C# using Win forms in .NET Framework.
• Used C# and ADO.NET to define and implement secure middle-tier components using both connected and disconnected environment strategies.
• Creating database tables, stored procedures, triggers, views and functions in SQL Server.
• Involved in developing automated unit tests for the developed functionalities.
• Involve in Preparation of Test Cases and Unit testing for the developed functionalities.

Environment: Visual Studio 2012/2010, C#, ASP.Net, SQL Server 2012, IIS, JavaScript, AJAX, jQuery, Angular JS, SSRS, SSIS, XML, SOAP, REST, WCF, EF, LINQ, Team Foundation Server (TFS)

Confidential

Member Technical Staff

Responsibilities:

• Involved in Software Development Life Cycle (SDLC) process included Analysis, Design, Development and Testing
• Involved in preparation of Technical Specification Documents (TSD) and Functional Specification Documents (FSD).
• Developed applications using C#.NET Framework.
• Worked on applications using ASP.Net and ADO.Net
• Used C# and ADO.NET to define and implement secure middle-tier components using both connected and disconnected environment strategies.
• Creating database tables, stored procedures, triggers, views and functions in SQL Server.

Environment: Visual Studio 2010/2008, C#, ASP.Net, SQL Server 2008, IIS, JavaScript, ASP.NET Framework, Team Foundation Server (TFS)