SUMMARY:

• Overall 8+ years of IT Experience with 5+ years of hands on experience in Hadoop, HDFS and HadoopEcosystem
• Highly experienced as Big DataEngineer with deep understanding of the HadoopDistributed File System and Eco System (Map Reduce, Spark, Sqoop, Apache Flume, Apache Kafka, Hive, Pig, HBase, Oozie).
• Hands - on experience on RDD architecture, implementing Spark operations, transformation and optimizing transformations.
• Extensive experience in applying business logic on Transformed Spark RDD's using Actions.
• Experience using Sqoop to import/export datainto HDFS from RDBMS and vice-versa.
• Extensive experience in writing Pig and Hive scripts for processing and analyzing large volumes of structured data.
• Experience in developing, debugging and tuning Map Reduce jobs in Hadoopenvironment.
• Experienced in importing streaming datainto HDFS using flume.
• Experience in developing customized UDF's in java to extend Hive and Pig Latin functionality.
• Implemented PIG join operations, custom user defined functions (UDF) to perform ETL operations.
• Used Apache kafka for streaming dataand worked on Producers, Brokers and customers.
• Hands-on experience in scheduling jobs on Oozie.
• Experience in NoSQL databases, such as HBase.
• Hands-on experience in J2EE technologies such as Servlets, JSP, JDBC and developing Web Services providers and consumers using SOAP, REST.
• Used Agile Development Methodology and Scrum for the development process.
• Experienced in using Integrated Development environments like Eclipse, NetBeans, IntelliJ.
• Experience in complete Software Development Life Cycle(SDLC).
• Adept in using Microsoft T-SQLto create and optimize database objects - Tables, Views, Indexes, Triggers, Stored Procedure, Transactions, CTEs, Joins and performance tuning.
• Have written complex T-SQLqueries, Dynamic SQL, Sub-queries and complex Joins.
• Good analytical, communication, problem solving and interpersonal skills.

TECHNICAL SKILLS:

Big Data Technologies: HDFS, Spark, Hive, MapReduce, Hbase, Pig, Sqoop, Oozie, Zookeeper, Flume, Kafka

Scripting Languages: HTML, CSS

Programming Languages: Java, Scala, SQL, TSQL

RDBMS: MySQL, MSSQL, Oracle 10G

IDE’s: NetBeans, Eclipse, IntelliJ, Microsoft Visual Studio

File Formats: Text, Avro, JSON, Parquet, Sequence

Virtual Machines: VMWare, Virtual Box

Operating Systems: Cent OS, Unix, Red Hat Linux, Ubuntu, Windows, Mac

PROFESSIONAL EXPERIENCE:

Spark Developer

Confidential, Basking Ridge, NJ

Responsibilities:

• Installed local access to Scala, Spark, Git, HDFS on DEV Environment using IntelliJ IDEA, FTP
• Experience in creating RDD’s, DataFrames & performed various Transformations, Actions
• Experience in Spark Core, SparkSQL API, and was also Involved in creating Spark UDF’s to create Map structures, Handling nulls, case classes and Spark UDAF for custom aggregations.
• Experience in creating different types of joins, perform Grouping, Methods, Property file for specify the schema, input paths specific to make the application run on different Environments
• Involved in Reading HBase tables using Spark and also Hive Tables by performing joins using HiveContext
• Experience in loading JSON RDD’s to Elastic Search Indices by following strict mapping, provided by the API Team and creation & deletion of aliases associated to the loaded Indices.
• Extensive Knowledge in creating generalized Shell Scripts for spark applications deployment on cluster.
• Worked closely with System Analyst in gathering requirements for creation of Visual Hierarchy and Development process.
• Extensively involved in Code tuning & Performance tuning by observing Spark Web URL and creating optimized code by using various functionalities like Repartitions, Coalesce, dropDuplicates ETC.
• Involved in utilizing various methods pertaining to different application by using JFROG Artifact Maven Dependency
• Experience in persisting and caching Dataframes depending on the size of the Dataframe and Memory.
• Refactored code from Spark 1.6 to Spark 2.0 for Production Deployment.
• Created Git Repositories, Branches and also Performed Push, Pull, Commit, Merge to Git from IntelliJ and Git cmd.

Environment: Hadoop, Hive, MySQL, IntelliJ, Scala, Spark 1.6, Spark 2.0, Sqoop, HBase, MapR Distribution, ElasticSearch, Linux Scripting, TWS(Tivoli Workflow Scheduler).

Hadoop/Spark Developer

Confidential, Rolling Meadows, IL

Responsibilities:

• Installed Hadoop, Map Reduce, and HDFS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
• Assisted in upgrading, configuration and maintenance of various Hadoop infrastructures like Pig, Hive, and Hbase.
• Performed near real time analysis on the incoming data using Spark.
• Performed transformations, cleaning and filtering on imported data using Hive, Map Reduce and loaded final data into HDFS.
• Loaded data into Hbase using Bulk Load and Non-bulk load.
• Developed Sparkscripts by using Pyspark commands as per the requirement.
• Used SparkAPI over Cloudera Hadoop YARN to perform analytics on data in Spark SQL/Hive.
• Developed Scala scripts, UDF’s using both Data frames/SQL and RDD’s in Spark for Data Aggregation, queries and writing data back into OLTP system directly or through Sqoop.
• Developed Sparkcode using Python and Spark-SQL/Streaming for faster testing and processing of real time data.
• Imported the data from different sources like HDFS/Hbase into SparkRDD.
• Developed a data pipeline using Kafka and creating multiple consumers and consumer groups to consume data and fetched it to dataframes of spark streaming API.

Environment: Hadoop, HDFS, MapReduce, YARN, Spark, Pig, Hive, Sqoop, Kafka, Hbase, Scala, Java, SQL scripting, Linux shell scripting, IntelliJ and Cloudera.

Big Data Developer - Hadoop

Confidential, Bloomington, IL

Responsibilities:

• Installed and configured Hadoopclusters and Hadooptools for application development including Hive, Pig, Sqoop, Flume and Oozie.
• Responsible for building scalable distributed datasolutions using Hadoop on Amazon EC2.
• Created multiple Hive tables with partitioning and bucketing for efficient dataaccess.
• Extracted and loaded customer datafrom databases to HDFS and Hive tables using Sqoop.
• Used Flume to transfer log source files to HDFS.
• Performed datatransformations, cleaning and filtering, using Pig and Hive.
• Developed workflow in Oozie to automate the tasks of loading the datainto HDFS and pre-processing with Pig.
• Performed unit testing using JUnit and MR Unit.

Environment: Hadoop, AWS, HDFS, YARN, MapReduce, Sqoop, Flume, Hive, Pig.

Hadoop Developer

Confidential, Plymouth Meeting, PA

Responsibilities:

• Processed datainto HDFS by developing solutions, analyzed the datausing MapReduce, Pig, Hive and produce summary results from Hadoopto downstream systems.
• Involved in ETL, DataIntegration and Migration.
• Was responsible for managingdata from multiple source.
• Developed the Pig and Hive queries as well as UDF'S to pre-process thedatafor analysis.
• Imported and exported datainto HDFS and Hive using Flume.
• Was responsible for architectingHadoopclusters with CDH4 on CentOS, managing with Cloudera Manager.
• Used Sqoop widely in order to importdatafrom various systems/sources (like MySQL) into HDFS.
• Applied Hive queries to performdataanalysis on HBase using Storage meet the business requirements.
• Created components like Hive UDFs for missing functionality in HIVE for analytics.
• Used different file formats like Text files, Sequence Files, JSONetc.
• Installed and configured Hadoop, Mapreduce, HDFS, Developed multiple MapReduce jobs in java for datacleaning and preprocessing.

Environment: Hadoop, HDFS, MapReduce, Pig, Hive, Sqoop, HBase, Java, Zookeeper.

Java/J2EE Developer

Confidential , Newtown Square, PA

Responsibilities:

• Designed and developed web based application & the front end in JSP with J2EE, MySQL, JQuery & CSS.
• Wrote Action Classes, Service Classes, Form Beans and XML files.
• Wrote UNIT Test cases and implanted to find bugs in fixed code.
• Fixed defects in existing project.
• Used Struts MVC to develop the applications.
• Developed RESTful web services for Insight applications.
• Implemented business cases on client to client basis.
• Was responsible for coding SQL Statements and Stored procedures for back end communication.
• Designed and implemented MVC architecture using Struts Framework.
• Used IntelliJ Idea IDE for development for versioning.
• Was involved in Feature Enhancements, Researching, Evaluating and Solving Production Issues.
• Worked on database design and creating database objects like tables, and retrieving data from tables in MS SQL.

Environment: Java, J2EE, JSP, Servlet, HTML, CSS, XML, JavaScript, MS SQL, Intelli

Confidential

SQL Developer

Responsibilities:

• Was responsible for designing logical as well as physical database.
• Defined data types, created database and database objects such as tables and views.
• Tested the table with complex queries such as Joins, Sub-queries and Temp Tables.
• Converted complex business logic into T-SQLStored Procedures and User Defined Functions.
• Created indexes (Clustered & Non-clustered) to reduce data retrieval time.
• Used Try Catch in stored procedure to perform error log mechanism.
• Created ETL packages to migrate data from different data sources to data warehouse by using SSIS.
• Created Custom Logging in SSIS packages.
• Wrote queries to create drill down, drill through and ad-hoc reports on regional analysis on pharmacy, nursing and benefits services using SSRS.
• Actively supported business users for changes in reports as and when required using SSRS.
• Used complex expressions to group data, filter and parameterize reports.
• Deployed reports on server using SSRS and created linked reports and snapshots.