PROFESSIONAL SUMMARY:

• Over 7+ years of work experience in IT field, involved in all phases of software development lifecycle while working in different projects
• Very strong experience in processing, analyzing large sets of structured, semi - structured and unstructured data and supporting systems application architecture
• Extensive experience in writing Hadoop jobs for data analysis as per the business requirements using Hive and Pig
• Expertise in creating Hive Internal/External Tables/Views using shared Meta store
• Developed custom UDFs in Pig and Hive to extend their core functionality
• Hands on experience in transferring incoming data from various application servers into HDFS, Hive, HBase using Apache Flume
• Stored Data in Vertica EDW
• Have experience of working on Snow - flake and Vertica data warehouse.
• Worked extensively on SQOOP to import and export data from RDBMS to HDFS and vice-versa
• Performed Data Ingestion from multiple disparate sources and systems using Kafka
• Proficient in big data ingestion and streaming tools like Apache Flume, Sqoop, Kafka, Storm and Spark.
• Experience of working on data formats like Avro, Parquet
• Hands on experience in Sequence files, RC files, Combiners, Counters, Dynamic Partitions, Bucketing for best practice and performance improvement
• Good experience of AWS Elastic Block Storage (EBS), different volume types and use of various types of EBS volumes based on requirement
• Experience creating real-time data streaming solutions using Apache Spark core, Spark SQL, Kafka, spark streaming and Apache Storm.
• Worked on Oozie to manage and schedule the jobs on Hadoop cluster
• Implemented AWS provides a variety of computing and networking services to meet the needs of applications
• Knowledge of developing analytical components using Scala
• Experience in managing and reviewing Hadoop log files
• Worked with NoSQL database HBase to create tables and store data
• Experience in setting up Hive, Pig, HBase, and SQOOP on Ubuntu Operating system
• Strong experience in design and development of relational database concepts with multiple RDBMS databases including Oracle 10g, MySQL, MS SQL Server & PL/SQL
• Proficient in using data visualization tools like Tableau, Raw and MS Excel
• Developed applications using Java, RDBMS and UNIX Shell scripting
• Experience of working on Servlets, JSP, JSF, Spring, Hibernate, JPA and JDBC
• Experience in developing web interfaces using technologies like XML, HTML, DHTML and CSS
• Implemented functions, stored procedures, triggers using PL/SQL
• Good understanding of ETL processes and Data warehousing
• Strong experience in writing UNIX shell scripts
• Working in different projects provided exposure and good understanding of different phases in SDLC

TECHNICAL SKILLS:

Hadoop/Big Data: Hadoop 1x/2x(Yarn), HDFS, Map Reduce, Hive, Pig, Sqoop, Flume, Kafka, Spark, Storm, Zookeeper, Scala, Oozie, Ambari, Tez, R

Development Tools: Eclipse, IBM DB2 Command Editor, TOAD, SQL Developer, VM Ware

Programming/Scripting Languages: Java, C++, Unix Shell Scripting, Python, SQL, Pig Latin, Hive QL

Databases: Oracle 11g,10g,9i, MySQL, SQL Server 2005,2008, PostgreSQL& DB2

NoSQL Databases: HBase, Cassandra, Mongo DB

ETL: Informatics

Visualization: Tableau, Raw and MS Excel

Frameworks: Hibernate, JSF 2.0, Spring

Version Control Tools: Sub Version (SVN), Concurrent Versions System (CVS) and IBM Rational Clear Case

Methodologies: Agile/ Scrum, Waterfall

Operating Systems: Windows, Unix, Linux and Solaris

PROFESSIONAL EXPERIENCE:

Confidential, Ashburn

Hadoop Developer

Responsibilities:

• Using Sqoop to import and export data from Oracle and DB2 into HDFS so as to use it for the analysis
• Migrated Existing MapReduce programs to Spark Models using Python.
• Migrating the data from Data Lake (hive) into s3 Bucket.
• Done data validation between data present in data lake and s3 bucket.
• Used Spark Data Frame API over Cloudera platform to perform analytics on hive data.
• Designed batch processing jobs using Apache Spark to increase speeds by ten-fold compared to that of MR jobs.
• Have experience of working on Snow - flake data warehouse.
• Developed Spark code using Scala and Spark-SQL/Streaming for faster processing of data
• Designed custom Spark REPL application to handle similar datasets
• Used Hadoop scripts for HDFS (Hadoop File System) data loading and manipulation
• Performed Hive test queries on local sample files and HDFS files
• Used Kafka for real time data ingestion.
• Created different topic for reading the data in Kafka
• Read data from different topics in Kafka.
• Moved data from s3 bucket to snowflake data warehouse for generating the reports.
• Written Hive queries for data analysis to meet the business requirements
• Migrated an existing on-premises application to AWS.
• Created Hive tables and worked on them using Hive QL
• Assisted in loading large sets of data (Structure, Semi Structured, and Unstructured) to HDFS
• Extensively used Pig for data cleaning and optimization
• Developed Spark SQL to load tables into HDFS to run select queries on top.
• Used AWS services like EC2 and S3 for small data sets.
• Developed the application on IntelliJ IDE
• Create data Frames using Scala.
• Developed Hive queries to analyze data and generate results
• Used Spark Streaming to divide streaming data into batches as an input to spark engine for batch processing.
• Exported data from HDFS to RDBMS via Sqoop for Business Intelligence, visualization and user report generation.
• Used Scala to write code for all Spark use cases.
• Analyzed user request patterns and implemented various performance optimization measures including implementing partitions and buckets in HiveQL
• Assigned name to each of the columns using case class option in Scala.
• Developed multiple Spark Sql jobs for data cleaning
• Developed PIG Latin scripts to extract the data from the web server output files and to load into HDFS
• Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting
• Created many Spark UDF and UDAFs in Hive for functions that were not preexisting in Hive and Spark Sql.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala.
• Implementing different performance optimization techniques such as using distributed cache for small datasets, partitioning and bucketing in hive, doing map side joins etc.
• Good knowledge on Spark platform parameters like memory, cores and executors
• By using Zookeeper implementation in the cluster, provided concurrent access for hive tables with shared and exclusive locking
• Developed analytical component using Scala, Spark and Spark Stream.
• Used Visualization tools such as Power view for excel, Tableau for visualizing and generating reports.
• Worked on the NoSQL databases HBase and mongo DB.

Environment: Linux, Apache Hadoop Framework, HDFS, YARN (MR2.X), HIVE, HBASE, AWS (S3, EMR), Scala, Spark, SQOOP

Confidential, Plano

Spark Developer

Responsibilities:

• Experienced in development using Cloudera distribution system.
• As a Hadoop Developer, my responsibility is manage the data pipelines and data lake.
• Performing Hadoop ETL using hive on data at different stages of pipeline.
• Worked in an agile technology with Scrum
• Sqooped data from different source systems and automating them with oozie workflows.
• Generation of business reports from Data Lake using Hadoop SQL (Impala) as per the Business Needs.
• Automation of Business reports using Bash scripts in UNIX on Data Lake by sending them to business owners.
• Developed Spark scala code to cleanse and perform ETL on the data in data pipeline in different stages.
• Worked in different environments like DEV, QA, Data Lake and Analytics Cluster as part of Hadoop Development.
• Snapped the cleansed data to the Analytics Cluster for reporting purpose to Business.
• Developed pig scripts, python to perform Streaming and created tables on the top of it using hive.
• Developed multiple POCs using Scala and deployed on the Yarn cluster, compared the performance of Spark, and SQL
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, and Scala.
• Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.
• Developed Oozie workflow engine to run multiple Hive, Pig, sqoop and Spark jobs.
• Handled importing of data from various data sources, performed transformations using Hive, Spark and loaded data into HDFS.
• Developed pig, hive, sqoop, Hadoop streaming, spark actions in Oozie in the workflow management.
• Supported Map Reduce Programs those are running on the cluster.
• Experienced in collecting, aggregating, and moving large amounts of streaming data into HDFS using Flume and Kafka.
• Responsible for developing multiple Kafka Producers and Consumers from scratch as per the software requirement specifications.
• Good Understanding of Workflow management process and in implementation.

Environment: Hadoop, AWS, Java, HDFS, MapReduce, Spark, Pig, Hive, Impala, Sqoop, Flume, Kafka, HBase, Oozie, Java, SQL scripting, Linux shell scripting, Eclipse and Cloudera

Confidential, Chicago

Hadoop Developer

Responsibilities:

• Responsible to manage data coming from different sources and involved in HDFS maintenance and loading of structured and unstructured data
• Developed MapReduce programs to parse the raw data and store the refined data in tables
• Designed and modified database tables and used HBase queries to insert and fetch data from tables
• Involved in moving all log files generated from various sources to HDFS for further processing through Flume
• Involved in loading and transforming large sets of structured, semi structured and unstructured data from relational databases into HDFS using Sqoop imports
• Responsible for analyzing and cleansing raw data by performing Hive queries and running Pig scripts on data
• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS that were further used for analysis
• Used Cloud watch logs to move application logs to S3 and create alarms based on a few exceptions raised by applications
• Created Hive tables, loaded data and wrote Hive queries that run within the map
• Used Oozie operational services for batch processing and scheduling workflows dynamically
• Developed and updated social media analytics dashboards on regular basis
• Performed data mining investigations to find new insights related to customers
• Managed and read viewed Hadoop log files
• Used Vertica as Enterprise data warehouse.
• Analyzed the web log data using the HiveQL to extract number of unique visitors per day
• Developed and generated insights based on brand conversations, which in turn helpful for effectively driving brand awareness, engagement and traffic to social media pages
• Involved in identification of topics and trends and building context around that brand
• Involved in the identifying, analyzing defects, questionable function error and inconsistencies observed in the output

Environment: HBase, Hadoop, HDFS, Map Reduce, Hive, Sqoop, Flume 1.3, Oozie, Zookeeper, MySQL, and Eclipse

Confidential

Java Developer

Responsibilities:

• Used XML for ORM mapping relations with the java classes and the database
• Worked in Analysis, Design and Coding for client development using J2EE stack using Eclipse platform
• Involved in creating web-based java components like client Applets and client side UI using JFC in Eclipse
• Developed PL/SQL stored procedures to perform complex database operations
• Used Struts in presentation tier
• Used Subversion as the version control system
• Played key role in the design and development of application using J2EE, Struts, Spring
• Involved in various phases of Software Development Life Cycle
• Configured Struts framework to implement MVC design patterns
• Designed and developed GUI using JSP, HTML, DHTML and CSS
• Generated the Hibernate XML and Java Mappings for the schemas
• Used Rational Application Developer (RAD) as Integrated Development Environment (IDE)
• Extensively used Core Java, Servlets, JSP and XML
• Used Oracle WebLogic workshop to generate the web service artifacts from the given WSDL for JAX-WS specification

Environment: Java, Struts, Servlets, spring, Tomcat, Hibernate, HTML, JSP, XML, SQL, J2EE, Junit, Oracle 11g, Windows

Confidential

Java Developer

Responsibilities:

• Implemented server side programs by using Servlets and JSP
• Designed, developed and validated User Interface using HTML, Java Script, XML and CSS
• Implemented MVC using Struts Framework
• Implemented Controller Servlet to handle the access to database
• Participated in code walkthroughs, Debugging and defect fixing
• Involved in the co-ordination of end to end production release process
• Used SVN for versioning control system
• Used JDBC prepared statements to call from Servlets for database access
• Designed and documented the stored procedures
• Involved in writing JUnit Test Cases and done unit testing for various components
• Worked on database interaction layer for insertions, updating and retrieval operations of data from oracle database by writing stored procedures
• Used Spring Framework for Dependency Injection and integrated with Hibernate
• Used Log4J for any errors in the application

Environment: Java, J2EE, JSP, Servlets, HTML, DHTML, XML, JavaScript, Struts, Eclipse, WebLogic, PL/SQL and Oracle