SUMMARY

• 5 Years of IT experience which includes providing solutions for Bigdata and software development.
• Expertise in various components of Hadoop Ecosystem - Map Reduce, Hive, Pig, Sqoop, Impala,Oozie, HBase, Spark, Kafka, YARN.
• Experienced in Map Reduce code tuning and performance optimization.
• Proficient in Hive Query language and experienced in hive performance optimization using Static-Partitioning, Dynamic-Partitioning, Bucketing and Parallel Execution concepts.
• Extended Hive and Pig core functionality by writing custom UDFs.
• Expertise in developing Pig Scripts for data transformations, joins and pre-aggregations on the raw datasets.
• Hands on experience in Data Import/Export using Hadoop Data Management tool Sqoop between HDFS and other Relational Database Systems like SQL server and MySQL.
• Experience in working with Flume to load the log data from multiple sources directly into HDFS.
• Used Oozie job scheduler to schedule Map Reduce, Hive, Pig and Sqoop jobs. Experience in automating the job execution.
• Used Impala for data analysis.
• Having good knowledge on HadoopAdministration like Cluster configuration, Single Node Configuration, Multi Node Configuration, Data Node Commissioning and Decommissioning, Name Node Backup and Recovery, HBase, HDFS and Hive Configuration, Monitoring clusters, Access control List.
• Use of NoSQL databases like HBASE and Cassandra.
• Developed Spark SQL scripts and involved in converting hive UDF’s to Spark SQL UDF’s
• Excellent T-SQL developer skills including Stored Procedures, Indexed Views, User Defined Functions and others.
• Administering SQL Server High Availability features like Log shipping, Database Mirroring and Always On.
• Extensive technical knowledge of data visualization techniques using TABLEAU.
• Good knowledge of Amazon webservices (AWS), in particular working with ec2 instances to create demand elastic clusters for distributed computing.
• Experience in creating high availability web servers using ELB.
• Hands-on experience with AWS Lambda workflow implementation using python to interact with application deployed on EC2 instance and S3 bucket
• Good knowledge of python libraries like Numpy, Scikit.
• Highly proficient and organized, reliable, ability to work in fast-paced environment by handling multiple tasks,and sensitive to deadlines.

TECHNICAL SKILLS

Big data: Map Reduce, Hive, Pig, Impala, Sqoop, Flume, HDFS, Oozie, Hue, HBase, Spark, Kafka

NoSQL: HBase, Cassandra

Java & J2EE technologies: Core Java, Servlets, JSP, JDBC

Operating Systems: Windows, Ubuntu, RedHat Linux

Frameworks: Struts, Hibernate

Databases: SQL server, MySQL

Programming: Java, Shell scripting, Python, PL/SQL, C#

Web Technologies: Java script, HTML, XML, CSS

Methodologies: Waterfall, Agile

PROFESSIONAL EXPERIENCE

Confidential

Hadoop developer

Responsibilities:

• Involved in gathering the business requirements, designing (Low-level and High-level), development and testing
• Written Map Reduce jobs in java to process the log data
• Created Hive tables to store the processed results in a tabular format
• Implemented external and managed tables using HIVE
• Used Partitioning, Dynamic-Partitioning and bucketing concepts for performance optimization in hive
• Used flume to collect all the web log from the online ad-servers and push into HDFS
• Written the Apache PIG scripts for loading, filtering and storing the data in HDFS, which later will be used by Hive or MR jobs
• Implemented Pub/Sub model using Apache Kafka for real-time transactions to load in HDFS
• Developed the Sqoop scripts to make the interaction between HDFS and RDBMS (Oracle, MySQL)
• Made use of Distributed cache to improve the performance of the jobs and reduce the load on cluster
• Developed Spark SQL scripts and involved in converting hive UDF’s to Spark SQL UDF’s
• Implemented POC to migrate Map Reduce programs into Spark transformations using Spark and Scala
• Automated workflows using Oozie jobs
• Worked on file formats Avro, Sequence files, Text files, csv, xml, JSON
• Used SerDe (Avro, CSV and Regex) and compressions formats (bzip2, Avro)
• Implemented Spark using Scala and Spark SQL for faster testing and processing of data.
• Generated reports using tableau by mapping HDFS data to tableau.

Confidential

Hadoop Developer/Admin

Responsibilities:

• Participate in requirement gathering and analysis phase of the project in documenting the business requirements by conducting workshops/meetings with various business users
• Worked as a part of AWS build team.
• Configured demand elastic Hadoop clusters using ec2 and, also managed S3 buckets.
• Cluster maintenance, commissioning and decommissioning of data nodes, cluster monitoring using cloudera manager services.
• Loading unstructured log data of user clicks into HDFS by using automated shell scripts
• Created Kafka based messaging system to create events and alters for different systems
• Developed the OOZIE workflows for the Application execution
• Configured Fair scheduler for development environments
• Created HBase tables and column families to store the user event data
• Written automated HBase test cases for data quality checks using HBase command line tools
• Created java programs using HBase java API to read and write HBase tables
• Analyze Gigabytes of logs generated by the Confidential 's OLAP/OLTP using real time data processing system - Apache Storm.
• Worked with personalized recommendation engine of Confidential, which analyses the viewer events in real time using Apache Storm and directs the contextual, personalized ads to subscribers

Confidential

Graduate Research assistant

Responsibilities:

• Migrated annual data from Access database to SQL server and Integrated using SQL server integration services (SSIS).
• Created multiple data warehouses to store and analyze historic data in various projects.
• Developed ETL packages to support data integration and perform analysis.
• Created Stored Procedures, Indexed Views, User Defined Functions, and Triggers.
• Involved in cleaning and organizing data by writing stored procedures and Ad-hoc queries helping in performing racial profiling by Department of public safety.
• Performed basic DBA functions like optimization, indexing, performance tuning.
• Achieved an increase in query performance about 30% by using tools like Tuning advisor and SQL profiler.
• Responsible for database tuning, backup/recovery etc.
• Created user roles to provide/restrict access to database.
• Developed data analysis scripts which helped in better understanding of oceanic traffic and helped in optimizing routes and traffic.
• Highlighted the key insights by creating dashboards using Tableau.
• Generated and maintained statistical reports and implemented data mining activities.
• Created multi node Apache Hadoop cluster using AWS cloud platform.
• Used Sqoop to import data from SQL server to HDFS.
• Implemented external and internal hive tables on the data in HDFS and analyzed data using hive queries.
• Used partitioning and bucketing concepts in hive to optimize the query performance.

Confidential

Java Hadoop Developer

Responsibilities:

• Developed multiple Map Reduce jobs in java for data cleaning and preprocessing and scheduled the job using Oozie to aggregate the log data on hourly basis
• Performance tuning of the Map Reduce jobs
• Developed workflows using custom Map Reduce, Pig, Hive and Sqoop
• Worked with pig scripts to extract, transform and load the data into HBase and Hive
• Built reusable Hive UDF libraries for business, which enabled users to use these UDF's in Hive Querying
• Used Sqoop to import from different database sources and file systems to HDFS and vice versa
• Developed workflow in Oozie to automate tasks of loading data into HDFS and preprocessing with pig
• Used Apache storm to process the input message streams and identify the fraud detection in real-time
• Implemented ApacheSolr for indexing
• Implemented Flume agents to collect the data from various sources and load into HDFS
• Used Maven extensively for building jar files of Map Reduce programs and deployed to Cluster
• Modelled Hive partitions extensively for data separation and faster data processing and followed Pig and Hive best practices for tuning
• Worked on file formats Avro, Sequence files, Text files, csv, xml, JSON
• Implemented Kafka Custom encoders to store custom objects, build storm topologies as part of pre-processing
• Configuring Kafka to read and write messages from external programs
• Conducted technical and domain level knowledge transfer sessions

Confidential

Java Developer

Responsibilities:

• Involved in the implementation of design using vital phases of the Software development life cycle (SDLC) that includes Development, Testing, Implementation and Maintenance Support.
• Used Agile methodology named SCRUM for all the work performed.
• Developed code for user interface by utilizing OOPS concepts like multithreading, inheritance, polymorphism and collection API.
• Developed JDBC code for backend processing.
• Extensively used java multi-threading to implement batch Jobs with JDK features.
• Used Java Script for Client-side validations.
• Used Struts Framework for developing Web applications.
• Worked on J2EE/Java, HTML and web application.
• Created user-friendly GUI interface and Web pages using HTML, JSP.
• Designed the module from scratch including database schema design.
• Experienced in debugging the java code with all the testing tools and logic flow using Eclipse UML plug-in and rational testing tools.
• Created and implemented PL/SQL queries, views stored procedures and triggers.
• Worked with QA team to validate the test cases if the test cases are meeting the business requirements.