SUMMARY

• Over 11 years of experience in Development, Analysis, Design, Integration, and Presentation with ETL tools along with 3 years of Big Data /Hadoop experience in hadoop ecosystem such as Hive, Pig, Flume, Sqoop, Zookeeper, Hbase, SPARK, Kafka, Storm.
• Worked in INDIA, SWEDEN, DENMARK and currently working in USA on various technical, functional and business assignments.
• Cloudera Certified Developer for Apache Hadoop(CCD - 410).
• Solid understanding of Hadoop MRV1 and Hadoop MRV2 (or) YARN Architecture.
• Hands on experience on core Java and solid experience in debugging MapReduce applications.
• Importing data using sqoop into HDFS from various Relational Database Systems.
• Worked on standards and proof of concept in support of CDH4 and CDH5 implementation using AWS cloud infrastructure.
• Hands on experience in implementing complex business logic and optimizing the query using Hive QL and controlling the data distribution by partitioning and bucketing techniques to enhance performance.
• Good knowledge in Architecting, Designing, re-Engineering and Performance Optimization.
• Experienced in building highly scalable Big-data solutions using Hadoop and multiple distributions, i.e. Cloudera, Horton works and NoSQL platforms (Hbase & Cassandra).
• Experience with Oozie Workflow Engine in running workflow jobs with actions that run Hadoop Map Reduce and Pig jobs.
• Very Good Knowledge in Logical and Physical Data modeling, creating new data models, data flows and data dictionaries.
• Hands-on experience in using relational databases like Oracle and MS-SQL Server.
• Experienced with version control systems like SVN and Serena.
• Extensively worked on ETL processing tools like Pentaho and SSIS.
• Knowledge on different reporting tools (SSRS, Cognos).
• Exceptional ability to learn and master new technologies and to deliver outputs in short deadlines.
• Quick Learner, with high degree of passion and commitment in work.
• Has domain expertise in Health Insurance, Telecom, and Finance & Airline domain.
• Proficiency with mentoring and on-boarding new engineers who are not proficient in Hadoop and getting them up to speed quickly.
• Hands on experience in troubleshooting MapReduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.

TECHNICAL SKILLS

Platforms/ cloud services: Windows, Ubuntu, CentOS 7, EMR,S3

Big Data Technologies: Apache Hadoop, HDP 2.2.4, CDH5, Hive, Sqoop, Pig, Flume, Zookeeper, Hbase, Solr, Spark, Storm, Kafka

Machine Learning Platforms: Apache Mahout

Web Servers: Apache 1.3/2.x, IIS 5.0/6.0

Application Servers: BEA Weblogic,Apache Tomcat 5.x/7.x

Languages: Shell scripting, HTML, XML, SQL,C#

Packages: J2EE, JDK

Databases: Oracle, SQL Server, MySQL

Tools: Apache Solr,Jenkins, Matlab, JIRA,Visual Studio 2010/2008

Version Control: SVN,GITHUB,CVS

BI Tools: SSIS,SSRS,SSAS & Cognos Report studio 10.

PROFESSIONAL EXPERIENCE

Confidential, Waltham, MA

Big Data Sr. Consultant

Responsibilities:

• Defined the logic for identifying visitor stictching use case scenarios and identify Visit and SubVisit.
• Design and implement ETL solution in spark for visitor stictching .
• Involved in processing data from S3 buckets and loaded into EDW tables.
• Worked on implementing optimizations techniques, persistence/caching, compression concepts which are gained through development experience.
• Responsible for developing column mapping, transformations and data profiling tasks.
• Responsible for scheduling ETL tasks in Airflow which is an ETL task scheduler and dependency manager.
• Worked extensively on debugging and making changes ETL spark solution.
• Configured git repositories and used different sbt plugins to manage the build dependencies.
• Developed a testing frame work in scala to test different use cases in visitor stitching.
• Deploy the developed jar files in to Qubole and make sure spark environment is correctly spinning up with latest changes.

Confidential, Jersey City, NJ

Big Data Sr. Consultant

Responsibilities:

• To react to fraud in a timely manner, developed a look up the history of any user in few milliseconds with options of using HBase NoSQL databases and caches for quick retrieval of information.
• Worked on various machine learning models and applied on data to find correlation between different entities.
• Hands on experience in various models like Linear Regression, Decision Trees, Naïve Bayes, Random Forests and K-means Clustering and worked on MLLib, GraphX libraries and spark streaming.
• Involved in loading data into HBase using HBase Shell, HBase Client API.
• Very good understanding of Spark core concepts and internals like partitions, memory management, functions of executors and drivers, shuffling, optimizations techniques, persistence/caching, compression concepts which are gained through development experience.
• Responsible for developing data pipeline using flume and Kafka to extract the data and store in HDFS.
• Used Maven extensively for building jar files of Java MapReduce programs and deployed to Cluster.
• Worked extensively on debugging and making changes in the MapReduce programs which are written in Java.
• Created Multimode projects using SBT and developed the build script and used various sbt plugins to manage the build dependencies.

Confidential

Senior Big Data Developer

Responsibilities:

• Responsible for ingesting data in to HDFS using Sqoop from relational databases Sql Server and DB2.
• Responsible for reconciliation between current records in HIVE and new change records.
• Responsible for performing extensive data validation using Hive.
• Sqoop jobs, PIG and Hive scripts were created for data ingestion from relational databases to compare with historical data.
• Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
• Hands on experience in adjusting the memory settings for Map or Reduce and any other performance optimizations needed to make the job execute in the agreed SLA time.
• Responsible for debugging failed jobs in production.

Confidential

ETL Architect

Responsibilities:

• Utilized SQL*Loader to load flat files database tables.
• Responsible for SQL tuning and optimization using Analyze, Explain Plan, TKPROF utility and optimizer hints.
• Responsible for performing code reviews.
• Designed and created different ETL packages using SSIS from Oracle source to SQL Server as destination
• Installed upgraded and migrated databases from Oracle database to MS SQL Server 2008.
• Transformed data from various data sources using OLE DB connection by creating various SSIS packages

Environment: Oracle 8i/9i, PL/SQL, SQL*PLUS, SQL, SQL*Loader, PL/SQL Developer, Explain Plan and TKPROF tuning utility, Microsoft SQL Server 2008, MS Windows Server 2003/2008,SSIS

Confidential

Senior Database Developer and Analyst

Responsibilities:

• Responsible for analyzing business functionality for the back end processes
• SQL Server 2008 RDBMS database development using Confidential -SQL programming, queries, stored procedures, views.
• Performed tuning and optimization on SQL queries using Explain Plan and optimizer hints.
• Created SQL reports, data extraction and data loading scripts for different databases and schemas
• Developed SSIS packages using different type’s task and with Error Handling.
• Developed reports on SSRS on SQL Server (2005).
• Used SSIS to create ETL packages to Validate, Extract, Transform and Load data to Data Warehouse and Data Mart Databases.

Environment: MS SQL Server 2005 Enterprise Edition, Windows 2003 Enterprise Edition, Windows 2000 Advanced Server, SSIS, SSRS

Confidential

ETL Developer

Responsibilities:

• Estimation, Analyzing FRD’s, Preparing Design Documents, writing Unit Test cases, Coding, Unit testing & guiding team members
• Responsible for package deployment and development
• Responsible for trouble shooting back end production problems Writing queries to front end designers.
• Used DTS for data population from different sources and create DTS packages for data conversions and load data from flat flies or excel files.
• Fine tune complex SQL queries and Stored Procedures to increase performance.
• Worked as a developer in creating complex Stored Procedures, DTS packages, triggers, cursors, tables, and views and other SQL joins and statements for applications

Environment: Sql Server 2000,MS.Net 2.0 MS SQL Server 2000, Windows NT/2000 Server, MS Access, Query Analyzer, DTS, BCP, SQL Profile.