SUMMARY:

• An accomplished and innovative IT professional with 15+ years of experience offers extensive experience in building a production ready Big Data ingestion platforms and developing Big Data analysis applications using Scala, Python, Apache Spark.

TECHNICAL SKILLS:

Languages: Scala, Python, Java, Pig, Hive, Spark, SQL,T - SQL, Oracle SQL, PL/SQL, PostgreSQL

Databases: Elasticsearch, DynamoDB, Mongodb, Redshift, SQL Server 2000/2005/2008 R2/2012, MySQL, Oracle 9i/10g/11g

Tools: and Frameworks: Apache Spark, AWS, Amazon EMR, Hadoop, Talend, Talend Enterprise Big data, SSRS, SSIS, SSAS

Operating Systems: Linux (Red Hat, CentOS), Solaris

PROFESSIONAL EXPERIENCE:

Confidential

Software Engineer

Responsibilities:

• Working on a POC for building a movie recommendation engine based on Fandango ticket sales data using Scala and Spark Machine Learning library.
• Performing data wrangling on various dataset for machine learning and downstream teams using Scala, Python, and Spark.
• Designed and developed Python ETL framework to extract Facebook, Youtube, and Twitter video metrics data for upcoming Universal studio movie trailers and store near real-time data into MySql using Facebook Graph API, Google Youtube API, and Spark Streaming.
• Converted long running Hive queries to Spark jobs by rewriting Java Hive UDFs into native Spark code using Scala, creating a performance-efficient Scala programs, and tuning Spark jobs for optimal performance.
• Developed big data ingestion framework to process multi TB data including data quality checks, transformation, and stored as efficient storage formats like parquet and loaded into Amazon S3 using Spark Scala API and Spark.
• Built a POC sentiment analyzer engine using NLP (Natural Language Processing) and Python.
• Developed an internal search engine to look up news transcripts for NBC news shows, by creating a python parser to convert those transcripts to JSON documents and loading into Elasticsearch; implemented automated data ingestion process for indexing new transcripts as they become available.
• Created data transformation processes for NBC social media posts from Facebook, Twitter, YouTube, and Instagram, including repairing text encoding discrepancies and keeping character representation consistent using Python.

Technologies Used: Scala, Python, Java, Hive, Pig, NLP, pandas, Unix shell, Apache Spark,PySpark, Amazon EMR, Elasticsearch, Red Hat Enterprise Linux.

Confidential

Lead Big Data Engineer

Responsibilities:

• Designed and implemented Big Data analytics platform for handling data ingestion, compression, transformation and analysis of 30+ TB payroll and HR data. Implement automation, traceability, and transparency for every step of the process to build trust in data and streamline data science efforts using Python, Java, Hadoop streaming, Apache Spark, Spark SQL, Scala, Hive, and Pig.
• Designed highly efficient data model for optimizing large-scale queries utilizing Hive complex data types and Parquet file format.
• Performed data validation and transformation using Python and Hadoop streaming.
• Developed highly efficient Pig Java UDFs utilizing advanced concept like Algebraic and Accumulator interface to populate Confidential Benchmarks cube metrics.
• In order to improve performance, scalability,and memory usage ofprocessing large volume of payroll data, adopt Spark and Spark SQL to build Confidential Benchmarks cubes and populate cube annual and quarter metrics using Scala.
• Perform Big Data analysis using Scala, Spark, Spark SQL, Hive, Mlib, Machine Learn algorithms.

Technologies Used: Scala, Python, Java, Pig, Hive, Unix shell, Apache Spark 1.2, Spark SQL, PySpark, CDH 5.3, Red Hat Enterprise Linux.

Confidential

Big Data Engineer/Data Architect (Consultant)

Responsibilities:

• Designed and developed an automatic data processing pipeline, including data ingestion, compression, transformation, loading data into Hive database to process CDR (call detail records) using Unix Shell, Hadoop streaming, Python, Java, Hive, Pig, and UDFs.
• Performed data profiling and transformation on the raw data using Pig, Python, and Java
• Designed efficient data model for loading transformed data into Hive database.
• Created analytics reports using Hive.
• Lead architecture and design of data processing, warehousing and analytics initiatives.
• Analyzed the business requirements and data, designed, developed and implemented highly effectively, highly scalable ETL processes for a fast, scalable data warehouses.
• Developed various Oracle SQL scripts, PL/SQL packages, procedures, functions, and java code for dataExtraction, transformation, and data load.
• Performed SQL Query and Database tuning for high BI reporting performance.

Technologies Used: Python, Java, Hive, Unix shell, Hadoop, ETL(Talend), Oracle SQL, PL/SQL, Oracle 11.2g, Red Hat Enterprise Linux

Confidential

Big Data Engineer/ETL Architect

Responsibilities:

• Built Big Data analytical framework for processing healthcare data for medical research using Python, Java, Hadoop, Hive and Pig. Integrated R scripts with MapReduce jobs.
• Perform data transformation using Pig, Python, and Java
• Designed efficient data model for loading transformed data into Hive database.
• Created analytics reports using Hive.
• Designed, developed, and implemented conceptual, logical and physical data models for highly scalable and high performance relational database systems.
• Designed, developed, and implemented highly scalable and efficient ETL flows using SSIS.
• Extensive experience in tuning SQL query and multi-terabyte databases for high performance; Utilized new features of SQL Server 2012, such as Columnstore indexes, table partitioning to increase performance in data warehousing.
• Developed various T-SQL Scripts, Stored Procedures, and UDFs, C# code for data extraction, transformations and data load.
• Performed SSIS tuning for efficiency ETL and utilized best practices SSIS design pattern to improve ETL performance and scalability.

Technologies Used: Python, Java, Hadoop Pig and Hive, Unix shell, T-SQL, Oracle SQL, PL/SQL, Oracle11g, SSIS/SSRS/SSAS 2012/2008R2, Microsoft SQL Server 2012/20008R2, Visual C#.NET, Sybase15.5, CentOS

Confidential, New York

Senior BI/Database Developer/DBA

Responsibilities:

• Designed, developed, and implemented SSIS framework and templates for ETL efficiency.
• Designed, developed, and implemented robust, efficient ETL for processing of healthcare data from various sources into Data Warehouse, ODS, Data Marts using SSIS.
• Led design, development, implementation of SSRS reports and development and management of the Microsoft BI reporting environment.
• Extensive experience in SQL programming, stored procedure creation and optimization as well as tuning and maintenance of highly available and highly transactional databases.
• Designed, developed, and implemented conceptual, logical and physical data models for analytics reporting with scalability in mind.
• Developed various SQL scripts, Stored Procedures, UDFs, views, triggers, and tables for data extraction, transformations and data loads.
• Extensive experience in tuning SQL query and Multi-Terabyte database for high performance.

Technologies Used: T-SQL, SSIS/SSAS/SSRS 2012/2008R2, Windows PowerShell, Visual C#.NET, ASP.NET, SQL CLR, XML, Microsoft SQL Server 2000/2005/2008 R2/2012, Crystal Report

Confidential

Software Engineer

Responsibilities:

• Developed Radioisotope Inventory Web application using Java, JSP, and Oracle; designed a user-friendly Web interface using HTML and JavaScript.
• Performed the logical and physical data model design to support the development of Web Based Radioisotope Inventory Control System.
• Performed SQL, database tuning and optimization to increase the performance of application and data retrieval.
• Developed ETL processes and implemented Perl and SQL*Loader scripts to load data from legacy Informix database to Oracle database
• Implemented reporting solutions using Crystal Report
• Designed and developed software using C, java, and Perl to facilitate manipulation of brain images created with PET (Position Emission Tomography) and MRI (Magnetic Resonance Imaging) modalities for laboratory’s research in brain aging and development.
• Performed administration duties of SunOS/Solaris UNIX systems that supported PET and MRI modalities; Developed automation shell scripts for system and data management.

Technologies Used: Java, JavaScript, C, JSP, Perl, HTML, CSS, SQL, PL/SQL, Oracle 9i/10g, Crystal Report, Unix shell, Red Hat Linux, SunOS/Solaris