SUMMARY:

• About 6 years of professional IT experience in Big Data Technologies, Data Management/Analysis and Java Based enterprise application using Java/Java EE technologies working with clients from financial services, banking, insurance and Information Technologies industries
• 2+ years of hands on experience in providing solutions for Big data using Hadoop 2.x, HDFS, Map Reduce, Hive, HBase, Sqoop, Flume, Zookeeper, Pig, YARN, Spark (SparkCore, SparkSQL, Spark Streaming), Oracle SQL
• Experience in developing MapReduce jobs with Java 8, Python 2.7/3.x in Hadoop
• Strong experience in writing custom UDFs in Java for Hive and Pig to extend functionality
• Exposure on the HBase distributed database and Zookeeper distributed configuration service
• Extensive experience in writing Pig scripts and Hive Queries for processing and analyzing large volumes of data structured in different level
• Extensive experience in importing and exporting data using Sqoop from HDFS/Hive/HBase to Relational Database Systems (RMDBS) and vice versa, capable of processing large sets of unclean, structured, semi - structured data and supporting systems applications architecture
• Experience in designing both time driven and data driven automated workflows with MapReduce and Pig jobs using Oozie
• Hands on experience in Zookeeper in managing and configuring in Name Node failure scenarios
• Experience in Hadoop Administration activities such as installation, configuration and management of clusters in Cloudera (CDH4, CDH5) & Hortonworks (HDP2.0-HDP2.4) Distributions using Cloudera Manager & A
• Experience in monitoring systems and services, architecture design and implementation of Hadoop deployment, configuration management, backup and disaster recovery systems and procedures
• Experience with Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Spark MLlib, Data Frame, Pair RDD’s, Spark YARN
• Experience in developing Spark applications with Scala and Python 2.7/3.x (Spark core-RDD, Spark Data Frames and Spark Streaming APIs)
• Experience in collecting, aggregating, and moving large amounts of streaming data with Flume, Kafka, Spark Streaming; Good conceptual knowledge and hands on experience with streaming Apache Kafka APIs
• Proficient in Java, Python, R, and Scala in Apache Spark, knowledge of numpy, scipy, pandas, scikit-learn
• Strong experience in core Java, data structure, algorithm design, Object-Oriented Design (OOD) and Java components like Collections Framework, Exception handling, I/O system.
• Hands on experience in managing multi-tenant Hadoop cluster on Amazon Web Services (AWS EC2, EMR)
• Strong Database experience on RDBMS (SQL Server, MySQL) with SQL programming skills in creating Packages, Stored Procedures, Functions, Triggers & Cursors
• Knowledge on architecture and functionality of NoSQL DB like HBase and Cassandra
• Experience in Extraction, Transformation and Loading (ETL) of data from multiple sources like Flat files, XML files, JSON, AVRO, Parquet and relational tables
• Strong experience in UNIX shell scripting to automate file preparation and database loads
• Experience in setting up the Linux environments, Passwordless SSH, creating file systems, disabling firewalls, Swappiness, Selinux and installing Java
• Strong experience in Machine learning and Data Mining, knowledge of variety of machine learning techniques (NLP, predictive modeling, recommendation systems, artificial neural networks, etc.)
• Experience in implementation Machine learning and Data Mining algorithms such as Support Vector Machine (SVM), decision trees, logistic regression, clustering, neural networks, graphical models, etc. and extending open-source libraries such as Mahout, Spark MLLib, GraphLab.
• Experience in data analysis and visualization with Tableau, SPSS, MATLAB, Stata and Weka
• Excellent communication, Analytical and Management skills; Ability to blend technical expertise with strong Conceptual and Analytical skills to provide quality solutions and result-oriented problem-solving technique and leadership skills

TECHNICAL SKILLS:

Big data Technologies\ Databases\: Hadoop 2.x, Spark v1.6, MapReduce \ HBase v0.98.0, MySQL, SQL Server, \ v2.6.x, HDFS, YARN, Apache Pig 0.15.0, \ Oracle, DB2, Mongo DB\ Hive 2.0.0, HBase 1.1.4, Flume 1.6.0, \: Sqoop 1.4.6, Oozie 4.2, Zookeeper 3.5.2, \Spark Kafka 0.10.1

Programming & Query languages\ Operating Systems: Core Java 1.7/1.8, Scala, SQL, Python \ Windows, Linux/Unix, Mac OS, Ubuntu\: 2.7/3.x, R

Data Analysis & Viz \ Cloud Platform\: Tableau 9.x, matplotlib, MATLAB\ Amazon Web Service, Heroku\

PROFESSIONAL EXPERIENCE:

Confidential, New York City, NY

Big Data Developer

Responsibilities:

• Extracted, transformed, and loaded ( ETL ) data from multiple federated data sources with DataFrames in Spark.
• Applied daily ad-hoc data analysis and pulled data from Hadoop with Hive (HiveQL).
• Utilized SparkSQL to extract and process data by parsing with Datasets or RDD’s in Hive Context, with transformations and actions (map, flatMap, filter, reduce, reduceByKey).
• Read data from JSON file format and Parquet file format to run SQL queries on data with SparkSQL .
• Created end to end Spark applications using Scala to perform various data cleansing, validation, loaded the data into Spark RDD and do in memory data Computation to generate the Output response.
• Involved in using Spark API over Hadoop YARN as execution engine for data analytics using Hive and submitted the data to BI team for generating reports, after the processing and analyzing of data in Spark SQL.
• Developed and maintained Workflow Scheduling Jobs in Oozie for importing data from RMDBS to Hive.
• Designed workflows and coordinators in Oozie to automate and parallelize Hive jobs on Apache Hadoop environment by Hortonworks (HDP 2.4)
• Developed shell scripts to periodically perform incremental import of data from third party API to Amazon AWS.
• Applied Spark MLlib to train analytical models including: linear regression, decision trees, logistic regression and k-means. Evaluated parallelized Machine Learning model accuracy and computed metrics using evaluators.
• Optimized existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD’s
• Integrated Visualizations into Spark application using Databricks and popular visualization libraries ( ggplot, matplotlib ).
• Developed interactive Dashboards using Tableau.

Environment: Hadoop 2.0, MapReduce v2.6x, Yarn, HBase, Pig, Sqoop, Oozie, Core Java, Scala, Kafka 0.8, Python, Eclipse, Spark MLlib, Tableau 9.2, Hortonworks (HDP 2.4)

Confidential, Newark, NJ

Data Modeler/Architect

Responsibilities:

• Installed and configured Hadoop HDFS developed MapReduce jobs in java for data cleaning.
• Used MapReduce and Spark for cleaning, pre-processing, extracting relevant fields, performing pre-aggregations, joins of raw stream data and converting text data into suitable file formats.
• Developed Pig Latin scripts to extract the data from the web server output files to load into HDFS .
• Installed and configured Zookeeper for Hadoop cluster. Worked on setting up high availability for cluster and designed automatic failover using zookeeper.
• Used Kafka, Flume for building robust and fault tolerant Data Ingestion pipeline for transporting streaming web log data into HDFS.
• Loaded data into the cluster from dynamically generated files using Flume and from relational database management systems using Sqoop.
• Created processing pipelines including transformations, estimations and evaluations of analytical models.
• Configured Hive metastore, which stores the metadata for Hive tables and partitions in a relational database.
• Created multiple Hive tables, implemented partitioning, dynamic partitioning and bucketing in Hive for efficient data access.
• Extended the capabilities of DataFrames using User Defined Functions in Scala
• Developed HBase tables to store variable data format of input data coming from different sources.
• Performed tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
• Loaded the data into Spark RDD and do in memory data Computation to generate the Output response.
• Handled large scale datasets with Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & Efficient Joins, Transformations and other during ingestion processes.
• Developed multiple POCs with PySpark and deployed on the YARN cluster, compared the performance of Spark with Hive and SQL.
• Applied Spark MLlib to build statistical model to classify and predict.

Environment: Zookeeper, Sqoop 1.4.6, Hadoop 2.0, YARN, Spark 1.0, MapReduce v2.6x, Kafka 0.8, Flume 1.5, Hive, Pig, HBase 0.9, RMDBS, Eclipse, Tableau 9.2, Scala

Confidential

Risk Management Data Scientist

Responsibilities:

• Defined and applied enterprise level standards and influence business strategy.
• Coordinated solutions from technical perspective and minimize technical risk.
• Transferred the analyzed data across relational database from HDFS using Sqoop enabling BI team to visualize analytics.
• Developed workflows for ETL and Data Science to generate the risk indicators and risk scores. Calculated composite risk scores on daily basis and generate alerts for Insider Threat and Bankers Supervision Program.
• Constructed data model of financial products including credit scoring model, credit risk model, credit decision process, risk management.
• Responsible for project management, collaborated with products, operation and research & development departments, ensured accurate implementation of quantification models and strategies, monitored and optimized models after projects going live .
• Predicted customer behavior by cleansing and analyzing big data for risk management, credit scoring, business decisions, etc.
• Applied machine learning, text mining and other technologies to mine the company’s massive datasets and discover potential patterns.
• Tested data mining models using clustering, association rules, classification, time series, text mining and text classification using Mahout and Python.
• Built different time series techniques like freehand technique, least square technique and moving average technique to predict future data points using R.

Environment: HDFS, MapReduce, Sqoop, RMDBS, Python, Java, Spark, Hive, SQL

Confidential

Data Warehouse Architect

Responsibilities:

• Responsible for data strategy and design of high volume transactional, operational, analytical and reporting solutions for consumer and company business.
• Designed jobs involving various cross reference lookups and joins, shared containers which can be used in multiple jobs.
• Worked side by side with t business and development teams to develop data domain strategies and conceptual and logical data models.
• Partnered with end users, business analysist, ETL developers and database administrators to design and develop high performing databases and maintain consistent data element definitions in modeling tool, database and enterprise data dictionary.
• Developed Hive queries to process data and generate cubes for data visualization.
• Implemented schema extraction for Parquet, Avro and JSON file formats in Hive.
• Created Data Models in Hive including tables, partitions and buckets.
• Developed MapReduce and data cleansing jobs using Pig and Python.
• Assisted in defining and documenting patterns and solutions supporting large-scale database platforms including big data (Hadoop), relational, and No-SQL data solutions.
• Led development of transaction analysis extension models relying on decision trees, random forest, logistics regression.
• Defined and enforce consistency in design, storage, movement, security, quality, & usage of information required to support the business.
• Partnered with key business and IT stakeholders to define and drive adoption of data strategies including big data, microservices, and event driven architecture
• Designed and developed data solutions and implement controls/governance across all data domains.
• Identified and governed appropriate data sourcing, data lineage, and metadata.

Environment: MapReduce, Pig, Python, Hive, SQL

Confidential

Java/J2EE Developer

Responsibilities:

• Analyzed the requirements and communicate to both Development and Testing teams, worked closely with business analysts in understanding the technical requirements of the project and prepared the use cases for different functionalities and designs.
• Developed and implemented business logic using Java, JSP, Servlets, XML.
• Wrote SQL queries for complex operations.
• Developed and supported of application using Core Java & J2EE technologies
• Defined Controller, action mappings using Spring framework.
• Created Hibernate mapping files, sessions, transactions, Query & Criteria’s to fetch the data from MySQL Database.
• Designed, developed and maintained the data layer using Hibernate.
• Used MySQL data query and JDBC for database connectivity.
• Created database program in SQL server to manipulate data accumulated by internet transactions.
• Responsible for coding SQL Statements and Stored procedures for back end communication using JDBC.
• Involved in working for Development & Maintenance of web applications using Java, J2EE, HTML, JavaScript.
• Implemented TFIDF classification algorithm using Lucene API to classify business data in JSON files into categories.
• Used Junit to do the code reviews and Unit testing for the application.
• Improved the coding standards, code reuse, and performance of the Extend application by making effective use of various design patterns (Business Delegate, View Helper, DAO, Value Object).
• Developed Sentiment Analysis approach using Stanford NLP Library to rank businesses based on customer reviews.

Environment: Core Java, JSP, MySQL, J2EE, Eclipse, JDBC, HTML, JavaScript