SUMMARY:

• Over 6 years of experience on Data manipulation, wrangling, model building and visualization with large data sets
• Design, build and maintain Big Data workflows to process billions of records in and out of data lake and Identity Graph
• Sound noledge on Hadoop ecosystem and hands - on experience with Hive, Spark, Kafka, Sqoop, HDFS
• Worked on NoSQL Databases such as HBase, Mongo Db and Cassandra
• Involved in complete Bigdata flow of the application starting from data ingestion from upstream to HDFS, processing and analyzing the data in HDFS
• Created end to end data pipelines using Big Data tools
• Productionizing Big Data Applications
• Experience in creating the data lakes in consultation with Data Warehousing teams. Defining the data layouts.
• Developing spark applications using python
• Experience in importing and exporting data from different RDBMS like MySql, Oracle and SQL Server into HDFS and Hive using Sqoop
• Good Knowledge about scalable, secure cloud architecture based on Amazon Web Services (AWS cloud services: EC2, EMR and S3)
• Worked on Machine Learning models like Linear, Logistic Regression, Decision Trees, Naive Bayes, SVM, Neural Networks, K-Nearest Neighbours, clustering (K-means, Hierarchical)
• Used Kafka to provide real-time analytics in real-time streaming data architectures
• Utilized partitioning and bucketing on large datasets to optimize the hive query based on the requirement
• Experience with different schedulers like TWS and Oozie
• Strong Experience with Scheduling, Hadoop cluster management, Administrative, operations using Oozie, Yarn
• Expertise in designing concise, pertinent visualizations using Tableau and publishing and presenting dashboards on web and desktop platforms
• Experience with onboarding new tools or technologies by carrying out different proof of concepts and defining different metrics for evaluation of tools or technologies
• Worked on multiple stages of Software Development Life Cycle including Development, Component Integration, Performance Testing, Deployment and Support Maintenance
• Strong communication skills, analytic skills, good team player and quick learner, organized and self-motivated

TECHNICAL SKILLS:

Programming: Python, R, SCALA

Python: Data Manipulation, NumPy, Pandas, Seaborn, Scikit learn (machine learning libraries and others)

Big Data: Spark, Hive, Kafka, Pig, Oozie, Flume, Sqoop, HBase, Hadoop, HDFS

Spark: Spark Core, Spark SQL, Spark Streaming, MLlib, GraphX, PySpark

Platforms: Ubuntu, Linux, MacOS

Analytical Tools: SQL, Jupyter Notebook, Apache Zeppelin, MS Excel

Methodologies: Agile

NoSQL: MongoDB, Cassandra

Others: AWS, S3, EC2, EMR, MySQL, PostgreSQL

EXPERIENCE:

Confidential - Malvern, PA

Big Data Engineer

Responsibilities:

• Involved in complete Bigdata flow of the application starting from data ingestion from upstream to HDFS, processing and analyzing the data in HDFS
• Defining the requirements for data lakes/pipe lines
• Developed end to end data pipelines
• Creating and loading the data into Hive tables and scheduling the data ingestion jobs
• Transforming the data received from source systems using python and creating the files to load into hive
• Perform data cleaning and transformations that is suitable for applying models using Pandas, NumPy
• Worked with analytics team in building predictive models using Logistic regression, Decision Tree, Support Vector Machine and KNN to predict customer churn
• Transforming the data to make it available for analytics jobs
• Migrate existing data processing from standalone or legacy technology scripts to Hadoop framework processing.
• Contribute to best practice for data extraction, integration and analysis
• Compile competitive information and external benchmarking data for Development.
• Structured analysis of portfolio, making recommendations to maximize value creation within budget and resourcing constraints
• Used Spark and Spark-SQL to read the parquet data and create the tables in hive using the Python API
• The Hive tables are created as per requirement were Internal or External tables defined with appropriate static, dynamic partitions and bucketing, intended for efficiency
• Writing Hive queries for data analysis to meet business requirements and experienced in Sqoop to import and export the data from Oracle & MySQL
• Developed Hive queries for the analysts by loading and transforming large sets of structured, semi structured data
• Worked on Spark SQL for joining multiple hive tables and write them to a final hive table and stored them on S3.

Environment: Hadoop, HDFS, Spark, Kafka, Hive, Python

Confidential - Windsor, CT

Data Engineer

Responsibilities:

• Responsible for data engineering functions such as data extraction, injection and transformation
• Importing and exporting data into HDFS and HIVE from a Oracle database using Sqoop
• Optimized Hive pipelines in Data lake by implementing Partitioning, and bucketing concepts for improving performance
• Exported the analysed data to the RDBMS using SQOOP for visualization and to generate reports for the BI team
• Stored the resultant data from transformation into HBase, MongoDB and also in parquet file format
• Worked closely with data scientists to assist on feature engineering, model frameworks, and model deployments at scaleWorked with application developers and DBAs to diagnose and resolve query performance problems
• Collaborated with Marketing, Finance, Business Development, Product & other teams to halp them uncover the insights from the data

Environment: HDFS, PIG, HIVE, Linux, HBase, Flume, Sqoop, Cloudera, Python, MongoDB, Cassandra, MySQL

Confidential

Data Engineer

Responsibilities:

• Extracted and Loaded customer data from databases to HDFS and HIVE tables using Sqoop
• Loading the data into Hive managed tables with partitions and buckets
• Performed data transformations, cleaning and filtering, using Hive
• Analyzed and studied customer behavior by running Hive queries
• Stored the resultant from transformation into parquet, avro file format
• Work closely with the business and analytics teams in gathering the system requirements
• Documentation of the day to day tasks

Environment: Hadoop, HDFS, Hive, Sqoop, Linux, Python

Confidential

SQL Developer

Responsibilities:

• Worked closely with all teams within the organization to understand business processes, gather requirements, understand complexities and end goals to come up with the best plan of execution
• Creating database objects like Tables, Stored Procedures, Views, User Defined Functions, Cursors and Triggers.
• Developed Report Services using SSRS
• Assisted managers and business analysts in developing reports, presentations, and analysis for upper management

ENVIRONMENT: Python, MYSQL, SQL