PROFESSIONAL SUMMARY:

• Over 8+ years of experience in Data Engineering, Data Pipeline Design, Development and Implementation as a Sr. Data Engineer/Data Developer and Data Modeler.
• Strong experience in Software Development Life Cycle (SDLC) including Requirements Analysis, Design Specification and Testing as per Cycle in both Waterfall and Agile methodologies.
• Strong experience in writing scripts usingPythonAPI, PySpark API and Spark API for analyzing the data.
• Extensively usedPythonLibraries PySpark, Pytest, Pymongo, cxOracle, PyExcel, Boto3, Psycopg, embedPy, NumPy and Beautiful Soup.
• Migrated an existing on - premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
• Hands On experience on Spark Core, Spark SQL, Spark Streaming and creating the Data Frames handle in SPARK with Scala.
• Experience in NoSQL databases and worked on table row key design and to load and retrieve data for real time data processing and performance improvements based on data access patterns.
• Extensive experience in Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, and Map Reduce concepts.
• Experience in building large scale highly available Web Applications. Working knowledge of web services and other integration patterns.
• Developed Simple to complex Map/reduce and Streaming jobs using Java and Scala language.
• Developed Hive scripts for end user / analyst requirements to perform ad hoc analysis.
• EMR with Hive to handle less important bulk ETL jobs.
• Hands-on use of Spark andScalaAPI's to compare the performance of Spark with Hive and SQL, and Spark SQL to manipulate Data Frames inScala.
• Expertise in Python andScala, user-defined functions (UDF) for Hive and Pig using Python.
• Experience in developing Map Reduce Programs using Apache Hadoop for analyzing the big data as per the requirement.
• Hands on Spark MLlib utilities such as including classification, regression, clustering, collaborative filtering, dimensionality reduction.
• Hands-on experience in developing and deploying enterprise-based applications using major Hadoop ecosystem components like MapReduce, YARN, Hive, HBase, Flume, Sqoop, Spark MLlib, Spark GraphX, Spark SQL, Kafka.
• Proficient with Spark Core, Spark SQL, Spark MLlib, Spark GraphX and Spark Streaming for processing and transforming complex data using in-memory computing capabilities written in Scala. Worked with Spark to improve efficiency of existing algorithms using Spark Context, Spark SQL, Spark MLlib, Data Frame, Pair RDD's and Spark YARN.
• Experience in application of various data sources like Oracle SE2, SQL Server, Flat Files and Unstructured files into a data warehouse.
• Able to use Sqoop to migrate data between RDBMS, NoSQL databases and HDFS.
• Experience in Extraction, Transformation and Loading (ETL) data from various sources into Data Warehouses, as well as data processing like collecting, aggregating and moving data from various sources using Apache Flume, Kafka, PowerBI and Microsoft SSIS.
• Worked with various text analytics libraries like Word2Vec, GloVe, LDA and experienced with Hyper Parameter Tuning techniques like Grid Search, Random Search, model performance tuning using Ensembles and Deep Learning.
• Skilled in System Analysis, E-R/Dimensional Data Modeling, Database Design and implementing RDBMS specific features.
• Knowledge of working with Proof of Concepts (PoC's) and gap analysis and gathered necessary data for analysis from different sources, prepared data for data exploration using data munging and Teradata.
• Well experience in Normalization and De-Normalization techniques for optimum performance in relational and dimensional database environments.
• Experience in developing customizedUDF’sin Python to extend Hive and Pig Latin functionality.
• Expertise working with AWS cloud services like EMR, S3,Redshift, EMR cloud watch, for big data development.
• Good working knowledge of Amazon Web Services(AWS) Cloud Platform which includes services likeEC2,S3,VPC,ELB, IAM, DynamoDB, Cloud Front, Cloud Watch, Route 53, Elastic Beanstalk (EBS), Auto Scaling, Security Groups, EC2 Container Service (ECS), Code Commit, Code Pipeline, Code Build, Code Deploy,DynamoDB, Auto Scaling, Security Groups, Red shift, CloudWatch, CloudFormation, CloudTrail, Ops Works, Kinesis, IAM, SQS, SNS, SES.
• Experience in Data Analysis, Data Profiling, Data Integration, Migration, Data governance and Metadata Management, Master Data Management and Configuration Management.
• Experience in developing customizedUDF’sin Python to extend Hive and Pig Latin functionality.
• Expertise in designing complex Mappings and have expertise in performance tuning and slowly changing Dimension Tables and Fact tables
• Extensively worked with Teradata utilities Fast export, and Multi Load to export and load data to/from different source systems including flat files.
• Experienced in building Automation Regressing Scripts for validation of ETL process between multiple databases like Oracle, SQL Server, Hive, and Mongo DB usingPython.
• Proficiency in SQL across several dialects (we commonly write MySQL, PostgreSQL, Redshift, SQL Server, and Oracle)
• Expert in building Enterprise Data Warehouse or Data warehouse appliances from Scratch using both Kimball and Inmon’s Approach.
• Experience in designing star schema, Snowflake schema for Data Warehouse, ODS architecture.
• Skilled in System Analysis, E-R/Dimensional Data Modeling, Database Design and implementing RDBMS specific features.
• Well experience in Normalization and De-Normalization techniques for optimum performance in relational and dimensional database environments.
• Good knowledge of Data Marts, OLAP, Dimensional Data Modeling with Ralph Kimball Methodology (Star Schema Modeling, Snow-Flake Modeling for FACT and Dimensions Tables) using Analysis Services.
• Strong analytical and problem-solving skills and the ability to follow through with projects from inception to completion.
• Ability to work effectively in cross-functional team environments, excellent communication, and interpersonal skills.

TECHNICAL SKILLS:

Big Data Tools: Hadoop Ecosystem Map Reduce, Spark 2.3, Airflow 1.10.8, Nifi 2, HBase 1.2, Hive 2.3, Pig 0.17 Sqoop 1.4, Kafka 1.0.1, Oozie 4.3, Hadoop 3.0

BI Tools: SSIS, SSRS, SSAS.

Data Modeling Tools: Erwin Data Modeler, ER Studio v17

Programming Languages: SQL, PL/SQL, and UNIX.

Methodologies: RAD, JAD, System Development Life Cycle (SDLC), Agile

Cloud Platform: AWS, Azure, Google Cloud.

Cloud Management: Amazon Web Services (AWS)- EC2, EMR, S3, Redshift, EMR, Lambda, Athena

Databases: Oracle 12c/11g, Teradata R15/R14.

OLAP Tools: Tableau, SSAS, Business Objects, and Crystal Reports 9

ETL/Data warehouse Tools: Informatica 9.6/9.1, and Tableau.

Operating System: Windows, Unix, Sun Solaris

PROFESSIONAL EXPERIENCE:

Confidential, Dallas TX

Sr. Data Engineer

Responsibilities:

• Performed data analysis and developed analytic solutions. Data investigation to discover correlations / trends and the ability to explain them.
• Worked with Data Engineers, Data Architects, to define back-end requirements for data products (aggregations, materialized views, tables - visualization)
• Developed frameworks and processes to analyze unstructured information. Assisted in Azure Power BI architecture design
• Experienced with machine learning algorithm such as logistic regression, random forest, XGboost, KNN, SVM, neural network, linear regression, lasso regression and k - means
• Implemented Statistical model and Deep Learning Model (Logistic Regression, XGboost, Random Forest, SVM, RNN, CNN).
• Designing and Developing Oracle PL/SQL and Shell Scripts, Data Import/Export, Data Conversions and Data Cleansing.
• Responsible for importing data from PostgreSQL to HDFS, HIVE using SQOOP tool.
• Experienced in migrating HiveQL into Impala to minimize query response time.
• Implemented Avro and parquet data formats for apache Hive computations to handle custom business requirements.
• Designed and implemented Sqoop for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2.
• Used Sqoop to channel data from different sources of HDFS and RDBMS
• Developing Spark applications using Scala and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats. Using Kafka and integrating with the Spark Streaming. Developed data analysis tools using SQL andPythoncode.
• Authoring Python (PySpark) Scripts for custom UDF’s for Row/ Column manipulations, merges, aggregations, stacking, data labeling and for all Cleaning and conforming tasks. Migrate data from on-premises to AWS storage buckets.
• Developed various Mappings with the collection of all Sources, Targets, and Transformations using Informatica Designer
• Developed Mappings using Transformations like Expression, Filter, Joiner and Lookups for better data messaging and to migrate clean and consistent data
• Designed and implemented Sqoop for the incremental job to read data from DB2 and load to Hive tables and connected to Tableau for generating interactive reports using Hive server2.
• Used Sqoop to channel data from different sources of HDFS and RDBMS.
• Developed Spark applications using Pyspark and Spark-SQL for data extraction, transformation and aggregation from multiple file formats.
• Used Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS using Python and NoSQL databases such as HBase and Cassandra
• Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS.
• Used Apache NiFi to copy data from local file system to HDP.
• Worked on Dimensional and Relational Data Modeling using Star and Snowflake Schemas, OLTP/OLAP system, Conceptual, Logical and Physical data modeling using Erwin.
• Automated the data processing with Oozie to automate data loading into the Hadoop Distributed File System
• Built the machine learning model include: SVM, random forest, XGboost to score and identify the potential new business case with Python Scikit-learn.
• Experience in Converting existing AWS Infrastructure to Server less architecture(AWS Lambda, Kinesis),deploying viaTerraformand AWS Cloud Formation templates.
• Worked onDocker containerssnapshots, attaching to a running container, removing images, managing Directory structures and managing containers.
• Experienced in day - to-day DBA activities includingschema management, user management(creating users, synonyms, privileges, roles, quotas, tables, indexes, sequence),space management(table space, rollback segment),monitoring(alert log, memory, disk I/O, CPU, database connectivity),scheduling jobs, UNIX Shell Scripting.
• Expertise in usingDocker to run and deploy the applications in multiple containers likeDocker SwarmandDocker Wave.
• Developed complexTalend ETL jobstomigratethe data from flat files to database. DevelopedTalend ESBservices and deployed them onESBservers on different instances.
• Architect and design serverless application CI/CD by using AWS Serverless (Lambda) application model.
• Developedstored procedures/views in Snowflakeand use inTalendfor loading Dimensions and Facts.
• Developed merge scripts toUPSERTdata intoSnowflakefrom an ETL source.

Environment: Hadoop, Map Reduce, HDFS, Hive, Sqoop, Spring Boot, Cassandra, Swamp, Data Lake, Sqoop, Oozie, Kafka, Spark, Scala, Java, AWS, GitHub, Docker, Talend Big Data Integration, Solr, Impala, Oracle, Sql Server, MySQL, No SQL, MongoDB, Hbase, Cassandra, Unix, Shell Scripting

Confidential

Big Data Engineer

Responsibilities:

• Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines
• Designed several DAGs (Directed Acyclic Graph) for automating ETL pipelines
• Performed data extraction, transformation, loading, and integration in data warehouse, operational data stores and master data management
• Strong understanding of AWS components such as EC2 and S3
• Created yaml files for each data source and including glue table stack creation
• Worked on a python script to extract data from Netezza databases and transfer it to AWS S3
• Developed Lambda functions and assigned IAM roles to run python scripts along with various triggers (SQS, EventBridge, SNS)
• Created a Lambda Deployment function, and configured it to receive events from S3 buckets
• Writing UNIX shell scripts to automate the jobs and scheduling cron jobs for job automation using commands with Crontab.
• Developed various Mappings with the collection of all Sources, Targets, and Transformations using Informatica Designer
• Developed Mappings using Transformations like Expression, Filter, Joiner and Lookups for better data messaging and to migrate clean and consistent data
• Installing IBM Http Server, WebSphere Plugins and WebSphere Application Server Network Deployment (ND).
• Worked on setting up high availability for major production cluster and designed automatic failover control using zookeeper and quorum journal nodes.
• Provide troubleshooting and best practices methodology for development teams.
• This includes process automation and new application onboarding.
• Produce unit tests for Spark transformations and helper methods. Design data processing pipelines.
• Configuring IBM Http Server, WebSphere Plugins and WebSphere Application Server Network Deployment (ND) for user work-load distribution.
• Multiple batch jobs were written for processing hourly and daily data received through multiple sources like Adobe, No-SQL databases.
• Leveraged cloud and GPU computing technologies for automated machine learning and analytics pipelines, such as AWS, GCP
• Designed and implemented configurable data delivery pipeline for scheduled updates to customer facing data stores built with Python
• Proficient in Machine Learning techniques (Decision Trees, Linear/Logistic Regressors) and Statistical Modeling
• Compiled data from various sources to perform complex analysis for actionable results
• Measured Efficiency of Hadoop/Hive environment ensuring SLA is met
• Optimized the TensorFlow Model for efficiency
• Analyzed the system for new enhancements/functionalities and perform Impact analysis of the application for implementing ETL changes
• Implemented a Continuous Delivery pipeline with Docker, and Git Hub and AWS
• Built performant, scalable ETL processes to load, cleanse and validate data
• Collaborate with team members and stakeholders in design and development of data environment
• Preparing associated documentation for specifications, requirements, and testing

Environment: AWS, Gcp, Bigquery, Gcs Bucket, G-Cloud Function, Apache Beam, Cloud Dataflow, Cloud Shell, Gsutil, Bq Command Line Utilities, Dataproc, Cloud Sql, Mysql, Posgres, Sql Server, Python, Scala, Spark, Hive, Spark-Sql

Confidential, Bowie, MD

Data Engineer

Responsibilities:

• Gathered business requirements, definition and design of the data sourcing, worked with the data warehouse architect on the development of logical data models.
• Created sophisticated visualizations, calculated columns and custom expressions anddeveloped Map Chart, Cross table, Bar chart, Tree map and complex reports which involves Property Controls, Custom Expressions.
• Investigated market sizing, competitive analysis and positioning for product feasibility. Worked on Business forecasting, segmentation analysis and Data mining.
• Automated Diagnosis of Blood Loss during Emergencies and developed Machine Learning algorithm to diagnose blood loss.
• Extensively used Agile methodology as the Organization Standard to implement the data Models. Used Micro service architecture with Spring Boot based services interacting through a combination of REST and Apache Kafka message brokers.
• Created several types of data visualizations using Python and Tableau. Extracted Mega Data from AWS using SQL Queries to create reports.
• Performed reverse engineering using Erwin to redefine entities, attributes, and relationships existing database.
• Analyzed functional and non-functional business requirements and translate into technical data requirements and create or update existing logical and physical data models. Developed a data pipeline using Kafka to store data into HDFS.
• Performed Regression testing for Golden Test Cases from State (end to end test cases) and automated the process usingpythonscripts.
• DevelopedSparkjobs using Scala for faster real-time analytics and usedSparkSQL for querying
• Generated graphs and reports using ggplot package in RStudio for analytical models. Developed and implemented R and Shiny application which showcases machine learning for business forecasting.
• Developed predictive models using Decision Tree, Random Forest, and Naïve Bayes.
• Used pandas, NumPy, seaborne, SciPy, Matplotlib, Scikit-learn, NLTK in Python for developing various machine learning algorithms. Expertise inR, Matlab, pythonand respective libraries.
• Research on Reinforcement Learning and control (TensorFlow, Torch), andmachinelearning model (Scikit-learn).
• Hands on experience in implementing Naive Bayes and skilled inRandom Forests, Decision Trees, Linear,and Logistic Regression, SVM, Clustering, Principal Component Analysis.
• Performed K-means clustering, Regression andDecision Treesin R. Worked on data cleaning and reshaping, generated segmented subsets using NumPy and Pandas in Python.
• Implemented various statistical techniques to manipulatethe datalike missingdataimputation, principal component analysis and sampling.
• Worked on R packages to interface with Caffe Deep Learning Framework. Perform validation on machine learning output from R.
• Applied different dimensionality reduction techniques like principal component analysis (PCA) and t-stochastic neighborhood embedding(t-SNE) on feature matrix.
• Performed univariate and multivariate analysis on the data to identify any underlying pattern in the data and associations between the variables.
• Responsible for design and development of Python programs/scripts to prepare transform and harmonize data sets in preparation for modeling.
• Worked with Market Mix Modeling to strategize the advertisement investments to better balance the ROI on advertisements.
• Implemented clustering techniques like DBSCAN, K-means, K-means++ and Hierarchical clustering for customer profiling to design insurance plans according to their behavior pattern.
• Used Grid Search to evaluate the best hyper-parameters for my model and K-fold cross validation technique to train my model for best results.
• Worked with Customer Churn Models including Random forest regression, lasso regression along with pre-processing of the data.
• Used Python 3.X (NumPy, SciPy, pandas, scikit-learn, seaborn) and Spark 2.0 (PySpark, MLlib) to develop variety of models and algorithms for analytic purposes.
• Performed Data Cleaning, features scaling, features engineering using pandas and NumPy packages in python and build models using deep learning frameworks
• Implemented application of various machine learning algorithms and statistical modeling like Decision Tree, Text Analytics, Sentiment Analysis, Naive Bayes, Logistic Regression and Linear Regression using Python to determine the accuracy rate of each model
• Implemented Univariate, Bivariate, and Multivariate Analysis on the cleaned data for getting actionable insights on the 500-product sales data by using visualization techniques in Matplotlib, Seaborn, Bokeh, and created reports in Power BI.
• Used PCA to reduce dimension and compute eigenvalue and eigenvector and used OpenCV to analysis the CT scan pictures to figure out the disease in CT scan.
• Processed the image data through the Hadoop distributed system by using Map and Reducethen stored into HDFS.
• Created Session Beans and controller Servlets for handling HTTP requests from Talend
• Performed Data Visualization and Designed Dashboards with Tableau and generated complex reports including chars, summaries, and graphs to interpret the findings to the team and stakeholders.
• Wrote documentation for each report including purpose, data source, column mapping, transformation, and user group.
• Utilized Waterfall methodology for team and project management.
• Used Git for version control with Data Engineer team and Data Scientists colleagues.

Environment: Spark, YARN, HIVE, Pig, Scala, Mahout, NiFi, TDD, Python, Spring Boot, Hadoop, Azure, Dynamo DB, Kibana, NOSQL, Sqoop, MYSQL.

Confidential

Data Analyst / Hadoop Developer

Responsibilities:

• Imported Legacy data from SQL Server and Teradata into Amazon S3.
• Created consumption views on top of metrics to reduce the running time for complex queries.
• Exported Data into Snowflake by creating Staging Tables to load Data of different files from Amazon S3.
• As a part of Data Migration, wrote many SQL Scripts for Mismatch of data and worked on loading the history data from Teradata SQL to snowflake.
• Developed SQL scripts to Upload, Retrieve, Manipulate and handle sensitive data (National Provider Identifier Data I.e. Name, Address, SSN,Phone No) in Teradata, SQL Server Management Studio and Snowflake Databases for the Project
• Worked on to retrieve the data from FS to S3 using spark commands
• Built S3 buckets and managed policies for S3 buckets and used S3 bucket and Glacier for storage and backup on AWS
• Created performance dashboards in Tableau/ Excel / Power point for the key stakeholders
• Incorporated predictive modeling (rule engine) to evaluate the Customer/Seller health score using python scripts, performed computations and integrated with the Tableau viz.
• Worked with stakeholders to communicate campaign results, strategy, issues or needs.
• Analyzed marketing campaigns from various perspectives including CTR, conversion rates, seasonal/geographical trends, search queries, landing page, conversion funnel, quality score, competitors, distribution channel, etc. to achieve maximum ROI for clients.
• Understood Business requirements to the core and Came up with Test Strategy based on Business rules
• Implemented Defect Tracking process using JIRA tool by assigning bugs to Development Team
• Involved in Functional Testing, Integration testing, Regression Testing, Smoke testing and performance Testing. Tested Hadoop Map Reduce developed in python, pig, Hive
• Created Metric tables, End user views in Snowflake to feed data for Tableau refresh.
• Generated Custom SQL to verify the dependency for the daily, Weekly, Monthly jobs.
• Using Nebula Metadata, registered Business and Technical Datasets for corresponding SQL scripts
• Experienced in working with spark ecosystem using Spark SQL and Scala queries on different formats like text file, CSV file.
• Developed spark code and spark-SQL/streaming for faster testing and processing of data.
• Closely involved in scheduling Daily, Monthly jobs with Precondition/Postcondition based on the requirement.

Environment: Snowflake,Hadoop, Map Reduce,Spark SQL, Python, Pig,AWS, GitHub, EMR, Nebula Metadata,Teradata, SQL Server, Apache Spark, Sqoop

Confidential

Java/Hadoop Developer

Responsibilities:

• Involved in review of functional and non-functional requirements.
• Installed and configured Hadoop MapReduce, HDFS, Developed multiple MapReduce
• jobs in java for data cleaning and preprocessing.
• Installed and configured Pig and also written Pig Latin scripts.
• Wrote MapReduce job using Pig Latin. Involved in ETL, Data Integration and Migration.
• Imported data using Sqoop to load data from Oracle to HDFS on regular basis.
• Developing Scripts and Batch Job to schedule various Hadoop Program.
• Written Hive queries for data analysis to meet the business requirements.
• Creating Hive tables and working on them using Hive QL. Experienced indefining jobflows.
• Utilized various utilities like Struts Tag Libraries, JSP, JavaScript, HTML, & CSS.
• Build and deployed war file in WebSphere application server.
• Implemented Patterns such as Singleton, Factory, Facade, Prototype, Decorator, Business Delegate and MVC.
• Involved in frequent meeting with clients to gather business requirement & converting them to technicalspecification for development team.
• Adopted agile methodology with pair programming technique and addressed issues during system testing.
• Involved in Bug fixing and Enhancement phase, used find bug tool.
• Version Controlled using SVN.
• Developed application in Eclipse IDE. Experience in developingspring Bootapplications for transformations.
• Primarily involved in front-end UI using HTML5, CSS3, JavaScript, jQuery, and AJAX.
• Used struts framework to build MVC architecture and separate presentation from business logic.

Environment: Hadoop, MapReduce, HDFS, Hive, Java, Hadoop distribution of Cloudera, Pig, HBase, Linux, XML, Java 6, Eclipse, Oracle 10g, PL/SQL, MongoDB, Toad