SUMMARY

• Data Engineering professional having around 8+ years of experience in a variety of data platforms, with hands on experience in Big Data Engineering and Data Analytics.
• Strong Expertise with tools in Hadoop Ecosystem and Big data stack which includes HDFS, Spark, Pyspark, MapReduce, Kafka, Hive, Oozie, Zookeeper, Ambari, Sqoop, HBase, and Impala.
• Strong Experience in Data Engineering, Data Pipeline Design, Development, Documentation, Deployment and Integration as a Sr. Data Engineer/Data Developer and Data Modeler.
• Experience in layers of Hadoop Framework - Storage (HDFS), Analysis (Pig and Hive), Engineering (Jobs and Workflows), extending the functionality by writing custom UDFs.
• Strong experience on Hadoop distributions like Horton works and Cloudera.
• Hands-on experience with Hadoop architecture and various components such as Hadoop File System HDFS, Job Tracker, Task Tracker, Name Node, Data Node and Hadoop MapReduce programming.
• Extensive knowledge of data architecture including designing pipelines, data ingestion, Hadoop/Spark architecture, data modeling, data mining, machine learning and advanced data processing.
• Involved in building Data Models and Dimensional Modeling with 3NF, Star and Snowflake schemas for OLAP and Operational data store (ODS) applications.
• Extensive experience in developing Data warehouse applications using Hadoop, Informatica, Oracle, Teradata, MS SQL server on UNIX and Windows platforms and experience in creating complex mappings using various transformations and developing strategies for Extraction, Transformation and Loading (ETL) mechanism by using Informatica.
• Proficient with Spark Core, Spark SQL, Spark MLlib, Spark GraphX, for processing and transforming complex data using in-memory computing capabilities written in Scala.
• Worked with Spark to improve efficiency of existing algorithms using Spark Context, Spark SQL, Spark MLlib, Data Frame, Pair RDD's and Spark YARN.
• Strong experience in writing scripts using Python API, PySpark API and Spark API for analyzing the data.
• Experience in analyzing/manipulating huge and complex data sets and finding insightful patterns and trends within structured, semi-structed and unstructured data.
• Extensively used Python Libraries PySpark, Pytest, Pymongo, cxOracle, PyExcel, Boto3, Psycopg, embedPy, Pandas, NumPy and Beautiful Soup.
• Experience in using Machine Learning algorithms and Predictive Modeling such as Linear Regression, Logistic Regression, Naïve Bayes, Decision Tree, Random Forest, KNN, Neural Networks, and K-means Clustering.
• Expert in working with Hive data warehouse tool-creating tables, data distribution by implementing partitioning and bucketing, writing, and optimizing the HiveQL queries.
• Experience in collection the real time streaming data and creating the pipeline for data from different source using Kafka and store data into HDFS and NoSQL using Spark.
• Extensive Shell/Python scripting experience and setup production systems on UNIX/LINUX environment.
• Experience in providing application support for Jenkins.
• Experience working with continuous integration framework, building regression-able code within data world using GitHub, Jenkins, and related applications.
• Experience in working with Amazon Web Services(AWS) Using EC2 for computing, Redshift Spectrum, Glue and S3 as storage mechanism.
• Strong working knowledge of Amazon Web Services (AWS) Cloud Platform which includes services like EC2, S3, VPC, ELB, IAM, DynamoDB, Cloud Front, Cloud Watch, Route 53, Elastic Beanstalk (EBS), Auto Scaling, Security Groups, EC2 Container Service (ECS), Code Commit, Code Pipeline, Code Build, Code Deploy, Dynamo DB, Auto Scaling, Security Groups, Red shift, CloudWatch, CloudFormation, CloudTrail, Ops Works, Kinesis, IAM, SQS, SNS, SES.
• Primarily involved in Data Migration using SQL, SQL Azure, Azure Storage, and Azure Data Factory, SSIS, PowerShell.
• Implemented medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB).
• Design, develop, and test dimensional data models using Star and Snowflake schema methodologies under the Kimball method.
• Experience in Google Cloud components, Google container builders and GCP client libraries and cloud SDK's.
• Experience in working with Flume for loading log files into Hadoop.
• Experience in working with NoSQL databases like HBase and Cassandra.
• Experience in designing star schema, Snowflake schema for Data Warehouse, ODS architecture.
• Hands on experience in developing SPARK applications using Spark tools like RDD transformations, Spark core, Spark MLlib and Spark SQL.
• Good knowledge of Data Marts, OLAP, Dimensional Data Modeling with Ralph Kimball Methodology (Star Schema Modeling, Snow-Flake Modeling for FACT and Dimensions Tables) using Analysis Services.
• Good working experience on using Sqoop to import data into HDFS from RDBMS and vice-versa.
• Develop effective working relationships with client teams to understand and support requirements, develop tactical and strategic plans to implement technology solutions, and effectively meet client expectations.
• An excellent team member capable to work efficiently in cross functional team. Excellent communication skills, strong analytical skills, problem solving skills and decision-making skills.

TECHNICAL SKILLS

Big Data/Hadoop Technologies: HDFS, Spark, Pyspark, YARN, Map Reduce, Hive, Pig, Sqoop, Kafka, Airflow, Flume, Impala, HBase, Zookeeper and Ozzie.

Programming: Python, Scala, Java, UNIX Shell script, SQL, PL/SQL

Data modelling Tools: ER Studio v17, Erwin Data Modeler

Databases: Snowflake(cloud), Oracle, Postgres, Teradata, SQL Server, MySQL IBM DB2

No SQL Database: HBase, Cassandra, Mongo DB

Operating Systems: UNIX, LINUX, Windows

Version Control: SVN, GIT, UCD, Ansible

Cloud Platform: AWS, Azure.

Cloud Management: Amazon Web Services (AWS)- EC2, EMR, Lambda, Redshift, S3, Glue

BI Tools: Power BI, Tableau

ETL /Datawarehouse Tools: Snowflake, Informatica, Talend

Security Management: Hortonworks Ambari, Cloudera Manager, Apache Knox, XA Secure, Kerberos

Development Methodologies: Agile/Scrum, Waterfall, RAD, JAD, System Development Life Cycle (SDLC)

PROFESSIONAL EXPERIENCE

Confidential, CA

Sr. Data Engineer

Responsibilities:

• Involved in designing and deploying multi-tier applications using all the AWS services like (EC2, Route53, S3, RDS, Dynamo DB, SNS, SQS, IAM) focusing on high-availability, fault tolerance, and auto-scaling in AWS Cloud Formation.
• Supporting Continuous storage in AWS using Elastic Block Storage, S3. Created Volumes and configured Snapshots for EC2 instances.
• Used Data Frame API in Scala for converting the distributed collection of data organized into named columns, developing predictive analytic using Apache Spark Scala APIs.
• Developed Python scripts using both Data frames/SQL/Data sets and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop.
• Developed Hive queries to pre-process the data required for running the business process.
• Worked on ETL Migration services by developing and deploying AWS Lambda functions for generating a serverless data pipeline which can be written to Glue Catalog and can be queried from Athena.
• Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
• Implementations of generalized solution model using AWS SageMaker.
• Extensive expertise using the core Spark APIs and processing data on an EMR cluster.
• Programmed in Hive, Spark SQL, Java and Python to streamline the incoming data and build the data pipelines to get the useful insights, and orchestrated pipelines.
• Worked on ETL pipeline to source these tables and to deliver this calculated ratio data from AWS to Datamart (SQL Server) & Credit Edge server.
• Experience in using and tuning relational databases (e.g. Microsoft SQL Server, Oracle, MySQL) and columnar databases (e.g. Amazon Redshift, Microsoft SQL Data Warehouse).

Environment: Hortonworks, Hadoop, HDFS, AWS Glue, AWS Athena, EMR, Pig, Sqoop, Hive, NoSQL, HBase, Shell Scripting, Python, Scala, Spark, SparkSQL, AWS, SQL Server, Tableau, ETL

Confidential, St. Louis, MO

Sr. Data Engineer

Responsibilities:

• Built S3 buckets and managed policies for S3 buckets and used S3 glacier for storage and backup on AWS
• Designed, built, and coordinate an automated build & release CI/CD process using Gitlab, Jenkins and Puppet on hybrid IT infrastructure.
• Involved in designing and developing Amazon EC2, Amazon S3, Amazon RDS, Amazon Elastic Load Balancing, Amazon SWF, Amazon SQS, and other services of the AWS infrastructure.
• Running build jobs and integration tests on Jenkins Master/Slave configuration.
• Conduct systems design, feasibility and cost studies and recommend cost-effective cloud solutions such as Amazon Web Services (AWS).
• Involved in maintaining the reliability, availability, and performance of Amazon Elastic Compute Cloud (Amazon EC2) instances.
• Managed Servers on the Amazon Web Services (AWS) platform instances using Puppet configuration management.
• Integrated services like GitHub, AWS Code pipeline, Jenkins, and AWS Elastic Beanstalk to create a deployment pipeline.
• Involved in complete SDLC life cycle - Designing, Coding, Testing, Debugging and Production Support.
• Coordinate/assist developers with establishing and applying appropriate branching, labeling/naming conventions using Git.
• Worked on JIRA for defect/issues logging & tracking and documented all my work using CONFLUENCE
• Branching, Merging, Release Activities on Version Control Tool GIT. Used GitHub as version control to store source code and implemented Git for branching and merging operations.

Environment: Jenkins, JIRA, Maven, GIT, AWS EMR/EC2/S3/Redshift, Python, Cassandra, Web logic, Unix Shell Scripting, SQL.

Confidential, Indianapolis, IN

Data Engineer

Responsibilities:

• Responsible for building scalable distributed data solutions using Hadoop.
• Demonstrated a strong comprehension of project scope, data extraction, design of dependent and profile variables, logic and design of data cleaning, exploratory data analysis and statistical methods.
• Used spark steaming APIs to perform necessary transformations for building the common learner data model which gets data from Kafka in near real time and persists into Hive.
• Developed Spark scripts by using Python as per the requirements.
• Developed real time data pipeline using Spark to ingest customer events/activity data into Hive and Cassandra from Kafka.
• Performed Spark jobs optimization and performance tuning to improve running time and resources.
• Worked on reading and writing multiple data formats like JSON, AVRO, Parquet, ORC on HDFS using Pyspark.
• Designed, developed, and did maintenance of data integration in Hadoop and RDBMS environment with both traditional and non-traditional source system as well as RDBMS and NoSQL data stores for data access and analysis.
• Involved in recovery of Hadoop clusters and worked on cluster size of 310 nodes.
• Worked on creating Hive tables, loading, and analyzing data using Hive queries.
• Experience in proving application support for Jenkins.
• Developed a data pipeline with AWS to extract the data from weblogs and store in HDFS.
• Used Hive QL to analyze the partitioned and bucketed data and compute various metrics for reporting.
• Used reporting tools like Tableau and Power BI to connect with Hive for generating daily reports of data.

Environment: s: Python, Bigdata, Hadoop, HBase, Hive, Spark, Pyspark, Cloudera, Kafka, Sqoop, Jenkins, Unix Shell scripting, GitHub, SQL, Tableau, Power BI.

Confidential

Data Engineer

Responsibilities:

• Data sources are extracted, transformed, and loaded to generate CSV data files with Python programming and SQL queries.
• Developed Spark, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for big data resources.
• Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself.
• Designed, developed and did maintenance of data integration programs in a Hadoop and RDBMS environment with both traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis.
• Worked on Cluster of size 130 nodes.
• Worked extensively with Sqoop for importing metadata from SQL server.
• Analyzed the SQL scripts and designed the solution to implement using Pyspark.
• Responsible for developing data pipelines with AWS cloud to extract the data from weblogs and store in HDFS.
• Involved in creating Hive tables and loading and analyzing data using hive queries.
• Developed Hive queries to process the data and generate the data cubes for visualizing.
• Implemented Partitioning, Dynamic Partitions, Buckets in Hive.
• Used Reporting tools like Tableau to connect with Hive for generating daily reports of data.
• Collaborated with the infrastructure, network, database, application, and BI teams to ensure data quality and availability.

Environment: Spark (Pyspark, SparkSQL, SparkMLIib), Python, Scala, Tableau, Power Bi, Linux, SQL server, GitHub, AWS EMR/EC2/S3/Redshift, Hive, HDFS, Sqoop, Pig, Shell Scripting.

Confidential

Data Analyst

Responsibilities:

• Environment: MS SQL server, MS Excel, ETL, Power BI, Designed, built, and maintained Power Bi and Tableau data visualization dashboards that helped business users make decisions based on key metrics and performance measures.
• Collaborated with multiple business units to gather requirements and develop data reporting infrastructure to provide real-time business insights.
• Developed Power Bi Data Model using datasets from multiple data sources (Postgres, Oracle, FAST) by establishing meaningful relationships between them, improving performance by 60%.
• Presented data insights to executive stakeholders using dashboards to provide them with data-based recommendations.
• Modeled a new data mart on SQL server to hold subset of data from multiple data sources by diverse use of SQL queries with working knowledge of RDBMS and ETL tools.
• Built the infrastructure required for optimal extraction, transformation and loading of data from a wide variety of data sources using SQL.
• Collaborated with QA analysts to troubleshoot data quality issues and worked with data source owners to resolve them; Monitoring dashboard helped improve data quality by 80%
• Utilized Excel features - Pivot Tables, Vlookup to reconcile account values for Actuarial department with Policy Management System reducing risk of invalid cash disbursements by 85%
• Used REST API endpoints to interact with Power Bi web services to automate report deployment across different environments, increasing efficiency by 40%.

Environment: Tableau, Azure, Informatica, Oracle server, PL/SQL, Linux, Python.