SUMMARY

• Data Engineer with Data Science, around 9+ years of hands - on experience in software development.
• Experience in building data pipelines and leveraging machine learning models.
• Experience in building data pipelines on different cloud services.
• Experience in developing ETL pipelines for Batch and Real-Time Processing. Developed Spark Streaming applications for Real Time Processing.
• Expertise in Data collection, cleaning, processing using Python.
• Working experience in Hadoop Ecosystem major components like Sqoop, Kafka, Hive, Spark, HBase, Airflow, and Hadoop distributions Cloudera, Horton Works.
• Good understanding of Hadoop v1/v2 architecture and experience with components such as Job Tracker, Task Tracker, Name Node, Secondary Name Node, Data Node, and YARN architecture which includes Node manager, Resource manager and App Master.
• Expertise in data modeling for data warehouse/data mart development, SQL, and analysis of online transactional Processing (OLTP), data warehouse (OLAP).
• Developed workflowschedulers torun jobs independently with time and data availability.
• Experience inusing CloudServices such as Azure, AWS, Google Cloud Platform.
• Worked with different file formats such as JSON, Flat files, Parquet.
• Developed a data pipeline to fetch and pre-process data and make it available for modeling.
• Worked with both structured and semi structured data using Snowflake.
• Experience in implementing Spark using Scala and Spark SQL for faster processing of data.
• Strong experience in extracting and loading data using complex business logic’s using Hive from different data sources and built the ETL pipelines to process tera bytes of data daily.
• Experienced in transporting, and processing real time event streaming using Kafka and Spark Streaming.
• Hands on experience with importing and exporting data from Relational databases to HDFS, Hive and HBase using Sqoop.
• Experienced in processing real time data using Kafka 0.10.1 producers and stream processors and implemented stream process using Kinesis and data landed into Data Lake S3.
• Designed and developed spark pipelines to ingest real time event-based data from Kafka and other message queue systems and processed huge data with spark batch processing into data warehouse hive.
• Experienced in creating and analyzing Software Requirement Specifications (SRS) and Functional Specification Document (FSD).
• Working with Bootstrap twitter framework to Design single page application.
• Strong working knowledge developing Cross Browser Compatibility (IE, Firefox, Safari, Chrome etc.) for dynamic web applications.
• Experienced in developing the unit test cases using Junit, Mockitoand Scala Test.
• Capable of organizing, coordinating and managing multiple tasks simultaneously.
• Excellent communication and inter-personal skills, self-motivated, organized and detail-oriented, able to work well under deadlines in a changing environment and perform multiple tasks effectively and concurrently.
• Strong analytical skills with ability to quickly understand client’s business needs. Involved in meetings to gather information and requirements from the clients.

TECHNICAL SKILLS

Big Data tools: Cloudera Ambari, HDFS, Sqoop, Hive, Spark, Scala, Kafka, HBase, Airflow.

AWS: AWS Glue, EMR, EC2, S3, RDS, Redshift, Kinesis, Sage Maker, DynamoDB, CI/CD.

Azure: Azure ML Studio, Synapse, Blob Storage, SQL, Stream Analytics, Data Factory.

GCP: Big Query, Compute Engine, Composer, Cloud Storage, Cloud SQL, Elastic Search.

Database: Snowflake, MySQL, Oracle, PostgreSQL, PL/SQL, MS SQL, MongoDB.

ML Algorithms: Linear Regression, KNN, Decision Tree, SVM.

PROFESSIONAL EXPERIENCE

Confidential, Chicago, IL

Data Engineer

Responsibilities:

• Developed Airflow Workflows for incremental loads from Teradata and imported data into Hive tables as a part of migration from Teradata to Hadoop using Sqoop.
• Build data pipelines in airflow in GCP for ETL related jobs using different airflow operators.
• Experience in GCP Dataproc, GCS, Cloud functions, Big Query.
• Designed a data quality framework to perform schema validation and data profiling on spark.
• Wrote scripts in Hive SQL for creating complex tables with high performance metrics like partitioning, clustering, and skewing
• Work related to downloading Big Query data into pandas or Spark data frames for advanced ETL capabilities.
• Worked with google data catalog and other google cloud APIs for monitoring, query and billing related analysis for big Query usage.
• Worked on reading multiple data formats JSON, XML, AVRO, OCR using Scala.
• Ingested the data from different sources like Teradata, PL/SQL into HDFS using Sqoop.
• Sound understanding and working experience in JSON, JSON Schema, and REST API.
• Involved in importing the real time data using Kafka, spark streaming using PySpark jobs.
• Leveraged spark to manipulate unstructured data and apply text mining on user's table utilization data.
• Written queries in big Query to look up all the Customer, Product, and Order level data.
• Worked on Cloud Composer to deploy components to run Airflow.
• Used Elasticsearch (Big Data) to retrieve data into application as required.
• Historical data load to Cloud Storage using Hadoop utilities and load to Big Query.
• Worked on Google cloud platform (GCP) services like compute engine, cloud SQL.
• Captured customer-centric data and Performed data processing, data extraction, analysis including personal information, customer feedback, sentiment, and frequency of spend.
• Employed algorithms like Linear Regression, Decision Tree, and SVM to analyze large data sets and make individual customer’s portfolio Policy Recommender Engines.
• Experience in various types of testing such as Unit testing, Integration testing, Functional testing, and User acceptance testing.

Environment: Python (scikit-learn/NLTK), Linear Regression, Decision Tree, SVM, Big Data, ETL, Teradata, MongoDB, Hadoop, Hive, Kafka, Sqoop, Spark, Scala, Airflow, Cloudera Ambari, GCP Dataproc, GCS, Cloud functions, Google Big Query, compute engine, cloud SQL, Elastic Search, Composer, Git, Agile, CI/CD, Windows10, Linux.

Confidential, Tampa, FL

Data Engineer

Responsibilities:

• Worked with data sources like Blob Storage. Cosmos DB, RESTAPIs for ETL pipeline.
• Worked on Azure Data Lake Storage Gen 2 for big data analytics.
• Ingested data from different data sources such as Cosmos DB, SQL server, Oracle to Azure SQL Data Warehouse using Azure Data Factory.
• Performed Data processing, structuring the data using Spark on Data Bricks.
• Performed data aggregation, joining, and queries using Data Frames with PySpark.
• Have good experience working with Azure BLOB andData Lakestorage and loading data intoAzure SQL Synapse analytics (DW).
• Built near real-time streaming ETL pipeline using HDI Kafka, Spark using Data Bricks.
• Performed unit testing in Data Warehousing and created dashboards in Power BI.
• Worked with large data warehouse tables and turned them into powerful SQL queries that will extract data and compile it into meaningful reports using Azure SQL.
• Understanding and manipulating existing scripts for Procedure Optimization.
• Experience in Python modules regular expressions, collections, date and time, unit testing, Load testing.
• Built and performed hyperparameter-tuning the model on AzureML Studio.
• Utilized Logging Metrics to review metrics for comparison across runs on AzureML Studio.
• Registered and packaged model to Azure container Instance (ACI) and deployed model using Azure container Registry.
• Performed Data Augmentation technique created artificial training data to avoid overfitting.
• Adopted Agile practices and CI/CD for development to deployment.
• Worked with Agile software delivery methodologies Strong knowledge of requirement gathering, use case analysis and user acceptance testing.

Environment: Python (Scikit-learn/SciPy), Big Data, ETL, SQL Server, Cosmos DB, Oracle, Spark, PySpark, Kafka, Azure SQL, AzureML Studio, Jupyter Notebook, Data Bricks, Synapse, Data Factory, ADLS, Blob Storage, Power BI, Agile, Git, CI/CD, Linux.

Confidential, O’Fallon, MO

Data Engineer

Responsibilities:

• Developed Sqoop jobs to import and store massive volumes of data in HDFS and Hive.
• Designed and developed pig data transformation scripts to work against unstructured data from various data points and created a baseline.
• Experienced in implementing Spark RDD transformations, actions to implement the business analysis.
• Performed Batch and Streaming processes on server less cluster AWS Glue and Hadoop.
• Developed data pipeline to fetch and pre-process data and make it available for modeling using AWS services EMR, Glue, S3 and Redshift.
• Worked with data sources and backup storages such as S3, DynamoDB, RDS, PostgreSQL.
• Extracted real time feed using Kinesis and Spark Streaming, converted it to RDD, processed data in the form of Data Frame, and saved the data in Redshift.
• Implemented map-reduce programs to handle semi/unstructured data like XML, JSON, Avro data files and sequence files for log files.
• Created AWS lambda function to load the data based on availability in S3, integrated the SNS notification as a part of this process.
• Configured Spark streaming to get on-going information from the AWS Kinesisand stored the stream information in S3.
• Optimized Spark/Scala scripts and SQL/Streaming for faster testing and processing of data.
• Loaded data into storages such as Snowflake, HDFS for analytics purposes.
• Worked on AWS Sage Maker, Jupyter notebook IDEs for data exploration, cleaning, and pre-processing. ran these on compute instances.
• Created jobs to quickly build, train, and perform hyperparameter tuning on machine learning models using AWSSage Maker.
• Used Spark SQL with Scala for creating data frames and performed transformations on data frames.
• Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
• Developed code to read data stream from Kafka and send it to respective bolts through respective stream.
• Creating Spark clusters and configuring high concurrency clusters using Azure Databricks to speed up the preparation of high-quality data.
• Developed Spark applications using Scala for easy Hadoop transitions.
• Implemented applications with Scalaalong with Akka and Play framework.
• Optimized the code using PySpark for better performance.
• Used cloud shell SDK in GCP to configure the services Data Proc, Storage, BigQuery
• Worked on Spark streaming using Apache Kafka for real time data processing.
• Experienced in optimizing Hive queries, joins to handle different data sets.
• Involved in ETL, Data Integration and Migrationby writing Pig scripts.
• Developed MapReduce programs to cleanse the data in HDFS obtained from heterogeneous data sources.
• Worked hands on No-SQL databases like MongoDB for POC purpose in storing images and URIs.
• Designed and implemented MongoDB and associated RESTful web service.
• Involved in writing test cases and implement test classes using MRUnit and mocking frameworks.
• Containerized and deployed trained models using AWSElastic Container Service(ECS).
• Created Dashboards in Tableau for management review based on requirements.
• Analyzed and reported on data sets such as sales, point of sale data, supply chain logistics, merchandising, demographics, promotional marketing, and customer experience.
• Expertise in writing test cases using Pytest mock and performed integration testing.
• Implemented monitoring solutions in Docker, and Jenkins.
• Implemented pipelines with CI/CD and worked with version control GitHub.

Environment: Python, Redshift, DynamoDB, Big Data, ETL, SQL, PostgreSQL, Cloudera Ambari, Hadoop, Hive, MapReduce, Pig, Sqoop, Scala, Spark, AWS Glue, EMR, EC2, S3, RDS, Kinesis, ECS, Jupyter Notebook, Tableau, Docker, Jenkins, Git, Agile, CI/CD.

Confidential

Software Developer

Responsibilities:

• Used Agile Methodology (Scrum) to meet customer expectation, timelines with quality deliverables.
• Presentation layer (GUI) application has been created using .NETWeb Forms, ASP.NET, MS Visual Studio, HTML.
• Followed SOA Architecture for Web services.
• Created and consumed WCF Web Services (SOAP, WSDL), Restful WEB API services for communicating.
• Coded in Data Access Layer, and Business Logic Layer using C#.NETand SQL Server in backend.
• Created Entity Framework object model For data in SQL
• Implemented search and CRUD operations using Entity Framework object model.
• Implemented Asmx web services and consumed on the client side using AJAX client callbacks to for performance reasons.
• Used LINQ to Entity Framework using lambda expressions for communicating between business layer and database.
• Used ADO.NETobjects to access data from SQL Server 2008R2 database by using stored procedures and queries in T-SQL.
• Created Model classes, Data Transfer Objects (DTOs), Mappers, Controllers and Views while developing ASP.NET MVC 3.0 application using Razor View Engine.
• Implemented Object Oriented Concepts such as Polymorphism, Inheritance, Encapsulation.
• Created Stored Procedures, triggers, cursors, views, and user defined functions for maintaining business functionalities.
• Worked with the group which evaluated and recommended technologies including ASP .NETMVC

Environment: ASP.NET MVC 3.0, C#, Entity Framework, XML, JavaScript, ASMX Web Services, LINQ, Lambda Expression, CSS, Subversion, Visual Studio, IIS, SQL Server 2008, WCF, HL,Facets.

Confidential

Software Developer

Responsibilities:

• Involved in Documentation and Use case design using UML modeling include development of Class diagrams, Sequence diagrams, and Use Case Transaction diagrams.
• Design, develop, test, support and custom web application followed by Agile.
• Designed custom deployment and configuration automation systems to allow for hands-off management of clusters via Cobbler, FUNC, and Puppet.
• Designed, developed and did maintenance of data integration programs in a Hadoop and RDBMS environment with traditional and non-traditional source systems as well as RDBMS and NoSQL data stores for data access and analysis.
• Configured Spark Streaming to receive real time data from the Kafka and store the stream data to HDFS.
• Involved in converting Hive/SQL queries into Spark transformations using Spark data frames, Scala and Python.
• Experienced in developing Spark scripts for data analysis in both python and Scala.
• Wrote Scala scripts to make spark streaming work with Kafka as part of spark Kafka integration efforts.
• Built on premise data pipelines using Kafka and spark for real-time data analysis.
• Created reports in TABLEAU for visualization of the data sets created and tested Spark SQL connectors.
• Implemented Hive complex UDF's to execute business logic with Hive Queries
• Developed a different kind of custom filters and handled pre-defined filters on HBase data using API.
• Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.
• Handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive and then loading data into HDFS.
• Collecting and aggregating large amounts of log data and staging data in HDFS for further analysis.
• Experience in managing and reviewing Hadoop Log files.
• Prepared complete description documentation as per the Knowledge Transferred about the Phase-II Talend Job Design and goal and prepared documentation about the Support and Maintenance work to be followed in Talend.
• Worked on generic classes using C# to create Support Functions and Database Operations.
• Designed the user interfaces using JSPs, developed custom tags, and used JSTL.
• Implemented Price menu and Add to Cart within Ajax and JavaScript.
• Created SOAP based web services to get the list of users and their details.
• Performed Unit testing on every new version before sending it for User Acceptance Test.
• Executed test cases for Unit, System, and Integration Testing.

Environment: .Net, Serenity, Selenium WebDriver, Locust, Bitbucket, Jenkins, VersionOne, JIRA, Maven, SQL Server, SQL Server Management Studio (SSMS), Postman, SoapUI, Fiddler 4, TeamCity, Octopus.