SUMMARY

• 8+ years of IT experience in a variety of industries working on Big Data technology using technologies such as Cloudera and Hortonworks distributions. The Hadoop working environment includes Hadoop, Spark, MapReduce, Kafka, Hive, Ambari, Sqoop, HBase, and Impala.
• Building Data pipelines in cloud services like AWS.
• Fluent programming experience with Scala, Java, Python, SQL, T - SQL, R.
• Hands-on experience in developing and deploying enterprise-based applications using major Hadoop ecosystem components like MapReduce, YARN, Hive, HBase, Flume, Sqoop, Spark MLlib, Spark GraphX, Spark SQL, Kafka.
• Adept at configuring and installing Hadoop/Spark Ecosystem Components.
• Proficient with Spark Core, Spark SQL, Spark MLlib, Spark GraphX, and Spark Streaming for processing and transforming complex data using in-memory computing capabilities written in Scala. Worked with Spark to improve the efficiency of existing algorithms using Spark Context, Spark SQL, Spark MLlib, Data Frame, Pair RDD, and Spark YARN.
• Experience in the application of various data sources like Oracle SE2, SQL Server, Flat Files, and Unstructured files into a data warehouse.
• Able to use Sqoop to migrate data between RDBMS, NoSQL databases, and HDFS.
• Experience in Extraction, Transformation, and Loading (ETL) data from various sources into Data Warehouses and data processing like collecting, aggregating, and moving data from various sources using Apache Flume, Kafka, PowerBI, and Microsoft SSIS.
• Hands-on experience with Hadoop architecture and various components such as Hadoop File System HDFS, Job Tracker, Task Tracker, Name Node, Data Node, and Hadoop MapReduce programming.
• Comprehensive experience in developing simple to complex Map reduction and Streaming jobs using Scala and Java for data cleansing, filtering, and data aggregation. Also, possess detailed knowledge of the MapReduce framework.
• Used IDEs like Eclipse, IntelliJ IDE, PyCharm IDE, Notepad ++, and Visual Studio for development.
• Seasoned practice in Machine Learning algorithms and Predictive Modelings such as Linear Regression, Logistic Regression, Naïve Bayes, Decision Tree, Random Forest, KNN, Neural Networks, and K-means Clustering.
• Ample knowledge of data architecture including data ingestion pipeline design, Hadoop/Spark architecture, data modeling, data mining, machine learning, and advanced data processing.
• Experience working with NoSQL databases like Cassandra and HBase and developed real-time read/write access to very large datasets via HBase.
• Developed Spark Applications that can handle data from various RDBMS (MySQL, Oracle Database) and Streaming sources.
• Proficient SQL experience in querying, data extraction/transformations, and developing queries for a wide range of applications.
• Capable of processing large sets (Gigabytes) of structured, semi-structured, or unstructured data.
• Experience in analyzing data using HiveQL, Pig, HBase, and custom MapReduce programs in Java 8.
• Experience working with GitHub/Git 2.12 source and version control systems.
• Strong in core Java concepts including Object-Oriented Design (OOD) and Java components like Collections Framework, Exception handling, I/O system.

TECHNICAL SKILLS

Hadoop/Big Data Technologies: HDFS, Hive, Pig, Sqoop, Yarn, Spark, Spark SQL, Kafka, AWS

Hadoop Distributions: Horton works and Cloudera Hadoop

Languages: C, C++, Python, Scala, UNIX Shell Script, COBOL, SQL, and PL/SQL

Tools: Teradata SQL Assistant, Pycharm, Autosys

Operating Systems: Linux, UNIX, ZOS, and Windows

Databases: Teradata, Oracle 9i/10g, DB2, SQL Server, MySQL 4.x/5.x

ETL Tools: IBM InfoSphere Information Server V8, V8.5 & V9.1

Monitoring: Azure DevOps.

Reporting: Tableau.

PROFESSIONAL EXPERIENCE

Data Engineer

Confidential, CO

Responsibilities:

• Worked on data pipelines, migration from the data lake to the snowflake.
• Created an ETL workflow with Python and AWS services.
• Analyzed the data from different APIs and wrote a python code to call the API.
• Ingestion of data lake into a snowflake.
• Technical liaising between customers, development & support teams.
• Built and tested pipelines on cross-account in AWS.
• Used AWS services to implement data lake and data warehouse solutions (AWS Lambdas, AWS Glue, AWS S3, Event bridge, SQS, SNS, RDS, DynamoDB, AWS Kinesis, AWS Athena, etc.).
• Built tables using SQL in Athena(AWS service).
• Created, deployed, and tested integrations between the Snowflake data warehouse platform and third-party applications.
• Analyzed, and tracked the data in Athena and DynamoDB.
• Assist with the development of fast proof of concepts (POCs) to illustrate the AWS solution hypothesis with tangible work results.
• Pulled or pushed code in CI/CD pipeline in Azure DevOps.
• Added Environmental variables in lambda, set SQS trigger for lambda, worked on the batch window and batch size of SQS trigger.
• Added SNS subscriptions to S3 bucket, and cross-account lambda trigger of SNS.
• Configured the infrastructure of the data pipeline in both non-prod and prod.
• Updated the assigned tasks in ADO, daily standups, sprint meetings, and meetings with the team.
• Extracted the data from PDF using AWS Textract, in both raw and tables format.
• Scheduled the lambdas with the help of the Amazon Event bridge.
• Active team member in architectural and solution discussions while developing new methods, and services on the AWS platform and migrating current ones.

Data Engineer

Confidential, CA

Responsibilities:

• Worked on AWS Data pipeline to configure data loads from S3 to Redshift.
• Using AWS Redshift, I Extracted, transformed, and loaded data from various heterogeneous data sources and destinations
• Created Tables, Stored Procedures, and extracted data using T-SQL for business users whenever required.
• Performs data analysis and design, and creates and maintains large, complex logical and physical data models, and metadata repositories using ERWIN and MB MDR
• I have written a shell script to trigger data Stage jobs.
• Assist service developers in finding relevant content in the existing reference models.
• Like Access, Excel, CSV, Oracle, flat files using connectors, tasks, and transformations provided by AWS Data Pipeline.
• Utilized Spark SQL API in PySpark to extract and load data and perform SQL queries.
• Worked on developing a Pyspark script to encrypt the raw data by using Hashing algorithms concepts on client-specified columns.
• Responsible for the Design, Development, and testing of the database and Developed Stored Procedures, Views, and Triggers
• Developed Python-based API (RESTful Web Service) to track revenue and perform revenue analysis.
• Compiling and validating data from all departments and Presenting it to the Director of Operation.
• KPI calculator Sheet and maintain that sheet within SharePoint.
• Created Tableau reports with complex calculations and worked on Ad-hoc reporting using PowerBI.
• Creating a data model that correlates all the metrics and gives a valuable output.
• Worked on tuning SQL Queries to bring down run time by working on Indexes and Execution Plan.
• Performing ETL testing activities like running the Jobs, Extracting the data using necessary queries from database transform, and uploading into the Data warehouse servers.
• Pre-processing using Hive and Pig.
• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.
• Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
• Implemented Copy activity, Custom Azure Data Factory Pipeline Activities
• Primarily involved in Data Migration using SQL, SQL Azure, Azure Storage, and Azure Data Factory, SSIS, PowerShell.
• Architect & implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, and NoSQL DB).
• Migration of on-premise data (Oracle/ SQL Server/ DB2/ MongoDB) to Azure Data Lake and Stored (ADLS) using Azure Data Factory (ADF V1/V2).
• Developed a detailed project plan and helped manage the data conversion migration from the legacy system to the target snowflake database.
• Design, develop, and test dimensional data models using Star and Snowflake schema methodologies under the Kimball method.
• Implement ad-hoc analysis solutions using Azure Data Lake Analytics/Store, HDInsight
• Developed data pipeline using Spark, Hive, Pig, Python, Impala, and HBase to ingest customer
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python, and Scala.
• Ensure deliverables (Daily, Weekly & Monthly MIS Reports) are prepared to satisfy the project requirements cost and schedule
• Worked on a direct query using PowerBI to compare legacy data with the current data and generated reports and stored dashboards.
• Designed SSIS Packages to extract, transfer, load (ETL) existing data into SQL Server from different environments for the SSAS cubes (OLAP)
• SQL Server reporting services (SSRS). Created & formatted Cross-Tab, Conditional, Drill-down, Top N, Summary, Form, OLAP, Subreports, ad-hoc reports, parameterized reports, interactive reports & custom reports
• Created action filters, parameters, and calculated sets for preparing dashboards and worksheets using PowerBI
• Developed visualizations and dashboards using PowerBI
• Used ETL to implement the Slowly Changing Transformation, to maintain Historically Data in the Data warehouse.
• Performing ETL testing activities like running the Jobs, Extracting the data using necessary queries from database transform, and uploading into the Data warehouse servers.
• Created dashboards for analyzing POS data using Power BI

Environment: MS SQL Server 2016, T-SQL, SQL Server Integration Services (SSIS), SQL Server Reporting Services (SSRS), SQL Server Analysis Services (SSAS), Management Studio (SSMS), Advance Excel (creating formulas, pivot tables, Hlookup, Vlookup, Macros), Spark, Python, ETL, Power BI, Tableau, Hive/Hadoop, Snowflakes, Power BI, AWS Data Pipeline, IBM Cognos 10.1, Data Stage, Cognos Report Studio 10.1, Cognos 8 & 10 BI, Cognos Connection, Cognos office Connection, Cognos 8.2/3/4, Data stage and Quality Stage 7.5

Data Engineer

Confidential, Bothell, WA

Responsibilities:

• Implemented Apache Airflow for authoring, scheduling, and monitoring Data Pipelines
• Designed several DAGs (Directed Acyclic Graph) for automating ETL pipelines
• Performed data extraction, transformation, loading, and integration in data warehouse, operational data stores, and master data management
• Strong understanding of AWS components such as EC2 and S3
• Performed Data Migration to GCP
• Responsible for data services and data movement infrastructures
• Experienced in ETL concepts, building ETL solutions, and Data modeling
• Worked on architecting the ETL transformation layers and writing spark jobs to do the processing.
• Aggregated daily sales team updates to send reports to executives and to organize jobs running on Spark clusters
• Loaded application analytics data into a data warehouse at regular intervals of time
• Designed & build infrastructure for the Google Cloud environment from scratch
• Experienced dimensional modeling (Star schema, Snowflake schema), transactional modeling, and SCD (Slowly changing dimension)
• Leveraged cloud and GPU computing technologies for automated machine learning and analytics pipelines, such as AWS, GCP
• Worked on confluence and Jira
• Designed and implemented a configurable data delivery pipeline for scheduled updates to customer-facing data stores built with Python
• Proficient in Machine Learning techniques (Decision Trees, Linear/Logistic Regressors) and Statistical Modeling
• Compiled data from various sources to perform complex analysis for actionable results
• Measured Efficiency of Hadoop/Hive environment ensuring SLA is met
• Optimized the Tensorflow Model for efficiency
• Analyzed the system for new enhancements/functionalities and performed Impact analysis of the application for implementing ETL changes
• Implemented a Continuous Delivery pipeline with Docker, Git Hub, and AWS
• Built performant, scalable ETL processes to load, cleanse and validate data
• Participated in the full software development lifecycle with requirements, solution design, development, QA implementation, and product support using Scrum and other Agile methodologies
• Collaborate with team members and stakeholders in the design and development of the data environment
• Preparing associated documentation for specifications, requirements, and testing

Environment: AWS, GCP, BigQuery, Gcs Bucket, G-Cloud Function, Apache Beam, Cloud Dataflow, Cloud Shell, Gsutil, Bq Command Line Utilities, Dataproc, Cloud SQL, MySQL, Postgres, SQL Server, Python, Scala, Spark, Hive, Spark - SQL.

Data Engineer

Confidential, Greenwood Village, CO

Responsibilities:

• Experience in building and architecting multiple Data pipelines, end to end ETL, and ELT processes for Data ingestion and transformation in GCP
• Strong understanding of AWS components such as EC2 and S3
• Implemented a Continuous Delivery pipeline with Docker and Git Hub
• Worked with g-cloud function with Python to load Data into BigQuery for on arrival CSV files in the GCS bucket
• Process and load bound and unbound Data from Google pub/subtopic to BigQuery using cloud Dataflow with Python.
• Devised simple and complex SQL scripts to check and validate Dataflow in various applications.
• Performed Data Analysis, Data Migration, Data Cleansing, Transformation, Integration, Data Import, and Data Export through Python.
• Developed and deployed data pipelines in the cloud such as AWS and GCP
• Performed data engineering functions: data extract, transformation, loading, and integration in support of enterprise data infrastructures - data warehouse, operational data stores, and master data management
• Responsible for data services and data movement infrastructures good experience with ETL concepts, building ETL solutions, and Data modeling
• Architected several DAGs (Directed Acyclic Graph) for automating ETL pipelines
• Hands-on experience in architecting the ETL transformation layers and writing spark jobs to do the processing.
• Gather and process raw data at scale (including writing scripts, web scraping, calling APIs, writing SQL queries, writing applications)
• Experience dimensional modeling (Star schema, Snowflake schema), transactional modeling, and SCD (Slowly changing dimension)
• Devised PL/SQL Stored Procedures, Functions, Triggers, Views, and packages. Made use of Indexing, Aggregation, and Materialized views to optimize query performance.
• Developed logistic regression models (Python) to predict subscription response rate based on customer variables like past transactions, response to prior mailings, promotions, demographics, interests, hobbies, etc.
• Develop near real-time data pipeline using spark
• Process and load bound and unbound Data from Google pub/subtopic to Bigquery using cloud Dataflow with Python
• Hands of experience in GCP, Big Query, GCS bucket, G - cloud function, cloud dataflow, Pub/suB cloud shell, GSUTIL, BQ command-line utilities, Data Proc, Stack driver
• Implemented Apache Airflow for authoring, scheduling, and monitoring Data Pipelines
• Proficient in Machine Learning techniques (Decision Trees, Linear/Logistic Regressors) and Statistical Modeling
• Worked on confluence and Jira skilled in data visualization like Matplotlib and seaborn library
• Hands-on experience with big data tools like Hadoop, Spark, Hive
• Experience implementing machine learning back-end pipeline with Pandas, Numpy

Environment: Gcp, Bigquery, Gcs Bucket, G-Cloud Function, Apache Beam, Cloud Dataflow, Cloud Shell, Gsutil, Docker, Kubernetes, AWS, Apache Airflow, Python, Pandas, Matplotlib, seaborn library, text mining, Numpy, Scikit-learn, Heat maps, Bar charts, Line charts, ETL workflows, linear regression, multivariate regression, Python, Scala, Spark

Spark Developer

Confidential

Responsibilities:

• Imported required modules such as Keras and NumPy on Spark session, also created directories for data and output.
• Read train and test data into the data directory as well as into Spark variables for easy access and proceeded to train the data based on sample submission.
• The images upon being displayed are represented as NumPy arrays, for easier data manipulation all the images are stored as NumPy arrays.
• Created a validation set using Keras2DML to test whether the trained model was working as intended or not.
• Defined multiple helper functions that are used while running the neural network in session. Also defined placeholders and number of neurons in each layer.
• Created neural networks computational graph after defining weights and biases.
• Created a TensorFlow session used to run the neural network and validate the accuracy of the model on the validation set.
• After executing the program and achieving acceptable validation accuracy a submission was created that is stored in the submission directory.
• Executed multiple SparkSQL queries after forming the Database to gather specific data corresponding to an image.

Environment: Scala, Python, PySpark, Spark, Spark ML Lib, Spark SQL, TensorFlow, NumPy, Keras, PowerBI.

Confidential

Data Analyst

Responsibilities:

• Developed stored procedures in MS SQL to fetch the data from different servers using FTP and processed these files to update the tables.
• Responsible for Designing Logical and Physical data modeling for various data sources on Confidential Redshift.
• Performed logical data modeling, and physical Data modeling (including reverse engineering) using the Erwin Data modeling tool.
• Created a dimensional model for the reporting system by identifying required dimensions and facts using Erwin.
• Designed and Developed ETL jobs to extract data from Salesforce replica and load it in data mart in Redshift.
• Involved in performance tuning, stored procedures, views, triggers, cursors, pivot, unpivot functions, CTEs
• Developed and delivered dynamic reporting solutions using SSRS.
• Extensively used Erwin for Data modeling. Created Staging and Target Models for the Enterprise Data Warehouse.
• Involved in Normalization / Denormalization techniques for optimum performance in relational and dimensional database environments.
• Resolved the data type inconsistencies between the source systems and the target system using the Mapping Documents and analyzing the database using SQL queries.
• Worked on ETL testing, and used SSIS tester automated tool for unit and integration testing.
• Designed and created SSIS/ETL framework from the ground up.
• Created new Tables, Sequences, Views, Procedures, Cursors, and Triggers for database development.
• Created ETL Pipeline using Spark and Hive to ingest data from multiple sources.
• Involved in using SAP and transactions done in SAP - SD Module for handling customers of the client and generating the sales reports.
• Creating reports using SQL Reporting Services (SSRS) for customized and ad-hoc Queries
• Coordinated with clients directly to get data from different databases.
• Worked on MS SQL Server, including SSRS, SSIS, and T-SQL.
• Designed and developed schema data models.
• Documented business workflows for stakeholder review.

Environment: ER Studio, SQL Server 2008, SSIS, Oracle, Business Objects XI, Rational Rose, Data stage, MS Visio, SQL, Crystal Reports 9