SUMMARY

• Data engineer with over 8+ years of professional experience in building end to end data pipelines on AWS cloud services using S3, Athena, Glue, EC2, Dynamo DB, Redshift and RDS.
• Experience working with teh business directly and partner with external and internal resources
• Strong experience in Microsoft Business Intelligence Tools including Integration Services, Reporting Services and Analysis Services
• Expertise in developing ETL applications on large volumes of data using different tools such as MapReduce, pySpark, Spark - SQL
• Experience in writingPL/SQLstatements - Stored Procedures, Functions, Triggers, Views, Cursors, Joins and indexes and packages.
• Experience in job/workflow scheduling and monitoring tools like Control-M, SQL Server Agent
• Involved in designing jobs using stages like Seq/Dataset File, Filter, Lookup, join,Aggregate etc.
• Helped build teh data warehouse from scratch on Snowflake and migrate data from SQL to Snowflake
• Expert in building ETL data pipelines using SQL Server Integration Services (SSIS), AWS Glue and programming languages like Python
• Experience in building data lake using Amazon S3, Glue and Athena

TECHNICAL SKILLS

Languages: SQL, PySpark, Kafka, Spark-SQL, java, Unix shell, Perl, cobal, Python 3.0

SQL Server Tools: BCP, DTS, Import/Export Wizard, SQL Mail, SQL Server Management Studio, SAS studio, SSIS, SSRS, SSAS, Index Analyzer

Databases: SQL Server, KQL Kusto, Cosmos 2014/2012/2008 R2/2008, Redshift, Aurora, Snowflake

Operating Systems: Windows Server 2008/2003/2000/ XP/95/98/7

ETL Tools: Data stage 11.7/11.5, DTS, SSIS, BCP, Import/Export Wizard, BI Studio, AWS Glue

OLAP Tools: SSAS, MS Excel

Reporting Tools: Power BI, SSRS, MS Excel, Crystal Report, Tableau

Business Tools: Word, Excel, PowerPoint, Access, Outlook, Slack, Teams.

PROFESSIONAL EXPERIENCE

Confidential, Malvern PA

SRE Data Engineer

Responsibilities:

• Evaluating client needs and translating their business requirement to functional specifications thereby onboarding them onto teh Hadoop ecosystem.
• Created various hive external tables, staging tables and joined teh tables as per teh requirement. Implemented static Partitioning, Dynamic partitioning, and Bucketing.
• Implemented Data Quality in ETL Tool Talend and having good knowledge in Data Warehousing.
• Worked withETLprocesses totransfer/migrate data from relational database and flat files common staging tables in various formats to meaningful data inOracle and MS-SQL.
• Developed Apache Spark applications by using spark for data processing from various streaming sources.
• Working on designing teh MapReduce and Yarn flow and writing MapReduce scripts, performance tuning and debugging.
• Installed application on AWS EC2 instances and configured teh storage on S3 buckets.
• Stored data in AWS S3 like HDFS and performed EMR programs on data stored.
• Data-intensive AWS Lambda applications which are aimed to do complex analysis creating analytical reports for end-to-end traceability, lineage, and definition of Key Business elements from Aurora.
• Worked on AWS Lambda functions in python for AWS Lambda which invokes python scripts to perform various transformations and analytics on large data sets in EMR clusters.
• Perform Performance tuning activities of IBM infosphere DataStage ETL Jobs, KEY areas.
• Workedon AWS Lambda, AWS Code Deploy, AWS cloud formation and Cloud Foundry.
• Worked on auto-scaling teh instances to design cost-TEMPeffective, fault-tolerant, and highly reliable systems.

Confidential, Bentonville AR

Database Engineer

Responsibilities:

• As a Big Data Developer, worked on Hadoop cluster scaling from 4 nodes in a development environment to 8 nodes in teh pre-production stage and up to 24 nodes in production.
• Involved in various phases of development analyzed and developed teh system going through Agile Scrum methodology.
• Responsible for data extraction and data ingestion from different data sources into Hadoop DataLake by creating ETL pipelines using Pig, and Hive.
• Built pipelines to move hashed and un-hashed data from XML files to Datalake.
• Developed Spark scripts using Python on Azure HDInsight for Data Aggregation, Validation and verified its performance over MR jobs.
• Involved in develop ETL Jobs using DataStage to load teh data into DWH/DM
• Extensively worked with Spark-SQL context to create data frames and datasets to preprocess teh model data.
• Data Analysis: Expertise in analyzing data using Pig scripting,Hive Queries, Sparks (python) and Impala.
• Developed NiFi workflow to pick up teh multiple files from theFTP location and move those to HDFS on daily basis.
• Scripting: Expertise in Hive, PIG, Impala, Shell Scripting, Perl Scripting, and Python.
• Worked with developer teams on NiFi workflow to pick up teh data from teh rest API server, from data lake as well as from SFTP server and send that to Kafka.

Confidential, Raleigh NC

Data Engineer/ ETL Developer

Responsibilities:

• Create database objects such as tables, views, stored procedures, indexes
• Build SSIS packages involving ETL process, extracting data from various flat files, Excel files, legacy systems and loading into SQL server
• Debug queries, error logging, error handling and production support for SSIS
• Implement and manage event handlers, package configurations, logging, system and user-defined variables, check points and expressions for SSIS Packages
• Code optimization and improving efficiency in databases including Re-indexing, Updating Statistics, recompiling Stored procedures and performing other maintenance tasks
• Scheduled teh SSIS jobs using JAMS, Windows Scheduler, Tasks on Snowflake and SQL server
• Help build ETL data pipelines using SSIS, AWS Glue and Python to migrate data from traditional data warehouses (SQL) to Redshift, Aurora and Snowflake
• Write static and dynamic SQL Scripts using JavaScript Procedures to transform data on Snowflake
• Using SnowSQL to perform ad-hoc file loads with teh help of Local/Table Stages on snowflake
• Setting up Snow PIPE and streams to enable streaming and data ingestions of real time and near real time supporting teh business to make decisions quick
• Use Python for parsing complex main frame files and extracting teh data Writing Python Spark scripts to transform teh data sitting on teh data lake (S3) and load into teh snowflake data warehouse to support teh business operations and analytic needs
• Creating workflows on AWS Glue to setup dependencies and defining teh tasks execution order for teh desired output
• Setting up job failure notifications using Events and SNS on AWS to alert teh users and teh support team
• Move data from SQL Server to Amazon Redshift and Amazon Aurora databases
• Create reports using Power BI to help teh users visualize teh data
• Workedon AWSLambda, AWS Code Deploy, AWS cloud formation.
• Write Python scripts to assist with repeated tasks and automate teh deployments.

Confidential, Columbus OH

SQL and BI Developer

Responsibilities:

• Create database objects such as tables, views, stored procedures, indexes
• Identify columns for primary keys in all teh tables at teh design time and create them
• Create functions to provide custom functionality as per teh requirements
• Participate in development and analysis of new data warehouse
• Develop and review SQL queries with use of joins clauses (inner, left, right) to validate static and dynamic data for data validation
• Design, develop and deploy SSIS Packages
• Build SSIS packages involving ETL process, extracting data from various flat files, Excel files, legacy systems and loading into SQL server
• Debug queries, error logging, error handling and production support for SSIS
• Implement and manage event handlers, package configurations, logging, system and user-defined variables, check points and expressions for SSIS Packages
• Code optimization and improving efficiency in databases including Re-indexing, Updating Statistics, recompiling Stored procedures and performing other maintenance tasks
• Develop reports in SQL Server Reporting Services
• Creating workflows on AWS Glue to setup dependencies and defining teh tasks execution order for teh desired output
• Setting up job failure notifications using Events and SNS on AWS to alert teh users and teh support team
• Move data from SQL Server to Amazon Redshift and Amazon Aurora databases
• Generate reports using SQL Server Reporting Services and Power BI to help teh users visualize teh data
• Configure and migrate SSIS packages from SQL Server 2012 to SQL Server 2016
• Convert teh stored procedures to new data warehouse and migrate to SQL Server 2012
• Scheduled teh SSIS jobs using SQL server agent for daily, weekly and monthly loads

Confidential

Software Engineer

Responsibilities:

• Worked for identity and access management team which involves developing applications that comprises identity management and authorizing teh identity to provide access to teh services.
• Involved in creation and maintenance of identities in one or more systems. Providing a web service with teh goal of providing interoperability with many different sending systems including clients that reside on different infrastructures.
• Worked closely with clients in identifying, investigating/troubleshooting and fixing defects raised on existing features of our applications.
• Involved in team collaborations while releasing services to teh client making sure that teh client requirements and teh deadlines are met.
• Actively mentored new engineers in teh team to help them on board quickly on teh team process, solutions and goals.