Sr. Data Engineer Resume
Memphis, TN
SUMMARY
- Over 7+ years of IT experience as a Data Engineer in Design, Development, Maintenance and support ofBigDataApplicationsover 3+ years of experience inDataEngineering,DataPipeline Design,Development and Implementation as a DataEngineer/DataDeveloper andDataModeler.
- Experienced wif buildingdatamining techniques, logistic/linear/nuclear regression curves, decision trees using R, Python,SASEnterprise Manager, Watson Analytics, and MicrostrategyHands On experience on Spark Core, Spark SQL, Spark Streaming and creating the DataFrames handle in SPARK wif Scala.
- In depth understanding of the principals and best practices of software configuration management (SCM) in Agile, Scrum andDevOpsmethodologies.
- Worked wif clients in setting up, and migrating from on - premise databases toSnowflakeenvironment.
- Strong experience in Big Data technologies and Hadoop ecosystem components like HDFS, Hive, Impala, Apache Pig, Sqoop, Oozie, Zookeeper, YARN, Apache Spark, Python, Pyspark, AWS (AmazonRedshift, Amazon Athena, Lambda, and Kinesis), and GCP.
- Experience in NoSQL databases and worked on table row key design and to load and retrievedatafor real timedataprocessing and performance improvements based ondataaccess patterns.
- Hands-on experience wifSnowflakeutilities such as SnowSQL, SnowPipe, Python, Tasks, Streams, Materialized Views, Optimizer, Metadata Manager, SecureDataSharing, and Stored Procedures.
- Expertise inSnowflakeadvanced concepts like setting up Resource Monitors, Role Based Access Controls, Virtual Warehouse Sizing, Clustering, Zero Copy Cloning and Time Trave
- Good working exposure to AWSGlue, EMR, Lambda, and Step functions, SNS, SQS andEC2.
- Experience in migratedatafrom on-premises Bigdata platform toAWScloud architecture.
- Extensive experience in Hadoop Architecture and various components such as HDFS,Job Tracker, Task Tracker, Name Node,DataNode, and Map Reduce concepts.
- Experience inDataIngestion toSnowflakefrom CSV, JSON, Parquet and XML files.
- Experience in creating and supporting CI/CD pipelines that are automated and reliable using Azure Yaml.
- Experience on developingRESTServices.
- Experience in web/ MySQL,Oracle, and MongoDB.
- Knowledge of managing databases using different RDBMS, including MS SQL Server,PostgreSQL, MySQL, NoSQL, and Elasticsearch.
- Experience in NoSQL databases and worked on table row key design and to load and retrievedatafor real timedataprocessing and performance improvements based ondataaccess patterns.
- Good working exposure to AWSGlue, EMR, Lambda, and Step functions, SNS, SQS andEC2.
- Experience in migratedatafrom on-premises Bigdata platform toAWScloud architecture.
- Extensive experience in Hadoop Architecture and various components such as HDFS,Job Tracker, Task Tracker, Name Node,DataNode, and Map Reduce concepts.
- Have experience in Dimensional Modeling usingSnowflakeand Starschemamethodologies ofDataWarehouse and Integration projects
- Experience in building large scale highly available Web Applications. Working noledge of web services and other integration patterns.
- Hands-on use of Spark and Scala API to compare the performance of Spark wifHive and SQL, and Spark SQL to manipulateDataFrames in Scala.
- Expertise docker container to azureKubernetesservice using kubectl commands using Azure CLI.
- Expertise in Python and Scala, user-defined functions (UDF) for Hive and Pig using Python.
- Hands on experience wifAWSIAM, S3, EC2, RDS, EMR, Athena, Lambda, RedShift, RedShiftSpectrum, RedShift Data share
- Able to use Sqoop to migratedatabetween RDBMS, NoSQL databases and HDFS.
- Experience in Extraction, Transformation and Loading (ETL)datafrom various sources intoDataWarehouses, as well asdataprocessing like collecting, aggregating and movingdatafrom various sources using Apache Flume, Kafka,Powerbase and Microsoft SSIS.
- Experience setting up instances behind Elastic Load Balancer inAWSfor high availability and cloud integration wifAWSusing ELASTIC MapReduce (EMR)
- Expertise indataloads,datatransformations, and fixing incorrectdatain theOracle andPostgreSQLtables.
- Worked on SSIS Package, DTS Import/Export for transferringdatafrom Database (Oracleand Text formatdata) to SQL Server.
- Storage,AzureSynapseAnalytics,AzureAnalytical services,AzureCosmos NO SQLDB, BigDataTechnologies (Hadoop and Apache Spark) andDataBricks.
- Have experience in Dimensional Modeling usingSnowflakeand Starschemamethodologies ofDataWarehouse and Integration projects
- Experience in building large scale highly available Web Applications. Working noledge of web services and other integration patterns.
- Hands-on use of Spark and Scala API to compare the performance of Spark wifHive and SQL, and Spark SQL to manipulateDataFrames in Scala.
- Knowledge in integrating differentdatasources into RDBMS such asOracleand SQL Server.
- Vast experience in creating apps that work wif Teradata,Oracle, SQL Server, and MySQL databases to processdata.application development by using Python, Django, HTML, XML CSS, JavaScript, jQuery, MySQL,PostgreSQLand SQLite.
- Built web application by using Python, Django, AWS, J2EE,PostgreSQL,
- Setup Spark cluster wif AKS (AzureKubernetesService) in Linux Virtual Machine
- Expertise in Python and Scala, user-defined functions (UDF) for Hive and Pig using Python.
- Hands on experience wifAWSIAM, S3, EC2, RDS, EMR, Athena, Lambda, RedShift, RedShiftSpectrum, RedShift Data share
- Able to use Sqoop to migratedatabetween RDBMS, NoSQL databases and HDFS.
- Hands - on experience inAzureCloud Services (PaaS & IaaS),AzureSynapseAnalytics, SQLAzure,DataFactory,AzureAnalysis services, Application Insights, andAzureDataLake.
PROFESSIONAL EXPERIENCE
Sr. Data Engineer
Confidential, MEMPHIS, TN
Responsibilities:
- Involved in Requirement gathering, Business Analysis and translated business requirements into technical design in Hadoop and BigData.
- Involved in testing Snowflake to understand best possible way to use the cloud resources.
- Created Hadoop installation, Commissioning, Decommissioning, Balancing,Troubleshooting, Monitoring and, debugging Configuration of multiple nodes usingHortonworks.
- Designed ETL Process using Informatica to loaddatafrom Flat Files, and ExcelFiles to targetOracleDataWarehouse database
- Worked wif team to develop ETL packages to movedatafrom disparate source systems such as flat files Excel, andOracledatabase into centralized SQL server enterprise database using ErwinDataModeler
- My other responsibilities were query conversion fromoracleto Mysql, creating stored procedures and preprocessing the files from the vendor which involves extracting required header and footer information from file using spark application and using Azuredatafactory the files will be placed inDataLake Storage
- Configured thedatamapping between sources and destination inOracleenvironment and tested performance accuracy related queries under SQL Server
- Created Database objects inSnowflake, setting up and migrating from on-premise databases toSnowflakeenvironment
- Created SnowSQL and Python scripts for mapping and to loaddataintoSnowflake.
- IntegratedSnowflaketo Tableau reporting tool for end user access.
- Worked onSnowflakewarehouse monitoring and performance.
- Worked onSnowflakeSchema,DataModeling and Elements, and Source to Target Mappings, Interface Matrix and Design elements. Performeddataquality issue analysis using Snow SQL by building analytical warehouses onSnowflake.
- Built web application by using Python, Django, AWS, J2EE, PostgreSQL, MySQL,Oracle, and MongoDB.
- Designed ETL Process using Informatica to loaddatafrom Flat Files, and ExcelFiles to targetOracleDataWarehouse database
- TuningOracledatabases and tuning applications (SQL), for tuning SQL working closely wif development Team
- Involved in importingdatafromOracleto HDFS using PYSPARK
- Created Spark applications that can processdatacoming from different RDBMSdatabases (MySQL,Oracle), as well as streaming sources
- My primary roles were to build applications using Apache Spark and Scala in Azure databricks and process large datasets by building pipelines in Azuredatafactory and finally storing the processeddatainto Azuredatalake storage andPostgreSQLdatabase.
- Built AWSPostgreSQLDataWarehouse TableDatasets for Power BI Reports.
- Importing thedatafrom various RDBMS like MySQL,PostgreSQLApache Cassandra to HDFS cluster using Sqoop
- Design and Implement Build and Release pipelines for Android, IOS and HealthEquity classic environment in both TFS and AzureDevops.
- Working on Terraform templates to automate the Azure Iaas virtual machines using terraform modules and deployed virtual machine scale sets in production environment. Working on migrating from TFS to AzureDevOps.
- Automate / Enforce AzureDevOpsGIT Master branch policies across all ADO GIT Repositories.
- Installing, configuring and administering AzureDevopsContinuous Integration (CI) tool on Windows machines along wif Azure Repos
- Involved in variousdatapipelines using Spark, Scala and Spark SQL for faster processing ofdata.
- Develop in Cloud based environment using Amazon Redshift and published on Tableau.
- AtNetflixI is a part of the product analytics team. I engineered high volumedatapipelines to ingest aggregatedatafrom theNetflixproduct apps (i.e. iOS apps, Web app etc) for analytics and visualization purposes. Thedatais primarily used to track theNetflixexperience.
- Explored and implemented Azure technologies like SQL Database,DataLake,Databricks,DataFactory, Virtual Machine, Database forPostgreSQL.
- Proficiency in SQL across several dialects (we commonly write MySQL,PostgreSQL Redshift, SQL Server, and Oracle)
- Involved in Cloud shell SDK inGCPto configure the servicesDataProc, Storage, Virtual Machines.
- Created in HiveQL to analyze the partitioned and bucketeddataand compute various metrics for reporting.
- The smallestdatasize I dealt wif atNetflixis 10 terabytes
- Amazon Web Services (AWS) EC2 instances while the resources are shutting down as necessary to generate cost savings; and Keep Metadata UpToDate for productiondataincluding migration of datasets to the latest in-house developed platform.
- Developed solutions on AWS technologies (EMR, EC2, and S3); as well as enterprisedataprocess for production environment.
- Modification of existing timedatacollection and indexing Creating unit test/regression test practices, tools & platforms.
- ManagedAWSenvironments in accordance wif our policies and security guidelines and assist wif building and improving on those policies.
- Involved inDataMigration from on premise databases toSnowflake.
- Developed ETL pipelines in and out ofdatawarehouses using a combination of Databricks /Python/ETL tools and Snowflake's SnowSQL.
- Implemented Snow SQL, Snow Pipe, Streams, Tasks and Stored Procedures inSnowflake.
- Responsible for Designing and Creating SSAS Cubes from theDataWare House.
- Performed batch and incremental loads wifinSnowflakeusing ETL tools SnowPipe Streams and Tasks.
- ImplementedSnowflakeadvanced concepts like setting up resource monitors, role-based access controls,datasharing, virtual warehouse sizing, query performance tuning.
- Designed and implemented effective Analytics solutions and models wifSnowflake.
- Created file storage/integration objects for Azure onSnowflake.
- Implemented on cloud technologies like Amazon web services, Azure andKubernetes.
- Loaded the historicalDatafrom SQL Managed Instance toAzureSynapseusingAzureDataFactory.
- Worked on migration ofdatafrom On-prem SQL server to Cloud databases (AzureSynapseAnalytics(DW) &AzureSQL DB).
- Deploy program written in Java wif spark framework toKubernetescluster by creating docker image.ss
- Utilization of Elasticsearch wif Kafka topics to enable near real
- Worked on SSIS Package, DTS Import/Export for transferringdatafrom Database (Oracleand Text formatdata) to SQL Server.
- Configure the appropriate VMs, storage and network inAzureto support the customer workloads moving toAzure.
- Assisted in Cluster maintenance, cluster monitoring, adding and removing cluster nodes and installed and configured Hadoop, HDFS, developed multipleMap Reduce jobs in java fordatacleaning and pre-processing.
Environment: HDFS, Hive, Scala, Glue, DataStage, Spark, Redshift, Power BI, AWS,Cloudera, SQL, Terraform, Splunk, RDBMS, Python, Elastic search,dataLake,Kerberos, Jira, Confluence, Databricks, DataLake, DataFactory, Git, Kafka.
Data Engineer
Confidential, Austin, TX
Responsibilities:
- Responsible for Building, publishing customized interactive reports and dashboards, report scheduling usingTableaudesktop/server.
- Involved in converting Hive/SQL queries into Spark transformations using SparkRDDs and Scala.
- Involved in Migrating the platform from Cloudera to EMR platform.
- Worked on BigDataHadoop cluster implementation anddataintegration in developing large-scale system software.
- Implemented cloud integrations toGCPand Azure for bi-directional flow setups fordatamigrations.
- Collecting and aggregating large amounts of logdatausing Apache Flume and stagingdatain HDFS for further analysis.
- Involved in creating Hive Tables, loading wifdataand writing Hive queries which will invoke and run MapReduce jobs in the backend.
- Importdatafrom sources like HDFS/HBase into Spark RDD.
- Designed SSIS packages were created to extract, transfer, and load (ETL) existingdatainto SQL Server from various environments for the SSAS cubes (OLAP).
- Handled importing ofdatafrom variousdatasources, performed transformations using Hive, MapReduce, loadeddatainto HDFS and extracted thedatafromOracleinto HDFS using Sqoop.
- Handled importing ofdatafrom variousdatasources, performed transformations using Hive, MapReduce, loadeddatainto HDFS and extracted thedatafromOracleinto HDFS using Sqoop.
- Involved in working wif AzureDevOpsfor CI/CD pipelining process and repositories.
- Creating workflows using genericGluejobs to automate the ETL process.
- Storeddatain AWS S3 like HDFS and performed EMR programs ondatastored in S3.
- Implemented value drivenDevOpsbased delivery model across GeneralInsurance (GI).
- Developed and configured Build and Release (CI/CD) processes using AzureDevOps, along wif managing application code using Azure GIT wif required security standards.
- Converted all Hadoop jobs to run in EMR by configuring the cluster according to thedatasize.
- Development of python scripts usingDataframes/SQL/Datasets and RDD in spark forDataAggregation,queries and writingdatato AWS Redshift staging tables using AWSGlueJobs & workflows.
- Created scripts to appenddatafrom temporary HBase table to target HBase table in Spark.
- Collecting and aggregating large amounts of logdatausing Apache and stagingdatain HDFS for further analysis.
- Involved in porting the existing on-premise Hive code migration to GCP (Google Cloud Platform) BigQuery.
- Implementation in designed system to collectdatafrom multiple portals using Kafka and then process it using spark.
- Developed Python Script to importdataSQL Server into HDFS & created Hive views ondatain HDFS using Spark.
- Created workflow in Oozie to automate the tasks of loadingdatainto HDFS and pre-processing wif Hive.
- Developed Storm topology to ingestdatafrom various source into HadoopDataLake.
- Created web application using HBase and Hive API to compare schema betweenHBase and Hive tables.
- Developed analytical component using Scala, Spark and Spark Streaming.
- Implementation a Python Script to load the CSV files into the S3 buckets and createdAWS S3buckets, performed folder management in each bucket, managed logs and objects wifin each bucket.
- Involved in creating Hive Tables, loading wifdataand writing Hive queries which will invoke and run MapReduce jobs in the backend.
- Connected to AWS s3 using SSH and ran spark-submit jobs.
- Created scripts to appenddatafrom temporary HBase table to target HBase table in Spark.
- Collecting and aggregating large amounts of logdatausing Apache and stagingdatain HDFS for further analysis.
- Involved in creating ETLusing Pig, loading wifdataand writing Pig Latin queries which will run internally in Map Reduce way.
Environment: Hadoop, HDFS, Hive, Python, Sqoop, Glue, Spark, Scala, Hive,Cloudera CDH4, Oracle, Elastic search, Kerberos, DataStage, SFTP,dataLake,Impala, Jira, Wiki, Alteryx, Teradata, Shell/Perl Scripting, Kafka, AWS (EC2, S3).
Data Analyst /Data Engineer
Confidential, Boston, MA
Responsibilities:
- Responsible for design and development of Python programs, transform and harmonizedatasets in preparation for modeling.
- Gather business requirements, definition and design of thedatasourcing, and worked wif thedatawarehouse architect on the development of logicaldatamodels.
- Involved in porting the existing on-premise Hive code migration to GCP (Google Cloud Platform) BigQuery.
- Involved in file movements between HDFS and AWS S3 and extensively worked wifS3 bucket in AWS.
- Automated and monitored complete AWS infrastructure wif terraform.
- Worked on several types ofdatavisualizations using Python - Matplotlib andTableau. Extracteddatausing SQL to create reports.
- Created wif both SQL and Non-SQL database andAWS.
- Worked wif Market Mix Modeling to strategize the advertisement investments to better balance the ROI on advertisements.
- Performed reverse engineering using Erwin to redefine entities, attributes, and relationships existing database.
- Analyze functional and non-functional business requirements and translate into technicaldatarequirements and create or update existing logical and physicaldatamodels. Developeda pipeline using Kafka to storedatainto HDFS.
- Performed Regression testing for Gloden Test Cases from State (end to end test cases) and automated the process using python scripts.
- Developed Spark jobs using Scala for faster real-time analytics and used SparkSQL for querying.
- Used pandas, NumPy, seaborne, SciPy, Matplotlib, Scikit-learn, NLTK in Python for developing various machine learning algorithms..
- Utilized Waterfall methodology for team and project management.
- Self-starter wif an in-depth hands-on work experience wif large-scale implementations of Google Cloud Platform (GCP) and/or wif AWS or Azure.
- Worked on google cloud platform (GCP) services like compute engine, cloud load balancing, cloud storage, cloud SQL, stack driver monitoring and cloud deployment manager
- Used Git for version control wif theDataEngineerteam andDataScientists colleagues.
- Processed the imagedatathrough the Hadoop distributed system by using Map andReduce then stored into HDFS.
- PerformedDataVisualization and Designed Dashboards wif Tableau and generated complex reports including chars, summaries, and graphs to interpret the findings to the team and stakeholders.
- Wrote documentation for each report including purpose,datasource, column mapping, transformation, and user group.
- Utilized Waterfall methodology for team and project management.
Environment: Spark, YARN, HIVE, Pig, Scala, Mahout, Knife, TDD, Python, Hadoop, DynamoDB, Kibana, NOSQL, Sqoop, MYSQL.
Data Analyst
Confidential, Woonsocket, RI
Responsibilities:
- Invovled in design the mappings, logic, sessions, workflows and worklets.
- Involved in integrated testing for various mappings, tested the data and data integrity among various sources and targets.
- Involved in SQL Server Stored process, Tuned SQL Queries using indexes.
- Took part in building database objects tables, indexes, sequences, and constraints.
- Involved in shell scripts to load the flat files into the database using SQL Loader.
- Involved in Pig Scripts for testing purposes.
- Developed JSON Scripts for deploying the Pipeline inAzureDataFactory (ADF) that process thedatausing the SQL Activity.
- Responsible in staging tables to handle import data and functions for duty calculations and validation of the inputs.
- Involved in Pl/SQL stored procedures, functions, triggers.
- Took part in creating Hive Tables to import large data sets from various relational databases and export the analyzed data back for visualization using Sqoop and report generation by the BI team.
Environment: SQL, Hive, Pig Script, Sqoop, Java, Pig Scripts AWS EMR, S3, Spark, Scala, Hive, Sqoop, ETL, Java, Athena, Glue Maven, GitHub HDFS, Map Reduce, Sqoop, Hive, Pig, Oozie, HBase, Python, Yarn Spark, Tableau
