Data Engineer Resume
Nyc, NY
SUMMARY
- Passionate Data Engineer/Scientist having strong background in statistics with 7+ years of experience, seeking to solve the bottlenecks of the business.
- Experience in Agile Methodology, participated in Sprints and daily Scrums to deliver software tasks on - time and with good quality on basis with onsite and offshore teams.
- Experience in development of BigData projects using Hadoop, Hive, Pig, Flume and MapReduce open source tools/technologies.
- Good understanding of Hadoop Gen1/Gen2 architecture, YARN architecture and its daemons Node manager, Resource manager, App Master and Map Reduce programming paradigm.
- Responsible for setup, configuration and migration of Excel based system toAzure SQL database.
- Experience in writing complex Pig scripts, Hive& Impala queries, implementation, importdatato Hadoop using Sqoop & vice versa.
- Good working with bigdata on AWS cloud services - EC2, S3, EMR, DynamoDB andRedshift.
- Strong experience with traditional ETL tools likeTalendand Apteryx
- Experience in Performance tunedETLBatch jobs, Reporting and Ad-hoc SQL.
- Automated jobs and data pipelines usingAWSStep Functions,AWSLambda, andAWSCloud Trail.
- Leveraging AzureSynapseto build analytics ML model data pipeline
- Good working experience in designing Oozie workflows for cleaning data and storing into Hive tables for quick analysis.
- Skilled in developing applications using Amazon Web Services like EC2, Virtual private clouds (VPCs), Storage models (EBS, S3, and instance storage), Elastic Load balancer ELBs.
- Good Understanding of otherAWSservices like S3, EC2, IAM, RDS Experience with Orchestration andDataPipeline likeAWSStep functions/DataPipeline/Glue.
- Experienced in usingAWSLambda and Glue for creating highly functionalDatapipelines
- Implemented AWS solutions using EC2, S3, RDS, EBS, Elastic Load Balancer, Auto scaling groups.
- Designed and engineered on premise to off-premise CI/CD Docker pipelines (integration and deployment) with ECS, Glue, Lambda, ELK, Spark data bricks, and firehose and kinesis stream.
- Snowflake Schema, and OLAP Cubes like MOLAP, ROLAP and HOLAP (hybrid). Executed various OLAP operations of Slicing, Dicing, Roll-Up, Drill-Down and Pivot in multidimensional data and analyzed reports in Analysis Tool Pak in MS Excel.
- Extensive experience with advanced J2EE Frameworks such as spring, Struts, JSF and Hibernate
- Experience in importing and exporting data using Sqoopfrom HDFS to Relational Database Systems and vice-versa.
- Extensively worked on statistical analysis tools and adept at writing code in Advanced Excel, R, MATLAB, and Python.
- Integrated Python with BigDataand Analytics based on Hadoop, Spark and No-SQL databases like Hbase and MongoDB.
- Excellent Understanding of Spark and its benefits in BigDataAnalytics
- Loaded application analytics data into datawarehouse using Batch processing.
- Experience in Extract, Transfer and Load process using ETL tools likeDataStage,DataIntegrator and SSIS forDatamigration andDataWarehouse projects.
- Expertise in Statistical analysis, Predictive modeling, Text mining, Supervised learning, Unsupervised Learning.
- Highly Experienced onDataAcquisition processes, handling complexdatasets with multiple file formats (CSV, JSON).
- Worked on building ETL pipelines usingAWSGlue and Python to create Dashboards in Tableau.
- Experience in BigData Analytics and developing data models using Hive, PIG, and Map reduce, SQL with strongdataarchitecting skills designingdata-centric solutions.
- Highly Skilled at python coding using SQL, NumPy, Pandas and Spark forDataAnalysis and Model building, Cognitive
- Used Azure Terraform and Azure Ops Works to deploy the infrastructure necessary to create development, test, and production environments for a software development project.
- Used Docker to enable faster software delivery cycles and for application portability.
PROFESSIONAL EXPERIENCE
Data Engineer
Confidential - NYC, NY
Responsibilities:
- Performed data visualization and Designed dashboards with Tableau, and generated complex reports, including charts, summaries, and graphs to interpret the findings to the team and stakeholders.
- Utilized AWSGlue to run ETL jobs and run aggregation on pyspark code.
- Developed and deployed production code using GitHub and Jenkins.
- Makes indent of required materials inSAP(Material Purchase Requisition) before commencing of work according to the planned dates.
- Schedule and monitor data pipelines and dashboards usingAirflow
- Transformed the rawdatainto actionable insights by incorporating various statistical techniques and usingdatamining tools such as Python (Scikit-Learn, Pandas, NumPy, Matplotlib) and SQL.
- Extending HIVE and PIG core functionality by using custom User Defined Function's (UDF), User Defined Table-Generating Functions (UDTF) and User Defined Aggregating Functions (UDAF) for Hive and Pig.
- Makes Service Purchase Requisition for the requirement of vendors inSAP.
- Developed and deployed successfully many modules on Spark, Hive,Sqoop, Shell, Pig, Scala and Python.
- Worked as a developer on ETL toolTalendto create jobs as per business requirement.
- Performed theDataMapping,Datadesign (DataModeling) to integrate thedataacross the multiple databases into EDW.
- Designed and Developed ETL Processes inAWSGlue to migrate Campaigndatafrom external sources like S3, ORC/Parquet/Text Files intoAWSRedshift.
- Provide data integration using ETLTalendOS.
- Ingested numerous types of data like XML, JSON, CSV files, and PDFs using Pyspark andTalend.
- Analysis of existing Hive queries and apply Optimization while migrating to Spark.
- In-depth knowledge of Snowflake Database, Schema and Table structures.
- Build servers usingAWS, importing volumes, Cloud Front, launching EC2, RDS, creating security groups, auto-scaling, load balancers (ELBs) in the defined virtual private connection and open stack to provision new machines for clients.
- DataExtraction, aggregations and consolidation of AdobedatawithinAWSGlue using PySpark.
- Created Airflow Scheduling scripts in Python.Importeddatafrom Oracle database to HDFS usingSqoop
- Check the vendor bills and make Service Entry sheet inSAP.
- Built models using Python and PySpark to predict the probability of attendance for various campaigns and events.
- I was responsible for designing and development of jobs to ingest data into clusters usingTalend.
- ImplementCI/CDpipelines using GitHub, Docker, Kubernetes and Python
- Locate and import theAWSS3 data files to the Hive warehouse, perform the transformations and actions using Spark RDD's
- Have implementedKafkaconsumer in Java and used Snowflake to load theKafkalogs
- Perform setting up workflow using Apache Airflow andOozieworkflow engine for managing and scheduling Hadoop jobs
- Automated statisticaldatareview anddatavisualization inShellScripting.
- DesigningETLpipelinesin Microsoft SQL Server Management Studio and Microsoft
- TestedETL, ELT, and ECTL processes with source systems as varied as oracle RDBMS, DB2, and flat files.
- Wrking in writing UNIXshellscripts and automation of the ETL processes using UNIXshellscripting.
Environment: Environment: Python, SQL, Sqoop, Tableau, MLlib, regression, Cluster analysis, SVM, Spark, MongoDB, logistic regression, Hadoop, PySpark, random forest, Apache Airflow, ShellScripting,AWSGlue.Built a workflow and deployed in AWS ECS as aKubernetesContainer.
Data Engineer
Confidential - Oaks, PA
Responsibilities:
- Developed Spark SQL to load tables into HDFS to run select queries on top.
- Developed analytical component using Scala, Spark, and Spark Stream.
- Used Visualization tools such as Power view for excel, Tableau for visualizing and generating reports.
- Designed and implemented customized linear regression model to predict the sales utilizing diversedatasources to predict demand, risk and price elasticity.
- Implemented automated Compliance Dashboard on AWS Elastic Search Dashboard with AWS Config with custom rules using AWS S3 Data Lake.
- Worked on Full Life Cycle implementation and technical knowledge to troubleshoot issues and enhance performance inSAPBusiness Intelligence (BI) landscapes.
- Utilized Crawler to populateAWSGlue data Catalog with metadata table definitions
- Responsible forDataCleaning,dataquality, features scaling, features engineering by using NumPy and Pandas in Python.
- Developed the tools using Python Django and used MongoDB for databases. Parsers written in Python for extracting usefuldatafrom the designdatabase. Used Parse kit (Enigma.io) framework for writing Parsers for ETL extraction.
- Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Python.
- Builtdatapipelines to movedatafrom source to destination scheduling byAirflow.
- Involved indataingestion from relational databases into HDFS usingSqoop
- Developed a fully automated continuous integration system using Git, Gerrit, Jenkins, MySQL and custom tools developed in Python and Bash.
- UseTalendto to import and transform structured data from internal databases.
- Implemented Error handling inTalendto validate the data Integrity and data completeness for the data from the Flat File.
- Write UNIXShellScriptingfor Informatics Pre-Session, Post-Session Scripts and also to run the workflows
- Technical oversight of electronic claims and EDI validation applications such as the declared tool to determine if the test or production EDI files met theHIPPArequirement.
- Developed POCs for bulk load options, web service API with inTalend.
- Involved in importing the real timedatato Hadoop using Kafka and implemented theOoziejob for daily imports Engineer Docker files to create portable services across AWS EC2 andKubernetes
- Airflowto schedule ETL jobs and Glue and Athena to extract the data from AWS data warehouse
- Designed and developed architecture fordataservices ecosystem spanning Relational, NoSQL, and Big Data technologies.
- Worked with Hadoop eco system covering HDFS, HBase, YARN and Map Reduce.
- Performed theDataMapping,Datadesign (DataModeling) to integrate thedataacross the multiple databases into EDW.
- Used Spark on Hive to process large amount of user's banking data.
- Writing bashshellscripts on local setups to automate jobs
- Monitored and debug performance issues on different OSLinux / UNIX.
- Design and develop the architecture for data flow in snowflake
- Developing the ETL mappings, Automation of the process thruAirflow
Environment: Python Apache Spark/ pyspark, Airflow, Sqoop, Unix, Apache Hadoop, Alteryx, DOMO, MongoDB,Apache Hive, AWS EMR, AWS EC2, ShellScripting, AWS S3, Pandas, NumPy, Databricks, Terraform, Blob storage, Teradata and Tableau, MS Excel,SQL
Data Engineer
Confidential - Hartford, CT
Responsibilities:
- Generated Sub-Reports, Drill down reports, Drill through reports and Parameterized reports using SSRS.
- Developed story telling dashboards in Tableau Desktop and published them on to Tableau Server which allowed end users to understand thedataon the fly with the usage of quick filters for on demand needed information.
- Migratingdatafrom Legacy system to SQL Server using SQL Server Integration Services 2012.
- Used C# scripts to map records.
- Build back-end application with Python / Django, work with Docker, RabbitMQ, Celery, and Jenkins.UsedTerraformto automate the configuration and setting up Azure services.
- Involved in writing complex SQL Queries, Stored Procedures, Triggers, Views, Cursors, Joins, Constraints, DDL, DML and User Defined Functions to implement the business logic and created clustered and non-clustered indexes.
- Created and modified Stored Procedures, Functions, and Indexes.
- Imported/exporteddatafrom RDBMS to HDFS usingDataIngestion tools likeSqoop.
- Automated Repository Migration from bit bucket toGitHub(over 200 repositories) using a python script
- Worked on OS like Microsoft Windows / where Apache is the Web server and MySQL handles the database components, while PHP, Python, or PERL represents the dynamicscriptinglanguages.
- Responsibilities include requirement gathering, Data modelling, developingETLapplications, E2E Testing.
- Involved in various phases of the project like Analysis, Design, Development, and Testing.
- Transformed data and loaded them into Amazon RDS database; wrote SQL queries to manage and process data
- Collaborated with team to build data pipelines and UI for website’s data analysis modules using AWS and Git.
- Created various types of data visualizations using R, and Tableau.
- Designed front end and backend of the application utilizing Python on Django Web Framework.
Environment: AWS, Chef, Ansible, Jenkins, DOMO, pyspark,, Sqoop, Unix, MongoDB, Maven, Git, Alteryx, Docker, Nexus, Jira, Linux, python, ruby, WebLogic, Nagios, Splunk, ShellScripting, terraform, Apache., MS Excel
