Data Engineer Resume
Evansville, IN
SUMMARY
- Over 8+ years of strong experience in Data Analyst, Data mining with large data sets of Structured and Unstructured data, Data Acquisition, Data Validation, Predictive modeling, Statistical modeling, Data modeling, Data Visualization, Web Crawling, Web Scraping. Adept in statistical programming languages like R and Python, SAS, Apache Spark, Matlab including Big Data technologies like Hadoop, Hive, Pig.
- Excellent knowledge of Machine Learning, Mathematical Modeling and Operations Research. Comfortable with R, Python, SAS and Weka, MATLAB, Relational databases. Deep understanding & exposure of Big Data Eco - system.
- Experienced on data architecture including data ingestion pipeline design, Hadoop information architecture, data modeling and data mining, machine learning and advanced data processing.
- Deep analytics and understanding of Big Data and algorithms using Hadoop, MapReduce, NoSQL and distributed computing tools.
- Expertise in synthesizing Machine learning, Predictive Analytics and Big data technologies into integrated solutions.
- Experienced in Dimensional Data Modeling experience using Data modeling, Relational Data modeling, ER/ Studio, Erwin, and Sybase Power Designer, Star Join Schema/Snowflake modeling, FACT & Dimensions tables, Conceptual, Physical & logical data modeling.
- Hands on experience in implementing LDA, Naive Bayes and skilled in Random Forests, Decision Trees, Linear and Logistic Regression, SVM, Clustering, neural networks, Principle Component Analysis.
- Expertise in managing entire data science project life cycle and actively involved in all the phases of project life cycle including data acquisition, data cleaning, data engineering, features scaling, features engineering, statistical modeling (decision trees, regression models, neural networks, SVM, clustering), dimensionality reduction using Principal Component Analysis and Factor Analysis, testing and validation using ROC plot, K- fold cross validation and data visualization.
- Experienced in writing Pig Latin scripts, MapReduce jobs and HiveQL.
- Extensively used SQL, Numpy, Pandas, Scikit-learn, Spark, Hive for Data Analysis and Model building.
- Extensively worked on ERWIN tool with all features like REVERSE Engineering, FORWARD Engineering, SUBJECTAREA, DOMAIN, Naming Standards Document etc.
- Experience in using various packages in Rand python like ggplot2, caret, dplyr, Rweka, gmodels, RCurl, tm, C50, twitteR, NLP, Reshape2, rjson, plyr, pandas, numpy, seaborn, SciPY, MatPlotLib, scikit-learn, Beautiful Soup, Rpy2.
- Excellent and experience and knowledge of Machine Learning, Mathematical Modeling and Operations Research. Comfortable with R, Python, SAS and Weka, MATLAB, Relational databases. Deep understanding & exposure of Big Data Eco-system.
- Experienced in importing and exporting the data using Sqoop from HDFS to Relational Database systems/ mainframe and vice-versa.
- Extensively worked on Sqoop, Hadoop, Hive, Spark, Cassandra to build ETL and Data Processing systems having various data sources, data targets and data formats
- Strong experience and knowledge in Data Visualization with Tableau creating: Line and scatter plots, Bar Charts, Histograms, Pie chart, Dot charts, Box plots, Time series, Error Bars, Multiple Charts types, Multiple Axes, subplots etc.
- Experienced with Integration Services (SSIS), Reporting Service (SSRS) and Analysis Services (SSAS)
- Expertise in Normalization to 3NF/De-normalization techniques for optimum performance in relational and dimensional database environments.
- Extensive experienced on ER Modeling, Dimensional Modeling (Star Schema, Snowflake Schema) and Data warehousing and OLAP tools.
- Expertise in data base programming (SQL, PLSQL) XML, DB2, Informix, Teradata, Data base tuning and Query optimization.
- Experience in designing, developing, scheduling reports/dashboards using Tableau and Cognos.
- Expertise in performing data analysis and data profiling using complex SQL on various sources systems including Oracle and Teradata.
TECHNICAL SKILLS
SDLC: Agile/Scrum, Python, VBA, R-Studio, Hive, Pandas, NumPy, T-SQL, Visio, Matplotlib, Seaborn, R-Base, UNIX, Linux, Windows, Git, Scala, AWS Lambda, Kinesis(streams), Oracle, NoSQL, PostgreSQL, DataStage, PySpark, Power BI, Tableau, SQL Server, OLAP, SharePoint, OLTP, ETL, SQL Query, Python, SQL, PL/SQL, OLAP, OLTP, Tableau, Excel, T-SQL, SSAS/SSIS, R studio, XML, SharePoint, SSIS, SAS, SSRS, PySpark, Numpy. Hadoop, MapReduce,, Hive, Java, R, PostgreSQL, ETL, Informatica, R studio, Scikit-Learn, Seaborn, Numpy, SciPy, MySQL, SQL, PL/SQL, SSAS/SSIS, UAT, Informatica, ETL, CSV file, Excel, HTML, Tableau, SQL.
PROFESSIONAL EXPERIENCE
Data Engineer
Confidential, Evansville, IN
Responsibilities:
- Created and run advanced SQL queries to pinpoint client's data issue.
- Created issue tracking system using BI dashboard.
- Documented business rules and implemented data transformations using Pandas.
- Analysis of ETL Mappings based on Facts & Dimensions from Source to target tables for directs moves and indirect moves based on transformation rules & lookup tables.
- Building, publishing customized interactive reports and dashboards, report scheduling using Tableau Desktop.
- Developed Scala scripts using both Data frames/SQL/Data sets and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop.
- Analyzed the business requirements for web applications and SharePoint application.
- Utilized Apache Spark with Python to develop and execute Big Data Analytics and Machine learning applications, executed machine Learning use cases under Spark ML and Mllib.
- Developed and deployed various Lambda functions in AWS with in-built AWS Lambda Libraries and also deployed Lambda Functions in Scala with custom Libraries.
- Automated resulting scripts and workflow using Apache Airflow and shell scripting to ensure daily execution in production.
- Actively involved in writing T-SQL Programming for implementing Stored Procedures and Functions and cursors, views for different tasks.
- Used Apache Falcon for mirroring of HDFS and HIVE data.
- Participate in the design, build and deployment of NoSQL implementations like MongoDB.
- Extensively used Apache Kafka, Apache Spark, HDFS and Apache Impala to build a near real time data pipelines that get, transform, store and analyze click stream data to provide a better personalized user experience.
- Worked on Big data on AWS cloud services i.e., EC2, S3, EMR and DynamoDB.
- Performed analysis of implementing Spark using Scala and wrote spark sample programs using PySpark
- Analyzed sales of past few years using data mining tools like R Studio, Python.
- Developed SSRS reports, SSIS packages to Extract, Transform and Load data from various source systems.
- Worked on ETL testing and used SSIS tester automated tool for unit and integration testing.
- Responsible for building-scalable distribution data solution using Hadoop.
- Developed AWS Lambda to invoke glue job as soon as a new file is available in Inbound S3 bucket.
- Created monitors, alarms, notifications and logs for Lambda functions, Glue Jobs, EC2 hosts using Cloudwatch.
- Design and implement secure data pipelines into a Snowflake data warehouse from on-premise and cloud data sources
- Creation of best practices and standards for data pipelining and integration with Snowflake data warehouses.
- Functioned as SME and managed new system that streamlined customer information process and increased productivity.
- Involved with all the phases of Software Development Life Cycle (SDLC) methodologies throughout the project life cycle.
- Worked extensively with Dimensional modeling, Data migration, Data cleansing, ETL Processes for data warehouses.
- Developed data Mart for the base data in Star Schema, Snow-Flake Schema.
- Used Hive, Impala and HBase as part of Cloudera Hadoop/big data testing.
- Used Web services (SOAP) for transmission of large blocks of XML data over HTTP.
- Developed Test Cases for Report, SharePoint, and Web applications for UAT environment.
- Extensively worked on to develop test plans and test strategies for ETL, Report, and SharePoint and Web application projects.
- Created Use Case Diagrams, Activity Diagrams, Sequence Diagrams and ER Diagrams in MS Visio.
- Involved in writing T-SQL working on SSIS, SSAS, Data Cleansing, Data Scrubbing and Data Migration.
- Worked on claims data and extracted data from various sources such as flat files, Oracle, and Mainframes.
- Designed and generated various dashboards, reports using various Tableau Visualizations.
Environment: SDLC- Agile/Scrum, Python, VBA, R-Studio, Hive, Pandas, NumPy, T-SQL, Visio, Matplotlib, Seaborn, SQL, R-Base, UNIX, Linux, Windows, Git, Scala, AWS Lambda, Kinesis(streams), Oracle, NoSQL, PostgreSQL, DataStage, PySpark, Power BI, Tableau, SQL Server, OLAP, SharePoint, OLTP, ETL, SQL Query.
Data Engineer
Confidential, Columbus, Indiana
Responsibilities:
- Worked with applications like R and Python to develop neural network algorithms, cluster analysis.
- Conducted a range of statistical analyses to provide valuable data-driven insights for business decision making.
- Worked with packages like ggplot2 and shiny in R to understand data and developing applications.
- Prepared ETL technical Mapping Documents along with test cases for each Mapping for future developments to maintain SDLC and Migration process.
- Have actively taken part in Data Profiling, Data Cleansing, Data Migration, Data Mapping and actively helped ETL developers to Compare data with original source documents and validate Data accuracy.
- Worked on Tableau, to create dashboards and visualizations.
- Designed and developed the framework to consume the web services hosted in Amazon EC2 instances.
- Analyze customer data in Python and R to track correlations in customer behavior, define user segments to implement process and product improvements.
- Encoded and decoded json objects using PySpark to create and modify the dataframes in Apache Spark.
- Real time streaming of data using Spark with Kafka.
- Install and configure Apache Airflow for S3 bucket and Snowflake data warehouse and created dags to run the Airflow.
- Optimized the performance of queries with modifications in T-SQL queries, removed unnecessary columns, eliminated redundant and inconsistent data, normalized tables, established joins and created indexes whenever necessary.
- Extensively used Apache Sqoop for efficiently transferring bulk data between Apache Hadoop and relational databases (Oracle, MySQL) for predictive analytics
- Implemented AWS lambda functions, python script that pulls the privacy files from AWS S3 buckets to post to it the Malibu data privacy endpoints.
- Created Hive tables on top of HBase using Storage Handler for effective OLAP analysis.
- Designed Setup maintain Administrator the Azure SQL Database, Azure Analysis Service, Azure SQL Data warehouse, Azure Data Factory, Azure SQL Data warehouse.
- Extensively worked on OLAP cubes using SSAS and created and Managed OLAP Cubes using SSAS.
- Worked on cleaning, exploring and manipulation of source data and transform to target system using Python and tools such as Pandas, Numpy, Matplotlib and PostgreSQL.
- Developed the statistical models designed to forecast market variables under stress scenarios within Financial Models using R.
- Gathered and analyzed business requirements, interacted with various business users, project leaders, developers and took part in identifying different data sources.
- Implemented large Lamda architectures using Azure Data platform capabilities like Azure Data Lake, Azure Data Factory, HDInsight, Azure SQL Server, Azure ML and Power BI.
- Creating workflow and process-flow diagrams in MS Visio for various projects
- Used MongoDB to stored data in JSON format and developed and tested many features of dashboard using Python, Bootstrap, CSS, and JavaScript.
- Analyzed the business requirements and designed Conceptual and Logical Data models using Erwin and generated database schemas and DDL (Data Definition Language) by using Forward and Reverse Engineering.
- Well versed experienced in creating pipelines in Azure data factory using different activities like Move &Transform, Copy, filter, for each, Data bricks etc.
- Created Hive tables on HDFS to store the data processed by Apache Spark on the Cloudera Hadoop Cluster in Parquet format.
- Worked as data modeler/analyst by creating and developing relational and dimensional data models by using Erwin.
- Implemented Normalization and De-Normalization Techniques to build the tables, indexes, views and maintained and implemented stored procedures as per requirements. created pipeline jobs, schedule triggers using Azure data factory.
- Changed in design in SharePoint for data types and added Metrics.
- Imported and Exporting Reports from SharePoint and Access
- Involved in data migration processes to migrate historical data from legacy system into AWS Redshift (to S3 as flat files and Redshift as tables) using Python.
- Involved in creating charts and graphs of the data from different data sources by using matplotlib and scipy libraries in python.
- Used ad hoc queries for querying and analyzing the data, participated in performing data profiling, data analyzing, data validation and data mining.
- Developed complex ETL mappings for Stage, Dimensions, Facts and Data marts load.
- Involved in Data Extraction for various Databases & Files using Talend.
- Worked on Tableau for Data Analysis, Digging the data for source systems for analysis and deeply dive in the data for Predictive findings and for various data Analysis by using dash boards and visualization.
Environment: Windows, Linux, Java, R, R studio, Python, ETL, Talend, Pandas, Numpy, R-Base, T-SQL, Visio, Azure Databricks, R-Shiny, Matplotlib, SSIS, MongoDB, Git, SharePoint, PostgreSQL, Hive, Nifi,Tableau, Azure data factory, Azure analysis services, ETL, SQL, SDLC, Agile, Scrum, MySQL.
Data Engineer
Confidential, New York, NY
Responsibilities:
- Understood and articulated business requirements from user interviews and then convert requirements into technical specifications.
- Gathered and analyzed business requirements, interacted with various business users, project leaders, developers and took part in identifying different data sources.
- Involved with Data Analysis primarily Identifying Data Sets, Source Data, Source Meta Data, Data Definitions and Data Formats.
- Generated new data mapping documentations and redefined the proper requirements in detail.
- Used pandas, NumPy, seaborn, SciPy, MatPlotLib, scikit-learn in Python for developing various machine learning algorithms.
- Created and deployed data dashboards on docker to visualize the inquiries querying in SQL to retrieve data as per client inquiries providing daily reports using production data and experimental analysis using development data.
- Wrote test cases, developed Test scripts using SQL and PL/SQL for UAT.
- Worked with Apache Spark SQL and data frame functions to perform data transformations and aggregations on complex semi structured data.
- Extracted data from Twitter using Java and Twitter API. Parsed JSON formatted twitter data and uploaded to database.
- Identified issues within the data by querying the source data and identifying the data patterns.
- Visually plotted the data using MatPlotLib and Seaborn after performing analysis with pandas.
- Developed Spark Applications by using Spark, Java and Implemented Apache Spark data processing project to handle data from various RDBMS and Streaming sources.
- Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
- Using pandas Data Frame performed Group by, merging and joining operations like in SQL.
- Collected business requirements to set rules for proper data transfer from Data Source to Data Target in Data Mapping, ETL tools like Informatica for loading data into the staging tables in the database.
- Read date from different sources like CSV file, Excel, HTML page and SQL and performed data analysis and written to any data source like CSV file, Excel, or database.
- Fetched twitter feeds for certain important keyword using python -twitter library.
- Performed data visualization with Matplotlib, R Shiny, ggplot2 and Tableau.
- Implementing advanced procedures like text analytics and processing using the in-memory computing capabilities like Apache Spark written in Scala.
- Performed data testing, tested ETL mappings (Transformation logic), tested stored procedures, and tested the XML messages.
- Visualizing the data with the help of box plots and scatter plots to understand the distribution of data using Tableau, Python libraries.
- Developed Merge jobs in Python to extract and load data into MySQL database, also worked on Python ETL file loading and use of regular expression.
- Worked on data pre-processing and cleaning the data to perform feature engineering and performed data imputation techniques for the missing values in the dataset using Python.
- Used extracted data for analysis and carried out various mathematical operations for calculation purpose using python library - NumPy, SciPy.
- Data wrangling and scripting in Python, database cleanup in SQL, advanced model building in R/ Python, and expertise in data visualization.
Environment: Python, PostgreSQL, ETL, Informatica, R studio, Scikit-Learn, Seaborn, Numpy, SciPy, MySQL, SQL, PL/SQL, SSAS/SSIS, UAT, Informatica, ETL, CSV file, Excel, HTML, Tableau, SQL.
Data Analyst
Confidential
Responsibilities:
- Performed complex data analysis in support of ad-hoc and standing customer requests.
- Involved with data profiling for multiple sources and answered complex business questions by providing data to business users.
- Written a complex SQL, PL/SQL, Procedures, Functions, and Packages to validate data and testing process.
- Used SAS to mine, alter, manage, and retrieve data from a variety of sources and perform statistical analysis.
- Involved in data mining, transformation and loading from the source systems to the target system.
- Developed Ad-hoc reports using Tableau Desktop, Excel.
- Have good experience in creating Star schema cubes using SSAS.
- Developed company dashboards to integrate with SharePoint sites and provide users with one-stop shopping for reporting.
- Defined specifications like use case documentation, activity diagram, and business process flow using Microsoft Visio.
- Designing the User Interface using HTML/DHTML, Java script
- Worked with data investigation, discovery, and mapping tools to scan every single data record from many sources.
- Worked on database design for OLTP and OLAP systems
- Translated business concepts into XML vocabularies by designing XML Schemas with UML
- Designed security model of reporting system, creating groups, folders, and users in Business Objects.
- Created SSIS packages to populate data from various data sources.
- Performed the Data Accuracy, Data Analysis, Data Quality checks before and after loading the data.
- Developed database triggers and stored procedures using T-SQL cursors and tables.
- Deployed reports created report schedules and subscriptions. Managing and securing reports using SSRS.
- Created Crystal reports depend upon end user needs.
Environment: Python, SQL, PL/SQL, OLAP, OLTP, Tableau, Excel, T-SQL, SSAS/SSIS, R studio, XML, SharePoint, SSIS, SAS, SSRS, PySpark, Numpy.
Sr. Hadoop Developer
Confidential
Responsibilities:
- Importing and sending out information into HDFS and Hive utilizing Sqoop.
- Experienced in running Hadoop stream jobs to process terabytes of xml group information with the help of Map Reduce programs.
- Used parquet file format for published tables and created views on the tables.
- In-charge of managing data coming from different sources.
- Support in running MapReduce Programs in the cluster.
- Cluster coordination services through Zoo Keeper.
- Involved in loading information from UNIX document framework to Hadoop Distributed File System.
- Installed, configured Hive and furthermore composed Hive UDFs.
- Automated every one of the jobs for pulling information from FTP server to stack information into Hive tables, utilizing Oozie work processes.
- Writing data to parquet tables both non-partitioned and partitioned tables by adding dynamic data to partitioned tables using Spark.
- Wrote User Defined functions (UDFs) for special functionality for Spark.
- Used SQOOP Export functionalities and scheduled the jobs on daily basis with Shell scripting in Oozie.
- Worked with SQOOP jobs to import the data from RDBMS and used various optimization techniques to optimize Hive and SQOOP.
- Used SQOOP import functionality for loading Historical data present in a Relational Database system into Hadoop File System (HDFS).
Environment: - Hadoop, MapReduce, HDFS, Hive, Java, R, Sqoop, Spark, Sqoop
