Sr. Bigdata/ Data Engineer Resume
Patskala, OhiO
SUMMARY
- Around 8 years of professional experience in IT and Analysis, Design, Development, Documentation, Deployment and Integration using SQL and BigData technologies.
- Worked on various Azure Services like Data Lake, Data Lake Analytics, SQL Database, Synapse, Data Bricks, Data factory, Analysis services, Logic Apps and SQL Data warehouse
- Experienced in building modern data warehouses in Azure Cloud, including building reporting in PowerBI.
- Experienced in migrating existing databases to Azure for better reporting experience.
- Created Azure Data Lake Storage (ADLS) (Gen 1 and Gen 2) and Ingested the Data from Flat files, CSV Files, Json Files and On - Premise Database Tables using Azure Data Factory V2(ADF)
- Created Azure SQL Data Warehouse and Loaded the data from On-Premise Databases, Azure SQL Databases, ADLS Gen 1, ADLS Gen 2, Azure BLOB Storage using Both ETL and ELT Fashion using Azure Data factory and Azure DataBricks (ADB) and PolyBase feature in SQL 2016 using External Tables
- Implemented and gained knowledge on many New ADF Mapping Data Flow components like New Branch, Join, Conditional Split, Exists, Union, Lookup, Derived Column, Surrogate Key, Window, Filter, Sort, Alter Row
- Vastly experienced in creating complex data pipelines using Azure data factory, Air Flow, T-SQL scripts, SSIS packages and Python scripts.
- Broadly utilized Python Libraries PySpark, Pytest, Pymongo, cxOracle, PyExcel, Boto3, Psycopg, embedPy, NumPy and Beautiful Soup.
- Obtained extensive experience in design and development of SSIS Packages to extract data from flat files, excel files, and operational databases.
- Gained Hands on experience in building CI/CD Azure pipelines using the Microsoft agent and running python file related tests.
- Have a practical understanding of the Data modeling concepts like Star-Schema, Snowflake Schema, Fact and Dimension tables.
- Extensive experience in Feature Engineering, Statistical Modeling, Machine Learning, Text Analytics, Data Mining with Large Data Sets of Structured and Unstructured Data and Performed Data Acquisition, Data Validation, Predictive Modeling and Data Visualization.
- Worked on BigData technologies like Hadoop/ HDFS, Spark, MapReduce, Pig, Hive, Scoop to extract and load data of various heterogeneous sources like Oracle, flat files, XML, other streaming data sources into EDW and transform for analysis (ETL/ELT).
- Hands on experience in implementing Regression and Classification techniques like GLM/Linear and Logistic Regression, Regularization, Random Forest, Decision Trees, K-NN, SVM, Clustering, Neural Networks, Principal Component Analysis, and good knowledge on Recommender Systems.
- Validated machine learning classifiers using Accuracy, AUC, ROC Curves and Lift Charts. • Extensive working experience in RDBMS technologies like Oracle, MS SQL Server, Excel and MYSQL, with good knowledge of SQL, TOAD, SQL Plus, Win SQL. Good at working with SQL Assistant in Teradata environment.
- Highly skillful in using visualization tools such as Tableau, Power BI, Microsoft Excel -, Pivot Tables, DAX.
- Good ability to work on multiple tasks and strong interpersonal, analytical, presentation skills, communication and problem-solving skills and work independently as well as in a team.
TECHNICAL SKILLS
BigData Technologies: AWS, Azure, EMR, S3, EC2-Fleet, Spark-2.2, 2.0 and 1.6, Hortonworks HDP, Hadoop, MapReduce, Pig, Hive, Apache Spark, Spark SQL, Informatica Power Center 9.6.1/8.x, Kafka, NoSQL, Elastic MapReduce(EMR), Hue, YARN, Nifi, Impala, Sqoop, Solr, Oozie.
Data Bases: Cloudera Hadoop CDH 15.x, Hortonworks HDP, Oracle 10g/11g, Teradata, DB2, Microsoft SQL Server, MySQL, NoSQL, SQL databases.
Platforms (O/S): Red-Hat LINUX, Ubuntu, Windows NT/2000/XP.
Programming Languages: Java, Scala, SQL, UNIX shell script, JDBC, Python, Perl.
Security Management: Hortonworks Ambari, Cloudera Manager, Apache Knox, XA Secure, Kerberos.
Web Technologies: DHTML, HTML, XHTML, XML, XSL (XSLT, XPATH), XSD, CSS, JavaScript, SOAP, RESTful, Agile, Design Patterns
Data Warehousing: Informatica PowerCenter/Powermart/Dataquality/Bigdata, Pentaho, ETL Development, Amazon Redshift, IDQ.
Database tools: JDBC, HADOOP, Hive, No-SQL, SQL Navigator, SQL Developer, TOAD, SQL Plus, SAP Business Objects
Data Modeling: Rational Rose, Erwin 7.3/7.1/4.1/4.0
Code Editors: Eclipse, Intellij
PROFESSIONAL EXPERIENCE
Sr. BigData/ Data Engineer
Confidential, Patskala, Ohio
Responsibilities:
- Enhanced and optimized Spark/Pyspark jobs to aggregate, group and run data mining tasks using the Spark framework.
- Worked on AWS tools like EMR, Redshift, and S3.
- Importing and exporting data into HDFS and hive using Sqoop and Kafka with batch and streaming.
- Worked on Oracle DB as backend database to store monitoring information.
- Used Service now platform to open and close tickets.
- Involved in complete BigData flow of the application data ingestion from upstream to HDFS, processing the data in HDFS and analyzing the data using several tools.
- Experienced on ingesting data from RDBMS sources like - Oracle, SQL Server and Netezza into HDFS using Sqoop.
- Utilized PySpark and Pandas to figure the moving normal and RSI score of the stocks and produced them into information stockroom.
- Built and planned ETL pipeline utilizing python to bring information from Redshift information distribution center and applications.
- Plan and Develop ETL Processes in AWS Glue to move Campaign information from outer sources like S3, ORC/Parquet/Text Files into AWS Redshift.
- Deployed Pyspark applications and developed in Databricks cluster.
- Experience in managing and reviewing huge Hadoop log files.
- Created Data Pipelines as per the business requirements and scheduled it using Oozie.
- Imported the data from various formats like JSON, Sequential, Text, CSV, AVRO and Parquet to HDFS cluster with compressed for optimization.
- Experienced on ingesting data from RDBMS sources like - Oracle, SQL Server into HDFS using Sqoop.
- Expertise in designing and creating various analytical reports and Automated Dashboards to help users to identify critical KPIs and facilitate strategic planning in the organization
- Experience in CI/CD tool Jenkins for code deployment and Automic scheduling of jobs.
- Good knowledge and interest in data science and Machine learning concepts, spent time understanding code and project details of machine learning teams
ENVIRONMENT: Hive, pyspark, Jenkins, Snowflake,Airflow, Gerrit, Kafka, Pyspark, Python, Spark, Sqoop, Maven, Automic, SQL, Scala, Junit, Intellij, Oracle, Databricks, Aws cloud.
Data Engineer
Confidential, El Segundo, CA
Responsibilities:
- Responsible for requirements gathering and analyzing the data sources.
- Responsible for building scalable distributed data solutions using Hadoop.
- Worked very closely with the Hadoop Administrator to set up the Hadoop cluster.
- Worked on analyzing Hadoop cluster and different BigData analytic tools including Pig, Hive HBase database and SQOOP.
- Importing and exporting data into HDFS and Hive using SQOOP.
- Generate Tableau reports to verify the Hive table’s data.
- Used Pig as ETL tool to do Transforillionions, even joins and some pre-aggregations before storing the data onto HDFS.
- Experience working in spark eco system using SparkSql queries on different formats like Text file, CSV file.
- Involved in loading data from LINUX file system to HDFS.
- Load and Transform large sets of structured and semi structured data.
- Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose.
- Accountable for PySpark code, making information outlines from tables in information administration layer and thinking of them to a Hive information stockroom.
- Created Pipelines in ADF utilizing Linked Services/Datasets/Pipeline/to Extract, Transform and burden information from various sources like Azure SQL, Blob stockpiling, Azure SQL Data stockroom, compose back apparatus and in reverse.
- Developed JSON Scripts for conveying the Pipeline in Azure Data Factory (ADF) that interaction the information utilizing the Sql Activity.
- Developed sparkle applications in python (PySpark) on dispersed climate to stack immense number of CSV records with various pattern in to Hive ORC tables.
- Made PySpark code that utilizations Spark SQL to produce information outlines from avro arranged crude layer and thinks of them to information administration layer inward tables as orc design.
- Created Hive tables to store variable data formats of PII data coming from different portfolios.
- Implemented a script to transmit data from Oracle to Hive using Sqoop.
- Designed and developed MapReduce jobs to analyze the data.
- Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
- Responsible to manage data coming from different sources.
- Involved in loading and transforming large sets of structured, semi structured and unstructured data.
- Experience using Amazon Web Services.
- Installed Oozie workflow engine to run multiple Hive and pig jobs.
- Analyzed large amounts of data sets to determine optimal way to aggregate and report on it.
- Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.
- Involved in email campaign launch and served as point of contact for anything related to data, pulling customers list, matching and finished training in Strong View email platform software.
Environment: Mapr, Hadoop MapReduce, Azure, Pyspark, Python, HDFS, Spark, Hive, Pig, SQL, Sqoop, Flume, Oozie, Java 8, Eclipse HBase, Shell Scripting, Scala.
Data Engineer
Confidential, Lake Success, NY
Responsibilities:
- Designing and building multi-terabyte, full end-to-end Data Warehouse infrastructure from the ground up on Confidential Redshift for large scale data handling Millions of records every day
- Worked on BigData on AWS cloud services i.e. EC2, S3, EMR and DynamoDB
- Managed security groups on AWS, focusing on high-availability, fault-tolerance, and auto scaling using Terraform templates. Along with Continuous Integration and Continuous Deployment with AWS Lambda and AWS code pipeline.
- Developed SSRS reports, SSIS packages to Extract, Transform and Load data from various source systems
- Implementing and Managing ETL solutions and automating operational processes.
- Optimizing and tuning the Redshift environment, enabling queries to perform up to 100 xs faster for Tableau and SAS Visual Analytics.
- Imported information from AWS S3 into Spark RDD, Performed changes and activities on Rdd's.
- Defined facts, dimensions and designed the data marts using the Ralph Kimball's Dimensional Data Mart modeling methodology using Erwin.
- Created Entity Relationship Diagrams (ERD), Functional diagrams, Data flow diagrams and enforced referential integrity constraints and created logical and physical models using Erwin.
- Created ad hoc queries and reports to support business decisions SQL Server Reporting Services (SSRS).
- Developed complex ETL Packages using SQL Server 2008 Integration Services to load data from various sources like Oracle/SQL Server/DB2 to Staging Database and then to Data Warehouse.
- Construct AI models to feature large information capacities utilizing Pyspark and MLlib.
- Analyze the existing application programs and tune SQL queries using execution plan, query analyzer, SQL Profiler and database engine tuning advisor to enhance performance.
- Optimization of Hive inquiries utilizing best practices and right boundaries and utilizing innovations like Hadoop, YARN, Python, PySpark.
- Involved in the Forward Engineering of the logical models to generate the physical model using Erwin and generate Data Models using ERwin and subsequent deployment to Enterprise Data Warehouse.
- Wrote various data normalization jobs for new data ingested into Redshift
- Created various complex SSIS/ETL packages to Extract, Transform and Load data
- Advanced knowledge on Confidential Redshift and MPP database concepts.
- Migrated on premise database structure to Confidential Redshift data warehouse
- Was responsible for ETL and data validation using SQL Server Integration Services.
- Defined and deployed monitoring, metrics, and logging systems on AWS.
- Implemented Work Load Management (WML) in Redshift to prioritize basic dashboard queries over more complex longer-running adhoc queries. This allowed for a more reliable and faster reporting interface, giving sub-second query response for basic queries.
- Worked publishing interactive data visualizations dashboards, reports /workbooks on Tableau and SAS Visual Analytics.
- Used Hive SQL, Presto SQL and Spark SQL for ETL jobs and using the right technology for the job to get done.
Environment: SQL Server, Erwin, Oracle, Redshift, Informatica, RDS, NOSQL, Pyspark, Python, Snow Flake Schema, MySQL, PostgreSQL.
Data Analyst
Confidential
Responsibilities:
- Developed stored procedures in MS SQL to fetch the data from different servers using FTP and processed these files to update the tables.
- Responsible for Designing Logical and Physical data modeling for various data sources on Confidential Redshift.
- Performed logical data modeling, physical Data modeling (including reverse engineering) using the Erwin Data modeling tool.
- Developed Complex database objects like Stored Procedures, Functions, Packages and Triggers using SQL and PL/SQL.
- Created dimensional model for the reporting system by identifying required dimensions and facts using Erwin.
- Generated server side PL/SQL scripts for data manipulation and validation and materialized views for remote instances.
- Designed and Developed ETL jobs to extract data from Salesforce replica and load it in data mart in Redshift.
- Involved in performance tuning, stored procedures, views, triggers, cursors, pivot, unpivot functions, CTE's
- Developed and delivered dynamic reporting solutions using SSRS.
- Extensively used Erwin for Data modeling. Created Staging and Target Models for the Enterprise Data Warehouse.
- Involved in Normalization / De normalization techniques for optimum performance in relational and dimensional database environments.
- Created report models from cubes as well as relational data warehouse to create ad-hoc reports and chart reports.
- Resolved the data type inconsistencies between the source systems and the target system using the Mapping Documents and analyzing the database using SQL queries.
- Worked on ETL testing, and used SSIS tester automated tool for unit and integration testing.
- Designed and created SSIS/ETL framework from ground up.
- Created new Tables, Sequences, Views, Procedure, Cursors and Triggers for database development.
- Created ETL Pipeline using Spark and Hive for ingest data from multiple sources.
- Involved in using SAP and transactions done in SAP - SD Module for handling customers of the client and generating the sales reports.
- Creating reports using SQL Reporting Services (SSRS) for customized and ad-hoc Queries.
- Coordinated with clients directly to get data from different databases.
- Worked on MS SQL Server, including SSRS, SSIS, and T-SQL.
- Designed and developed schema data models.
- Documented business workflows for stakeholder review.
Environment: ER Studio, SQL Server 2008, SSIS, Oracle, Business Objects XI, Rational Rose, Data stage, MS Visio, SQL, Crystal Reports 9
SQL Developer
Confidential
Responsibilities:
- Planning, designing, and implement application database code objects, such as stored procedures and views.
- Building and maintain SQL scripts, indexes, and complex queries for data analysis and extraction.
- Provide database coding to support business applications using Sybase T-SQL
- Performed quality assurance and testing of SQL server environment.
- Used Erwin tool for dimensional modeling (Star schema) of the staging database as well as the relational data warehouse.
- Developed new processes to facilitate import and normalization, including data file for counterparties.
- Worked with business stakeholders, application developers, and production teams across functional units to identify business needs and discuss solution options.
- Developed parameter and dimension based reports, drill-down reports, matrix reports, charts, and Tabular reports using Tableau Desktop.
- Retrieved data from data warehouse and generated a series of meaningful business reports using SSRS.
- Expertise in Client-Server application development using Oracle 12c/11g/10g/9i, PL/SQL, SQL PLUS, TOAD and SQL LOADER.
- Validated and tested reports, then published the reports to the report server.
- Designed coded, tested and debugged custom queries using Microsoft T-SQL and SQL Reporting Services.
- Experience in Oracle Dynamic SQL, Records Collections and PL/SQL Tables.
- Conducted research to collect and assemble data for databases - Was responsible for design/development of relational databases for collecting data
- Built data input and designed data collection screens - Managed database design and maintenance, administration and security for the company.
Environment: MS SQL Server, Windows Advanced Server, VB, and XML, DTS, Query Analyzer, SSRS, SQL Profiler and Enterprise Manager.
