Sr. Big Data Engineer Resume
SUMMARY:
- 8+ years of IT industry experience in analysis, design, testing, development, deployment and implementation of secure, scalable and high - performance BI and analytic solutions.
- Expertise in using major components of Hadoop ecosystem components like HDFS, Yarn, MapReduce, Hive, Pig, Sqoop, HBase, Spark, Spark SQL, Kafka, Spark Streaming, Oozie.
- Familiar with data architecture including data ingestion pipeline design, Hadoop information architecture, data modeling and advanced data processing.
- Experience covers areas of designing and development in the field of ETL methodologies, business intelligence applications and data Warehousing principles and architectures.
- Expertise in writing MapReduce Jobs in Java for processing large sets of structured, semi-structured and unstructured data sets and ingesting to various platforms like HDFS, S3 and Kafka.
- Extensively worked on Spark SQL and Automated queries using Python.
- Ability to write customized Kafka consumer code and alter existing producer code in python to push data to Spark-streaming jobs
- Built many successful ETL pipeline that involves data pull from different RDMS sources into HDFS using Sqoop and Oozie.
- Good understanding and knowledge of NoSQL databases like MongoDB, HBase and Cassandra.
- Experience in integration of NoSQL databases with Hadoop cluster and setting up environment required to execute various Pig and Hive jobs.
- Extensive knowledge in writing Hadoop jobs for data analysis as per the business requirements using Hive and worked on HiveQL queries for required data extraction, join operations, writing custom UDF's as required and having good experience in optimizing Hive Queries.
- Created Internal and External tables in Hive and Hive serialization and deserialization tables in HDFS and analyzed data with partitioning and bucketing concepts using Hive.
- Analyzed large amount of data with various formats including XML, Json, Parquet, Avro and ORC from different Data Sources, also use
PROFESSIONAL EXPERIENCE:
Confidential
Sr. Big Data Engineer
Responsibilities:
- Worked with the Spark for improving performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark - SQL, Spark MLlib, Data Frame, Pair RDD's, Spark YARN. Developed Sqoop and Oozie workflows to ingest data into Hadoop from various RDMS. Developed custom Kafka producer and consumer for different publishing and subscribing to Kafka topics. Worked on GIT, Jenkins and Udeploy for code moveup, build and deployment. Automated DDL creation using Liquibase files and used YAML to trigger jobs. Worked with AWS services such as EC2, S3, CloudWatch, VPCs,
- Security Groups, Route 53, ALB, ELB Migrated an existing on-premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR. Imported data from AWS S3 in Spark RDD, Performed transformations and actions on RDD's Used Hive QL to analyze the partitioned and bucketed data, Executed Hive queries on Parquet tables stored in Hive to perform data analysis to meet the business specification logic. Experience in using Avro, Parquet, RC File and JSON file formats. Configured, provisioned and updated resources on AWS using Terraform. Used various Project Management services like Rally for tracking issues, bugs related to code and GitHub for various code reviews and worked on various version control tools like GIT, SVN. Designed ETL workflows on Tableau, Deployed data from various sources to
- HDFS. Used JIRA for bug tracking, issue tracking and project management. Generated various kinds of knowledge reports using Qlik based on Business specification. Extensively researched for cause of errors in code and modified code according to requirements Good experience with use-case development, with Software methodologies like Agile and Scrum.
Environment: Spark, Sqoop, Oozie, Hive, Kafka, AWS- EC2, S3, RDS, EBS, ELB, VPC, Cloud Watch, Route 53, Dynamo DB, Redshift, Glue, SVN, Tortoise SVN, GITHUB, Terraform, Jenkins, Hadoop, Teradata, Postgresql, pyspark, python.
Confidential
Data Engineer
Responsibilities:
- Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark - SQL, PostgreSQL, Data Frame, pair RDD's. Collaborated with Business Analysts, SMEs across departments to gather business requirements, and identify workable items for further development. Partnered with ETL developers to ensure that data is cleaned, and the data warehouse is up to date for reporting purpose by Pig. Selected and generated data into csv files and stored them into AWS S3 by using AWS EC2 and then structured and stored in AWS Redshift. Involved in integration of
- Hadoop cluster with spark engine to perform BATCH and GRAPHX operations. Performed data preprocessing and feature engineering for further predictive analytics using Python Pandas. Generated report on predictive analytics using Python and Tableau including visualizing model performance and prediction results. Identified medical cost, service utilization, service intensity, unit cost, and trends to identify new opportunities for product expansion. Working knowledge of healthcare data and measurement standards such as ICD10, CPT/HCPCS, MS-DRG, DxCG, PMPM, and utilization per 1,000. Used PySpark and Pandas to calculate the moving average and RSI score of the stocks and generated them into data warehouse. Utilized Agile and Scrum methodology for team and project management. Used Git for version control with colleagues.
Environment: Spark, Hive, SQL, AWS - EC2, S3, VPC, Cloud Watch, Dynamo DB, Redshift, Glue, SVN, Tortoise SVN, GITHUB, Terraform, Jenkins, Udeploy, Hadoop, Tableau, Postgresql, Pyspark, python.
Confidential
Data & Analytics Engineer
Responsibilities:
- Worked on a Python ETL framework to bring data in and out of the Teradata database systems. Worked on Sqoop and Oozie to ingest data into Hadoop from various RDMS. Worked on GIT, Jenkins and Udeploy for code moveup, build and deployment. Automated DDL creation using Liquibase files and used YAML to trigger jobs. Worked with AWS services such as EC2, S3, CloudWatch, VPCs, Security Groups, Route 53, ALB, ELB Analyzed the Claims data, future trends and summarized results to generate meaningful recommendations for management. Created complex and reusable Macros and used existing macros for Data Cleansing, Validation analysis and Report generation. Sourced and analyzed data from a variety of sources like SAS data sets, MS Access, MS Excel, CSV, XML, and flat files etc. Analyzed data using various statistical procedures like PROC SUMMARY, PROC MEANS, PROC FREQ, PROC
- REG. Used SAS Stored Processes to deliver content to the portal and used SAS Web Services to deliver the same through the SAS Management Console. Extensively used SQL statements and PROC SQL for the analysis of large datasets (millions of records) and for data manipulation like Indexing and Appending various datasets. Extensively used procedures including TRANSPOSE, TABULATE, MERGE SORT, APPEND, etc. in Data transformation and Manipulation processes. Created Dashboard Reports in SAS Visual Analytics as per Clients Requirement. Coded, Modified and automated existing SAS programs to automate process, improve efficiency Extensively used SAS ODS tag sets to generate complex reports. Performed quality control (QC) checks on peer developed programs to ensure that the correct source data is used, correct code and logic are used. Creating SAS dataset from tables in
- Database using SAS/Access. Retrieved the data from flat files, Oracle database and converted to SAS data sets for Analysis using SAS/Stat procedures Coding SAS programs with the use of Base SAS and SAS/Macros for ad hoc Requirements. Used Enterprise Guide (EG) for creating, editing and submitting SAS code for stored processes to Application server, for running queries against Oracle to bring the data into the SAS environment. Extensively researched for cause of errors in code and modified code according to requirements
Environment: Sqoop, Oozie, Teradata, Python, SQL, SAS/BASE, SAS/ENTERPRISE GUIDE, SAS/STAT, SAS/SQL, SAS/GRAPH, SAS/ACCESS, DB2, SQL Server, Tortoise SVN, Tortoise GIT, Microsoft Office Applications, UNIX.
Confidential
Data Engineer
Responsibilities:
- Developed reports using SAS/BASE, SAS/CONNECT, SAS/MACROS, SAS/SQL, SAS/ODS, SAS/ACCESS and SAS/SQL. Designed, developed, tested and debugged new MACROS and modified the existing macros according to the enhancements in the business. Involved in detail Design, coding, configuration and testing of SAS programs/reports. Developed SAS Programs/MACROS to load data into DB2 tables. Extracted and transformed data from source system and created SAS DATA SETS. Developed reports using Visualization tools such as HP EXSTREAM. Exported SAS Results to Different
- Format, Such as XML, Excel Using SAS/Export, SAS/ODS for Reporting and Presentation. Developed Extracts and Data Sources for reports using PROC SQL, SAS/MACROS. Developed UNIX scripts to load, validate data. Involved in various Advance MACRO debugging options such as MPRINT, SYMBOLGEN, MLOGIC to Debug store process and SAS code. Use SAS, SQL, XML, SQL and Windows batch programming techniques to code the technical specifications apply business logic and produce automated reporting solutions. Involved in production release activities and also provided Runbooks, documentation. Worked with teams to examine and solve business problems by accurately analyzing the needs of clients and helping them achieve improved quality and value for their customers
Environment: QlikView 9.0/10.0/11.0/11.2, Qlik Sense, Tableau, IBM Cognos, SAS SAS/BASE, SAS/ENTERPRISE GUIDE, SAS/STAT, SAS/SQL, SAS/GRAPH, SAS/ACCESS, DB2, Toad, SQL Server, Tortoise SVN, Tortoise GIT, Microsoft Office Applications, Windows 2000, Windows NT, UNIX.
Confidential
Data Modeler
Responsibilities:
- Gathered and analyzed business requirements through continuous discussions with the business analysts, data analysts, and performed GAP analysis of the business rules, user administration and requirements. Involved in the study and understanding of existing system and data architecture and proposed changes necessary to in corporate the new data models. Reverse Engineered the existing database structure to understand the existing data models so that any changes in corporate would synchronize with current model. Created conceptual models and conducted several walkthrough meetings with Architects for approvals and also conducted team meetings for creating logical and physical models from the conceptual model. Created physical and logical models using Erwin to effectively translate conceptual model into logical and physical models conforming to the business and data requirements. Created a family of stars to effectively translate the physical model into a dimensional model using conformed dimensions, role playing dimensions mini and sub dimensions and fact less fact tables. Incorporated enhancements into DW in accordance to Ralph Kimball Design specifications and adhering to the business needs and industry standards.
- Development of Daily and Weekly inventory and sales reports using Micro strategy. Worked on integrating data from disparate sources and the was involved with data cleansing, validating and loading, client user and account information Implemented performance tuning strategies like sub query flattening, view merging and recreating indexes to speed up the performance of highly resource intensive queries. Performed logical and physical data modeling and database design for an Oracle Retail Data ODS designed to enable Retail LOB fraud detection data mining with the use of analytics tools.
Environment: Erwin, Oracle 10g, DB2, Toad, Windows XP, SQL Server 2000/ 2005, XML, Excel, Access.
