We provide IT Staff Augmentation Services!

Sr. Big Data Engineer Resume

New York, NY

SUMMARY:

  • Above 8+ years of experience as Big Data Engineer /Data Engineer and Data Analyst including designing, developing and implementation of data models for enterprise - level applications and systems.
  • Strong knowledge of Software Development Life Cycle (SDLC) and expertise in detailed design documentation.
  • Excellent working experience in Scrum / Agile framework and Waterfall project execution methodologies.
  • Expert in Amazon EMR, Spark, Kinesis, S3, Boto3, BeanStalk, ECS, Cloudwatch, Lambda, ELB, VPC, ElasticCache, DynamoDB, Redshit, RDS, Aethna, Zeppelin & Airflow.
  • Good experience in working with different ETL tool environments like SSIS, Informatica and reporting tool environments like SQL Server Reporting Services (SSRS).
  • Knowledge and working experience on big data tools like Hadoop, Azure Data Lake, AWS Redshift.
  • Experience in configuring and administering the Hadoop Cluster using major Hadoop Distributions like Apache Hadoop and Cloudera.
  • Expert level in Server side development using J2EE , EJB, J2SE, Spring, Servlets, J2SE, Python, C++ on Windows, Unix and Linux Platform.
  • Involve in writing SQL queries, PL/SQL programming and created new packages and procedures and modified and tuned existing procedure and queries using TOAD
  • Excellent technical and analytical skills with clear understanding of design goals of ER modeling for OLTP and dimension modeling for OLAP.
  • Extensive experience in using ER modeling tools such as Erwin and ER/Studio, Teradata, BTEQ and MDM.
  • Experience in cloud development architecture on Amazon AWS, EC2, Redshift and Basic on Azure.
  • Experience in Work on NoSQL databases - HBase, Cassandra & MongoDB, database performance tuning & data modeling.
  • Excellent Experience in job workflow scheduling and monitoring tools like Oozie and Zookeeper.
  • Experience in Working Dimensional Data modeling, Star Schema/Snow flake schema, Fact & Dimensions Tables.
  • Strong knowledge of Spark for handling large data processing in streaming process along with Scala.
  • Thorough Knowledge in creating DDL, DML and Transaction queries in SQL for Oracle and Teradata databases.
  • Experienced in writing Storm topology to accept the events from Kafka producer and emit into Cassandra DB.
  • Expertise in integration of various data sources like RDBMS, Spreadsheets, Text files, JSON and XML files.
  • Expertise in performing User Acceptance Testing (UAT) and conducting end user training sessions.
  • Experience in designing a component using UML Design-Use Case, Class, Sequence, and Development, Component diagrams for the requirements.
  • Good expertise knowledge with the UNIX commands like changing the permissions of the file to file and group permissions.
  • Experience in building reports using SQL Server Reporting Services and Crystal Reports.
  • Experience in Data transformation, Data mapping from source to target database schemas, Data Cleansing procedures.
  • Experience in writing stored procedures and complex SQL queries using relational databases like Oracle, SQL Server, and MySQL.
  • Experience in Text Analytics, developing different Statistical Machine Learning, Data Mining solutions to various business problems
  • Performing extensive data profiling and analysis for detecting and correcting inaccurate data from the databases and to track data quality.
  • Experience in importing and exporting Terabytes of data between HDFS and Relational Database Systems using Sqoop.

TECHNICAL SKILLS:

Data Modeling Tools: Erwin Data Modeler, Erwin Model Manager, ER Studio v17, and Power Designer 16.6.

Big Data / Hadoop Ecosystem: MapReduce, HBase 1.2, Hive 2.3, Pig 0.17, Flume 1.8, Sqoop 1.4, Kafka 1.0.1, Oozie 4.3, Hue, Cloudera Manager, Hadoop 3.0, Apache Nifi 1.6, Cassandra 3.11

Cloud Management: Amazon Web Services(AWS), Amazon Redshift

ETL/Data warehouse Tools: Informatica 9.6/9.1, SAP Business Objects XIR3.1/XIR2, Talend, Tableau, and Pentaho.

OLAP Tools: Tableau, SAP BO, SSAS, Business Objects, and Crystal Reports 9

Cloud Platform: AWS, Azure, Google Cloud, Cloud Stack/Open Stack

Programming Languages: SQL, PL/SQL, UNIX shell Scripting, PERL, AWK, ANSI SQL, SED

Databases: Oracle 12c/11g, Teradata R15/R14, MS SQL Server 2016/2014, DB2.

Testing and defect tracking Tools: HP/Mercury, Quality Center, Win Runner, MS Visio & Visual Source Safe

Operating System: Windows, Unix, Sun Solaris

Methodologies: RAD, JAD, RUP, UML, System Development Life Cycle (SDLC), Agile, Waterfall Model.

PROFESSIONAL EXPERIENCE:

Sr. Big Data Engineer

Confidential - New York, NY

Responsibilities:

  • Designed and developed software applications, testing, and building automation tools.
  • Involved in start to end process of Hadoop jobs that used various technologies such as Sqoop, PIG, Hive, MapReduce, Spark and Shells scripts (for scheduling of few jobs).
  • Involved in importing and exporting data between RDBMS and HDFS using Sqoop.
  • Performed querying of both managed and external tables created by Hive using Impala.
  • Designed/developed tables, views, various SQL queries, stored procedures, functions.
  • Involved in PL/SQL code review and modification for the development of new requirements.
  • Extracted data from existing data source and performed ad-hoc queries.
  • Utilized SAS and SQL extensively for collecting, validating and analyzing the raw data received from the client.
  • Executed data extraction programs/data profiling and analyzing data for accuracy and quality.
  • Analyzed the data using advanced excel functions like Pivot tables, VLOOK up, visualizations to get the descriptive analysis of the data.
  • Created Schema objects like Indexes, Views, and Sequences, triggers, grants, roles, Snapshots.
  • Used advanced Microsoft Excel to create pivot tables, and other excel functions to prepare reports and dashboard with user data.
  • Maintained numerous monthly scripts, executed on monthly basis, produces reports and submitted on time for business review.
  • Developed ad-hoc reports using Crystal reports for performance analysis by business users.
  • Implemented the Big Data solution using Hadoop, and hive to pull/load the data into the HDFS system.
  • Installed and configured Hadoop and responsible for maintaining cluster and managing and reviewing Hadoop log files.
  • Implemented and configured workflows using Oozie to automate jobs.
  • Migrated the needed data from MySQL in to HDFS using Sqoop and importing various formats of flat files into HDFS.
  • As a Sr. Big Data Engineer, provided technical expertise and aptitude to Hadoop technologies as they related to the development of analytics.
  • Expertise in writing Hadoop Jobs to analyze data using MapReduce, Hive, and Pig.
  • Experience in designing, building and implementing complete Hadoop ecosystem comprising of MapReduce, HDFS, Hive, Pig, HBase, MongoDB, and Spark.
  • Worked experience in Scrum / Agile framework and Waterfall project execution methodologies.
  • Managed data from various file system to HDFS using UNIX command line utilities.
  • Worked in Azure environment for development and deployment of Custom Hadoop Applications.
  • Managed and support of enterprise Data Warehouse operation, big data advanced predictive application development using Cloudera &.
  • Extensively worked on Shell scripts for running SAS programs in batch mode on UNIX.
  • Created partitioned tables in Hive, also designed a data warehouse using Hive external tables and also created hive queries for analysis.
  • Installed, Configured and Maintained Hadoop clusters for application development and Hadoop tools like Hive, Pig, Hbase, Zookeeper and Sqoop.
  • Worked on creating Hive tables and written Hive queries for data analysis to meet business requirements and experienced in Sqoop to import and export the data from Oracle & MySQL.
  • Developed pig scripts to transform the data into structured format and it are automated through Oozie coordinators.

Environment: Hadoop 3.0, MapReduce, Hive 2.3, Pig 0.17, HDFS, HBase, MongoDB, Agile, Azure, MySQL, Oozie, MySQL, Sqoop1.4, HBase1.2.

Confidential, Merrimack, NH

Sr. Big Data Engineer

Responsibilities:

  • Worked as a Big Data Engineer to Import and export data from different databases.
  • Prepared ETL technical Mapping Documents along with test cases for each Mapping for future developments to maintain Software Development Life Cycle (SDLC).
  • Wrote script for Location Analytic project deployment on a Linux cluster/farm & AWS Cloud deployment using Python.
  • Wrote Scala/Spark/AWS EMR cloud application to process & transform billions of Rest & mobile events generated on Realtor.com every hour.
  • Created Logical & Physical Data Modeling on Relational (OLTP), Dimensional Data Modeling (OLAP) on Star schema for Fact & Dimension tables using Erwin.
  • Involved in importing the real time data to Hadoop using Kafka and implemented the Oozie job for daily.
  • Involved in various phases of development analyzed and developed the system going through Agile Scrum methodology.
  • Used Python, Ruby, Pig, Hive, Sqoop to implement various tools & utilities for data import & export.
  • Utilize U-SQL for data analytics/ data ingestion of raw data in Azure and Blob storage
  • Performed thorough data analysis for the purpose of overhauling the database using ANSI-SQL.
  • Translated business concepts into XML vocabularies by designing XML Schemas with UML.
  • Worked with medical claim data in the Oracle database for Inpatient/Outpatient data validation, trend and comparative analysis.
  • Designed the data marts using the Ralph Kimball's Dimensional Data Mart modeling methodology using Erwin.
  • Implemented reporting in PySpark, Zeppelin & querying through Airpal & AWS Aethna.
  • Implemented Data Validation using MapReduce programs to remove un-necessary records before move data into Hive tables.
  • Worked on analyzing and examining customer behavioral data using MongoDB.
  • Involved in PL/SQL query optimization to reduce the overall run time of stored procedures.
  • Performed data analysis and data profiling using complex SQL on various sources systems including Oracle and Netezza
  • Worked with MDM systems team with respect to technical aspects and generating reports.
  • Worked on with importing and exporting data from different Relational Database Systems like DB2 into HDFS and Hive and vice-versa, using Sqoop
  • Supported various reporting teams and experience with data visualization tool Tableau.
  • Extensively used Pig for data cleansing using Pig scripts and Embedded Pig scripts.
  • Designed the data aggregations on Hive for ETL processing on Amazon EMR to process data as per business requirement
  • Developed and designed data integration and migration solutions in Azure.
  • Developed Sqoop scripts to extract the data from MYSQL and load into HDFS.
  • Prepared complex ANSI-SQL queries, views and stored procedures to load data into staging area.
  • Involved in Hive-HBase integration by creating hive external tables and specifying storage as HBase format.
  • Analyzed the data which is using the maximum number of resources and made changes in the back-end code using PL/SQL stored procedures and triggers
  • Deployed SSRS reports to Report Manager and created linked reports, snapshots, and subscriptions for the reports and worked on scheduling of the reports.
  • Integrated multiple sources data (SQL Server, DB2, TD) into Hadoop cluster and analyzed data by Hive-HBase integration.
  • Imported data from different relational data sources like Oracle, Teradata to HDFS using Sqoop.

Environment: Hadoop 3.0, Kafka 1.1, SQL, PL/SQL, OLAP, ANSI-SQL, Java, Python, OLTP, SDLC, MDM, Netezza, Oracle 12c, XML, MapReduce, SSRS, MYSQL, T-SQL, Teradata 15, Azure, AWS, ETL, HDFS, Hive 2.3, Sqoop 1.4, Tableau, Pig 0.17.

Confidential - West Point, PA

Sr. Data Engineer

Responsibilities:

  • Worked as a Data Engineer designed and Modified Database tables and used HBase Queries to insert and fetch data from tables.
  • Participated in design discussions and assured functional specifications are delivered in all phases of SDLC in an Agile Environment.
  • Designed & Developed tool like Ambari & Chef for MPact Software, Hadoop, Hbase & SparkDeployment, Configuration, Monitoring, HA, Load & Data Balancing, Scalability on AWS, ESXI, XEN & distributed cluster using Java 8, Spring, Scala, Python & Ruby.
  • Created data models for AWS Redshift and Hive from dimensional data models.
  • Executed change management processes surrounding new releases of SAS functionality
  • Prepared complex T-SQL queries, views and stored procedures to load data into staging area.
  • Performed Normalization of the existing (3rd NF), to speed up the DML statements execution time.
  • Participated in data collection, data cleaning, data mining, developing models and visualizations.
  • Worked with Sqoop to transfer data between the HDFS to relational database like MySQL and vice versa and experience in using of Talend for this purpose.
  • Developed Server side and client-side web applications using Spring 2.5, Struts 2, EJB, Hibernate, IBatis, JSF, JSTL, ExtJS and Web 2.0 Ajax frameworks. Developed small intranet sites using Python.
  • Developed automatic job flows and ran through Oozie daily and when needed which runs MapReduce jobs internally.
  • Used SSIS and T-SQL stored procedures to transfer data from OLTP databases to staging area and finally transfer into data-mart.
  • Extracted Tables and exported data from Teradata through Sqoop and placed in Cassandra.
  • Worked on analyzing and examining customer behavioral data using MongoDB.
  • Enforced referential integrity in the OLTP data model for consistent relationship between tables and efficient database design.
  • Worked on Amazon AWS concepts like EMR and EC2 web services for fast and efficient processing of Big Data.
  • Generated parameterized queries for generating tabular reports using global variables, expressions, functions, and stored procedures using SSRS.
  • Created Hive External tables to stage data and then move the data from Staging to main tables
  • Created jobs and transformation in Pentaho Data Integration to generate reports and transfer data from HBase to RDBMS.
  • Wrote DDL and DML statements for creating, altering tables and converting characters into numeric values.
  • Worked on Master data Management (MDM) Hub and interacted with multiple stakeholders.
  • Worked on Kafka and Storm to ingest the real time data streams, to push the data to appropriate HDFS or HBase.
  • Extensively involved in development and implementation of SSIS and SSAS applications.
  • Collaborated with ETL, and DBA teams to analyze and provide solutions to data issues and other challenges while implementing the OLAP model.
  • Implemented Star Schema methodologies in modeling and designing the logical data model into Dimensional Models.
  • Designed and Developed PL/SQL procedures, functions and packages to create Summary tables.
  • Worked on Performance Tuning of the database which includes indexes, optimizing SQL Statements.
  • Worked with OLTP to find the daily transactions and type of transactions occurred and the amount of resource used
  • Developed a Conceptual Model and Logical Model using Erwin based on requirements analysis.
  • Created reports analyzing large-scale database utilizing Microsoft Excel Analytics within legacy system.
  • Generated ad-hoc SQL queries using joins, database connections and transformation rules to fetch data from legacy oracle and SQL server database systems.

Environment: Hadoop 3.0, HDFS, HBase 1.2, SSIS, SSAS, OLAP, OLTP, ETL, Java, ANSI-SQL, AWS, SDLC, T-SQL, SAS, MySQL, HDFS, Sqoop 1.4, Cassandra 3.0, MongoDB, Hive 2.3, SQL, PL/SQL, Teradata 15, Oracle 12c, MDM.

Confidential - Lowell, AR

Sr. Data Analyst/Data Engineer

Roles & Responsibilities

  • Worked as a Sr. Data Analyst/Data Engineer to review business requirement and compose source to target data mapping documents.
  • Used Agile Central (Rally) to enter tasks which has the visibility to all the team and Scrum Master.
  • Connected to AWS Redshift through Tableau to extract live data for real time analysis.
  • Designed and developed architecture for data services ecosystem spanning Relational, NoSQL, and Big Data technologies.
  • Performed the Data Mapping, Data design (Data Modeling) to integrate the data across the multiple databases in to EDW.
  • Implemented Forward engineering to create tables, views and SQL scripts and mapping documents
  • Captured the data logs from web server into HDFS using Flume for analysis.
  • Involved in PL/SQL code review and modification for the development of new requirements.
  • Developed Data mapping, Transformation and Cleansing rules for the Data Management involving OLTP and OLAP.
  • Worked closely with the ETL SSIS Developers to explain the complex Data Transformation using Logic.
  • Worked with MDM systems team with respect to technical aspects and generating reports.
  • Worked on Data Mining and data validation to ensure the accuracy of the data between the warehouse and source systems.
  • Created logical and physical data models using Erwin and reviewed these models with business team and data architecture team.
  • Used SAS procedures like means, frequency and other statistical calculations for Data validation.
  • Developed and presented Business Intelligence reports and product demos to the team using SSRS (SQL Server Reporting Services).
  • Implemented Installation and configuration of multi-node cluster on Cloud using Amazon Web Services (AWS).
  • Performed data analysis and data profiling using complex SQL on various sources systems including Oracle.
  • Worked in importing and cleansing of data from various sources like Teradata, flat files, SQL Server with high volume data.
  • Handled importing of data from various data sources, performed transformations using Map Reduce, loaded data into HDFS and Extracted the data from My SQL into HDFS using Sqoop
  • Integrated various sources in to the Staging area in Data warehouse to Integrating and Cleansing data.
  • Cleansed, extracted and analyzed business data on daily basis and prepared ad-hoc analytical reports using Excel and T-SQL

Environment: Erwin 9.6, Teradata r14, Oracle 11g, SQL, T-SQL, PL/SQL, AWS, Agile, OLAP, OLTP, SSIS, HDFS, SAS, Flume, SSRS, Sqoop, Map Reduce, My SQL, HDFS.

Confidential - SFO, CA

Sr. Data Analyst/Data Modeler

Responsibilities:

  • Worked as a Sr. Data Analyst / Data Modeler I was responsible for all data related aspects of a project.
  • Participated in requirement gathering session, JAD sessions with users, Subject Matter experts, and BAs.
  • Reverse Engineered DB2 databases and then forward engineered them to Teradata using E/R Studio.
  • Worked with Business Analysts team in requirements gathering and in preparing functional specifications and translating them to technical specifications.
  • Improved performance on SQL queries used indexes for tuning created DDL scripts for database. Created PL/SQL Procedures and Triggers.
  • Extensively used of data transformation tools such as SSIS, Informatica or Data Stage.
  • Worked with Business users during requirements gathering and prepared Conceptual, Logical and Physical Data Models.
  • Performed data mining on data using very complex SQL queries and discovered pattern.
  • Designed both 3NF data models for OLTP systems and dimensional data models using star and snowflake Schemas
  • Worked on Normalization and De-Normalization techniques for OLAP systems.
  • Created Tableau scorecards, dashboards using stack bars, bar graphs, geographical maps and Gantt charts.
  • Analyzed the business requirements by dividing them into subject areas and understood the data flow within the organization.
  • Optimized and updated UML Models (Visio) and Relational Data Models for various applications.
  • Used E/R Studio for reverse engineering to connect to existing database and ODS to create graphical representation in the form of Entity Relationships and elicit more information
  • Created mappings using pushdown optimization to achieve good performance in loading data into Netezza.
  • Wrote complex SQL queries for validating the data against different kinds of reports generated by Business Objects.
  • Wrote T-SQL statements for retrieval of data and Involved in performance tuning of T-SQL queries and Stored Procedures.
  • Created and reviewed the conceptual model for the EDW (Enterprise Data Warehouse) with business user.
  • Generated periodic reports based on the statistical analysis of the data using SQL Server Reporting Services (SSRS).
  • Created a list of domains in E/R Studio and worked on building up the data dictionary for the company
  • Created a Data Mapping document after each assignment and wrote the transformation rules for each field as applicable
  • Performed data management projects and fulfilling ad-hoc requests according to user specifications by utilizing data management software programs.

Environment: E/R Studio V15, Teradata, SQL, PL/SQL, T-SQL, OLTP, SSIS, SSRS, OLAP, Tableau, OLTP, Netezza.

Hire Now