We provide IT Staff Augmentation Services!

Big Data Engineer Resume

5.00/5 (Submit Your Rating)

Dallas, TX

SUMMARY

  • Around 8 years of experience in IT industry with specialization Analysis, Design, Development, providing ETL solutions and Implementation as a Data Engineer
  • Experience in building big data applications and products using open - source frameworks like Hadoop, HIVE, Apache Spark, MapReduce, and Python.
  • Expertise in loading the data from the different Data sources like (Teradata and DB2) into HDFS using Sqoop and load into partitioned Hive tables.
  • Experienced on Hadoop cluster maintenance including data and metadata backups, file system checks, commissioning and decommissioning nodes and upgrades.
  • Strong experience in analyzing large amounts of data sets writing Pig scripts and Hive queries and extensive experienced in working with structured data using Hive QL, join operations, writing custom UDF’s and experienced in optimizing Hive Queries.
  • Experience using Sqoop to import data into HDFS from RDBMS.
  • Experience in working with Cloudera, Hortonworks, and Microsoft Azure HDINSIGHT Distributions.
  • Experience on Migrating SQL database to Azure data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and Controlling and granting database access and Migrating On premise databases to Azure Data lake store using Azure Data factory
  • Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
  • Played a lead role in the development of Confidential Data Lake and in building Confidential Data Cube on Microsoft Azure HDINSIGHT cluster.
  • Good knowledge of Hadoop Architecture and various components such as YARN, HDFS, NodeManager, ResourceManager, JobTracker, TaskTracker, NameNode, DataNode and MapReduce concepts.
  • Installing, configuring and managing of Hadoop Clusters and Data Science tools
  • Strong knowledge in NOSQL column-oriented databases like HBase, Cassandra, MongoDB and its integration with Hadoop cluster.
  • Expertise in developing Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
  • Experience in build scripts using Maven and do continuous integrations systems like Jenkins.
  • Hands on experience in installation, configuration, supporting and managing Cloudera, Hortonworks Hadoop platform along with CDH3 and CDH4 clusters.
  • Very good understanding in ecosystems with Cloudera CDH1, CHD2, CDH3, CDH4, CDH5, Hortonworks HDP2.1 and HadoopMR1 & MR2 Architectures.
  • Expertise in using Kafka as a messaging system to implement real-time Streaming solutions and implemented Sqoop for large data transfers from RDMS to HDFS/HBase/Hive and vice-versa.
  • Involve in the integration of the applications with different tools like Team city, GitLab, Bit bucket and JIRA for issues and stories tracking.
  • Experienced in using various SSIS control flow items like execute SQL task, Data Flow task, Send Mail Task, File System task, Bulk Insert Task, for-each Loop and Sequence Container.
  • Experienced in debugging SSIS packages using checkpoints, breakpoints and error handling by configuring send mail task using event handlers.
  • Proficient in logging the runtime information into different log providers like SQL Server, Text File using SSIS Logging to increase the performance of SSIS packages.
  • Good hands-on experience in implementing dashboards, Visualization and analytics using Tableau Desktop and server for the performance data metrics and Visualization in the BI environment.
  • Hands on experience in extracting data from heterogeneous sources, perform data cleansing and load them to the data model region.
  • Experience working with statistical and regression analysis, multi - objective optimization.
  • Worked with packages like ggplot2 and shiny in R to understand data and developing applications.
  • Ability to work in tight schedules, efficient in time management and a highly motivated team player with the intensity of delivering high quality outputs within the deadline.
  • Excellent communication, analytical & troubleshooting skills.
  • Possess good hands-on experience in processing tools/scripts like Spark, Pig, Hive and Shell.
  • Built mapping doc from scratch for the new/enhance requirements.
  • Handled complete responsibility on deploying/migrating the code to the production environments.
  • Supported QA team for all the end-to-end activities like coordinating with clients in getting the data for QA, preparing all the mockup data and clarifying all the Q’s raised in complete QA cycle.
  • Good hands-on experience on Phoenix DB which is integrated on top of HBase (No SQL DB).
  • Good ability to analyze and identify the root cause on the top priority Production issues and applying the fix.

TECHNICAL SKILLS

Programming Languages: C, C++, SQL, PL/SQL, UML, Python, Scala, R, Java

Hadoop Eco System: HDFS, MapReduce, Yarn, Pig, Hive, HBase, Sqoop, Flume, Oozie and TWS

Frameworks: Apache Spark, Spring, Hadoop, Databricks, Hive, Sqoop, Ambari, HBase Spring MVC, Hibernate, Struts, Junit, log4J

Design Patterns: Singleton, MVC, DAO, VO, Service locator

Java/J2EE Technologies: Servlets, JSP, JSF, Apache, EJB 2.0/3.0, JDBC, RMI, JMS, JNDI.

Tools: Maven, log4j, SVN, CVS, GIT, DB Visualizer, Team city, TWS (Tivoli Workload Scheduler), Putty, RStudio, WinSCP, Hortonworks, SQL Server Integration, Analysis Services and Reporting Services (SSIS/SSAS/SSRS)

IDE’s: Eclipse, Net Beans, My Eclipse, IntelliJ, Visual Studio

Web/Application Servers: Apache Tomcat, IBM WebSphere, WebLogic

Database Technologies: Oracle, MySQL, DB2, HBase, Mongo DB, MS Access, IBM Big SQL

Methodologies: Agile Scrum, Waterfall model

Operating Systems: Windows, Linux/Unix.

PROFESSIONAL EXPERIENCE

Confidential, Dallas TX

Big Data Engineer

Responsibilities:

  • Built mapping for new requirements and developing the design for enhancements of multiple projects.
  • Simultaneously worked on multiple projects handling the responsibilities like, developing the code, design discussions, production issues and Dashboard activities.
  • Enhanced the production tables aligning with the enhance requirements.
  • Coordinated with multiple teams for all the clarifications on the enhancements.
  • Analyzed the end-to-end implementation process to add-on possible solutions to the project.
  • Design and developed the code without any issues and deployed to all environments without any issues.
  • Assisted in upgrading, configuration and maintenance of various Hadoop infrastructures like Pig, Hive, and HBase.
  • Developed and scheduled the job to process the code developed.
  • Experience in developing MapReduce jobs using various Input and output formats.
  • Developed workflow in schedule to automate the tasks of loading the data into Hive and HBase.
  • Developed Spark scripts using Scala and shell commands as per the requirement.
  • Involved in loading data into HBase NoSQL Database. Transform and Load data from Sources Systems to Hadoop Data Lake.
  • Developed Spark code to move data into HBase tables from various tables to Hive/HBase.
  • Enhanced UDF using both Data frames and RDD in Spark for Data Aggregation, queries and writing data into HBase.
  • Used Pig and Hive in the analysis of data.
  • Load the data into Spark RDD and performed in-memory data computation to generate the output response.
  • Importing data into HDFS and Hive.
  • Monitoring Production jobs and its performance using Scheduler.
  • Written complex SQL queries.
  • Implemented ETL code to load data from multiple Hive table into HBase using Pig Scripts.
  • Implemented Pig as ETL tool to do Transformations, event joins and some pre-aggregations before storing the data onto HBase.
  • Exploring with Spark improving the performance and optimization of the Existing algorithms in Hadoop using Spark context, data frame pair RDD’s, Spark YARN.
  • Developed Spark code for faster testing and processing of data.

Environment: Hadoop, HDFS, Cloudera, Python, Spark, YARN, Map Reduce, Hive, Teradata SQL, Pig, Data Lake, HBase, Java, Scala, Python, UNIX Shell Scripting.

Confidential, Charlotte NC

Data/Big Data Engineer

Responsibilities:

  • Worked on a live 65 nodes Hadoop cluster running CDH4.7.
  • Worked with highly unstructured and semi structured data of 70 TB in size (210 TB with replication factor of 3).
  • Experience in AWS cloud environment on S3 storage and EC2 instances.
  • Hands on experience in using various Hadoop distros (Cloudera (CDH 4/CDH 5), Hortonworks, Map-R, IBM Big Insights, Apache and Amazon EMR Hadoop distributions.
  • Migrated an existing on-premises application to AWS. Used AWS services like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
  • Assisted in upgrading, configuration and maintenance of various Hadoop infrastructures like Pig, Hive, and HBase.
  • Configured Flume to capture the news from various sources for testing the classifier.
  • Experience in developing MapReduce jobs using various Input and output formats.
  • Assisted in upgrading, configuration and maintenance of various Hadoop infrastructures like Pig, Hive, and HBase.
  • Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing, analyzing and training the classifier using MapReduce jobs, Pig jobs and Hive jobs.
  • Developed Spark scripts by using Scala shell commands as per the requirement.
  • Involved in loading data into Cassandra NoSQL Database. Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
  • Developed Spark applications to move data into Cassandra tables from various sources like Relational Database or Hive.
  • Worked on Spark streaming collects the data from Kafka in near real time and performs necessary transformations and aggregations on the fly to build the common learner data model and persists the data in Cassandra.
  • Worked on Cassandra Data modelling, NoSQL Architecture, DSE Cassandra Database administration, Key space creation, Table creation, Secondary and Solr index creation, User creation & amp; access administration.
  • Experience in performance tuning a Cassandra cluster to optimize writes and reads.
  • Developed Python scripts, UDF using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop.
  • Used Pig and Hive in the analysis of data.
  • Load the data into Spark RDD and performed in-memory data computation to generate the output response.
  • Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
  • Performed Sqooping for various file transfers through the HBase tables for processing of data to several NoSQL DBs- Cassandra, MongoDB.
  • Implemented Spark Storm builder topologies to perform cleansing operations before moving data into Cassandra.
  • Developed ETL workflow which pushes webserver logs to an Amazon S3 bucket.
  • Implemented Cassandra connection with the Resilient Distributed Datasets (local and cloud).
  • Importing and exporting data into HDFS and Hive.
  • Created SSIS/DTS packages to copy tables, schemas, and views and to extract data from Excel and Oracle.
  • Migrated data from legacy system (text-based files, Excel spreadsheets, and Access) to SQL Server databases using SQL Server Integration Services (SSIS) and stored procedures to overcome the transformation constraints.
  • Monitoring SQL server performance using SQL Profiler to find performance and dead locks.
  • Written complex stored procedures and SQL queries.
  • Involved in building and maintaining SSIS Packages to import and export the data from various data sources using BIDS based on the design data models.
  • Implemented ETL code to load data from multiple sources into HDFS using Pig Scripts.
  • Implemented Pig as ETL tool to do Transformations, event joins and some pre-aggregations before storing the data onto HDFS.
  • Worked on Talend ETL scripts to pull the data from TSV Files/Oracle Data Base into HDFS.
  • Worked extensively on design, development and deployment of talend jobs to extract data, filter the data and load them into datalake.
  • Extract data from source system and transform into newer systems using Talend DI Components.
  • Worked on Storm to handle the parallelization, partitioning, and retrying on failures and developed a data pipeline using Kafka and Strom to store data into HDFS.
  • Exploring with Spark improving the performance and optimization of the Existing algorithms in
  • Hadoop using Spark context, Spark-SQl, data frame pair RDD’s, Spark YARN.
  • Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.
  • Supported Map Reduce Programs those are running on the cluster.

Environment: Hadoop, HDFS, Cloudera, Python, AWS, Spark, YARN, Map Reduce, Hive, Teradata SQL, PL/SQL, Pig, Talend, Data Lake, Data Integration 6.1/5.5.1 (ETL), Enterprise Edition, SSIS, SSAS, SSRS, T-SQL, DTS, SQL Profiler, Kafka, Sqoop, Oozie, HBase, Cassandra, Java, Scala, Python, UNIX Shell Scripting.

Confidential, Dallas TX

Data Engineer/ Big Data Developer

Responsibilities:

  • Involve in all phases of SDLC (Software Development Life Cycle) which includes requirement collection, design and analysis, development and deployment of the application.
  • Architecture and design of business requirements and to make Visio Diagrams for the design and to develop the application and deploy the application in various environments.
  • Develop Spark 2.1/2.4 Scala component to process the business logic and store the computation results of 10 TB data into HBase database to access the downstream web apps using Big SQL db2 database.
  • Uploaded and processed more than 10 terabytes of data from various structured and unstructured sources into HDFS using Sqoop and Flume.
  • Test the developed modules in the application using Junit Library and Junit testing Framework
  • Analyze structured, unstructured data, and file system data and loading the data to HBase tables based on the project requirement using IBM Big SQL with Sqoop mechanism and processing the data using Spark SQL in-memory computation &processing results to Hive, HBase
  • Handle importing other enterprise data from different data sources into HDFS using JDBC and Load Hadoop in Big SQL and perform transformations using Spark API to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from upstream in near real time and persists into HBase.
  • Working with different file structures with different Hive file formats like Text file, Sequence file, ORC file, Parquet and Avro to analyze the data to build data model and reading them from HDFS and processing through parquet files and loading into HBASE tables.
  • Develop the Batch jobs using Scala programming Language to process the data from files and tables, transform the data with the business logic and deliver it to the user
  • Work in Continuous Deployment module, which is used to create new tables or to update the existing table structure if needed in different environments along with DDL (Data Definition Language) creation for the tables
  • Wrote AZURE POWERSHELL scripts to copy or move data from local file system to HDFS Blob storage.
  • Created Pipelines in ADF (Azure Data Factory) using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
  • Developed Json Scripts for deploying the Pipeline in Azure Data Factory (ADF) that processes the data.
  • Loading data from Linux/Unix file system to HDFS and working with PUTTY for the better communication between Unix and Window system and for accessing the data files in the Hadoop environment;
  • Developed and Implement HBase capabilities for Big de-normalized data set and then apply transformation on the de-normalized data set using Spark/Scala.
  • Involved Spark tuning to improve the Jobs performance based on the Pepper Data monitoring tool metrics.
  • Worked in building application platforms in the Cloud by leveraging Azure Databricks
  • Develop shell scripts for configuration checks and files transformation which is to be done before loading the data into Hadoop Landing area HDFS
  • On demand secure EMR launcher with custom spark submit steps using S3 Event, SNS, KMS and Lambda function.
  • Hands on experience on AWS cloud services (VPC, EC2, S3, RDS, Redshift, Data Pipeline, EMR, DynamoDB, WorkSpaces, Lambda, Kinesis, RDS, SNS, SQS)
  • Created dimension model for reporting system by identifying required dimensions and facts using ERWIN.
  • Designed and developed data load jobs using SSIS package and scheduled in SQL Agent.
  • Created Grouping, sorting columns and writing expressions per the business requirements.
  • Created Cross-Tab, Drill-down, Drill-through and Sub drill down reports -Report using RDL. Promoted RDLs to Reporting Service Server.
  • Deploying and scheduling Reports using SSRS to generate all daily, weekly, monthly and quarterly Reports including status.
  • Extensively worked with SSIS tool suite, designed and created mappings using various SSIS transformations like OLEDB Command, Conditional Split, Lookup, Aggregator, Multicast and Lookup.
  • Wrote stored procedures and User Define Scalar Functions (UDFs) to be used in the SSIS packages and SQL scripts.
  • Developed and Implement Spark ETL custom component to extract the data from upstream systems and push the data to HDFS and finally store the data in HBase with wide row format
  • Work with Apache Hadoop environment by Hortonworks
  • Enhance the application with new features and make the performance improvement in all the modules of the application
  • Exposure to Microsoft Azure in the processing of moving the on-prem data to azure cloud.
  • Exploring and applying spark techniques like partitioning the data with Keys and writing it to parquet files which improve performance improvement.
  • Understand the Mapping documents, existing Source Data and preparing load strategies for different source systems and implement them using Hadoop technology
  • Work with Continuous integration tools like maven, Team city, IntelliJ and scheduling the jobs with TWS (Tivoli Workload Scheduler) tool,
  • Creating and cloning the jobs and Job streams in TWS tool and promoting them to higher environments.
  • Coordinating with co-developers, agile development and project management team and external systems and responsible for demos, presentation of developed modules to the project management team.
  • Perform Code review activities with peer developers in the team and architect of the team for delivering exception/error free application and testing it with real time test scenario’s and deploying it to the next level environments. Analyze and fix the data processing issues and troubleshooting the process.

Environment: Scala, Java, Spark framework, Linux, Jira, Bitbucket, IBM Big SQL, Hive, HBase, IntelliJ IDEA, Maven, Db2 Visualizer, ETL, TeamCity, Enterprise Edition, SSIS, SSAS, SSRS, T-SQL, DTS, SQL Profiler, WinSCP, PuTTY, IBM TWS (Tivoli Workload Scheduler), Windows, Azure Data Factory, Linux

Confidential, Costa Mesa, CA

Hadoop Developer

Responsibilities:

  • Handled importing of data from various data sources, performed transformations using Hive, Map reduce, loaded data into HDFS.
  • Extracted the data from MySQL into HDFS using Sqoop.
  • Exported the analyzed data to the Relational databases using Sqoop for visualization and to generate reports for the BI team.
  • Developed Simple to complex Map Reduce jobs.
  • Analyzed the data by performing Hive queries and running Pig Scripts to know user behavior.
  • Created partitioned tables in Hive.
  • Administered and supported distribution of Horton works.
  • Wrote Korn shell, Bash shell, Pearl scripts to automate most DB maintenance tasks.
  • Worked on Installed and configured Hadoop Map Reduce, HDFS, developed multiple Map Reduce jobs in Java for data cleaning and preprocessing.
  • Importing and exporting data into HDFS and HIVE using SQOOP.
  • Responsible to manage data coming from different sources.
  • Monitoring the running Map Reduce programs on the cluster.
  • Responsible for loading data from UNIX file systems to HDFS.
  • Installed and configured Hive and Created Hive UDFs.
  • Involved in creating Hive Tables, loading with data and Writing Hive queries which will invoke and run Map Reduce jobs in the backend.
  • Implemented the workflows using Apache Oozie framework to automate tasks.
  • Developed scripts and automated data management from end to end and sync up between the clusters.

Environment: Apache Hadoop, Java, Bash, ETL, Map Reduce, Hive, Pig, Horton works, Deployment tools, Data tax, Flat files, Oracle 11g/10g, MySQL, Window NT, UNIX, Sqoop, Oozie.

Confidential, Dallas TX

Big data Developer

Responsibilities:

  • Designing the applications from the ingestion to reports delivery to third party vendors using big data technologies flume, kafka, sqoop, map-reduce, hive, pig
  • Process the raw log files from the set top boxes using java map reduce code and shell scripts and stored them as text files in HDFS.
  • Ingesting the data from legacy and upstream systems to HDFS using apache Sqoop, Flume java map reduce programs, hive queries and pig scripts.
  • Generating the required reports using Oozie workflow and Hive queries for operations team from the ingested data.
  • Alert and monitoring mechanism for the oozie jobs for the failure conditions and successful conditions using email notifications involved in bluecoat proxy approach while sharing the data from the in-house Hadoop cluster to external vendors
  • Working with application teams to install operating system, Hadoop updates, patches, version upgrades as required.
  • Writing Map Reduce code to make un-structured and semi- structured data into structured data and loading into Hive tables.
  • Worked on debugging, performance tuning of Hive & amp; Pig Jobs
  • Analyzing system failures, identifying root causes, and recommended course of actions as part of operations support.
  • Involved in Spark streaming solution for the time sensitive revenue generating reports to match the pace with upstream (STB) systems data
  • Experience in working on Hbase with Apache phoenix as a data layer to serve the web requests to meet the SLA requirements
  • Created HBASE tables and load the data using spark-streaming application
  • Worked on SFDC ODATA connector to get the data from NodeJS services which in turns the fetch the data from HBASE
  • Utilized AWS S3 services to push/store and pull the data from AWS from external applications
  • Responsible for functional requirements gathering, code reviews, deployment scripts and procedures, offshore coordination and on-time deliverables

Environment: Hadoop, HDFS, Pig, Hive, Flume, Kafka, MapReduce, Sqoop, Spark, Oozie, LINUX, NodeJS, SFDC, ODATA and AWS

Confidential, OH

Hadoop/Bigdata Developer

Responsibilities:

  • Prepared design blueprints and application flow documentation
  • Gathering requirements from the Business patterns
  • Maintained the data in Data Lake (ETL) which is coming from Teradata Database.
  • Writing on an average of 80 GB on daily basis. Overall the data warehouse was having 5 PB of data and used 135 node cluster to process the data.
  • Responsible in creating Hive Tables to load the data which comes from MySQL by using Sqoop.
  • Writing java snippets in order to perform cleaning, pre-processing and data validation.
  • Experienced in creating Hive schema, external tables and managing views.
  • Worked on performing Join operations in Spark using hive.
  • Writing HQL statements as per the user requirements.
  • Involved in converting Hive/SQL queries into Spark transformations using Spark Data frames and working with Spark-Shell.
  • Developed Spark code using Java and Spark-SQL for faster testing and data processing.
  • Imported millions of structured data from relational databases using Sqoop import to process using Spark and stored the data into HDFS in parquet format.
  • Used Spark SQL to process the huge amount of structured data.
  • Implemented Spark Data Frames transformations, actions to migrate Map reduce algorithms.
  • Exploring with the Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
  • Used Data Frame API solutions to pre-process large sets of structured, with different file formats like Text file, CSV, Sequence files, Xml and JSON files and Parquet files and converting the distributed collection of data organized into named columns.
  • Experienced with batch processing of data sources using Apache Spark.
  • Developing predictive analytic using Apache Spark Java APIs.
  • Expert in implementing advanced procedures like text analytics and processing using the in-memory computing capabilities like Apache Spark written in Java.
  • Worked on the core and Spark SQL modules of Spark extensively.
  • Extensively used Broadcast Variables and Accumulators for better performances.

Environment: Hadoop, HDFS, Hive, Java 1.7, Spark 1.6, SQL, HBase, UNIX Shell Scripting, MapReduce, Putty, WinSCP, IntelliJ, Teradata, Linux.

Confidential

SQL Developer

Responsibilities:

  • Installed and configured SQL server 2005 and set-up Various Databases.
  • Installed and configured Oracle databases on Unix and Linux platform.
  • Created all administration tasks of SQL Server and designed database
  • Worked with development team in relation to application tuning.
  • Created tables, views, and indexes.
  • Developed various stored procedures, cursors, functions, and triggers.
  • Created tables and provided constraints and user defined rules and defaults.
  • Used system stored procedures and DBCC commands.
  • Created Database Maintenance Planner for the Performance of SQL Server which covers Database Integrity Checks, Update Database Statistics and Re-indexing.
  • Scheduled backups and performed restore operations for system test data depends on testers requirement.
  • Created logins and maintained user permissions appropriately.
  • Wrote complex T-SQL queries and procedures for retrieval of data.
  • Created DTS packages for migration of data from SQL Server to MS Access or Oracle databases.
  • Performed daily backups and developed recovery procedures.
  • Managed Disk space, memory, and User connections.
  • Performed DBCC checks and fixing data corruption in Application databases.

Environment: SQL Server 2005, DBCC, SQL Profiler, Enterprise Manager, Query Analyzer.

We'd love your feedback!