Big Data Engineer Resume
Memphis, TN
SUMMARY
- Overall, 8 years of professional IT experience in Application Development and Data Analytics using various languages and tools like SQL, Scala, Java, Python.
- Over 2+ years of experience in Apache Spark’s Core, Spark SQL, Spark Streaming.
- Around 2 years’ experience working on Apache Kafka.
- 5+ years of experience in design and development of Big Data Analytics using Hadoop ecosystems - related technologies.
- Expertise in Big Data technologies as a consultant, proven capability in project-based and as an individual developer with good communication skills.
- Over 3 of extensive experience working on big data ETL.
- Expertise in Microsoft Azure Cloud Services (PaaS & IaaS), Application Insights, Document DB, Internet of MYYThings (IoT), Azure Monitoring, Key Vault, Visual Studio Online (VSO) and SQL Azure.
- Experience in Implementing AWS solutions using EC2, S3 and EMR
- Extensive experience on Big Data Analytics with hands on experience in writing Map Reduce jobs on Hadoop Ecosystem including Hive, Pig, HBase, Sqoop, Impala, Oozie, Zookeeper, Spark, Kafka, Cassandra, and Flume.
- Very good understanding in ecosystems with Cloudera CDH1, CHD2, CDH3, CDH4, CDH5, Hortonworks HDP2.1 and Hadoop MR1 & MR2 Architectures.
- Strong knowledge of Rack awareness topology in the Hadoop cluster.
- Hands on experience in installation, configuration, supporting and managing Cloudera, Hortonworks Hadoop platform along with CDH3 and CDH4 clusters.
- Working on Data Lake set up with Hortonworks and AWS team.
- Experience in managing multi-tenant Cassandra clusters on public cloud environment - Amazon Web Services (AWS)-EC2.
- Hands on experience in Azure Development, worked on Azure web application, App services, Azure storage, Azure SQL Database, Virtual machines, Fabric controller, Azure AD, Azure search, and notification hub.
- Experience in execution of Batch jobs through the data streams to SPARK Streaming.
- Expert in working with Hive data warehouse tool-creating tables, data distribution by implementing Partitioning and Bucketing, writing and optimizing the HiveQL queries.
- Developed Reusable solution to maintain proper coding standard across different java project.
- Very good in Application Development and Maintenance of SDLC projects using different programming languages such as Java, C, Scala, SQL and NoSQL.
- Developing various cross platform products while working with different Hadoop file formats like Sequence File, RC File, ORC, AVRO & Parquet.
- Expert in importing and exporting data from different Relational Database Systems like MySQL and Oracle into HDFS and Hive using Sqoop.
- Strong in analyzing data using HiveQL, Pig Latin, HBase and Map Reduce programs in java.
- Expertise in Extending Hive and Pig core functionality by writing custom UDF’s.
- Experience with databases like DB2, MySQL, SQL, MongoDB.
- Configured Kerberos Security setup for authentication and integrated with Active Directory(AD)
- Configured Ranger and Knox services for authorization purposes
- Good technical Skills in SQL Server, ETL Development using Informatica tool.
- Written documentation to describe program development, logic, coding, testing, changes, and corrections.
- Expertise in writing SQL, PL/SQL to integrate of complex OLTP and OLAP database models and data marts, worked extensively on Oracle, SQL SERVER, and DB2.
- Experience in all the life cycle phases of the projects on large data sets and experience with performance tuning and troubleshooting.
- Extensive knowledge of UNIX and Shell scripting.
- Strong background in mathematics and have very good analytical and problem-solving skills.
- Proactive problem-solving mentality that thrives in an agile work environment.
- Ability to work effectively with associates at all levels within the organization.
- Have good knowledge on NoSQL databases like HBase, Cassandra and MongoDB.
- Experienced in Data Ingestion projects to inject data into Data Lake using multiple sources systems using Talend Bigdata.
- Extensively used ETL methodology for performing Data Migration, Extraction, Transformation and loading using Talend and designed data conversions from wide variety of source systems.
- Experience in creating complex SQL Queries and SQL tuning, writing PL/SQL blocks like stored procedures, Functions, cursors, Index, triggers and packages.
TECHNICAL SKILLS
Hadoop/Big Data: HDFS, MapReduce, Hive, HBase, Pig, Sqoop, Flume, Oozie, Cassandra, YARN, Zookeeper, Spark SQL, Apache Spark, Impala, Apache Drill, Kafka, Elastic MapReduce.
Hadoop Frameworks: Cloudera CDHs, Hortonworks HDPs, MAPR.
Java & J2EE Technologies: Core Java, Servlets, Java API, JDBC, Java Beans.
IDE and Tools: Eclipse, Net beans, Maven, ANT, Hue (Cloudera Specific), Toad, Sonar, JDeveloper.
Frameworks: MVC, Structs, Hibernate, Spring.
Programming Languages: C, C++, Java, Scala, Python, Linux shell.
Web Technologies: HTML, XML, DHTML, HTML5, CSS, JavaScript.
Databases: MYSQL, DB2, MS-SQL Server, Oracle.
NO SQL Databases: HBase, Cassandra, Mongo DB.
Methodologies: Agile Software Development, Waterfall.
Version Control Systems: GitHub, SVN, CVS, Clearcase.
Operating Systems: RedHat Linux, Ubuntu Linux, Windows XP/Vista/7/8/10, Sun Solaris, Suse Linux.
Data Visualization: Power Bi, Tableau, Qlik.
PROFESSIONAL EXPERIENCE
Confidential, Memphis, TN
Big Data Engineer
Responsibilities:
- Worked on a live 65 nodes Hadoop cluster running CDH4.7.
- Installed & configured multi-node Hadoop cluster for data store & processing.
- Design and implement database solutions in Azure SQL Data Warehouse, Azure SQL.
- Assisted in upgrading, configuration and maintenance of various Hadoop infrastructures like Pig, Hive, and HBase.
- Configured Flume to capture the news from various sources for testing the classifier.
- Experience in developing MapReduce jobs using various Input and output formats.
- Assisted in upgrading, configuration and maintenance of various Hadoop infrastructures like Pig, Hive, and HBase.
- Configure Snowflakes DW to be used for Tableau report by the Supply Chain Dept.
- Developed workflow in Oozie to automate the tasks of loading the data into HDFS and preprocessing, analyzing and training the classifier using MapReduce jobs, Pig jobs and Hive jobs.
- Developed Spark scripts by using Scala shell commands as per the requirement.
- Involved in loading data into Cassandra NoSQL Database.
- Configure Snowflakes DW to be used for Tableau report by the Supply Chain Dept.
- Developed Spark applications to move data into Cassandra tables from various sources like Relational Database or Hive.
- Created various pipelines to load the data from Azure data lake into Staging SQLDB and followed by to Azure SQL DB
- Performed data wrangling and developed data tables for excess inventory, stock outs, future high orders, forecast inaccuracy and delayed purchase orders on supply chain data.
- Performed extensive analyses on club-item and store level attributes to determine impacts on club-item replenishment.
- Used R programming to test data predictive models for supply chain division and develop R programming to identify Parts sourcing gaps.
- Worked on Spark streaming collects the data from Kafka in near real time and performs necessary transformations and aggregations on the fly to build the common learner data model and persists the data in Cassandra.
- Involved in developing data ingestion pipelines on Azure HDInsight Spark cluster using Azure Data Factory and Spark SQL. Also Worked with Cosmos DB (SQL API and Mongo API).
- Worked on Cassandra Data modelling, NoSQL Architecture, DSE Cassandra Database administration, Key space creation, Table creation, Secondary and Solr index creation, User creation & access administration.
- Experience in performance tuning a Cassandra cluster to optimize writes and reads.
- Developed Python scripts, UDF's using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into RDBMS through Sqoop.
- Used Pig and Hive in the analysis of data.
- Load the data into Spark RDD and performed in-memory data computation to generate the output response.
- Develop ETL packages with different data sources (SQL Server, Db2, Flat Files etc.) and load data into destination tables in Enterprise Data Warehouse (EDW) using SSIS 2017/2015 packages and SAP Data Services
- Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
- Performed Sqooping for various file transfers through the HBase tables for processing of data to several NoSQL DBs- Cassandra, MongoDB.
- Implemented Spark Storm builder topologies to perform cleansing operations before moving data into Cassandra.
- Developed ETL workflow which pushes webserver logs to an Amazon S3 bucket.
- Implemented Cassandra connection with the Resilient Distributed Datasets (local and cloud).
- Importing and exporting data into HDFS and Hive.
- Implemented ETL code to load data from multiple sources into HDFS using Pig Scripts.
- Implemented Pig as ETL tool to do Transformations, event joins and some pre-aggregations before storing the data onto HDFS.
- Worked on Talend ETL scripts to pull the data from TSV Files/Oracle Data Base into HDFS.
- Worked extensively on design, development and deployment of talend jobs to extract data, filter the data and load them into datalake.
- Extract data from source system and transform into newer systems using Talend DI Components.
- Worked on Storm to handle the parallelization, partitioning, and retrying on failures and developed a data pipeline using Kafka and Strom to store data into HDFS.
- Exploring with Spark improving the performance and optimization of the Existing algorithms in Hadoop using Spark context, Spark-SQl, data frame pair RDD’s, Spark YARN.
- Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.
- Supported Map Reduce Programs those are running on the cluster.
Environment: Hadoop, HDFS, Cloudera, Python, AWS, Spark, YARN, Map Reduce, Hive, Teradata SQL, PL/SQL, Pig, Talend, Data Lake, Data Integration 6.1/5.5.1 (ETL), Kafka, SAP, Sqoop, Oozie, HBase, Cassandra, Java, Scala, Python, UNIX Shell Scripting.
Confidential, Charlotte, NC
Sr. Hadoop Developer
Responsibilities:
- Prepared design blueprints and application flow documentation, gathering requirements from the Business patterns
- Maintained the data in Data Lake (ETL), coming from the Teradata Database, writing on an average of 80 GB daily. Overall, the data warehouse had 5 PB of data and used a 135-node cluster to process the data.
- Responsible for creating Hive Tables to load the data from MySQL by using Sqoop, writing java snippets to perform cleaning, pre-processing, and data validation.
- Knowledge and experience in Domains such as Banking and Manufacturing, Supply chain, and Healthcare.
- Experienced in creating Hive schema, external tables, and managing views. Worked on performing Join operations in Spark using hive. Writing HQL statements as per the user requirements.
- Involved in converting Hive/SQL queries into Spark transformations using Spark Data frames and working with Spark-Shell. Developed Spark code using Java and Spark-SQL for faster testing and data processing.
- Imported millions of structured data from relational databases using Sqoop import to process using Spark and stored the data into HDFS in parquet format.
- Experienced working with AWS services like EMR, Redshift, S3, Glue, Kinesis, and Lambda for serverless ETL.
- To process the massive volume of structured data, Spark SQL was used. In addition, spark Data Frames transformations were implemented and steps to migrate Map Reduce algorithms.
- Explored with Spark for improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
- Used Data Frame API solutions to pre-process massive volumes of structured data in various file formats, including Text files, CSV, Sequence files, XML and JSON files, and Parquet files, and then turn the data into named columns.
- Ensure necessary system security by using best-in-class AWS cloud security solutions. Additionally, experienced in deploying Java projects using Maven/ANT and Jenkins.
- DevOps and CI/CD pipeline knowledge - Mainly Teamcity, Selenium. Implement continuous integration/ delivery (CI/CD) pipelines in AWS when necessary.
- Experienced with batch processing of data sources using Apache Spark. Developed predictive analytic using Apache Spark Java APIs.
- Expert in implementing advanced procedures like text analytics and processing using in-memory computing capabilities like Apache Spark written in Java.
- Worked on the core and Spark SQL modules of Spark extensively. Extensively used Broadcast Variables and Accumulators for better performances.
- Designed, configured and deployed MicrosoftAzurefor a multitude of applications utilizing theAzurestack (Including Compute, Web & Mobile, Blobs, Resource Groups, Azure SQL, Cloud Services, and ARM), focusing on high - availability, fault tolerance, and auto-scaling.
Environment: Hadoop, HDFS, Hive, Java 1.7, Spark 1.6, SQL, HBase, UNIX Shell Scripting, MapReduce, Putty, WinSCP, IntelliJ, Teradata, Linux.
Confidential, Westlake, TX
Big Data Engineer
Responsibilities:
- Worked on loading disparate data sets coming from different sources to BDpaas (HADOOP) environment using Spark.
- Developed UNIX scripts in creating Batch load for bringing huge amount of data from Relational databases to BIGDATA platform.
- Delivery experience on major Hadoop ecosystem Components such as Pig, Hive, Spark Kafka, Elastic Search & HBase and monitoring with Cloudera Manager.
- Involved with the team of fetching live stream data from DB2 to HBase table using Spark Streaming and Apache Kafka.
- Validation of On Demand SAP and third-party solutions for Performance product standards in the production landscape, suggesting the real time improvement areas and thus acting for the solutions.
- Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the SQL Activity.
- Collected data on economic indicators, analyzed economic trends, and developed models to forecast growth, interest rates, inflation and investments using SAS.
- Implemented the Machine learning algorithms using Spark with Python and worked on Spark Storm, Apache and Apex and python.
- Used R programming to test data predictive models for supply chain division and develop R programming to identify Parts sourcing gaps.
- Assisted budget coordinator with preparing different community programs, allocating budget, preparing daily financial transactions and allocation reports using Microsoft Excel.
- Design Setup maintain Administrator the Azure SQL Database, Azure Analysis Service, Azure SQL Data warehouse, Azure Data Factory, Azure SQL Data warehouse.
- Build Complex distributed systems involving huge amount data handling, collecting metrics building data pipeline, and Analytics.
- Sound with the principles of lean six sigma, lean manufacturing, manufacturing analysis, decision making under uncertainty, in depth knowledge of statistics supply chain management, simulation techniques
- Involved in analyzing data coming from various sources and creating Meta-files and control files to ingest the data into the Data Lake.
- Involved in configuring batch job to perform ingestion of the source files into the Data Lake and developed Pig queries to load data to HBase.
- Leveraged Hive queries to create ORC tables and developed HIVE scripts for analyst requirements for analysis.
- Implemented Kafka consumers to move data from Kafka partitions into Cassandra for near real-time analysis and worked extensively on Hive to create, alter and drop tables and involved in writing hive queries.
- Created and altered HBase tables on top of data residing in Data Lake and Created external Hive tables on the Blobs to showcase the data to the Hive Meta Store.
- Involved in requirement and design phase to implement Streaming Architecture to use real time streaming using Spark and Kafka.
- Used Spark for interactive queries, processing of streaming data and integration with HBase database for huge volume of data.
- Use Spark API for Machine learning. Translate a predictive model from SAS code to Spark and used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
- Created Reports with different Selection Criteria from Hive Tables on the data residing in Data Lake.
- Worked on Hadoop Architecture and various components such as YARN, HDFS, Node Manager, Resource Manager, JobTracker, TaskTracker, NameNode, Data Node and MapReduce concepts.
- Deployed Hadoop components on the Cluster like Hive, HBase, Spark, Scala and others with respect to the requirement.
- Implemented the Business Rules in Spark/ SCALA to get the business logic in place to run the Rating Engine.
- Developed SQL packages and database triggers in both Oracle and Greenplum.
- Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
- Used Spark UI to observe the running of a submitted Spark Job at the node level and used Spark to do Property Bag Parsing of the data to get the required fields of data.
- Extensively used ETL methodology for supporting Data Extraction, transformations and loading processing, using Hadoop.
- Used both Hive context as well as SQL context of Spark to do the initial testing of the Spark job and used WINSCP and FTP to view the data storage structure in the server and to upload JARs which were used to do the Spark Submit.
- Developed code from scratch in Spark using SCALA according to the technical requirements.
Environment: Hadoop, Map Reduce, Yarn, Hive, Pig, HBase, Sqoop, Spark, Scala, MapReduce, SAS, Core Java, R Language, SQL, Python, Eclipse, Linux, Unix, HDFS, SAP, Map Reduce, Impala, Cloudera, SQOOP, Kafka, Apache Cassandra, Oozie, Impala, Zookeeper, MySQL, Eclipse, PL/SQL
Confidential, Nebreska
Hadoop Developer
Responsibilities:
- Developed workflows for complete end-to-end ETL process starting with getting data into HDFS, validating and applying business logic, storing clean data in hive external tables, exporting data from hive to RDBMS sources for reporting and escalating and data quality issues.
- Working as onsite coordinator and providing technical assistance, troubleshooting and alternative development solutions.
- Handled importing of data from various data sources performed transformations using Spark and loaded data into hive.
- Exclusive knowledge on the Single user performance testing for SAP Net weaver ABAP
- Involved in performance tuning of Hive (ORC table) for design, storage, and query perspectives.
- Developing and deploying using Horton works HDP 2.3.0 in production and HDP 2.6.0 in the development environment.
- Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems and vice-versa.
- Worked in developing Pig scripts to create the relationship between the data present in the Hadoop cluster.
- Performed advanced procedures like text analytics and processing using the in-memory computing capabilities of spark using Python and Scala.
- Worked on implementing data lake and responsible for data management in Data Lake.
- Developed Ruby Script to map the data to the production environment.
- Developed Hive UDFs and Pig UDFs using Python script.
- Experienced in working with IBM Data Science tool and responsible for injecting the processed data to IBM Data Science tool.
- Strong Knowledge of clinical information systems/EMRs/EHRs and health care files like PGF, HL7, 837and claims.
- Designed and provisioned the platform architecture to execute Hadoop and Machine Learning use cases under Cloud infrastructure, AWS, EMR, and S3.
- Worked on cloud computing infrastructure (e.g. Amazon Web Services EC2) and considerations for scalable, distributed systems and involved in file movements between HDFS and AWS S3 and extensively worked with S3 bucket in AWS and converted all Hadoop jobs to run in EMR by configuring the cluster according to the data size.
- Responsible for developing data pipeline with Amazon AWS to extract the data from weblogs and store in HDFS.
- Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary
- Transformations and Aggregations on the fly to build the common learner data model and persistence the data in HDFS and imported the data from different sources like AWS S3, Local file system into SparkRDD.
- Worked with importing metadata into Hive using Python and migrated existing tables and applications to work on AWS cloud (S3).
- Worked extensively with importing metadata into Hive and migrated existing tables and applications to work on Hive and AWS cloud and making the data available in Athena and Snowflake.
- Extensively used Stash Git-Bucket for Code Control and Worked on AWS Components such as Airflow,
- Elastic Map Reduce (EMR), Athena and Snowflake.
- Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL, and a variety of portfolios.
- Worked on Oozie Workflow Engine in running workflow jobs with actions that run Hadoop Map/Reduce and Pig jobs.
- Responsible for configuring the cluster in IBM cloud and maintaining the number of nodes as per requirement.
- Developed Kafka consumer to consume data from Kafka topics.
- Developed shell scripts for running Hive scripts in Hive and Impala.
- Responsible for optimization of data-ingestion, data-processing, and data-analytics.
- Expertise is developing a Pyspark application which builds connections between HDFS and HBase and allows data transfer between them.
- Worked on RDBMS like Oracle DB2 SQL Server and My SQL database.
- Developed workflows to cleanse and transform raw data into useful information to load it to a Kafka
- Queue to be loaded into HDFS and NoSQL databases.
- Responsible for sanity testing of the system once the code is deployed in production.
- Experienced in using IDEs like Eclipse and intellij to modify the code in GIt.
- Involved in quality assurance of the data mapped into production.
- Involved in code walk through, reviewing, testing and bug fixing.
Environment: Hadoop, MapReduce, HDFS, SAP, Sqoop, flume, kafka, Hive, Pig, HBase EclipseDBeaver, Datagrip, SQL Developer, intellij, GiT, SVN, JIRA, Unix.
Confidential
Data Analyst
Responsibilities:
- Involved in complete project life cycle starting from design discussion to production deployment.
- Developed a 16-node cluster in designing the Data Lake with the Cloudera Distribution.
- Responsible for building scalable distributed data solutions using Hadoop.
- Worked closely with the business team to gather their requirements and new support features.
- Developed Hive scripts to analyze data and PHI are categorized into different segments and promotions are offered to customer based on segments.
- Installed and configured Hadoop Clusters with required services (HDFS, Hive, HBase, Spark, Zookeeper).
- Implemented and configured High Availability Hadoop Cluster.
- Created Hive tables, partitions and loaded the data to analyze using HiveQL queries.
- Created different staging tables like ingestion tables and preparation tables in Hive environment.
- Optimized Hive queries and used Hive on top of Spark engine.
- Developed UDFs in Java as and when necessary to use in Pig and HIVE queries.
- Worked on Oozie workflow engine for job scheduling.
- Extensive experience in writing Pig scripts to transform raw data into baseline data.
- Tested Apache TEZ, an extensible framework for building high performance batch and interactive data processing applications, on Pig and Hive jobs.
- Experience in retrieving data from oracle using PHP and Java programming.
- Worked on Sequence files, Map side joins, Bucketing, Static and Dynamic Partitioning for Hive performance enhancement and storage improvement.
- Experience in managing and reviewing Hadoop log files.
- Followed Agile Methodologies while working on the project.
- Good understanding of ETL tools and how they can be applied in a Big Data environment.
- Bug fixing and 24-7 production support for running the processes.
- Followed Agile Methodologies while working on the project.
Environment: Hadoop, MapReduce, HDFS, Sqoop, flume, kafka, Hive, Pig, HBase, SQL, Shell Scripting, Eclipse, Deaver, Data grip, SQL Developer, IntelliJ, GiT, SVN, JIRA, Unix, Power Bi.
