Sr. Big Data Engineer Resume
Chicago, IL
SUMMARY
- 8+ years of overall software development experience in Big Data Technologies, Hadoop Ecosystem, and SQL Knowledge with programming experience in Python, Scala, and Java.++ and other coding languages as well as AWS/Azure Cloud.
- 4+ years of strong hands - on experience in the Hadoop Ecosystem and its components including HDFS, Yarn, MapReduce, Hive, Pig, Sqoop, HBase, Spark, Spark SQL, Kafka, Spark Streaming, Oozie, Ambari.
- Proven experience in leading teams and complex data analytics projects
- Experienced in designing and architecting continuous delivery models
- Good knowledge in using Apache Nifi to automate data transactions between different Hadoop systems
- Follows the company’s required standards and guidelines regarding the use of the tools including technical documentation
- Experience in architecting, designing, installing, configuring, managing, and building distributed data pipelines in the Hadoop infrastructure with advanced data processing.
- Deep knowledge of troubleshooting and tuning Spark applications and Hive scripts to achieve optimal performance.
- Profound experience in creating real-time data streaming solutions using Apache Spark /Spark Streaming, Kafka, and Flume.
- Experience in moving data into and out of the HDFS and Relational Database Systems (RDBMS) using Apache Sqoop and loading into HIVE tables that are partitioned.
- Experience developing Kafka producers and Kafka Consumers for streaming millions of events per second on streaming data.
- Hands-on Experience in GCP, Big query, GCS Bucket, G-Cloud function, Cloud dataflow, PUB/SUB cloud shell, GSUTIL, command-line utilities, Data proc, Stack Driver.
- Extensive experience writing custom UDFs in Hive and custom Input Formats in MapReduce.
- Knowledge of job workflow scheduling and monitoring tools like Oozie, and Airflow.
- Manage metadata alongside the data for visibility from where data came from to ensure the quick and efficient finding of data for customers using AWS data lakes with its functions like lambda and glue
- Experience using various Hadoop Distributions (Cloudera, Hortonworks, Amazon AWS EMR) to fully implement and utilize various Hadoop services.
- Build data pipelines in airflow in GCP for ETL-related jobs using airflow operators
- Experience in GCP data proc, GCS, Cloud function
- Experience in moving data between GCP and azure using Azure Data Factory
- Experience working with NoSQL databases like MongoDB, Cassandra, and HBase.
- Solid experience in working with various data formats like Parquet, Orc, Avro, JSON, etc.,
- Valuable experience in designing and implementing end-to-end data security and governance within the Hadoop Platform using Kerberos.
- Hands-on experience in developing end-to-end Spark applications using Spark APIs like RDD, Spark Data frame API, Spark MLLib, Spark Streaming, and Spark SQL.
- Good experience working with various data analytics and big data services in AWS Cloud-like EMR, Redshift, S3, Athena, Glue, etc.,
- Experienced on GCP services GCS, Virtual Engines, Cloud SQL, BigQuery.
- Expertise in Data Migration, Data Profiling, Data Ingestion, Data Cleansing, Transformation, Data Import, and Data Export using multiple ETL tools
- Good understanding of Spark ML algorithms such as Classification, Clustering, and Regression.
- Experienced in migrating data warehousing workloads into Hadoop-based data lakes using MR, Hive, Pig, and Sqoop.
- Worked on HQL for required data extraction and join operations as required and having good experience in optimizing Hive Queries.
- Experience in Partitions, bucketing concepts in Hive, and designing both Managed and External tables in Hive to optimize performance.
- Setting up the build and deployment automation for Java-based projects by using Jenkins.
- Developed spark applications in python (Pyspark) on the distributed environment to load a substantial number of CSV files with different schema into Hive ORC tables.
- Experience in maintaining an Apache Tomcat MYSQL, LDAP, and Web service environment.
- Designed ETL workflows on Tableau, Deployed data from various sources to HDFS.
- Experience converting SQL databases to Azure data lakes, azure data lake analytics, and Azure SQL databases, as well as managing database access and data migration using Azure data factory.
- Experience in SQL Server, SSIS, SSAS, star schema, and snowflake schema database design and development for business intelligence.
- Experienced in Software Development Lifecycle (SDLC) using SCRUM, Agile methodologies
TECHNICAL SKILLS
Big Data Technologies/ Hadoop Components: HDFS, Hue, MapReduce, Yarn, Sqoop, Pig, Hive, HBase, Oozie, Kafka, Impala, Zookeeper, Flume, Cloudera Manager, Airflow, NiFi
Spark: Spark SQL, Spark Streaming, Data Frames, YARN, Pair RDDs
Cloud Services: AWS (S3, EC2, EMR, Lambda, Redshift, Glue), Azure (Azure Data Factory / ETL / ELT / SSIS, Azure Data Lake Storage, Azure Databricks)
Programming Languages: SQL, PySpark, Python, Scala, Java
Databases: Oracle, MySQL, DB2, SQL Server, Teradata
NoSQL Databases: HBase, Cassandra, MongoDB
Web Technologies: HTML, JDBC, Java Script, CSS
Version Control Tools: GitHub, Bitbucket
Server-Side Scripting: UNIX Shell, Power Shell Scripting
IDE: Eclipse, PyCharm, Notepad++, IntelliJ, Visual Studio
Operating Systems: Linux, Unix, Ubuntu, Windows, Cent OS
PROFESSIONAL EXPERIENCE
Sr. Big Data Engineer
Confidential - Chicago, IL
RESPONSIBILITIES:
- Worked with business analyst to gather and define project requirements
- Develop a business logic framework for model implementation.
- Working as a Data Engineer utilizing Bigdata & Hadoop Ecosystems components for building highly scalable data pipelines.
- Worked in an Agile development environment and participated in the daily scrum and other design-related meetings.
- Develop and implement complex databases and data marts for the current production state for both RDBMS and Hadoop ecosystem with legacy applications and design solutions to in corporate new processes and implementations into the current environment
- Involved in converting Hive/SQL queries into Spark transformations using Spark.
- Implemented the workflows using the Apache Oozie framework to automate tasks
- Wrote Flume configuration files for importing streaming log data into HBase with Flume.
- Worked on Spark SQL, created data frames by loading data from Hive tables, and created prep data and stored in AWS S3.
- Responsible for loading the customer's data and event logs from Kafka into Redshift through spark streaming.
- Worked on Apache Nifi to automate the data movement between RDBMS and HDFS.
- Implement Spark Kafka streaming to pick up the data from Kafka and send it to the Spark pipeline.
- Developed batch scripts to fetch the data from AWS S3 storage and do required transformations in Scala using the Spark framework.
- Implemented the workflows using the Apache Oozie framework to automate tasks.
- Optimization of Hive queries using best practices and right parameters and using technologies like Hadoop, YARN, Python, and Pyspark.
- Develop Nifi workflow to pick up the data from rest API server, from data lake as well as from SFTP server and send that to Kafka broker.
- Acted for bringing in data under HBase using HBase shell also HBase client API.
- Worked on reading and writing multiple data formats like JSON, ORC, and Parquet on HDFS using Pyspark.
- Experience developing Scala applications for loading/streaming data into NoSQL databases (MongoDB) and HDFS.
- Analyzed the SQL scripts and designed them by using Pyspark SQL for faster performance for the client.
- Created Sqoop Scripts to import and export customer profile data from RDBMS to S3 buckets.
- Assisted senior-level data scientists in the design of ETL processes, including SSIS packages
- Built custom Input adapters to migrate click stream data from FTP servers to S3.
- Developed various enrichment applications in spark using Scala for performing cleansing and enrichment of click stream data with customer profile lookups.
- Experience in working with different join patterns and implementing both Map and Reduce Side Joins.
- Ensure the data warehouse was populated only with quality entries by performing regular cleaning and integrity checks.
- Troubleshooting Spark applications for improved error tolerance and reliability.
- Used Spark Data frame and Spark API using Scala, and Python for Data transformations, cleansing as well as validation using Spark API to implement batch processing of Jobs.
- Worked on fine-tuning and performance enhancements of various spark applications and hive scripts.
- Used various concepts in spark like broadcast variables, caching, dynamic allocation, etc. to design more scalable spark applications.
- Loading Scalable data incrementally in raw form into BigQuery and UDM layer using Google Dataproc, GCS buckets, Hive, Spark, Scala, Python and shell scripts.
- Worked on creating pipelines to ingest data from Hadoop to GCP BigQuery.
- Worked on formulating business metric queries on BigQuery and validating across multiple datasets.
- Write Python program to maintain raw file archival in GCS bucket
- Open SSH tunnel to google data proc to access yarn manager to monitor spark jobs
- Submit spark using gsutil and spark submission in data proc cluster
- Implemented continuous integration and deployment using CI/CD tools like Jenkins, GIT, and Maven.
Environment: AWS EMR, GCP, S3, Spark, Hive, Sqoop, Scala, Hadoop, Cloudera, Flume, HBase, HDFS, MapReduce, Kafka, YARN Java, MySQL, Oracle DB, Athena, Redshift.
Spark Developer
Confidential - Minneapolis, MN
RESPONSIBILITIES:
- Coordinated specifications and requirements for project technical needs
- Architecture, design, and implementation of enterprise logging system using Hadoop, Hbase, HDFS, Hive, and Cloudera Search
- DataIngestioninto the DataLake using Open-source Hadoop distribution to process Structured, Semi-Structured, and Unstructured datasets.
- Design Star schema in BigQuery.
- Build Scala and Spark based configurable framework to connect common data sources like MYSQL, Oracle, Postgres, SQL server and load it in Bigquery
- Experience monitoring Bigquery, DataProc, and cloud dataflow jobs for all environments.
- Expertise in Hive queries created user-defined aggregated function worked on advanced optimization techniques and have extensive knowledge on joins.
- Solved performance issues in Hive and Pig scripts with an understanding of Joins, Group, and Aggregation and how does it translate to MR jobs.
- Experience in Apache NiFi to copy thedatafrom the local file system to HDP and implement Nifi flow topologies to perform cleansing operations before movingdatainto HDFS.
- Create hive scripts to extract, transform, load (ETL), and store the data using Talend.
- Developed Sqoop scripts to extract data from DB2 EDW source databases onto HDFS.
- Used NiFi and Sqoop for moving data between HDFS and RDBMS.
- Worked with Oracle and Teradata for data import/export operations from different data marts.
- Developed Spark jobs, and Hive jobs to summarize and transform data.
- Worked extensively with Data migration, Data cleansing, Data profiling, and ETL Processes features for data warehouses.
- Scheduled different Snowflake jobs using NiFi.
- First-hand experience inAzureCloud stored the data in ADLS - Azure Data Lake Storage,App services,Databricks cluster for running the jobs,Azure SQL Database,Virtual machines,Fabric controller,Azure AD, Azure search, and notification hub.
- Designed, configured, and deployed MicrosoftAzurefor a multitude of applications utilizing theAzurestack (Including Compute, Web & Mobile, Blobs, Resource Groups, Azure SQL, Cloud Services, and ARM), focusing on high - availability, fault tolerance, and auto-scaling.
- Developed Spark Application by using Python (Pyspark).
- Developed Spark Streaming script which consumes topics from distributed messaging source Kafka and periodically pushes batch of data to spark for real-time processing.
- Developed and implemented API services using Python in spark.
- CreatedPartitions,and Bucketsbased on State to further process usingBucket-based Hive joins.
- Load the data into Spark RDD and Perform in-memory data computation to generate the output as per the requirements.
- Responsible for continuous monitoring and managing Elastic MapReduce cluster throughAWS console.
- ImplementedSparkusing Scala andSparkSQL for faster testing and processing of data.
- Created tables, indexes, views, constraints, sequences, triggers, synonyms, tablespaces, nested tables, and database links using SQL and PL/SQL.
- Developed data pipelines using Spark, Hive, and Sqoop to ingest, transform and analyze operational data.
- Knowledge of handling Hive queries using Spark SQL that integrate with the Spark environment.
- Using Rest API with python to ingest data from random sources into bigquery
- Worked on migrating MapReduce programs intoSparktransformations usingSparkand Scala.
- Used cloud Shell SDK in GCP to configure the service data proc, data storage and bigquery
- Experience working with SparkSQL and creating RDDs using PySpark. Extensive experience working with ETL of large datasets using PySpark in Spark on HDFS.
Environment: Hadoop, Hive, GCP, Talend, Map Reduce, Pig, Salesforce, SQOOP, Splunk, CDH5, Python, HDFS, Pig, DB2, Sqoop, Oozie, Putty, Java.
Data Engineer
Confidential - Scottsdale, AZ
RESPONSIBILITIES:
- Build the new universes in Business Objects as per the user requirements by identifying the required tables from Data mart and by defining the universe connections.
- Used Business Objects to create reports based on SQL queries. Generated executive dashboard reports with the latest company financial data by business unit and by-product.
- Improve and enhance the value using the Hadoop cluster, in conjunction with the Hadoop Architect, through the application of Apache Spark and other emerging technologies
- Performed the data analysis and mapping database normalization, performance tuning, query optimization data extraction, transfer, and loading (ETL), and clean-up.
- Design and Implement ETL to data load from Source to Confidential databases and for Fact and Slowly Changing Dimensions (SCD) Type1, Type 2 to capture the changes for Staging, Dimensions, Facts, and Data marts.
- Implemented Teradata RDBMS analysis with Business Objects to develop reports, interactive drill charts, balanced scorecards, and dynamic Dashboards.
- Responsible for requirements gathering, status reporting, creating various metrics, and project deliverables.
- Developed PL/SQL Procedures, Functions, and Packages and used SQL loader to load data into the database.
- Designed and developed Informatica Mappings to load data from Source systems. Worked on Informatics Power Center tool - Source Analyzer, Data warehousing designer, Mapping Mapplet Designer, and Transformation Designer.
- Involved in migrating warehouse database from Oracle 9i to 10g database.
- Scheduled different Snowflake jobs using NiFi.
- Involved in analyzing and adding new features of Oracle 10g like DBMS SCHEDULER create the directory, data pump, CONNECT BY ROOT in existing Oracle 9i application.
- Tuned Report performance by exploiting Oracle's new built-in functions and rewriting SQL statements.
- Write Python program to maintain raw file archival in GCS bucket.
- Worked on integrating GIT into the continuous Integration (CI) environment along with Jenkins.
- Using JIRA for issues and project tracking, GIT for version control, and Airflow for scheduling the jobs.
Environment: SQL Server, GCP, J2EE, UNIX, .Net, MS Project, Oracle, Web Logic, Shell script, JavaScript, HTML, Microsoft Office Suite 2010, Excel.
Hadoop Developer
Confidential
RESPONSIBILITIES:
- Involved in Requirements Analysis and design of an Object-oriented domain model.
- Designed Use Case Diagrams, Class Diagrams, and Sequence Diagrams and Objects.
- Involved in designing user screens using HTML as per user requirements.
- Used Spring-Hibernate integration in the back end to fetch data from Oracle and MYSQL databases.
- Used Spring Dependency Injection properties to provide loose coupling between layers.
- Implemented the Web Service client for login authentication, credit reports, and applicant information.
- Used Web services (SOAP) for transmission of large blocks of XML data over HTTP.
- Implemented the logging mechanism using the Log4j framework.
- Wrote test cases in JUnit for unit testing of classes.
- Developed application to be implemented on Windows XP.
- Created application using Eclipse IDE.
- Installed Web Logic Server for handling HTTP Request/Response.
- Used Subversion for version control and created automated build scripts.
