Senior Big Data Engineer Resume
SUMMARY
- Over 9+ years of IT experience in Analysis, design, development, implementation, maintenance, and support with experience in developing strategic methods for deploying big data technologies to efficiently solve Big Data processing requirement.
- Experienced working with various Hadoop Distributions (Cloudera, Hortonworks, Map R, Amazon EMR) to fully implement and leverage new Hadoop features.
- Implemented frameworks to extract, transform, load data from various sources.
- Deep knowledge of troubleshooting and tuning Spark applications and Hive scripts to achieve optimal performance
- Experience in Implement frameworks to import and export data from Hadoop to RDBMS.
- Deep knowledge of troubleshooting and tuning Spark applications and Hive scripts to achieve optimal performance.
- Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns
- Experience in manipulating/analysing large datasets and finding patterns and insights within structured and unstructured data.
- Strong experience with ETL and/or orchestration tools (e.g. Talend, Oozie, Airflow)
- Experience in using Teradata ETL tools and utilizes such as BTEQ, MLOAD, FASTLOAD, TPT, and Fast Export.
- Experience setting up AWS Data Platform - AWS CloudFormation, Development End Points, AWS Glue, EMR and Jupyter/Sagemaker Notebooks, Redshift, S3, and EC2 instances
- Experience in developing Spark Applications using Spark RDD, Spark - SQL and Data frame APIs.
- Worked with real-time data processing and streaming techniques using Spark streaming and Kafka.
- Experience in moving data into and out of the HDFS and Relational Database Systems (RDBMS) using Apache Sqoop.
- Solid experience in Snowflake data warehouse and good in experience in migration other databases to snowflake as well.
- Experience developing Kafka producers and Kafka Consumers for streaming millions of events per second on streaming data
- Experience in developingcustomUDFsfor Pig and Hive to in corporate methods and functionality of Python/Java intoPig LatinandHQL(HiveQL) and Used UDFs from Piggybank UDF Repository.
- Experience on Migrating SQL database to Azure Data Lake, Azure data lake Analytics, Azure SQL Database, Data Bricks and Azure SQL Data warehouse and Controlling and granting database access and Migrating On premise databases to Azure Data lake store using Azure Data factory
- Good understanding of the Data modelling (Dimensional & Relational) concepts like Star-Schema Modelling, a Schema Modelling, Fact and Dimension tables.
- Database design, modeling, migration and development experience in using stored procedures, triggers, cursor, constraints and functions. Used My SQL, MS SQL Server, DB2, and Oracle
- Experience working with NoSQL database technologies, including MongoDB, Cassandra and HBase.
- Expertise in working with HIVE data warehouse infrastructure-creating tables, data distribution by implementing Partitioning and Bucketing, developing and tuning the HQL queries.
- Replaced existing MR jobs and Hive scripts with Spark SQL & Spark data transformations for efficient data processing.
- Experience in writing complex SQL queries, creating reports and dashboards.
- Proficient in using Unix based Command Line Interface.
- Used Informatica Power Center for (ETL) extraction, transformation and loading data from heterogeneous source systems into target database
- Experience with Software development tools such as JIRA, Play, GIT.
- Experienced in using Agile methodologies including extreme programming, SCRUM and Test-Driven Development (TDD)
- Determined, committed and hardworking individual with strong communication, interpersonal and organizational skills.
TECHNICAL SKILLS
BigData/Hadoop Technologies: MapReduce, SparkSQL, Spark Streaming, Kafka, PySpark, AWS, Pig, HiveHBase, Flume, Yarn, Oozie, Zookeeper
Languages: Java, Scala, Python (NumPy, SciPy, Pandas, Gensim, Keras), Java ScriptShell Scripting
NO SQL Databases: Cassandra, HBase, MongoDB
Development Tools: Microsoft SQL Studio, IntelliJ, Azure Databricks, Eclipse, NetBeans.
Public Cloud: AWS, Azure, Snowflake
Build Tools: Jenkins, Toad, SQL Loader, PostgreSQL, Talend, Maven, ANT, RTC, RSAControl-M, Oozie, Hue, SOAP UI
Reporting Tools: MS Office (Word/Excel/Power Point/ Visio/Outlook), Crystal reports XISSRS, cognos.
Databases: Microsoft SQL Server, MySQL, Oracle 9i/10g/11g/12c, DB2, TeradataNetezza
Operating Systems: All versions of Windows, UNIX, LINUX, Macintosh HD, Sun Solaris
PROFESSIONAL EXPERIENCE
Confidential
Senior Big Data Engineer
Responsibilities:
- Worked on Ingesting data by going through cleansing and transformations and leveraging AWS Lambda, AWS Glue and Step Functions
- Created monitors, alarms, notifications and logs for Lambda functions, Glue Jobs, EC2 hosts using Cloudwatch.
- Worked on EMR clusters of AWS for processing Big Data across a Hadoop Cluster of virtual servers.
- Used Apache NiFi to copy data from local file system to HDP.
- Designed both 3NF Data models and dimensional Data models using Star and Snowflake Schemas Handling message streaming data through Kafka to s3.
- Implementing python script for creating the AWS Cloud Formation template to build EMR cluster with instance types.
- Creating Databricks notebooks using SQL, Python and automated notebooks using jobs.
- Successfully loading files to Hive and HDFS from Oracle, SQL Server usingSQOOP.
- Developed highly complexPythonandScalacode, which is maintainable, easy to use, and satisfies application requirements, data processing and analytics using inbuilt libraries.
- Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, PostgreSQL, Data Frame, Open Shift, Talend, pair RDD's.
- Experience with deploying Hadoop in a VM and AWS Cloud as well as physical server environment
- Monitor Hadoop cluster connectivity and security and File system management.
- Data engineering using Spark, Python and Pyspark.
- Supporting Continuous storage in AWS using Elastic Block Storage, S3, Glacier. Created Volumes and configured Snapshots for EC2 instances
- Data replacement and appending to the Hive database via pulling it using Sqoop processing tool to HDFS database from multiple data marts as source.
- Developed a python script to transfer data, REST API’s and extract data from on-premises to AWS S3. Implemented Micro Services based Cloud Architecture using Spring Boot.
- Worked onDocker containerssnapshots, attaching to a running container, removing images, managing Directory structures and managing containers.
- Collected data using Spark Streaming from AWS S3 bucket in near-real-time and performs necessary Transformations and Aggregation on the fly to build the common learner data model and persists the data in HDFS.
- Experience in change implementation, monitoring and troubleshooting of AWS Snowflake databases and cluster related issues.
- Expertise in usingDocker to run and deploy the applications in multiple containers likeDocker SwarmandDocker Wave.
- UtilizedSpark, Scala, Hadoop, HBase, Cassandra, MongoDB, Kafka, Spark Streaming, MLLib, Pythonand utilized the engine to increase user lifetime by 45% and triple user conversations for target categories.
- Involved inUnit Testingthe code and provided the feedback to the developers. PerformedUnit Testingof the application by usingNUnit.
- Created architecture stack blueprint for data access with NoSQL DatabaseCassandra;
- Brought data from various sources in to Hadoop and Cassandra usingKafka.
- Created multiple dashboards in tableau for multiple business needs.
- Works on loading data into Snowflake DB in the cloud from various sources.
- Installed and configured Hive and written Hive UDFs and used piggy bank a repository of UDF’s for Pig Latin
- Successfully implemented POC (Proof of Concept) in a Development Databases to validate the requirements and benchmarking the ETL loads.
- Worked on ETL Migration services by developing and deploying AWS Lambda functions for generating a serverless data pipeline which can be written to Glue Catalog and can be queried from Athena.
- Managed security groups on AWS, focusing on high-availability, fault-tolerance, and auto scaling using Terraform templates. Along with Continuous Integration and Continuous Deployment with AWS Lambda and AWS code pipeline.
- Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark databricks cluster
- Used Pandas in Python for Data Cleansing and validating the source data.
- Deployed the Big DataHadoop applicationusingTalendon cloudAWS (Amazon Web Sevices) and also on Microsoft Azure
- After the transformation of data is done, this transformed data is then moved to Spark cluster where the data is set to go live on to the application usingSpark streamingandkafka.
- Unit tested the data between Redshift and Snowflake.
- Developed highly complexPythonandScalacode, which is maintainable, easy to use, and satisfies application requirements, data processing and analytics using inbuilt libraries.
Environment: Hortonworks, Hadoop, HDFS, AWS Glue, AWS Athena, Spark Databricks, MR, Pig, Sqoop, Hive, NoSQL, HBase, Shell Scripting, Scala, Spark, SparkSQL, Snowflake, AWS, SQL Server, Tableau, Kafka
Confidential
Big Data Engineer
Responsibilities:
- Involved in analyzing business requirements and prepared detailed specifications that follow project guidelines required for project development.
- Used Pyspark for data frames, ETL, Data Mapping, Transformation and Loading in complex and high-volume environment
- Implemented Apache Airflow for authoring, scheduling and monitoring Data Pipelines
- Extensively used ApacheKafka, ApacheSpark,HDFSand ApacheImpalato build a near real time data pipelines that get, transform, store and analyze click stream data to provide a better personalized user experience.
- Performed data extraction, transformation, loading, and integration in data warehouse, operational data stores and master data management
- Experienced in ETL concepts, building ETL solutions and Data modeling
- Worked on architecting the ETL transformation layers and writing spark jobs to do the processing.
- Aggregated daily sales team updates to send report to executives and to organize jobs running on Spark clusters
- Creating Reports in Looker based on Snowflake Connections
- Optimized the Tensor Flow Model for efficiency
- Create Spark code to processstreaming datafromKafkacluster and load the data to staging area for processing.
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team UsingTableau.
- Built performant, scalable ETL processes to load, cleanse and validate data
- Implemented business use case in Hadoop/Hive and visualized in Tableau
- Create data pipelines to use for business reports and process streaming data by using Kafka on premise cluster.
- Performed end-to-end delivery of PySpark ETL pipelines on Azure-databricks to perform the transformation of data orchestrated via Azure Data Factory (ADF) scheduled through Azure automation accounts and trigger them using Tidal Scheduler.
- Created data sharing between two snowflake accounts.
- Implement medium to large scale BI solutions on Azure using Azure Data Platform services (Azure Data Lake, Data Factory, Data Lake Analytics, Stream Analytics, Azure SQL DW, HDInsight/Databricks, NoSQL DB).
- Implement ad-hoc analysis solutions using Azure Data Lake Analytics/Store, HDInsight
- Analyzed the system for new enhancements/functionalities and perform Impact analysis of the application for implementing ETL changes.
- Designed several DAGs (Directed Acyclic Graph) for automating ETL pipelines
- Process the data from Kafka pipelines from topics and show the real time streaming in dashboards
- Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.
- Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.
- Worked on developingETL Workflowson the data obtained using Scala for processing it in HDFS andHBaseusing Oozie
- Migration of on premise data (Oracle/ SQL Server/ DB2/ MongoDB) to Azure Data Lake and Stored (ADLS) using Azure Data Factory (ADF V1/V2).
- Writing UNIX shell scripts to automate the jobs and scheduling cron jobs for job automation using commands with Crontab.
- Creating Spark clusters and configuring high concurrency clusters using Azure Databricks to speed up the preparation of high-quality data.
- Created Metric tables, End user views in Snowflake to feed data for Tableau refresh.
- Compiled data from various sources to perform complex analysis for actionable results
- Measured Efficiency of Hadoop/Hive environment ensuring SLA is met
- Implemented Copy activity, Custom Azure Data Factory Pipeline Activities
- Primarily involved in Data Migration using SQL, SQL Azure, Azure Storage, and Azure Data Factory, SSIS, PowerShell.
- Designed and implemented configurable data delivery pipeline for scheduled updates to customer facing data stores built with Python
- Proficient in Machine Learning techniques (Decision Trees, Linear/Logistic Regressors) and Statistical Modeling
Environment: Kafka, Impala, Pyspark, Azure, HDInsight, Data factory, Databricks, Datalake, Apache Beam, Cloud Shell, Tableau, Cloud Sql, MySQL, Posgres, Sql Server, Snowflake, Python, Scala, Spark, Hive, Spark-Sql, No Sql, MongoDB, TensorFlow, Jira.
Confidential
Data Engineer
Responsibilities:
- Experience in Job management using Fair scheduler and Developed job processing scripts using Oozie workflow.
- Worked and learned a great deal from AWS Cloud services like EC2, S3, EBS, RDS and VPC.
- Responsible for implementing ETL process through Kafka-Spark-HBase Integration as per the requirements of customer facing API
- Helped maintain and troubleshoot UNIX and Linux environment
- Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.
- Built pipelines to move hashed and un-hashed data from XML files to Data lake.
- Expertise in analyzing data using Pig scripting, Hive Queries, Sparks (python) and Impala.
- Experienced in writing live Real-time Processing using Spark Streaming with Kafka.
- Created Cassandra tables to store various data formats of data coming from different sources.
- Involved in importing the real time data to Hadoop using Kafka and implemented the Oozie job for daily imports.
- Developed Pig program for loading and filtering the streaming data into HDFS using Flume.
- Experienced in handling data from different datasets, join them and pre-process using Pig join operations.
- Developed HBase data model on top of HDFS data to perform real time analytics using Java API.
- Created Source to Target Mappings (STM) for the required tables by understanding the business requirements for the reports
- Used Spark, Hive for implementing the transformations need to join the daily ingested data to historic data.
- Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time.
- Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
- Developed PySpark and SparkSQL code to process the data in Apache Spark on Amazon EMR to perform the necessary transformations based on the STMs developed
Environment: Spark, Kafka, Hadoop, HDFS, Spark-SQL, AWS, Python, Map Reduce, Pig, Hive, Oracle 11g, My SQL, MongoDB, Hbase, Oozie, Zookeeper, Tableau.
Confidential
Hadoop Developer
Responsibilities:
- Involved in review of functional and non-functional requirements.
- Installed and configured Hadoop MapReduce, HDFS, Developed multiple MapReduce
- Created consumption views on top of metrics to reduce the running time for complex queries.
- The custom File System plugin allows Hadoop MapReduce programs, HBase, Pig and Hive to work unmodified and access files directly.
- Using Nebula Metadata, registered Business and Technical Datasets for corresponding SQL scripts
- Developed spark code and spark-SQL/streaming for faster testing and processing of data.
- Installed and configured Pig and also written Pig Latin scripts.
- Designed and implemented MapReduce-based large-scale parallel relation-learning system
- Setup and benchmarked Hadoop/HBase clusters for internal use
- Wrote MapReduce job using Pig Latin. Involved in ETL, Data Integration and Migration.
- Importing and exporting data into HDFS from Oracle Database and vice versa using sqoop.
- Imported data using Sqoop to load data from Oracle to HDFS on regular basis.
- Written Hive queries for data analysis to meet the business requirements.
- Experienced in working with spark ecosystem using Spark SQL and Scala queries on different formats like text file, CSV file.
Environment: Hadoop, MapReduce, HDFS, Hive, Java, Hadoop distribution of Cloudera, Pig, HBase, Linux, XML, Tableau, Eclipse, Oracle 10g, Toad
Confidential
Data Analyst
Responsibilities:
- Experience in developing POCs and highly distributed applications.
- Worked with Data Scientists, Big Data Analysts and Business Analysts from ITS teams to gather requirements.
- Involved in analyzing, planning, coding, debugging, testing and go-live phases of Big Data applications.
- Designed and developed Use Cases, Activity Diagrams, Sequence Diagrams using Visio.
- Performed technical analysis, validating, and enhancing the processes by incorporating vendors' requirements.
- Worked with the business group to document functional requirements for migration project and process enhancements including new features, functionality, version upgrades, and reporting requirements.
- Conducted workflow, process, and GAP analysis to derive requirements for existing systems enhancements.
- Worked with the business group to document functional requirements for enhancements.
- Coordinated with the various team in the analysis, design and implementation of enhancements.
- Performed data analysis and used SQL extensively for analysis and business logic validation.
- Prepared High level test scenarios and helped QA team in preparing and reviewing the test cases.
- Performed User Acceptance Testing (UAT) for Enhancement phase 1 and phase 2.
Environment: Linux, SQL, MS Office, MS Excel, MS Visio, Windows
