SUMMARY:

• Experienced with a programming background in C#, Java, and HTML that has now transitioned to the field of big data engineer. Also been involved in projects that deals with integrating robotics with the Unity game engine. Now is involved with learning all the operations, functions, and architecture of Hadoop.
• Improving SQL skills and developing in the other Relational Database Management System which includes PostgreSQL and MSSQL Server.
• Grown accustomed to working on Linux using a Ubuntu machine as well as working with Google Cloud Platform.
• Created two instances in GCP where one is for development and the other for production.
• Experience with working in the Hadoop ecosystem, understanding the architecture, and performing a project involving transferring data.
• Research on data that involves calculating video game sales that spans across the world.
• Altered the video game sales data in SQL and Microsoft Excel to make comparisons and separate them in different RDBMs.
• Creation of a HDFS and implementing RDMBs into HDFS using the Sqoop tool.
• Using Spark to improve Python programming skills and understanding the architecture of Spark.
• Hands on experience with Hive and HBase integration, understanding the knowledge of how different they are from one another.
• Experience using Flume to collect a huge amount of data from Twitter.
• Being introduce to Scala and getting familiarity of programming in Java
• Developed a script that completes the Hadoop ecosystem from going to HDFS to HBase all in one execution.
• Experience with using Spark for a project that deals with going from HDFS to hive and back to a RDMBs
• Worked on other file formats such as Avro and ORC
• Hands - on experience with MongoDB and learned its differences from HBase and Hive
• Installed and used Airflow for my project that involves using Spark
• Worked on Kafka to gather API and displayed them in a terminal
• Getting hands-on wi

PROFESSIONAL EXPERIENCE:

Confidential

Big Data Engineer

Responsibilities:

• Configuration, assemble and dispatch new information pipelines underway utilizing Apache Spark. Attempted to help existing cycles running underway. Characterize to and oversee SLA for all data collections in designated zones of proprietorship. Work with information framework to emergency infra issues and drive to goal. Developed python scripts to move data from local computer to AWS S3 Bucket. Developed python scripts to retrieve data stored in AWS S3 Bucket. Designed an Apache Airflow Data Pipeline to automate the data ingestion and retrieval. Composing Python classes to stack information from Kafka to
• MongoDB according to the ideal model. Composing Scala classes to extricate information from MongoDB, since throughout the time mongo will turn into the wellspring of truth. Composing Scala classes for functions that is getting Kafka messages each time another information is added to or refreshed in MongoDB. Merge information among Oracle and MongoDB, utilizing the Spark system. Performing Unit Testing utilizing Pyunit, mock items. Composing and executing mongo queries. Working with business on prioritization of any upgrade solicitation or bug fix. Giving over the expectations to QE and UAT group.
• Worked on various real - time and batch processing applications using Spark/Scala, Kafka and Cassandra. Building Spark applications to perform data enrichments and transformations using Spark Data frames with Cassandra look ups. Used Data Stax Spark Cassandra Connector to extract and load data to/from Cassandra. Worked in a team to develop an ETL pipeline which involves extraction of Parquet serialized files from S3 and persist them in HDFS. Developed Spark application that uses Kafka Consumer and Broker libraries to connect to Apache Kafka and consume data from the topics and ingest them into Cassandra.

Confidential

Sr. Big Data Engineer

Responsibilities:

• Installed Kafka to gather data from disperse sources and store for consumption. Familiar with diverse IDE's like PyCharm for majority of project testing and coding. Consume rest APIs and develop added source code so it can be used for the Kafka program. Another python file was create called consumer which calls on the producer and prints the same data the producer displays. Worked on various real - time and batch processing applications using Spark/Scala, Kafka and Cassandra. Building Spark applications to perform data enrichments and transformations using Spark Data frames with Cassandra look ups.
• Used Data Stax Spark Cassandra Connector to extract and load data to/from Cassandra. Worked in a team to develop an ETL pipeline which involves extraction of Parquet serialized files from S3 and persist them in HDFS. Developed Spark application that uses Kafka Consumer and Broker libraries to connect to Apache Kafka and consume data from the topics and ingest them into Cassandra. Leveraged strong Skills in developing applications involving Big Data technologies like Hadoop, Spark, Map Reduce, Yarn, Hive, Pig, Kafka, Oozie, Sqoop, Hue. Experienced in Cloudera and Hortonworks Hadoop distributions.
• Experienced in Java, Scala and Python programming languages. Worked on Apache Airflow, Apache Oozie and Azkaban. Experience in designing and implementing the data ingestion framework by to load the data into the data lake for analytical purposes. Has good knowledge on ETL development for processing the large-scale data using Big data platform. Have developed the data pipelines using Hive, Pig and MapReduce. Hands on experience in writing Map Reduce jobs. Have experience in administrating clusters in the Hadoop ecosystem. Experience in installing and configuring the Hadoop Cluster of Major Hadoop Distributions. Designed the reporting application that uses the Spark SQL to fetch and generate reports on Hive. Good experience in analyzing data using Hive and have written User Defined Functions (UDF's).

Confidential

Big Data Engineer

Responsibilities:

• Used csv files to import RDBMS to Sqoop Pipeline project while keeping the same format. Created script to convert csv files into Avro while delivering them to HDFS. Generated an Avro schema for Hive external files/tables. Transferred the Avro files into the Hive external Raw database using a HQL. Converting
• Avro files into ORC files and placing them into the internal database. Create a new database in MySQL for that will be the location for the final phase in the project. Used Spark transfer the tables from Hive internal to MySQL. Installed and started Airflow for we will use airflow the run the script for everything that has been completed. Created a DAG folder and a python file that would read the script to do all commands that has been done throughout the project. Connect to airflow using "localhost:8080" and navigate through airflow to run the DAG script. The script ran faster in airflow than it would in a local terminal and the logs were displayed in a graph.
• Designed frameworks at High level and detailed level for sales and customer reports. Successfully Completed migration of on - premise multiple Hadoop and Spark jobs into AWS. Designed and Developed multiple Hourly, Daily, Weekly aggregates required for sales and customer dashboard in Redshift. Developed DAGs Implemented using Apache Airflow for multiple Spark jobs. Involved in the requirement gathering, Sprint planning, user story building session with Agile methodologies. Data optimization, Performance improvement, designing scalable application for further business analysis. Proficient knowledge on development, migration, enhancement and support Activities in Data Warehousing/BI. Automated the validation to analyse the transactional data. Work experience with CICD Pipeline using Git and Jenkins.

Confidential

Data Engineer

Responsibilities:

• Plan and build up a day by day cycle to do steady import of crude information from DB2 into Hive tables utilizing Sqoop. Engaged with troubleshooting Map Reduce work utilizing MR Unit. Broadly utilized Hive/HQL to query information in Hive Tables and stacked information into Hive. Created data pipeline utilizing Flume, Sqoop, Spark and MapReduce to ingest information into HDFS for examination. Utilized Oozie and Zookeeper for work process planning and monitoring. Adequately utilized Sqoop to move information from data sets (SQL, Oracle) to HDFS, Hive. Incorporated Apache Storm with Kafka to perform web examination. Transferred click stream information from Kafka to HDFS, Hbase and Hive by incorporating with Storm. Planned Hive external tables utilizing shared meta - store with dynamic partitioning and buckets. Taken a shot at Big Data Integration & Analytics dependent on Hadoop, ELKSpark, Kafka, and web Methods. Dealt with moving programs into Spark changes utilizing Spark and Scala. Plan and actualize ETL measure utilizing Talend to stack information from Worked widely with Sqoop for bringing in and trading the information from HDFS to Relational Database frameworks/centralized server and the other way around. Stacking information into HDFS. Made simultaneous access for hive tables with shared/elite locks. Actualized utilizing SCALA and SQL for quicker testing and preparing of information. Constant streaming the information utilizing with KAFKA. Utilized OOZIE for group preparing and booking work processes powerfully. Dealt with making End-End information pipeline organization utilizing Oozie. Populated HDFS and Cassandra with enormous measures of information utilizing Apache Kafka. Using Sqoop to transfer all the tables and their data into HDFS into specific directories that were created for each one. An extra table was added into HDFS and it was original just an ordinary csv file converted into a table with data. After Sqoop the tables were sent to Hive (a data warehouse software) in 2 databases that I created in it. One is called "raw" which is for external tables and "dsl" for internal tables. The last step in this project was to put the tables from internal tables into a distributed big data HBase. Executed commands to create tables in HBase and call back through Hive.