Sr. Bigdata Engineer Resume
3.00/5 (Submit Your Rating)
SUMMARY
- 14+ Years of extensive IT experience in Analysis, Architecture, Design, Development, Testing, Maintenance, and User training of software applications which including 4+ years of experience working on Apache Hadoop ecosystem and Apache Spark and over 10+ Years of experience in Java/J2EE.
- Hands on experience in developing and deploying enterprise - based applications using major components in Hadoop ecosystem such as Hadoop 2.x, YARN, Hive, Pig, Map Reduce, Sqoop, Spark, Scala, Kafka, Oozie.
- Good knowledge in handling messaging services using Apache Kafka.
- Experienced with the Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
- Expertise in using Spark-SQL with various data sources like JSON, Parquet and Hive.
- Experience in usage of Hadoop distribution like Cloudera and Horton Works distribution.
- Experience in transferring data from RDBMS to HDFS and HIVE table using Sqoop.
- Experience in creating tables, partitioning, bucketing, loading and aggregation using HIVE.
- Migrating the code from Hive to Spark/PySpark and Scala/Python using Spark-SQL and Spark Windows function.
- Extensive experience in Spring Core, Spring IOC, Spring MVC, Spring Web Flow, Spring Batch, Spring Security, Spring Boot for micro-services, Hibernate framework, iBatis and AJAX.
- Experience in writing Apache MAVEN, and Log4J and JUnit for unit testing.
- Extensive experience in developing Use Cases, Activity Diagrams, Sequence Diagrams and Class Diagrams using Visio.
- Sun Certified Java Programmer (SCJP 1.5).
- Work with business analysts to understand problems and provided architecture optimal solution.
- Experience working in environments using Agile (SCRUM) and BDD (Behavior-Driven Development), Test-Driven development methodologies.
- Excellent team player with good communication, people and leadership skills.
PROFESSIONAL EXPERIENCE
Confidential
Sr. Bigdata Engineer
Responsibilities:
- Responsible for developing data pipeline using Spark, Scala, Apache Kafka to ingestion the data from CSL source and store in HDFS protected folder.
- Implemented many Kafka ingestion jobs to consume the real time data processing and batch processing.
- Used HBase for storing the Kafka topic, partition number and Offsets value. Also used phoenix jar to connect HBase table.
- Used PySpark to creating batch job for merge multiple small files (Kafka stream files) into single larger files in parquet format.
- All Spark/PySpark jobs we are implemented Progtegrity API for writing & reading PCI/PII data from HDFS location or Hive table.
- Implemented multiple function in PySpark program like 'UnionAll' function to combine the two Dataset & remove duplicates.
- Implemented on spark using Scala/Java custom function for map object.
- Developed Autosys scripts to schedule the Kafka streaming and batch job.
- Involved in creating Hive tables and loading and analyzing data using hive queries.
- Used Ambari to monitor node’s health and status of the jobs in Hadoop clusters.
- Used Rally for user-story/bug tracking and Bit Bucket to check-in and checkout code changes.
Confidential
Bigdata Developer & Java Tech Lead
Responsibilities:
- Responsible for building scalable distributed data solutions using Hadoop Eco system and Spark.
- Developed Spark applications for the entire batch processing by using Scala.
- Imported data from different sources into Spark RDD for processing. Developed custom aggregate functions using Spark SQL and performed interactive querying.
- Utilized spark data frame and spark sqlapi extensively for all the processing.
- Integrated Kafka with Spark Streaming for real time data processing.
- Experience in managing and reviewing Hadoop log files. Experience in hive partitioning, bucketing and perform joins on hive tables.
- Importing and exporting the analyzed data to the relational databases into HDFS using Sqoop.
- New library development with microservices architecture using Rest APIs, spring boot, Pivotal Cloud Foundry and AWS.
- Create and configured the continuous delivery pipelines for deploying microservices using Jenkins.