We provide IT Staff Augmentation Services!

Sr. Big Data Engineer Resume

2.00/5 (Submit Your Rating)

Edison, NJ

SUMMARY

  • A dynamic professional with 7+ years of diversified experience in the field of Information Technology with an emphasis on BigData, Spark and Java/J2EE Technologies and tools using industry accepted methodologies and procedures
  • Strong Knowledge of Hadoop Architecture and Daemons such as HDFS, Yarn, Job Tracker, Task Tracker, Name Node, Data Node and Map Reduce concepts.
  • Hands on experience in working with Ecosystems like Hive, Pig, Sqoop, Map Reduce, Flume, Oozie, Teradata, Spark SQL, AWS S3, Maven, Jenkins, Kafka. Strong knowledge of Pig and Hive’s analytical functions, extending Hive and Pig core functionality by writing custom UDFs.
  • Good Experience in designing the Akka FrameWorks, NIFI Flows for data routing, Transformations and Mediation logic
  • Highly Knowledgeable in streaming data from different data sources like Log files, JMS, applications sources into HDFS using Kafka sources interceptors and destinations.
  • Experience in using design patterns such as MVC, Singleton, Decorator, Composite, Command, Facade, Business Objects, Data Transfer Objects, Entity Objects, Data Access Objects and Delegates.Experience in open - source technologies such as Spring Framework, Hibernate Framework, Struts Framework, Maven, SVN, Apache, JUnit.
  • Thorough understanding of Database application designs and implementation using Oracle Database, MS SQL Server and MySQL.
  • Experience in validating and cleansing the data using Pig statements and hands-on experience in developing Pig MACROS.
  • Used Agile(SCRUM) methodologies for Software Development.
  • Working with onsite and offshore team members, mentoring junior team members and ability to work in team.
  • Possesses excellent work ethics and always like to explore new technology trends and apply them when the opportunity presents.

TECHNICAL SKILLS

Bigdata Ecosystem: Spark SQL, Spark Streaming, Map Reduce, Hive, Nifi, Pig, Sqoop, Kafka, Zookeeper, Oozie, YARN

Operating Systems: Windows XP,Linux Distro(Ubuntu,Mint,Fedora)

Languages: Java, Scala, Python

Frame Works and WebTechnologies: Akka, Play, Spring, Hibernate and HTML5,XML,JavaScript

Scripting Language: Unix Shell Scripting, Python Scripting

RDBMS DB: Oracle,MySQL, Teradata

NOSQL DB: Hbase,Cassandra

Servers: Tomcat, Amazon Web Services(AWS)

Tools: Maven, Jenkins, SVN Repository, GIT

Web Services: Restful Services, SOAP Services

Testing Tools: Selenium Web Driver, Cucumber, TestNG, JUnit

PROFESSIONAL EXPERIENCE

Sr. Big Data Engineer

Confidential, Edison, NJ

Responsibilities:

  • Involved in writing Spark Core Applications and Spark-SQL/Streaming for faster testing and processing of Data.
  • Experience working with real time /near real time analytics and big data platforms like Hadoop, Spark using programming languages like Scala
  • Implemented the new household data fix in Hadoop tables using the data from Experian and Teradata tables.
  • Maintained the data in Teradata and Hadoop tables. Developed workflows to automate the row count differences Hadoop and Teradata tables.
  • Leveraged Sqoop to ingest data into the datalake.
  • Created topics in Kafka broker which gets the data from sources with the help of NIFI and Spark job consumes it and pushes it into HDFS.
  • Experience in selenium automation using Selenium Web Driver, Cucumber, JUnit, TestNG, Jenkins.
  • Experience in preparation of Test procedures, Test Scenarios, Test Cases and Test Data.
  • Exploring with Spark improving the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frame, Pair RDD’s, Spark YARN
  • Responsible for design and creation of Hive tables, partitioning, bucketing, loading data and writing hive queries. Implemented existing Hive Script in Spark Scala for better Performance.
  • Created RESTful services,converted data formats to make it consumable by the created services.
  • Implemented innovative solutions using various Hadoop ecosystem tools like Pig, Hive, Sqoop, Kafka, HBase and Spark
  • Plan, deploy, monitor and maintain Amazon AWS S3 cloud infrastructure consisting of multiple EC2 nodes and VMWare Vm’s as required in the environment.
  • Expertise in AWS data migration between different database platforms like SQL Server to Amazon Aurora using RDS tool.
  • Configured Spark streaming to receive real time data from the Kafka and Store the Stream data to HDFS using Scala.
  • Experience integration of Kafka with Spark for real time data processing.
  • Experience in validating and cleansing the data using Pig statements and hands-on experience in developing Pig MACROS.
  • Good experience in Hive partitioning, bucketing and perform different types of joins on Hive tables and implementing Hive serdes like REGEX, JSON and Avro.
  • Used Agile(SCRUM) methodologies for Software Development.

Environment: Spark SQL, Scala, Kafka, Nifi, Amazon Web Services(AWS),Selenium, Teradata, Hive, Pig, HDFS, RESTful Services, Oozie, Maven, Jenkins

Big Data/Spark Consultant

Confidential, New Jersey

Responsibilities:

  • Developed high Performance distributed queueing system using Scala, Redis, Akka, Closure, Json
  • Kerberos security was implemented to safeguard the cluster, no compromise on data security and security polices.
  • Execution of Selenium Test Cases and Reporting Defects.
  • Involved in Regression Testing and Automation Infrastructure Development using Selenium.
  • Developed multiple POCs using PySpark and deployed on the YARN Cluster, compared the performance of Spark with Hive and SQL/Teradata
  • Moving Bulk amount of data into HBase using Map Reduce Integration.
  • Developed Pig Programs for loading and filtering the streaming data into HDFS using Flume.
  • Developed HBase data model on top of HDFS data to perform real time analytics using Scala API.
  • Used tools like MapReduce and Spark with Scala Streaming for performing operations like ClickStream Analysis and to perform Analysis on batch Data.
  • Experienced in handling Avro data files by passing schema into HDFS using Avro tools and Map Reduce.
  • Optimizing the Hive queries using Partitioning and Bucketing techniques, for controlling the data distribution.
  • Visualised data by preparing viz boards using tools like Platfora and Tableau.
  • Wrote custom Map reduce programs to analyse the data and provide insights
  • Extensively used DataStage Manager to Export/Import DataStage job components and Import plug in table Definitions from DB2UDB, Oracle and Sybase databases.
  • Used Oozie scheduler to automate the pipeline workflow and orchestrate the Map Reduce, Nifi, hive and pig jobs that extract the data on a timely manner.
  • Involved in story-driven agile development methodology and actively participated in daily scrum meetings.

Environment: MapReduce, Spark with Scala, Hive, Pig, Nifi, Teradata, Oozie, HBase, Platfora, Redis, RestFul Services, Linux, Kerberos, Maven, Selenium, Akka, Jenkins, HDFS.

Hadoop Developer

Confidential

Responsibilities:

  • Experience in Importing and exporting data into HDFS and Hive using Sqoop.
  • Developed Pig program for loading and filtering the streaming data into HDFS using Flume.
  • Experienced in handling data from different data sets, join them and pre process using Pig join operations.
  • Moving Bulk amount data into HBase using Map Reduce Integration.
  • Developed Map-Reduce programs to clean and aggregate the data
  • Developed HBase data model on top of HDFS data to perform real time analytics using Scala API.
  • Developed different kind of custom filters and handled pre-defined filters on HBase data using API.
  • Imported and exported data from Teradata to HDFS and vice-versa.
  • Strong understanding of Hadoop eco system such as HDFS, MapReduce, HBase, Zookeeper, Pig, Hadoop streaming, Sqoop, Oozie and Hive
  • Implement counters on HBase data to count total records on different tables.
  • Experienced in handling Avro data files by passing schema into HDFS using Avro tools and Map Reduce.
  • Worked on custom Pig Loaders and Storage classes to work with a variety of data formats such as JSON, Compressed CSV, etc.
  • We used Amazon Web Services to perform big data analytics.
  • Implemented Secondary sorting to sort reducer output globally in map reduce.
  • Implemented data pipeline by chaining multiple mappers by using Chained Mapper.
  • Created Hive Dynamic partitions to load time series data
  • Experienced in handling different types of joins in Hive like Map joins, bucker map joins, sorted bucket map joins.
  • Created tables, partitions, buckets and perform analytics using Hive ad-hoc queries.
  • Experienced import/export data into HDFS/Hive from relational data base and Tera data using Sqoop.
  • Handling continuous streaming data comes from different sources using flume and set destination as HDFS.
  • Integrated spring schedulers with Oozie client as beans to handle cron jobs.
  • Experience with CDH distribution and Cloudera Manager to manage and monitor Hadoop clusters
  • Actively participated in software development lifecycle (scope, design, implement, deploy, test), including design and code reviews.
  • Involved in story-driven agile development methodology and actively participated in daily scrum meetings.

Environment: Hadoop, HDFS, Map Reduce, Hive, Pig, Hbase, Sqoop, RDBMS/DB, Flat files, Teradata, Mysql, Oozie, Jenkins, Maven, Svn Repository.

Java Developer

Confidential

Responsibilities:

  • Understanding andanalyzingthe requirements.
  • Designed, developedand validatedUser Interface using HTML, Java Script, XML.
  • Handled the database access by implementing Controller Servlet.
  • Implemented PL/SQL stored procedures and triggers.
  • Used JDBC prepared statements to call from Servlets for database access.
  • Designed and documented of the stored procedures.
  • Expertise in implementation of Automation FrameWork using Selenium, LoadRunner, UFT.
  • Ability to develop BDD/TDD scripts with Cucumber and writing step definitions for behavior.
  • Experience in writing unit test cases using Junit, Mockito, PowerMock using Cucumber and Gherkin.
  • Used cucumber to implement behavior driven design for acceptance tests and unit testing.
  • Prepared test case scenarios and internal documentation for validation and reporting.
  • Coordinating with QA team and resolving the QA defects
  • Widely used HTML for web based design.
  • Involved in Unit testing for various components.
  • Worked on database interaction layer for insertions, updating and retrieval operations of data from oracle database by writing stored procedures.
  • Used Data Access Object (DAO) pattern to introduce an abstraction layer between the business logic tier (Business object) and the persistent storage tier (data source).
  • Used Spring Framework for Dependency Injection and integrated with Hibernate.
  • Involved in writing JUnit Test Cases.
  • Used Log4J for any errors in the application

Environment: Java,Spring Framework, J2EE, HTML, JUnit, XML,JavaScript, Eclipse, WebLogic, PL/SQL and Oracle,Selenium, Cucumber.

We'd love your feedback!