We provide IT Staff Augmentation Services!

Big Data Engineer Resume

0/5 (Submit Your Rating)

Malvern, PA

SUMMARY

  • An Experienced Bigdata Engineer with Solid Foundation in Computer Science and Engineering (BTech, MS) with specialization in technologies such as Big Data, Data science and Cloud Computing. Certified on Spark and AWS (Amazon Web Services).
  • Looking for an opportunity where I can leverage my Experience, Expertise and Skills.
  • 8+ years of professional experience in IT in Analysis, Design, Development, Testing, Documentation, Deployment, Integration, and Maintenance of web, Client/Server and distributed applications using Java, Big Data (Hadoop & Spark) and Cloud Computing (AWS) technologies.
  • 4+ years of experience on Big data technologies such as HDFS, MapReduce, YARN, Pig, Hive, Sqoop, HBase, Flume, Kafka, Oozie, Zookeeper.
  • 6 months hands on experience on AWS (EMR, S3, EC2, CloudFormation).
  • 2+ years hands on experience on Spark (Spark SQL, Spark Streaming).
  • 1+ year hands on experience on Tableau, Zeppelin.
  • 1+ year of experience on Impala.
  • Databricks Developer Certification on Apache Spark.
  • AWS (Amazon Web Services) Certified Developer.
  • Expertise in components of Hadoop ecosystem like HDFS, MapReduce, YARN, Pig, Hive, HBase, Sqoop, Oozie, Zookeeper, Impala, Hue.
  • Good Understanding of Hadoop architecture and experience with Hadoop components such as Job Tracker, TaskTracker, NameNode, DataNode and MapReduce concepts and HDFS Framework.
  • Experience in writing Spark SQL scripts and implementing Spark RDD transformations and actions using Scala/ Python.
  • Hands on experience in importing data using Sqoop from Enterprise DB2 to AWS S3.
  • Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice - versa.
  • Worked on Scala, Python and Java to create frameworks for processing data pipelines throughSpark.
  • Hands on experience in writing Spark SQL scripts and implementing Spark RDD transformations and actions using Scala/ Python.
  • Hands on experience working on Tableau and Zeppelin for Data Visualization and Analytics.
  • Hands on experience in developing customized UDFs to extend Hive functionality.
  • Good knowledge on Spark GraphX and Spark MLlib.
  • Hands on experience in developing MapReduce programs Hadoopfor analyzing the Big Data.
  • Hands on experience with data analytics using R and Python.
  • Experience in CoreJavawith strong understanding and working knowledge of Object Oriented programming and concepts like Collections, Multi-Threading, Exception Handling and Polymorphism.
  • Involved in pre-sales activities like requirement gathering & effort estimation.

TECHNICAL SKILLS

Big Data Technologies: HDFS, MapReduce, YARN, Pig, Hive, Sqoop, HBase, Oozie, Impala, Zookeeper, Flume, Kafka, Hue

Spark: SQL, RDDs, Dataframes

AWS Cloud Services: EMR, S3, EC2, CloudFormation

Languages: Scala, Python, Perl, R, ABAP/4, C/C++, Java

Databases: Oracle 8.0 & 9i, SQL, NoSQL

Operating Systems: Linux, UNIX, Ubuntu, Windows 9x/ XP / NT / 2000, MS DOS

Web Technologies: JSP, HTML, JavaScript

Design Tools: ER-Diagram, Sequence Diagrams, Business Process Models, Flow Charts etc.

Other Tools: Eclipse, GitHub, Spring Tool Suite, NoMachine, PuTTY, Bitbucket, Bamboo, Antiphony, SQL Developer, Jenkins

Data Visualization / Analytics: Tableau, Zeppelin

Environment: CDH 4 & 5

ERP: SAP R/3

PROFESSIONAL EXPERIENCE

Confidential, Malvern, PA

Big Data Engineer

Responsibilities:

  • Identifying and validating the fine-grained requirements along with Technical and Business Users, developing data mapping and transformation documents, preparing detailed design document, developing the software using Big Data processing engine Spark API in Python programming language, performing integration testing, supporting User Acceptance testing, preparing go-live checklists and hand-off release documents to infrastructure support team.
  • Creating and scheduling Control-M jobs to spin up and spin down Amazon EMR clusters for the Share of Wallet (SOW), Flagship Personal Advisory Services (Flagship PAS), Retail Personal Advisory Services (Retail PAS) and Retirement Engagement Indicator (REI) applications.
  • Installing, configuring and setup Bitbucket code repository. Creating the source code branches and resolving the conflicts. Using the branches to validate and deploy the code in production environment.
  • Developing Python programs in Spring Tool Suite (STS) java platform using Maven project management and pushing the code from STS to Bitbucket.
  • Developing Infrastructure related code in Bitbucket INF repository to spin up the Amazon EMR cluster.
  • Developing Application related code in Bitbucket APP repository to run the application in the Amazon EMR cluster.
  • Importing the data using Sqoop from Enterprise DB2 to AWS S3.
  • Managing Amazon EMR cluster connection by creating and configuring Amazon EC2 key pairs.
  • Spinning up Amazon EMR clusters as necessary by creating the stack in Bamboo.
  • Spinning down Amazon EMR clusters as necessary by deleting the stack in Bamboo to clean up the resources that are used for the application.
  • Developing shell scripts to run the automated Oozie jobs.
  • Installing and Setup Splunk software to monitor the events and send necessary notifications/email alerts.
  • Developing SQL queries in Py-Saprk and run the applications using spark-submit command in NoMachine remote desktop or PuTTY terminal emulator when the Amazon EMR cluster is up.
  • Monitoring the Amazon EMR cluster using Antiphony to proactively identify and fix the issues.
  • Monitoring Spark web User Interface (UI) for performance, carryout Memory Tuning, Garbage Collection (Java GC) tuning, inspect the logs and make the necessary changes to the code.
  • Developing Spark Scripts using Py-Saprk, Spark SQL to access hive tables in spark for faster data processing.
  • Developing Py-Saprk program which converts test data from Impala to Hive (Impala to Hive conversion utility).
  • Connecting to Hue to check Hive queries and table schema details.

Environment: AWS (EMR, S3, EC2, CloudFormation), Hive, Hue, Spring Tool Suite, NoMachine, PuTTY, Bitbucket, Bamboo, Antiphony, SQL Developer.

Confidential, Phoenix, AZ

Senior Bigdata Engineer

Responsibilities:

  • Gathering the requirements from the business users and performing the requirement analysis.
  • Experience in leading a team technically and assigning the tasks to the team and updating the development and the testing status to the manager in a day-to-day basis.
  • Worked on analyzing Hadoop cluster and different big data tools including Pig, HBase and Sqoop.
  • Used Sqoop to import and export data from HDFS to RDBMS and vice-versa.
  • Created HBase tables to store various data formats of data coming from different portfolios.
  • Used Impala for performance upgrade for hive scripts.
  • Used Cloudera Manager for installation and management of Hadoop Cluster.
  • Wrote Pig scripts to run ETL jobs on the data in HDFS.
  • Involved in scheduling Oozie workflow engine to run multiple Hive and pig jobs.
  • Extensively used Spark for optimizing of existing process in Hadoop using Spark Context, Spark SQL, Data Frames and pair RDDs.
  • Developed Spark core and Spark SQL scripts using Scala for faster data processing.
  • Developed Spark scripts to import large files from Amazon S3 buckets.
  • DevelopedSparkapplications in python(Py-Saprk) on distributed environment.
  • Configured Spark Streaming to receive real time data from Kafka and store the stream data to HDFS.
  • Extensively worked on Spark by using notebook-style Zeppelin for data visualization.
  • Worked with BI teams in generating the reports in Tableau.
  • Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.

Environment: Hadoop, MapReduce, Hive, HDFS, Pig, Sqoop, Oozie, Spark, Flume, Kafka, HBase, Impala, ZooKeeper, Tableau, Zeppelin, CDH 5, Oracle, NoSQL and Unix/Linux.

Confidential, Princeton. NJ

Hadoop Consultant

Responsibilities:

  • Worked on Cloudera CDH 5.4.0 cluster for application development and Hadoop tools like Hive, Pig, HBase, Zookeeper and Sqoop.
  • Internally trained team members on Hadoop Ecosystem.
  • Developed data pipeline using Sqoop, Pig and Hive to structure customer behavioral data and financial histories in HDFS for analysis.
  • Did optimization of Hive by converting input data from Text file to ORC achieving 78% compression and faster execution.
  • Involved in collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis.
  • Involved in planning infrastructure and design to move from Flume to Kafka in future incorporating fault tolerance and multiple data sources.
  • Used Pig as ETL tool to do transformations, event joins and some pre-aggregations and storing the data onto HDFS.
  • Installed Oozie workflow engine to run multiple Sqoop, Hive and Pig Jobs.
  • Used Sqoop to import and export data from HDFS to RDBMS and vice-versa.
  • Created Hive tables and involved in data loading and writing Hive UDFs.
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
  • Supported in setting up QA environment and updating configurations for implementing scripts with Pig and Sqoop.

Environment: Hadoop, MapReduce, Hive, HDFS, Pig, Sqoop, Oozie, Flume, Kafka, Zookeeper, CDH 5, Oracle, NoSQL and Unix/Linux.

Confidential, Atlanta, GA

Hadoop Consultant

Responsibilities:

  • Worked on analyzing Hadoop cluster and different big data tools including Pig, HBase and Sqoop.
  • Installed and configured Hadoop MapReduce, HDFS, Developed multiple MapReduce jobs in java for data cleaning and pre-processing.
  • Developed MapReduce programs for applying business rules on the data.
  • Used Sqoop to import and export data from HDFS to RDBMS and vice-versa.
  • Created Hive external tables and loaded the data into tables and query data using HQL.
  • Wrote Pig scripts to run ETL jobs on the data in HDFS.
  • Created Pig UDF's to implement business logic in Hadoop.
  • Used Cloudera Manager for installation and management of Hadoop Cluster.
  • Involved in Installing the Oozie workflow engine to run multiple Hive and Pig jobs.
  • Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required.

Environment: Hadoop, MapReduce, Hive, HDFS, Pig, Sqoop, HBase, Oozie, CDH 4 & 5, Oracle, NoSQL and Unix/Linux.

Confidential, Atlanta, GA

Senior Java Developer

Responsibilities:

  • Implemented MVC architecture to develop web application using Struts, Hibernate framework
  • Planned and involved in the Unit and QA testing’s.
  • Prepared the WS design and DDD (Detailed Design Document) Documents.
  • Used Struts validation framework, customized default error messages by creating message resource bundle.
  • Conducted the performance testing using the JProbe tool to identify the memory leaks, unused and useless java objects.
  • Daily and Weekly status/progress to the IT manager.
  • Developed the project using Agile methodologies and Test-Driven Development.
  • Designed XML configuration files for Action Mapping as well as Tiles Definition.
  • Designed and developed Struts Configuration files as well as Controller classes for the implementation of the business logic in the Struts Framework.
  • Designed and Developed Java persistence classes objects for storing the data in the database using Hibernate OR mappings.
  • Utilized Hibernate for Object3/Relational Mapping purposes for transparent persistence onto the database using the Stateless and session beans (EJB 3.0).
  • Developed SOA architecture based WebServices using SOAP and WSDL.
  • Wrote and used SOAP for sending and getting data from the external interface.
  • Consistently produced the highest quality code through disciplined and rigorous unit testing using Junit.
  • Extensively used Eclipse IDE while writing code & used Sub Version for version control.
  • Used Log4j for logging and debugging.
  • Deployed application in Tomcat Server.

Environment: Java 1.5, SOA, Web Services, JSP, JavaScript, XML, DB2, Eclipse IDE, SOAP, WSDL, UNIX, Sub Version, Junit, Log4j, JProbe, Tomcat Server, Jenkins, Opsware.

Confidential

Java Developer

Responsibilities:

  • Gathering the requirements from the business and performing the requirement analysis.
  • Designed and developed all the Web pages using Struts Tags, JSP, HTML and validating the client data using java script.
  • Internationalized the application for Canada and Spanish population through resource bundle.
  • Developed Action Form classes as well as Dispatch Action classes to process each logical request and forwarded through appropriate Action Mapping.
  • Designed XML configuration files for Action Mapping as well as Tiles Definition.
  • Designed and developed controller classes to implement all the business logic.
  • Developed SOA architecture based WebService calls to make communication with external systems.
  • Deployed the applications on Tomcat Server 6.0 application server in the AD and QA regions.
  • Maintained the version control of the source code using IBM Clear Case.
  • Performed the unit testing, integration testing and system testing of the applications.
  • Coordinate with the QA team for the QA testing of the applications and defect fixes.
  • Software configuration management using Clear case and Clear Quest.
  • Deployed the packages in the AD and QA regions using Opsware SAS client.

Environment: Java, J2EE, JSP 2.0, Web Services (Apache Axis), XML, Log4j, XML, Eclipse 3.5, Junit, Find Bugs, JProbe, Oracle, Tomcat Server 6.0, Jenkins, Opsware.

We'd love your feedback!