We provide IT Staff Augmentation Services!

Big Data Engineer Resume

4.00/5 (Submit Your Rating)

Cincinnati, OhiO

SUMMARY:

  • Professional IT experience this includes recent experience in Big Data/Hadoop Ecosystem
  • Competence in using various Hadoop components such as MapReduce (MR1),YARN(MR2), HDFS, MR2, Kafka, Pig, Hive, Sqoop, HBase, Impala, Spark Streaming, Flume, Spark SQL, ZooKeeper, Oozie, Hue
  • Extensive knowledge on Hadoop architecture and Hands - on experience with Hadoop components such as Job Tracker, Task Tracker, Name Node, Data Node and Map Reduce concepts and HDFS Framework
  • Experienced in using Spark API over Hadoop Map Reduce to perform analytics on data sets
  • Experience in converting Hive queries into Spark transformations using Spark RDDs
  • Good experience in data loading from Oracle and MySQL databases to HDFS system using Sqoop (Structure Data) and Flume (Log Files & XML)
  • Hands on experience with in-memory data processing using Apache Spark applications with Python utilizing the data frame and Spark SQL API
  • Experience working with Relational Database Management Systems (RDMS)
  • Capable of processing large sets of structured, semi-structured and unstructured data and supporting systems application architecture
  • Experience in tools such as GIT, Subversion
  • Experience working with Horton Works Distribution and Cloudera Distribution
  • Experience with web-based UI development using JQuery, CSS, HTML, HTML5, XHTML JavaScript and bootstrap
  • Good experience working in Agile/Scrum development environment participated in technical discussions with client and contributed to project analysis and development specs
  • Exposure in GraphX, Mlib (Machine Learning Algortithm K-Means, Naïve Bayes)
  • Exposure to AWS stack (EC2, S3, RDS, and Lambda) focused on high-availability, fault tolerance environment
  • Excellent communication & problem solving skills with the ability to work independently or in a team environment
  • Experience in using CI tools such as Hudson, Jenkins, Nagios, Docker for continuous integration and for End to End automation for all build and deployments
  • Highly motivated and passionate to learn new technologies quickly
  • Exposure to Machine learning like collaborative filtering, clustering, classification

TECHNICAL SKILLS:

NOSQL: HBASE, Cassandra

Java Technologies and Framework: J2EE, JDBC, JSP, Struts, Hibernate

Languages: Java, Python, Scala, R, SQL

Web Technologies: JavaScript, CSS, CSS3, HTML, HTML5, Bootstrap, JQuery

Databases: Oracle, MySQL

Web Servers: Weblogic, Apache Tomcat, Glassfish, Weblogic

IDE Tools: Eclipse, NetBeans, Intellij

Build Tools: Maven, Scala Build Tools (SBT), ANT

Operating Systems: Linux (Red Hat, Ubuntu, Centos), Windows7

WORK EXPERIENCE:

Big Data Engineer

Confidential, Cincinnati, Ohio

Responsibilities:

  • Created scripts to load data from UNIX local file system to HDFS
  • Extracted data from different databases and copied into HDFS using Sqoop
  • Implemented Flume, Spark framework for real time data processing
  • Created the Spark streaming code to take the source files as input
  • Used Oozie workflow to automate all the jobs
  • Exported the analyzed data into relational databases using Scoop for visualization and to generate reports for BI team
  • Used Hive to Partition and Bucketing, for optimizing data storage and compute-various metrics for reporting
  • Created tables, oplogs and source logs for data ingestion in BIGSQL(IBM)
  • Involved in developing Hive DDLs to create, drop and alter Hive tables
  • Created hive queries and UDFs for loading and processing data
  • Developed Spark programs using Scala; involved in creating Spark SQL queries and developed Oozie workflow for Spark jobs
  • Responsible for cluster maintenance, monitoring, managing, commissioning and decommissioning data nodes.
  • .88;8/;7
  • Troubleshooting and review data backups. Manage & review log files.
  • Worked on weekend in L3 for production issue and monitored data lake using HDP(Horton Works)
  • Used Agile methodology in developing the application; participated in standup meetings and Sprints

Tools: and Technologies: Hadoop MapReduce, HDFS, Hive, Spark, BIGSQL, Sqoop, Impala, SQL, Scala, Java (JDK 1.8), Hadoop (Horton Works), Eclipse, Zookeeper 3.4.6, HUE, HDP 2.7.3, Ranger,Kerbos.RHEL 7.x,AWS(Cloud computing)

Big Data/Hadoop Developer

Confidential

Responsibilities:

  • As a part of Data acquisition used Sqoop and Flume to inject the data from server to Hadoop using incremental import
  • Used Sqoop to import data from RDMS sources into HDFS
  • Developed scripts for automating Job Queues and workflows
  • Performed transformation, cleaning and filtering on imported data using Hive, Map Reduce, and loaded data into HDFS
  • Worked on performing transformation & actions on RDDs and Spark streaming data
  • Real time streaming of data was developed using Spark with Kafka
  • Implemented Spark using Scala and utilized Data frames / Spark SQL API for faster processing of data
  • Collected large amount of log data using Flume and staged data in HDFS for further analysis
  • Configured Kafka to read and write messages from external programs
  • Converted Hive queries into Spark transformations using Spark RDDs
  • Responsible for creating shell script to run Hive jobs
  • Import/Export data into HDFS and Hive using Sqoop
  • Worked with different file formats like Sequence file, ORC file and Parquet
  • Wrote ETL scripts for automation the ETL process
  • Worked with Nagios for monitoring and Jenkins for continuous integrations
  • Worked as Devops responsible for configuring Puppet, GIT, Ansible and Nagios
  • Developed Spark scripts by using Scala and Python shell commands as per the requirement
  • Tools and Technologies: Hadoop, Sqoop, Spark (1.5), Spark SQL, Spark Streaming, Hive, HBase, Linux, Scala, IntelliJ, Tableau, UNIX, Shell Scripting, Putty, Oozie, Zookeeper

Software Engineer

Confidential, NY

Responsibilities:

  • Created scripts to load data from UNIX local file system to HDFS
  • Extracted data from different databases and copied into HDFS using Sqoop
  • Implemented Flume, Spark framework for real time data processing
  • Created the Spark streaming code to take the source files as input
  • Used Oozie workflow to automate all the jobs
  • Exported the analyzed data into relational databases using Scoop for visualization and to generate reports for BI team
  • Used Hive to Partition and Bucketing, for optimizing data storage and compute-various metrics for reporting
  • Involved in developing Hive DDLs to create, drop and alter Hive tables
  • Created hive queries and UDFs for loading and processing data
  • Worked with HBase to conduct quick look ups (Update, Inserts and Deletes) in Hadoop
  • Developed Spark programs using Scala; involved in creating Spark SQL queries and developed Oozie workflow for Spark jobs
  • Performed analytics on structured/unstructured data and managed ingestion of large data by using Flume, Kafka and Sqoop
  • Used Bit Bucket as code repository and version control
  • Managing build and deployment using Maven, Nagios, Jenkins, Chef SCM tools
  • Involved in deployment enhancements & issues
  • Created Branches, tags, labels and performed merges in Stash and GIT
  • Used Agile methodology in developing the application; participated in standup meetings and Sprints
  • Tools and Technologies: Hadoop MapReduce, HDFS, Hive, Spark, Flume, Kafka, Sqoop, Impala, SQL, Scala, Java (JDK 1.6), Hadoop (Cloudera), Tableau, Eclipse, CDH 5, Zookeeper, HUE, Chef, Jenkins

Software Engineer

Confidential, NY

Responsibilities:

  • Used Spring MVC as Model, View & Controller design pattern
  • Used Spring support for JDBC for all CRUD (Create, Update, Delete) operations
  • Experience in J2EE framework Spring MVC
  • Generating WSDL for publishing the web services
  • Used Maven build tool to build the project
  • Used Log4j for logging errors, message and performance logs
  • Worked in Agile environment with active scrum participation
  • Tools and Technologies: J2EE, Spring MVC Framework, WS, JavaScript, Oracle, SQL Server, Web Logic 11g, UNIX, Version One, OXYGEN XML Editor, Maven, SVN

Front End Developer

Confidential, Ohio

Responsibilities:

  • Wrote UI/Business validations for the owned use cases
  • Executed SQL queries to perform crud operations on customer records
  • Assist developers throughout the Scrum and help team in understanding the requirements access the design, code and review the solution while building portals
  • Used maven for build tool and added external dependencies
  • Involved in configuring Jenkins for many jobs and handled lot of issues on Jenkins side
  • Solved new defects from end users and worked with new change requests
  • JQuery and JavaScript used to handle all functionality of the front page along with CSS and HTML
  • Tools and Technologies: Core Java, J2EE, Spring, WS, JavaScript, Oracle, SQL Server, Windows7, Web Logic 11g, UNIX, Version One, FogBugz, OXYGEN XML Editor, Maven, JQuery, SVN

Software Engineer

Confidential

Responsibilities:

  • Used MySQL database for storing all data
  • Generating all types of necessary reports from database
  • Implemented web services (WSDLs) using Jax-Ws
  • Used Custom Exception handling for carrying perfect message to the end user
  • Bug fixing and maintenance of the product
  • Created presentation layer with Java script and HTML
  • Involved in daily stand up and weekly sprint meetings
  • Tools and Technologies: Core Java, J2EE, JDBC, Oracle 11g, Hibernate, Spring, Hibernate, Tomcat Server, Windows 7, Maven, SVN, Putty, MySQL

We'd love your feedback!