We provide IT Staff Augmentation Services!

Spark Developer Resume

0/5 (Submit Your Rating)

Charlotte, NC

SUMMARY

  • Over 8 years of experience in application development and design using emerging technologies like Hadoop, NoSQL, Big data and Java/J2EE Technologies.
  • Around 4+ years of strong working experience with Big Data and Hadoop Ecosystems.
  • Experience in Machine Learning and Data Science and in using new tools and technologies developments to drive improvements throughout entire software development lifecycle.
  • Experience in installing, configuring and using Apache Hadoop ecosystem components like Hadoop Distributed File System (HDFS), MapReduce, PIG, HIVE, HBASE, ZOOKEEPER, SQOOP, Hue, Spark, Storm, Kafka Solr, Git, Maven, AVRO, JSON and CHEF.
  • Exposure in analyzing data using HiveQL, Pig Latin, HBase and custom Map Reduce programs in Java.
  • Experience in importing and exporting data using Sqoop from HDFS to RDBMS, Teradata and vice versa.
  • Good working experience on using Sqoop to import data into HDFS from RDBMS and vice - versa.
  • Expertise in writing Apache Spark Streaming API on Big Data distribution in the active cluster environment.
  • Good experience in Apache Flume for efficiently collecting, aggregating, and moving large amounts of log data.
  • Hands-on experience in performing analytics using Apache Spark.
  • Involved in upgrading existing MongoDB instances from version 2.4 to version 2.6 by upgrading the security roles and implementing newer features.
  • Responsible for performing reads and writes in Cassandra from web services by using java JDBC connectivity.
  • Worked on Data Serialization formats for converting Complex objects into sequence bits by using AVRO, PARQUET, JSON, CSV formats.
  • Experience in integrating Apache Kafka with Apache Storm and created Storm data pipelines for real time processing.
  • Designed and implemented Hive and Pig UDF's using Python for evaluation, filtering, loading and storing of data.
  • Proficient in writing the MapReduce jobs on Hadoop ecosystem including Hive and Pig in Cloudera.
  • Excellent working experience in Scrum / Agile framework and Waterfall project execution methodologies.
  • Good understanding of job workflow scheduling and monitoring tools like Oozie and ControlM.
  • Extensive experience in working with SOA based architectures using Rest based web services using JAX-RS and SOAP based web services using JAX-WS.
  • Experience working on Version control tools like SVN, Git, GitHub and JIRA to track issues and crucible for code reviews.
  • Knowledge on delivering Microsoft Azure product with agile methodology.
  • Excellent Java development skills using J2EE, Spring, J2SE, Servlets, JSP, JDBC.
  • Knowledge in installation, configuration, supporting and managing Hadoop clusters using Apache Cloudera (CDH3, CDH4) distributions and Amazon web services (AWS).
  • Experience on Amazon Web Services(AWS), Amazon Cloud Services like Elastic Compute Cloud(EC2), Simple Storage Service(S3), Elastic Map Reduce(EMR) Amazon Simple DB, Amazon Cloud Watch, SNS, SQS, LAMBDA.
  • Proficiency in multiple databases likeMongoDB, Cassandra, MYSQL, Oracle 9i, 10g, 11g and MS SQL Server.
  • Experience on Microservices, Spring boot and Spring cloud.
  • Experienced in developing applications using application/Web servers such as Web Sphere, Web logic and Tomcat.
  • Good understanding of all aspects of Testing such as Unit, Regression, Agile, White & Black-box.
  • Exposure to requirements gathering, design and development, application migration and maintenance phases of the Software Development Lifecycle (SDLC).
  • Expertise in web Technologies like HTML, CSS, PHP, XML, JSP, Ajax, JQuery .
  • Good Inter personnel skills and ability to work as part of a team. Exceptional ability to learn, master new technologies and to deliver outputs in short deadlines.

TECHNICAL SKILLS

Hadoop /Big Data: HDFS, Hadoop Map Reduce, Hive, Pig, Storm, Sqoop, Flume, Oozie, Scala, kafka, Spark, Spark SQL, Spark Streaming, Spark MLib

Languages: C, C++, Java, J2EE, PL/SQL, Python, Pig Latin, HiveQL, Unix shell scripts, R

Java/J2EE Technologies: Core Java, Applets, Swing, JDBC, JSON, JSTL, JMS, Java Script, JSP, Servlets, EJB, JSF

Methodologies: Agile, Waterfall Model, TDD

Frameworks: MVC Struts, Hibernate, Spring

ETL: IBM WebSphere/Ascential DataStage 8.7/8.1.2/8.5

Databases: HBase, Cassandra, MongoDB, Oracle, MySQL, SQL server

Application Servers: Apache Tomcat, Web Logic, Web Sphere

Web Technologies: HTML, XML, CSS, JavaScript, JQuery, AJAX, Bootstrap

Operating Systems: Windows, Linux, Unix, Mac

Tools: and IDE: Eclipse, NetBeans, IntelliJ

Network Protocols: TCP/IP, UDP, HTTP, DNS.

PROFESSIONAL EXPERIENCE

Confidential - Charlotte,NC

Spark Developer

Responsibilities:

  • Responsible for developing prototypes and proof of concepts for the selected solutions and implementing complex big data projects with a focus on collecting, parsing, managing, analyzing and visualizing large sets of data using multiple platforms.
  • Expertise in designing and deployment of Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase, Oozie, Sqoop, Flume, Spark, with Cloudera distribution.
  • Developed Hive UDF's to bring all the customers email id into a structured format.
  • Involved in converting Hive/SQL queries into Spark transformations using Spark data frames, Scala and Python.
  • Expertise in implementing Spark Scala application using higher order functions for both batch and interactive analysis requirement.
  • Performed unit testing for Sparkand Spark Streamingwith Scala Test and JUnit
  • Leverage Tableau to perform visualizations on the collected data.
  • Performed importing data from various sources to the Cassandra cluster using Sqoop. Worked on creating data models for Cassandra from Existing Oracle data model.
  • Used Spark - Cassandra connector to load data to and from Cassandra.
  • Having experience in developing a data pipeline using Kafka to store data into HDFS. Good Experience on SDLC (Software Development Life cycle).
  • Exposure on usage of Apache Kafka to develop data pipeline of logs as a stream of messages using producers and consumers.
  • Designed and built the Reporting Application, which uses the Spark SQL to fetch and generate reports on Cassandra table data.
  • Set up Solr for distributing indexing and search.
  • Created AirflowScheduling scripts in Python to automate the process of sqooping wide range of data sets.
  • Designed and Built Spark Streaming application which analyses and evaluates the Streaming data against the business rules through Rules Engine and then send alerts to the business users to address the customer preferences and do product promotions.
  • Developed Spark scripts by using Scala Shell commands as per the requirement.
  • Developed UDF's using both Data Frames/ SQL and RDD in Spark for data Aggregation queries and reverting into OLTP through Sqoop.
  • All the bash scripts are scheduled using Resource Manager Scheduler.
  • Developed Oozie Workflows for daily incremental loads, which gets data from Teradata and then imported into hive tables.
  • Using Hive join queries to join multiple tables of a source system and load them toElasticsearch tables.
  • Implementation of custom design and verification of Spark Machine Learning algorithms.
  • Created Mappings using Talend Open Studio for Evaluation and POC.
  • Automated the data flow using Nifi / ControlM.
  • Written various key queries inElasticsearchfor retrieval of data effectively.
  • Extensively worked onElasticsearchquerying and indexing to retrieve the documents in high speeds.
  • Developing Spring Boot based Microservices using the annotations & implementing architecture patterns, Used Spring Rest/JSON to expose microservices APIs.
  • Spearheaded the POCs for the AWS ecosystem via the AWS Management console.
  • WrittenElasticsearchtemplate for the index patterns.
  • Used microservices modules to create the online applications used for continuous delivery.
  • Developed the GUI using JSP, spring web flow following spring web MVC pattern and Spring Boot.
  • Loaded two different datasets sources like Oracle, MySQL to HDFS and Hive respectively on daily basis.
  • Experience in cloud computing on Azure and AWS.
  • Optimized Hive queries to extract the customer information from HDFS or Cassandra.
  • Developed Pig Latin scripts to aggregate the log files of the business clients.

Environment: Hadoop, HDFS, Spark, Strom, Kafka, Map Reduce, Hive, Pig, Sqoop, Oozie, Airflow, Talend Studio, DB2, Java, Spring Boot, Python, Splunk, UNIX Shell Scripting.

Confidential -Chicago, IL

Hadoop Developer/Data Engineer

Responsibilities:

  • Good understanding and related experience with Hadoop stack-internals, Hive, Pig and Map/ Reduce. Experience in setting up of clusters utilizing cloud era manager.
  • Wrote the MapReduce jobs to parse the web logs which are stored in HDFS.
  • Involved in optimizing Hive Queries, joins to get better results for Hive ad-hoc queries.
  • Exploring with the Spark for improving the performance and optimization of the existing algorithms in Hadoop.
  • Installed and configured Hadoop MapReduce, HDFS, developed multiple MapReduce jobs in Java for data cleaning and pre-processing.
  • Processed the Web server logs by developing Multi-hop flume agents by using Avro Sink and loaded into MongoDB for further analysis, also extracted files from MongoDB through Flume and processed.
  • Involved in creating Oozie workflow and Coordinator jobs for Hive jobs to kick off the jobs on time for data availability.
  • Generated reports from this hive table for visualization purpose.
  • Working on Airflowto schedule the spark jobs.
  • Experience in setting up Fan-out workflow in flume to design v shaped architecture to take data from many sources and ingest into single sink.
  • Understanding of managed distributions of Hadoop, like Cloudera and Hortonworks.
  • Used Scala to develop Scala coded Spark projects and executed using Spark-submit.
  • Hands on experience in developing Map Reduce programs using Apache Hadoop for analyzing the Big Data.
  • Expertise in implementing ad-hoc Map Reduce programs using Pig Scripts.
  • Experience in importing and exporting data from RDBMS to HDFS, Hive tables and HBase by using Sqoop.
  • Wrote Flume configuration files for importing streaming log data into Cassandra with Flume.
  • Imported several transactional logs from web servers with Flume to ingest the data into
  • HDFS. Using Flume and Spool directory for loading the data from local system (LFS) to HDFS.
  • Written and Implemented Teradata Fast load, Multiload and Bteq scripts, DML and DDL.
  • Experience in importing streaming data into HDFS using flume sources, and flume sinks and transforming the data using flume interceptors.
  • Knowledge in handling Kafka cluster and created several topologies to support real-time processing requirements.
  • Hands on experience migrating complex map reduce programs into Apache Spark RDD transformations.
  • Design and Programming experience in developing Internet Applications using Java, J2EE, JSP, MVC, Servlets, Struts, Hibernate, JDBC, JSF, EJB, XML, AJAX and web based development tools.

Environment: Hadoop, HDFS, MapReduce, Spark, Pig, Hive, Sqoop, Flume, Kafka, HBase, Oozie, Spark, Airflow, Java, SQL scripting, Linux shell scripting, Eclipse and Cloudera.

Confidential -Dulles, VA

Hadoop Developer

Responsibilities:

  • Experience in Importing and exporting data into HDFS and Hive using Sqoop.
  • Involved in the design team for designing the flow architecture.
  • Developed Pig program for loading and filtering the streaming data into HDFS using Flume.
  • Experienced in handling data from different data sets, join them and preprocess using Pig join operations.
  • Moving Bulk amount data into HBase using Map Reduce Integration.
  • Developed HBase data model on top of HDFS data to perform real time analytics using Java API.
  • Developed different kind of custom filters and handled pre-defined filters on HBase data using API.
  • Strong understanding of Hadoop ecosystem such as HDFS, MapReduce, HBase, Zookeeper, Pig, Hadoop streaming, Sqoop, Oozie and Hive .
  • Implement counters on HBase data to count total records on different tables.
  • Experienced in handling Avro data files by passing schema into HDFS using Avro tools and Map Reduce.
  • Worked on custom Pig Loaders and Storage classes to work with a variety of data formats such as JSON, Compressed CSV, etc.
  • Created Hive Dynamic partitions to load time series data .
  • Created tables, partitions, buckets and perform analytics using Hive ad-hoc queries.
  • Experienced import/export data into HDFS/Hive from relational data base and Tera data using Sqoop.
  • Handling continuous streaming data comes from different sources using flume and set destination as HDFS.
  • Integrated spring schedulers with Oozie client as beans to handle cron jobs.
  • Experience with CDH distribution and Cloudera Manager to manage and monitor Hadoop clusters.
  • Actively participated in software development lifecycle (scope, design, implement, deploy, test), including design and code reviews.
  • Involved in story-driven agile development methodology and actively participated in daily scrum meetings.

Environment: Hadoop, HDFS, Map Reduce, Hive, Pig, HBase, Sqoop, RDBMS/DB, Flat files, MySQL, CSV, Avro data files.

Confidential

Java Developer

Responsibilities:

  • Extensively used Core Java, Servlets, JSP and XML.
  • Used Struts 1.2 in presentation tier.
  • Used DB2 Database to store the system data
  • Used Apache log 4j Logging framework for logging of trace and Auditing.
  • Used Asynchronous JavaScript and XML (AJAX) for better and faster interactive Front-End.
  • Developed RESTful Web services client to consume JSON messages using Spring JMS configuration. Developed the message listener code.
  • Developed the business components using EJB Session Beans.
  • Create JSP pages for the Customer module of the application.
  • Involved in developing in all the tiers of J2EE application.
  • Developing Hibernate XML Java object-to-database mapping documents.
  • Performed code reviews. Used unit testing for all the components using JUnit
  • Developed and deployed projects in a team that following Software management procedure as Rational Unified Process(RUP) combined with Pair Programming and Test Driven Development(TDD).

Environment: Java, JSP, Servlets, XML, IBM Rational Application Developer(RAD), Web Sphere, J Boss, SQL Server 2003, PLSQL, Windows XP, SVN, ANT.

Confidential

Software Engineer

Responsibilities:

  • Involved in various phases of Software Development Life Cycle (SDLC) such as requirements gathering, analysis, design and development.
  • Refined application framework for data flow and data handling.
  • Generated the application using Eclipse IDE.
  • Meeting stringent Deadlines and analyzing the specific process requirements.
  • Involved in AJAX implementation.
  • Developed JavaScript validations to validate form fields.
  • Performed unit testing for the developed code using JUnit.
  • Worked on defining and improving a home grown testing framework to support integration and performance testing.
  • Implemented Waterfall model practices according to the application requirements.

Environment: Eclipse, Java, JSP, Spring, PHP, MySQL, HTML, CSS, JavaScript, jQuery.

We'd love your feedback!