We provide IT Staff Augmentation Services!

Spark Developer Resume

Boston, MA

SUMMARY:

  • Around 8 years of professional IT experience, 4+ years Big Data Ecosystem experience in ingestion, querying, processing and analysis of big data.
  • Hands on experience with Spark Core, Spark SQL, Spark Streaming using Scala and Python.
  • Good Understanding in Hadoop Ecosystem including Spark, Hive, Pig, HBase, Oozie, Sqoop and Kafka.
  • Experience in Data Frame, RDD architecture and implementing spark operations on RDD, Data Frame and optimizing Transformations and Actions in Spark.
  • Experience with processing large sets of structured, semi - structured and unstructured data using Spark and Scala.
  • Expertise knowledge on Object Oriented Programming Concepts and valuable experience in Exception Handling, Debugging and Tracing concepts.
  • Experience with Hadoop architecture and its components such as HDFS, Name Node, Data Node and MapReduce programming paradigm.
  • Experience with creating Hive tables, Hive joins & HQL for querying the databases eventually leading to complex Hive UDFs.
  • Experience in work with Cassandra for fast retrieval of data.
  • Worked with Several File Formats such as Avro, Parquet, CSV, JSON, Sequential, ORC etc.
  • Knowledge of ETL (Extract, Transform, Load) methods for data extraction, transformation and loading in corporate-wide ETL Solutions and Data warehouse tools for data analysis.
  • Understanding of several programing languages such as Scala, Python, Java, C++.

TECHNICAL SKILLS:

Data Ingestion: Sqoop, Kafka, Flume, Apache Hadoop Eco Systems

Data Processing: Spark, Impala, YARN, Map Reduce

Distributed Storage and Computing: HDFS, Zookeeper, S3.

Programming Languages: Scala, SQL, Python

ETL: IBM-Data Stage, Talend, AB Initio

Relational Databases: Oracle, MYSQL, Oracle SQL & ACCESS

NOSQL Databases: MONGODB, Cassandra, HBASE, DYDB

Cloud AWS: EMR, EC2, S3

Build tools: Jenkins, Maven, Gradle

Version Control: GIT, IntelliJ, Eclipse

Operating System: LINUX, Windows, UNIX

Data Formats: Parquet, Sequence, AVRO, ORC, CSV, JSON

Monitoring: Ambari, Cloudera Manager

PROFESSIONAL EXPERIENCE:

Spark Developer

Confidential, Boston, MA

Responsibilities:

  • Developed programs in Spark based on the application for faster data processing than standard MapReduce programs.
  • Written extensive Hive queries to do transformations on the data to be used by downstream models.
  • Developed Hive queries to do analysis of the data and to generate the end reports to be used by business users.
  • Developed the Oozie workflows with Sqoop actions to migrate the data from relational databases like Oracle, Teradata to HDFS.
  • Worked on scalable distributed computing systems, software architecture, data structures and algorithms using Hadoop, Apache Spark etc. and ingested streaming data into Hadoop using Spark and Scala.
  • Experience in writing Sqoop Scripts for importing and exporting data from RDBMS to HDFS.
  • Developed Scala scripts using both Data frames and RDD in Spark for Data Aggregation, queries and writing data through Sqoop.
  • Developed Spark code and Spark-SQL/Streaming for faster testing and processing of data.
  • Worked with various HDFS file formats like Avro, ORC, Sequence File and various compression formats like Snappy, gzip etc.
  • Extensively used Maven, SVN as a code repository and Version One for managing day agile project development process and to keep track of the issues and blockers.
  • Used Spark for interactive queries, processing of streaming data and integration with popular NoSQL database for huge volume of data.

Environment: Hadoop, Spark, Hive, Oracle, Maven, Scala, Python, Pig, Sqoop, Oozie, MongoDB, SVN.

Hadoop Developer

Confidential, Reston, VA

Responsibilities:

  • Ingested incremental Batch Data from MySQL database and Teradata Using Sqoop.
  • Ingested real time data into the HDFS using Kafka and Oozie.
  • Worked with Elastic map reduce (EMR) for data processing on Amazon Web Services (AWS).
  • Worked with Amazon S3 for storage on Amazon Web Services (AWS).
  • Worked with Different Spark Modules like Spark core, Spark SQL, Spark Streaming, Spark Data sets and Data frames.
  • Involved in converting the files in HDFS into RDD's from multiple data formats and performing Data Cleansing using RRD Operations.
  • Very good understanding of Partitions, bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.
  • Wrote complex queries and User Defined functions (UDFs) for custom functionality in hive using Scala.
  • Worked with different HDFS file formats like Avro, ORC, parquet, Sequence File
  • Integrated Oozie with the rest of Hadoop stack supporting several types of jobs and the system specific jobs (such as Java programs and shell scripts).

Environment: HDFS, Spark, Hive, Sqoop, Kafka, AWS EMR, AWS S3, Oozie, Spark Core, SPARK SQL, Maven, Scala, SQL, Linux, YARN, IntelliJ, Agile Methodology

Big Data Developer

Confidential, Pittsburgh, PA

Responsibilities:

  • Utilized SQOOP, Kafka, Flume and Hadoop File System API’s for implementing data ingestion pipelines from heterogeneous data Sources.
  • Created storage with Amazon S3 for storing data and Worked on transferring data from Kafka topic into AWS S3 storage.
  • Worked on real time streaming and performed transformations on the data using Kafka and Spark Streaming.
  • Implemented Spark Scripts using Scala and used Spark SQL to access hive tables into spark for faster processing of data.
  • Created data pipeline for several events of ingestion, aggregation and load consumer response data from AWS S3 bucket into Hive external tables.
  • Worked on various data formats like AVRO, Sequence File, JSON, Map File, Parquet and XML.
  • Used Apache NiFi to automate data movement between different Hadoop components and perform conversion of raw XML data into JSON, AVRO.

Environment: Hadoop, HDFS, AWS, Scala, Kafka, Map Reduce, YARN, Spark, Pig, Hive, Python, Java, NiFi, HBase, IMS Mainframe, Maven.

Java Developer

Confidential

Responsibilities:

  • Designed and developed User Interface using HTML5, CSS, JavaScript, jQuery, AJAX and JSON.
  • Implemented spring security features using AOP Interceptors for the authentication.
  • Developed Spring Framework based Restful Web Services for handling and persisting of requests and Spring MVC for returning response to presentation tier
  • Used multithreading in programming to improve overall performance using Singleton design pattern in Hibernate Utility class.
  • Wrote complex SQL query to pull data from different tables to build the report.
  • Used Log4j for error handling, to monitor the status of the service and to filter bad loans.
  • Development and debugging done using Eclipse IDE.

Environment: Java, Hibernate, spring, HTML5, CSS, JavaScript, J Query, AJAX, JSON, Web Logic, Oracle, PL/SQL.

Java/ J2EE Developer

Confidential

Responsibilities:

  • Developed the application using Spring Framework that leverages Model View Controller (MVC) architecture, spring security and Java API.
  • Implemented design patterns like Singleton, Factory pattern and MVC.
  • Deployed the applications on IBM Web Sphere Application Server.
  • Worked on Java script, CSS Style Sheet and JQuery.
  • Worked one-on-one with client to develop layout, color scheme for his website and implemented it into a final interface design with the HTML5/CSS3 & JavaScript.
  • Used advanced level of HTML5, JavaScript, JQuery, CSS3 and pure CSS layouts (table less layout)
  • Wrote SQL queries to bring data from the Oracle & MySQL databases.

Environment: Java, Oracle 11g Express, CVS, Struts, spring, HTML, CSS, JavaScript, Apache Tomcat, Eclipse IDE, Maven, Junit.

Hire Now