Spark Developer Resume
Cherry Hill, NJ
SUMMARY
- 6 years senior developer with 3 years as a Hadoop/Spark Developer in Big Data domain and 3 years as a Java developer.
- Experienced in analyzing the different types of data flow from source to Hadoop Clusters.
- Good at with Big Data Hadoop core and Eco - System components (HDFS, MapRedue, Hive, Yarn, Sqoop, Oozie, Hbase and Pig).
- Hands on Cloudera and Hortonworks Distribution System of Hadoop.
- Experienced in manipulating the streaming data to clusters through Kafka and Spark-Streaming.
- Extensive experience in partitioning Data according to the business requirements using Hive.
- Working with data importing and exporting in Sqoop.
- Good knowledge in data extraction, transformation and loading between Hive, HDFS and RDBMS.
- Proficient to work with data transformation from HDFS, HIVE, HBase, and RDBMS.
- Experienced in creating UDF's, UDAF's for Hive.
- Acumen in choosing an efficient ecosystem in Hadoop and providing the best solutions to Big Data problems.
- Extensive Knowledge in Spark, Spark-Streaming, SparkSQL and Scala.
- Proficient in working with NoSQL database like Cassandra and HBase.
- Strong expertise in algorithm and data structure.
- Strong ability to write complex sql query.
- Prolific in generating the splendid and informative dashboards for Business Intelligence teams.
- Experienced in using Front-end technologies like JavaScript, HTML, CSS, HTML5 and CSS3.
- Experienced in Test such as JUnit
- Proficient in relational databases like Oracle, MySQL and SQL Server.
- Used to work in Agile environment.
- Comfortable to program in Linux, UNIX environments. And Expertise in Unix-Shell.
- Extensive experience with Front-end and Back-end IDE's like Eclipse, Visual Studio.
- Proficient communication skills with an ability to lead a team & keep them motivated.
- Good knowledge to work on AWS, including S3, EMR, EC2.
- Highly motivated and versatile team player with the ability to work independently & adapt quickly to new emerging technologies.
TECHNICAL SKILLS
Hadoop Ecosystem Framework: MapReduce, Hive, Pig, Hadoop, Spark1.6 and 2.0 HBase, SqoopHDFS, Kafka
Databases Methodologies: Oracle, MySQL, SQL Server, Agile Scrum Hbase, Cassandra, MongoDB
Languages: Java, Scala, C++, SQL, HiveQL, HTML 5, JSON
Web Technologies: Pig Latin, JavaScript, Shell-Scripting, XML, CSS 3, Angular.JS
Systems Other: Windows, Linux, UNIX Eclipse, Maven, JUnit, Git
PROFESSIONAL EXPERIENCE
Confidential, Cherry Hill, NJ
Spark Developer
Responsibilities:
- Involved in designing hadoop ecosystem pipeline.
- Designed real-time data importing method as producer to pull the data from the sources to Kafka clusters.
- Pulled the consumption records form Kafka to Spark.
- Realized collecting incoming data in real-time and processing them within few seconds at the same time using Spark-Streaming and SparkSQL.
- Implemented ML model to filter the records and get the fraud events.
- Aggregated and stored the data result into HDFS and Cassandra .
- Accomplished different goals of the management team by implement algorithms and business logics of data process in Spark related technologies.
- Involved in using RDBMS, sqoop and Hive to do ETL jobs and some data analytics on historical records stored in RDBMS.
- Designed some UDFs with Java to improve query performance in Hive.
- Strong problem solving experience on test, identify, address, debug and then resolve technical issues that affect the integrity of the application.
- Expertized in improving the performance of application by analytical skills.
- Worked in the agile Environments, effectively communicated at all levels of an organization in Management and Technical roles.
Environment: Hadoop, Spark, Apache Kafka, RDBMS, Hive, HDFS, Cassandra, Scala, Parquet, Linux.
Confidential, NYC, NY
Hadoop/Spark Developer
Responsibilities:
- Developed a data pipeline using Kafka, Spark, RDBMS, HDFS, Scala, MapReduce, Hive and Sqoop to ingest, transform and analyzing customer behavioral data.
- Imported the data from RDBMS into HDFS using Sqoop.
- Designed dynamically storage logic of data and improved performance with HDFS and Hive.
- Designed and Developed MapReduce in Java to process large volume of data.
- Used Kafka to collect streaming data from customers' consumption records. Configured Kafka to read and write messages from external data sources.
- Realized process of real time streaming data using Spark Streaming and Kafka, storage data into HDFS.
- Exported the analyzed data from HDFS to Hive using Sqoop to further visualize and generate reports for the BI team.
- Utilized Spark Core, Spark Streaming and Spark SQL API for faster processing of data instead of using MapReduce in Java.
- Save the final result back into HDFS and then imported to Hive.
Environment: Kafka, Spark, Scala, Cloudera, Hadoop, RDBMS, Hive, Sqoop, Java, HDFS, Linux, HBase
Confidential, NY
BigData Developer
Responsibilities:
- Implemented Pig scripts to clean and accomplish data preprocessing.
- Involved in importing and exporting data using Sqoop from HDFS to RDBMS and vice-versa.
- In order to improve performance, utilize Hive scripts to reorganize data.
- Migrated data from the sources such as HDFS and Hive into Map Reduce.
- Developed Map Reduce programs in Java to process data.
- Wrote various Queries on data according to business requirement.
- Effectively communicated at all levels of an organization’s Management and Technicians.
- Working closely with BI team.
Environment: Oracle, Linux, Hadoop, Hive, Pig, Sqoop, HBase, Java, JSON, Hortonworks
Confidential
Web Developer
Responsibilities:
- Developed database schema and SQL queries for querying, inserting and managing database
- Developed pages using HTML5, CSS3 and JavaScript as part of the presentation layer
- Hibernate framework is used in persistence layer for mapping an object-oriented domain model to database.
- Used Maven scripts to fetch, build, and deploy application to development environment
- Used JUnit for functional and unit testing code.
Environment: JavaScript, MySQL, JUnit, CSS 3, HTML 5, RDBMS, Hibernate
Confidential
Java Developer
Responsibilities:
- Used Apache Tomcat as the web server of the house reservation records.
- Implemented JDBC in Java to connect Apache Tomcat and Oracle.
- Implemented JDBC to connect User interface and Apache Tomcat.
- Created and developed database with Oracle, participated in manipulation of data transaction and improve the performance of data
- Developed of application using Spring.MVC to handle the data communication from User interface to Server then to Database and vice-versa.
- Involved in exchange of data between the Client, Server and Database in Hibernate.
Environment: Eclipse, JDBC, Spring.MVC, JavaScript, JUnit, Java6, Servlets, CSS3, HTML 5, Oracle.