Hadoop Developer Resume
0/5 (Submit Your Rating)
Iselin, NJ
SUMMARY
- Around 3 years of overall IT experience in a variety of industries, this includes hands on experience on Big Data Analytics, and Development.
- Experience in doing bash, writing shell scripting.
- Experience spark Framework and Map Reduce for Hadoop to debug code.
- Hands on experience in Big data development using AWS EMR, Redshift, RDS, Hive, Presto, Hadoop cluster set up, performance fine - tuning, monitoring and administration
- J2EE technologies such as Spring, Spring MVC, Spring JDBC, Web services, SOAP, WSDL.
- Having good experience in Bigdata related technologies like Hadoop frameworks, Map Reduce, Hive, HBase, PIG, Sqoop, Spark, Kafka, Flume, ZooKeeper, Oozie, EMR Clusters and Storm.
- Experienced as Hadoop, expertise in providing end to end solutions for real time big data problems by implementing distributed processing concepts such as map reduce on Hadoop frameworks such as HDFS and Hadoop Ecosystem components.
- Experience in working on large scale big data implementations and in production environment
- Hands on experience on Data Migration from Relational Database to Hadoop Platform\ SQOOP.
- Good knowledge of database structures, theories, principle and RDS or other columnar Databases.
- Working experience Spark 1.6 and 2.10 Development and remediating code to new version.
- Hands on Experience with back-end programming using Scala.
- Experience wih EMR,Hive and knowledge in Shell Scripting.
- Worked with Writing ETL jobs through Spark and ability to handle performance and Memory Issues.
- Experience with Database Structures,principles and theories.
- Experienced in using Pig scripts to do transformations, event joins, filters and some pre-aggregations before storing the data onto HDFS
- Understanding of managed distributions of Hadoop, like Cloudera and Hortonworks.
- Experience with Agile development methodology.
- Having experience with Java Programming like writing Javascript, Node.js and OOAD.
- Developed analytical components using Scala, Spark, Spark Stream and Cloudera.
- Good experience in writing Spark applications using Python and Scala.
- Experience processing Avro data files using Avro tools and MapReduce programs.
- Implemented pre-defined operators in spark such as map, flat Map, filter, reduceByKey, groupByKey, aggregateByKey and combineByKey etc.
- Used Scala sbt to develop Scala coded spark projects and executed using spark-submit
- Developed multiple MapReduce jobs to perform data cleaning and preprocessing.
- Designed HIVE queries & Pig scripts to perform data analysis, data transfer and table design.
- Excellent communications skills possess strong problem solving, analytical, time management skills.
- Experience analyzing and resolving performance, scalability and reliability issues.
- Having experience in developing a data pipeline using Kafka to store data into HDFS. Good Experience on SDLC (Software Development Life cycle).
- Exceptional ability to learn new technologies and to deliver outputs in short deadlines.
- Monitoring Map Reduce Jobs and Yarn Applications.
- Possess strong Communication skills of written, oral, interpersonal and presentation.
- Ability to perform at a high level, meet deadlines, adaptable to ever changing priorities.
TECHNICAL SKILLS
Hadoop Technologies: HDFS, YARN, MapReduce, Hive, Pig, Sqoop, Flume, Spark, Kafka, Zookeeper, and Oozie.
Languages: SQL, Unix Shell Scripting, HTML, PL/SQL
Java and J2EE Technologies: Core Java, Servlets, Hibernate, Spring, Struts.
Databases: MySQL, Oracle, DB2
Operating Systems: Windows, Unix
Development/Methodologies: Agile/Scrum, Waterfall.
PROFESSIONAL EXPERIENCE
Confidential, Iselin, NJ
Hadoop Developer
Responsibilities:
- Responsible for design development of Spark SQL Scripts based on Functional Specifications.
- Responsible for Spark Streaming configuration based on type of Input Source
- Wrote the Map Reduce jobs to parse the web logs which are stored in HDFS.
- Developed the services to run the Map-Reduce jobs as per the requirement basis.
- Importing and exporting data into HDFS and HIVE,PIG using Sqoop.
- Responsible to manage data coming from different sources.
- Built data pipeline using Pig and Java/Scala Map Reduce to store onto HDFS.
- Designing ETL Data Pipeline flow to ingest the data from RDBMS source to Hadoop using shell script, sqoop, package and mysql.
- Developing business logic using scala.
- Responsible for loading data from UNIX file systems to HDFS. Installed and configured Hive and also written Pig/Hive UDFs.
- Involved in creating Hive Tables, loading with data and writing Hive queries which will invoke and run MapReduce jobs in the backend.
- Writing MapReduce (Hadoop) programs to convert text files into AVRO and loading into Hive (Hadoop) tables
- Implemented the workflows using Apache Oozie framework to automate tasks.
- Worked with NoSQL databases like HBase in creating HBase tables to load large sets of semi structured data coming from various sources.
- Developing design documents considering all possible approaches and identifying best of them.
- Loading Data into HBase using Bulk Load and Non-bulk load.
- Developed scripts and automated data management from end to end and sync up b/w all the clusters.
- Exploring with the Spark improving the performance and optimization of the existing algorithms in Hadoop.
- Import the data from different sources like HDFS/HBase into SparkRDD.
- Experienced with Spark Context, Spark-SQL, Data Frame, Pair RDD's, Spark YARN.
- Import the data from different sources like HDFS/HBase into Spark RDD.
- Involved in converting Hive/SQL queries into Spark transformations using Spark RDD, Scala and Python.
- Involved in gathering the requirements, designing, development and testing.
- Developing traits and case classes etc in scala.
- Ability to work with onsite and offshore team members.
- Able to work on own initiative, highly proactive, self-motivated commitment towards work and resourceful.
- Strong debugging and critical thinking ability with good understanding of frameworks advancement in methodologies and strategies.
Confidential
Responsibilities:
- Load and transform large sets of structured, semi structured data using hive.
- Developed Hive queries for the analysts.
- Managing and scheduling Jobs to remove the duplicate log data files in HDFS using Oozie workflows.
- Involved in creating Hive tables, loading data and running hive queries in those data.
- Extensive Working knowledge of partitioned table, UDFs, performance tuning, compression-related properties, thrift server in Hive.
- Used Hive to find correlations between customer's browser logs in different sites and analyzed them to build risk profile for such sites.
- End-to-end performance tuning of Hadoop clusters and Map Reduce routines against very large data sets.
- Monitored Hadoop cluster job performance and performed capacity planning and managed nodes on Hadoop cluster.
- Very good understanding Cassandra cluster mechanism that includes replication strategies, snitch, gossip, consistent hashing and consistency levels.
- Performed Java web application development using J2EE and Net beans.
- Developing software architecture using J2EE.
- Setup the application in development, test and Production environments
- Participate in Daily scrum calls and track day to day activities using Rally
- Expertise in working in agile environment.
- Created Hive Generic UDF's to process business logic that varies based on policy.
- Experienced with using different kind of compression techniques to save data and optimize data transfer over network using Lzo, Snappy etc in Hive tables.
- Creating Hive tables, loading with data and writing Hive queries which will run internally in Map Reduce way
Environment: HDP 2.3.4, Hadoop, Hive, HDFS, HPC, WEBHDFS, J2EE, Net Beans, KAFKA, AWS, Java, Scala, Web Server's, Maven Build and SBT build, Rally
