We provide IT Staff Augmentation Services!

Hadoop Developer Resume

0/5 (Submit Your Rating)

Cincinnati, OH

SUMMARY

  • Result oriented professional with 7 years of IT experience which includes Analysis, Design, Development, Integration, Deployment and Maintenance of quality software applications using Java/J2EE Technologies and Big dataHadooptechnologies.
  • Around 3 Years of experience in Big DataHadoopEcosystems with ingestion, storage, querying, processing and analysis of big data.
  • Excellent understanding ofHadooparchitecture and its components such as Job Tracker, Task Tracker, Name Node, Secondary Name Node, Data Node and MapReduce programming paradigm.
  • Hands on experience in installing, configuring, monitoring and integration ofHadoopecosystem components like MapReduce, HDFS, HBase, Pig, Hive, Oozie, Sqoop, Flume, Spark, Splunk and ZooKeeper.
  • Experience in building, maintaining multipleHadoopclusters (prod, dev etc.,) of different sizes and configuration and setting up the Rack Topology for large clusters.
  • Experience using variousHadoopDistributions (Cloudera, Hortonworks, MapR etc) to fully implement and leverage newHadoopfeatures.
  • Expertise in deployment of Hadoop, Yarn,Sparkand Storm integration with Cassandra, Ignite and RabbitMQ, Kafka etc.
  • Good exposure to ApacheHadoopMap Reduce programming, Hive, PIG scripting and HDFS.
  • Experience in managing and reviewingHadooplog files.
  • Extensively used Apache Kafka to load the log data from multiple sources directly into HDFS.
  • Expertise in writing custom UDFs and UDAFs in Pig & Hive Core Functionality.
  • Experience in Data Load Management, importing and exporting data from HDFS to RDBMS and vice versa using Sqoop and Flume.
  • Experience in Elastic search technologies in creating custom Lucene/Solr Query components.
  • Good experience in using Graph databases such as Neo4j and Tableau.
  • Good knowledge in Machine Learning algorithms and used Apache Mahout and Spark MLib.
  • Good experience in Apache Storm based streaming analytics.
  • Expertise in Amazon Web Services AWS EMR and EC2 web services which provides fast and efficient processing of Big Data.
  • Experience in using Cloudera Manager for installation and management of single - node and multi-node Hadoop cluster (CDH3, CDH4 & CDH5).
  • Experience in cloud stack such as Amazon AWS Redshift and VMWARE stack.
  • Exported the analyzed data to various Databases like Teradata (Sales Data Warehouse), SQL-Server using Sqoop.
  • Experience in developingsplunkdashboards, creating data models, summary indexes and forwarder management.
  • Very good experience with both MapReduce 1 (Job Tracker) and MapReduce 2 (YARN) setups
  • Very good experience in monitoring and managing the Hadoop cluster usingCloudera Manager.
  • Excellent understanding and knowledge of NoSQL databases like MongoDB, HBase, and Cassandra.
  • ImplementedKerberosauthentication for Hadoop services.
  • Experience in usingImpalato analyze stored data.
  • Hands on experience in ETL tools such as Informatica 8.6.1 and Talend.
  • Strong understanding of data warehousing concepts, OLTP and OLAP data models.
  • Strong experience in working with UNIX/LINUX environments, writing shell scripts.
  • Good in writing Python, Shell and Perl scripts to automate activities via Jenkins and Monitoring Build System.
  • Good experience working withScalaand Python.
  • Extensive experience in Programming, Deploying, Configuring on Web Servers such as IBM Web sphere, BEA's Web Logic, Junit, Apache Tomcat Server, Jbuilder and Amazon EC2.

TECHNICAL SKILLS

Hadoop/Big Data: HDFS, MapReduce, HBase, Pig, Hive, Sqoop, Flume, Storm, MongoDB, Oozie, Zookeeper, Cassandra, Spark, Splunk, & Kafka

Java Technologies: Java, J2EE, JSTL, JDBC 3.0/2.1, JSP 1.2/1.1, Java Servlets, JMS, JUnit, Log4j

IDE Tools: Eclipse 3.5, Net Beans, My Eclipse, Oracle JDeveloper 10.1.3, SOAP UI, Ant

Big data Analytics: Datameer 2.0.5

Frameworks: MVC, Struts, Hibernate, Spring

Visualization Tools: Tableau, Talend

Programming languages: C, C++, Java, Python, Scala, Ant scripts, Linux shell scripts

Databases: Oracle 9i/10g, MySQL, DB2, MS-SQL Server, Teradata

Web Servers: IIS 7.0, AWS, Redshift, EC2, S3, RDS, ELB

Web Technologies: HTML, XML, JavaScript, AJAX, SOAP, WSDL

ETL Tools: Informatica, Pentaho, SSRS, SSIS, BO, Crystal reports, Cognos.

PROFESSIONAL EXPERIENCE

Hadoop Developer

Confidential, Cincinnati, OH

Responsibilities:

  • Worked on a live60 nodes Hadoop clusterrunningCDH4.4
  • Worked with highlyunstructured and semi structured data of 90 TBin size (270 TB with replication factor of 3)
  • Used SQOOP to move the structured data from Teradata, Oracle and SQL.
  • Worked and createdSqoop (version 1.4.3)jobs with incremental loadto populate Hive External tables.
  • Extensive experience in writingPig (version 0.11)scripts to transform raw datafrom several data sources into forming baseline data.
  • DevelopedHive(version 0.10) scripts for end user / analyst requirements to perform ad hoc analysis
  • Involved in capacity planning, configuration of Cassandra Cluster on DATASTAX.
  • Real time streaming the data using Spark with Kafka.
  • Involved in using components of Spark including Spark SQL, Spark Stream and MLib.
  • Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala.
  • Involved in data modeling the tables on Cassandra.
  • Worked in converting Hive/SQL queries into Spark transformations using Spark RDDs,Pythonand Scala.
  • Experience in using Sqoop to import the data on to Cassandra tables from different RDBMS.
  • Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scale and databases such as HBase, andMongoDB.
  • Very good understanding ofPartitions, Bucketingconcepts in Hive and designed both Managed and Externaltables in Hive to optimize performance
  • Solved performance issuesin Hive and Pig scripts with understanding of Joins, Group and aggregation and how does it translate to MapReduce jobs.
  • Worked on debugging, performance and tuning of Hive & Pig Jobs.
  • Prepared, arranged and testedSPLUNKsearch strings and operational strings.
  • DevelopedUDFsin Java as and when necessary to use in PIG and HIVE queries
  • Experience in usingSequence files, RCFile, AVRO and HARfile formats.
  • ImplementApacheStormwith Kafka Spout for streaming and messaging process.
  • DevelopedOozieworkflow for scheduling and orchestrating the ETL process.
  • DevelopedSparkcode using Scala andSpark SQLfor faster testing and processing of data.
  • Documented the data flow formStormand Hive tables
  • Worked with the admin team in designing and upgrading CDH 3 to CDH 4.
  • Involved in Cluster coordination services through Zookeeper and adding new nodes to an existing cluster.
  • Developed Enterprise Lucene/Solrbased solutions to include custom type/object modeling and implementation into the Lucene/Solranalysis (Tokenizers/Filters) pipline.
  • Created customSolrQuery components to enable optimum search matching.
  • Plan, deploy, monitor, and maintain Amazon AWS cloud infrastructure consisting of multiple EC2 nodes and VMWare Vm's as required in the environment.
  • Developed Simple to complex MapReduce streaming jobs usingPythonlanguage that are implemented using Hive and Pig.
  • Administrating Tableau Server backing up the reports and providing privileges to users.
  • Worked on Tableau for generating reports on HDFS data.
  • Extracted and updated the data into Monod using MongoDB import and export command line utility interface.
  • Experienced in implementing POC's to migrate iterative MapReduce programs into Spark transformations using Scala.
  • Moving data from HDFS to RDBMS and vice-versa using SQOOP.
  • Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis
  • Installed and configured Hadoop cluster in Test and Production environments
  • Performed both major and minor upgrades to the existing CDH cluster.
  • Implemented POC for usingAPACHEIMPALA for data processing on top of HIVE.
  • Configured a High Availability in your cluster and Configured security with Kerberos

Environment: Hadoop, HDFS, Pig, Hive, MapReduce, Sqoop, Splunk, Spark, AWS EC2, Redshift, Kafka, Solr, LINUX, Cloudera, Big Data, Scala, Python, SQL, NoSQL, Cassandra, Tableau, Teradata, HBase.

Hadoop Developer

Confidential, Philadelphia, PA

Responsibilities:

  • Worked on a live50 nodes Hadoop clusterrunningCDH 3.
  • Worked with highlyunstructured and semi structured data of 60 TBin size (180 TB with replication factor of 3).
  • Handling structured and unstructured data and applying ETL processes.
  • Developed data pipeline using Flume, Sqoop, Pig and Java mapreduce to ingest customer behavioral data and financial histories into HDFS for analysis.
  • Involved in writing MapReduce jobs.
  • Involved in extracting the data fromTeradatainto HDFS using Sqoop.
  • Involved in using Sqoop, HDFS Put or Copy from Local to ingest data.
  • Acquired good understanding and experience of NoSQL databases such as MongoDB and Cassandra.
  • Used Pig to do transformations, event joins, filter boot traffic and some pre-aggregations before storing the data onto HDFS.
  • Involved in developing Pig UDFs for the needed functionality that is not out of the box available from Apache Pig.
  • Involved in developing Hive DDLs to create, alter and drop tables.
  • Involved in Cluster coordination services through Zoo Keeper.
  • AppliedApacheStormwith Kafka Spout and Hbase/Redis for real time analytics, designing Spouts, Bolts and Topology.
  • Worked on AWS cloud services like Redshift, EC2, S3, RDS, ELB, and EBS for installing, configuring and troubleshooting on various Amazon images for server migration from physical into cloud.
  • Worked on data warehouse product Amazon Redshift which is a part of the AWS (Amazon Web Services).
  • Discussed the implementation level of concurring programming in spark usingpythonwith message passing.
  • Good development experience inInformatica(9.5) Power Center - ETL tool andInformaticaData Quality (IDQ) tool.
  • Developed multiple POCs using PySpark and deployed on the Yarn cluster, compared the performance of Spark, with Hive and SQL/Teradata.
  • Configured and integrated Lucene/Solrfor ALDOT's documents full-text search
  • Installed and configured Cloudera Manager for easy management of existing Hadoop Cluster.
  • Involved in developing Hive UDFs for the needed functionality that is not out of the box available from Apache Hive.
  • Worked on implementation of a log producer in SCALA that watches for application logs, transforms incremental logs and sends them to a Kafka and Zookeeper based log collection platform.
  • Involved in using HCATALOG to access Hive table metadata from MapReduce or Pig code.
  • Created Workbooks and dashboards for analyzing statistical billing data using Tableau 8.2
  • Developed ComplexTalendETL jobs to migrate the data from flat files to database
  • Developed Daily Test engine using Python for continuous tests.

Environment: Hadoop, MapReduce, Cassandra, MongoDB, Yarn, Hive, Pig, HBase,Kafka, Oozie, Sqoop,AWS EC2,Redshift, Spark, Solr, Flume, Oracle 11g, Teradata, Scala, Cloudera, HDFS, Eclipse, Python

Hadoop Developer

Confidential - Dallas, TX

Responsibilities:

  • Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase and Sqoop.
  • Installed Hadoop, MapReduce, HDFS, and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
  • Importing and exporting data into HDFS and Hive using Sqoop.
  • Involved in developing Pig Scripts for change data capture and delta record processing between newly arrived data and already existing data in HDFS.
  • Created Hive tables and worked on them using Hive QL.
  • Worked extensively with Flume for importing data from various webservers to HDFS.
  • Populated HBase and Cassandra with huge amounts of data using Apache Kafka.
  • Used Python for pattern matching in build logs to format errors and warnings.
  • Used Informatica as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data on to HDFS.
  • Installed Oozie workflow engine to run multiple Mapreduce jobs.
  • Extensively used Oozie and Zookeeper to automate the flow of jobs and coordination in the cluster respectively.
  • Involved in loading data from LINUX file system to HDFS.
  • Recommendation engine for Portfolio and Research articles using Apache Spark andMongoDB.
  • Experienced with batch processing of data sources using Apache Spark and Elastic search.
  • Experienced in implementing Spark RDD transformations, actions to implement business analysis migrated HiveQL queries on structured into SparkQL to improve performance.
  • Experienced in migrating the huge volume of data from EDW to IDW Environment.
  • Wrote build and deployed scripts using Shell scripts and involved in performance analysis of the application and fixed problems/suggest solutions.
  • Storing and loading the data from HDFS to Amazon AWS S3 and backing up the Namespace data into NFS Filers.
  • Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.
  • Worked with Linux systems and RDBMS database on a regular basis in order to ingest data using Sqoop.
  • Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.
  • Created many Java UDF and UDAFs in hive for functions that were not preexisting in Hive like the rank, Scum, etc.

Environment: Hadoop, Hive, MapReduce, Pig, Sqoop, Kafka, Cloudera, Spark, HBase, HDFS, Python, Scala, Solr, Zookeeper, Cassandra, MongoDB

Confidential

Java/J2EE Developer

Responsibilities:

  • Involved in Analysis, Design and Implementation/translation of Business User requirements.
  • Designed, Developed & implemented Web services on State street's cloud platform.
  • Designed and developed various modules of the application with J2EE design architecture.
  • Developed Reporting framework usingJAVAand J2EE which will generate daily, monthly and yearly reports and Prepared different framework design documents.
  • Analyzed business requirements and existing software for High Level Design.
  • Worked in an agile development process, monthly Confidential and daily Scrum.
  • Used spring framework for building the application based on MVC design paradigm.
  • Used Springs AOP to implement security, where cross cutting concerns were identified.
  • Developed JSPs, Servlets and custom tags for creating user interfaces.
  • CreatedJavaJ2EE application to View Data inMongoDB.
  • Developed Business logic with the help of Spring and Data Access was implemented using Hibernate.
  • Developed SQL queries and executed them by using JDBC Template provided by Spring.
  • Developed HQL queries to retrieve data using Hibernate. The Data Manipulation operations were implemented using Hibernate Template provided by Spring
  • Implemented OR Mapping with the tables in the Oracle database for one-to-one relation and many-to-one relation with the tables.
  • Created applications inJava/Python to do the data migration from Oracle toMongoDB.
  • Utilized Apache Tomcat server integrated with Eclipse for debugging and Unit testing
  • Developed RESTFul services using Spring and Used JAXB API for XML parsing.
  • Worked on Restful API and invoked web services (consumed) that are based on EJBs.
  • Did Packaging and Deployment of builds through ANT script.
  • Test Driven Development (TDD) approach was utilized and the test coverage was always maintained and validated using Clover and Cruise Control.
  • Created LDAP services for user authentication and authorization.
  • Web logic Application Server is being used as the Business Service Tool in middle tier.
  • Worked with NoSQL database MongoDB and worked with it to perform many different operations
  • Used Log4j for tracking the applications and Used Harvest as version control tool and Clear Quest for defect management.
  • Consumed and created REST Web services for Quick Quote Details
  • Involved in exposing, consuming and packaging Web services using Spring Framework
  • Involved in the code review process and updating the best practices document.

Environment: Java1.6, Jersey REST, Web logic, Oracle 11, MongoDB, Spring MVC, IOC, Spring AOP, Hibernate, Scrum, NoSql, ANT, SVN, Jdeveloper, Putty.

Confidential

Java Developer

Responsibilities:

  • Responsible for design and development of Web Application using Struts Framework.
  • Written Action Classes, Form Bean Classes and configure the Application using Struts Configuration file.
  • Did technical design to conform to STRUTS (MVC) framework.
  • Wrote server side programs by using Servlets and JSP.
  • Designed and developed the HTML front end screens and validated forms using JavaScript.
  • Made use of Object Oriented concepts like Inheritance, polymorphism and Abstraction.
  • Application and user level configurations have been maintained by using XML Files.
  • Widely used HTML for web based design.
  • Implemented MVC using Struts Framework.
  • Utilized Servlets to handle various requests from the client browser and send responses.
  • Created and implemented PL/SQL stored procedures, triggers.
  • Designing and documenting of the stored procedures.
  • Coding Test Classes using Junit for unit testing, Performed functional integration system and validation testing.
  • Customized RESTful Web Service using Spring RESTful API, sending JSON format data packets between front-end and middle-tier controller.
  • Used JDBC Prepared statements to call from Servlets for database access
  • Implemented design patterns MVC, Session Facade for developing the application.
  • Developed Message Driven Beans for asynchronous processing of alerts.

Environment: Java, J2EE, JSP, Servlets, HTML, DHTML, XML, JavaScript, Struts 2.2, Eclipse, Apache Tomcat, PL/SQL, Oracle9i.

We'd love your feedback!