Hadoop Developer Resume
Cincinnati, OH
SUMMARY
- Result oriented professional with 7 years of IT experience which includes Analysis, Design, Development, Integration, Deployment and Maintenance of quality software applications using Java/J2EE Technologies and Big dataHadooptechnologies.
- Around 3 Years of experience in Big DataHadoopEcosystems with ingestion, storage, querying, processing and analysis of big data.
- Excellent understanding ofHadooparchitecture and its components such as Job Tracker, Task Tracker, Name Node, Secondary Name Node, Data Node and MapReduce programming paradigm.
- Hands on experience in installing, configuring, monitoring and integration ofHadoopecosystem components like MapReduce, HDFS, HBase, Pig, Hive, Oozie, Sqoop, Flume, Spark, Splunk and ZooKeeper.
- Experience in building, maintaining multipleHadoopclusters (prod, dev etc.,) of different sizes and configuration and setting up the Rack Topology for large clusters.
- Experience using variousHadoopDistributions (Cloudera, Hortonworks, MapR etc) to fully implement and leverage newHadoopfeatures.
- Expertise in deployment of Hadoop, Yarn,Sparkand Storm integration with Cassandra, Ignite and RabbitMQ, Kafka etc.
- Good exposure to ApacheHadoopMap Reduce programming, Hive, PIG scripting and HDFS.
- Experience in managing and reviewingHadooplog files.
- Extensively used Apache Kafka to load the log data from multiple sources directly into HDFS.
- Expertise in writing custom UDFs and UDAFs in Pig & Hive Core Functionality.
- Experience in Data Load Management, importing and exporting data from HDFS to RDBMS and vice versa using Sqoop and Flume.
- Experience in Elastic search technologies in creating custom Lucene/Solr Query components.
- Good experience in using Graph databases such as Neo4j and Tableau.
- Good knowledge in Machine Learning algorithms and used Apache Mahout and Spark MLib.
- Good experience in Apache Storm based streaming analytics.
- Expertise in Amazon Web Services AWS EMR and EC2 web services which provides fast and efficient processing of Big Data.
- Experience in using Cloudera Manager for installation and management of single - node and multi-node Hadoop cluster (CDH3, CDH4 & CDH5).
- Experience in cloud stack such as Amazon AWS Redshift and VMWARE stack.
- Exported the analyzed data to various Databases like Teradata (Sales Data Warehouse), SQL-Server using Sqoop.
- Experience in developingsplunkdashboards, creating data models, summary indexes and forwarder management.
- Very good experience with both MapReduce 1 (Job Tracker) and MapReduce 2 (YARN) setups
- Very good experience in monitoring and managing the Hadoop cluster usingCloudera Manager.
- Excellent understanding and knowledge of NoSQL databases like MongoDB, HBase, and Cassandra.
- ImplementedKerberosauthentication for Hadoop services.
- Experience in usingImpalato analyze stored data.
- Hands on experience in ETL tools such as Informatica 8.6.1 and Talend.
- Strong understanding of data warehousing concepts, OLTP and OLAP data models.
- Strong experience in working with UNIX/LINUX environments, writing shell scripts.
- Good in writing Python, Shell and Perl scripts to automate activities via Jenkins and Monitoring Build System.
- Good experience working withScalaand Python.
- Extensive experience in Programming, Deploying, Configuring on Web Servers such as IBM Web sphere, BEA's Web Logic, Junit, Apache Tomcat Server, Jbuilder and Amazon EC2.
TECHNICAL SKILLS
Hadoop/Big Data: HDFS, MapReduce, HBase, Pig, Hive, Sqoop, Flume, Storm, MongoDB, Oozie, Zookeeper, Cassandra, Spark, Splunk, & Kafka
Java Technologies: Java, J2EE, JSTL, JDBC 3.0/2.1, JSP 1.2/1.1, Java Servlets, JMS, JUnit, Log4j
IDE Tools: Eclipse 3.5, Net Beans, My Eclipse, Oracle JDeveloper 10.1.3, SOAP UI, Ant
Big data Analytics: Datameer 2.0.5
Frameworks: MVC, Struts, Hibernate, Spring
Visualization Tools: Tableau, Talend
Programming languages: C, C++, Java, Python, Scala, Ant scripts, Linux shell scripts
Databases: Oracle 9i/10g, MySQL, DB2, MS-SQL Server, Teradata
Web Servers: IIS 7.0, AWS, Redshift, EC2, S3, RDS, ELB
Web Technologies: HTML, XML, JavaScript, AJAX, SOAP, WSDL
ETL Tools: Informatica, Pentaho, SSRS, SSIS, BO, Crystal reports, Cognos.
PROFESSIONAL EXPERIENCE
Hadoop Developer
Confidential, Cincinnati, OH
Responsibilities:
- Worked on a live60 nodes Hadoop clusterrunningCDH4.4
- Worked with highlyunstructured and semi structured data of 90 TBin size (270 TB with replication factor of 3)
- Used SQOOP to move the structured data from Teradata, Oracle and SQL.
- Worked and createdSqoop (version 1.4.3)jobs with incremental loadto populate Hive External tables.
- Extensive experience in writingPig (version 0.11)scripts to transform raw datafrom several data sources into forming baseline data.
- DevelopedHive(version 0.10) scripts for end user / analyst requirements to perform ad hoc analysis
- Involved in capacity planning, configuration of Cassandra Cluster on DATASTAX.
- Real time streaming the data using Spark with Kafka.
- Involved in using components of Spark including Spark SQL, Spark Stream and MLib.
- Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala.
- Involved in data modeling the tables on Cassandra.
- Worked in converting Hive/SQL queries into Spark transformations using Spark RDDs,Pythonand Scala.
- Experience in using Sqoop to import the data on to Cassandra tables from different RDBMS.
- Configured Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scale and databases such as HBase, andMongoDB.
- Very good understanding ofPartitions, Bucketingconcepts in Hive and designed both Managed and Externaltables in Hive to optimize performance
- Solved performance issuesin Hive and Pig scripts with understanding of Joins, Group and aggregation and how does it translate to MapReduce jobs.
- Worked on debugging, performance and tuning of Hive & Pig Jobs.
- Prepared, arranged and testedSPLUNKsearch strings and operational strings.
- DevelopedUDFsin Java as and when necessary to use in PIG and HIVE queries
- Experience in usingSequence files, RCFile, AVRO and HARfile formats.
- ImplementApacheStormwith Kafka Spout for streaming and messaging process.
- DevelopedOozieworkflow for scheduling and orchestrating the ETL process.
- DevelopedSparkcode using Scala andSpark SQLfor faster testing and processing of data.
- Documented the data flow formStormand Hive tables
- Worked with the admin team in designing and upgrading CDH 3 to CDH 4.
- Involved in Cluster coordination services through Zookeeper and adding new nodes to an existing cluster.
- Developed Enterprise Lucene/Solrbased solutions to include custom type/object modeling and implementation into the Lucene/Solranalysis (Tokenizers/Filters) pipline.
- Created customSolrQuery components to enable optimum search matching.
- Plan, deploy, monitor, and maintain Amazon AWS cloud infrastructure consisting of multiple EC2 nodes and VMWare Vm's as required in the environment.
- Developed Simple to complex MapReduce streaming jobs usingPythonlanguage that are implemented using Hive and Pig.
- Administrating Tableau Server backing up the reports and providing privileges to users.
- Worked on Tableau for generating reports on HDFS data.
- Extracted and updated the data into Monod using MongoDB import and export command line utility interface.
- Experienced in implementing POC's to migrate iterative MapReduce programs into Spark transformations using Scala.
- Moving data from HDFS to RDBMS and vice-versa using SQOOP.
- Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis
- Installed and configured Hadoop cluster in Test and Production environments
- Performed both major and minor upgrades to the existing CDH cluster.
- Implemented POC for usingAPACHEIMPALA for data processing on top of HIVE.
- Configured a High Availability in your cluster and Configured security with Kerberos
Environment: Hadoop, HDFS, Pig, Hive, MapReduce, Sqoop, Splunk, Spark, AWS EC2, Redshift, Kafka, Solr, LINUX, Cloudera, Big Data, Scala, Python, SQL, NoSQL, Cassandra, Tableau, Teradata, HBase.
Hadoop Developer
Confidential, Philadelphia, PA
Responsibilities:
- Worked on a live50 nodes Hadoop clusterrunningCDH 3.
- Worked with highlyunstructured and semi structured data of 60 TBin size (180 TB with replication factor of 3).
- Handling structured and unstructured data and applying ETL processes.
- Developed data pipeline using Flume, Sqoop, Pig and Java mapreduce to ingest customer behavioral data and financial histories into HDFS for analysis.
- Involved in writing MapReduce jobs.
- Involved in extracting the data fromTeradatainto HDFS using Sqoop.
- Involved in using Sqoop, HDFS Put or Copy from Local to ingest data.
- Acquired good understanding and experience of NoSQL databases such as MongoDB and Cassandra.
- Used Pig to do transformations, event joins, filter boot traffic and some pre-aggregations before storing the data onto HDFS.
- Involved in developing Pig UDFs for the needed functionality that is not out of the box available from Apache Pig.
- Involved in developing Hive DDLs to create, alter and drop tables.
- Involved in Cluster coordination services through Zoo Keeper.
- AppliedApacheStormwith Kafka Spout and Hbase/Redis for real time analytics, designing Spouts, Bolts and Topology.
- Worked on AWS cloud services like Redshift, EC2, S3, RDS, ELB, and EBS for installing, configuring and troubleshooting on various Amazon images for server migration from physical into cloud.
- Worked on data warehouse product Amazon Redshift which is a part of the AWS (Amazon Web Services).
- Discussed the implementation level of concurring programming in spark usingpythonwith message passing.
- Good development experience inInformatica(9.5) Power Center - ETL tool andInformaticaData Quality (IDQ) tool.
- Developed multiple POCs using PySpark and deployed on the Yarn cluster, compared the performance of Spark, with Hive and SQL/Teradata.
- Configured and integrated Lucene/Solrfor ALDOT's documents full-text search
- Installed and configured Cloudera Manager for easy management of existing Hadoop Cluster.
- Involved in developing Hive UDFs for the needed functionality that is not out of the box available from Apache Hive.
- Worked on implementation of a log producer in SCALA that watches for application logs, transforms incremental logs and sends them to a Kafka and Zookeeper based log collection platform.
- Involved in using HCATALOG to access Hive table metadata from MapReduce or Pig code.
- Created Workbooks and dashboards for analyzing statistical billing data using Tableau 8.2
- Developed ComplexTalendETL jobs to migrate the data from flat files to database
- Developed Daily Test engine using Python for continuous tests.
Environment: Hadoop, MapReduce, Cassandra, MongoDB, Yarn, Hive, Pig, HBase,Kafka, Oozie, Sqoop,AWS EC2,Redshift, Spark, Solr, Flume, Oracle 11g, Teradata, Scala, Cloudera, HDFS, Eclipse, Python
Hadoop Developer
Confidential - Dallas, TX
Responsibilities:
- Worked on analyzing Hadoop cluster and different Big Data analytic tools including Pig, Hive, HBase and Sqoop.
- Installed Hadoop, MapReduce, HDFS, and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
- Importing and exporting data into HDFS and Hive using Sqoop.
- Involved in developing Pig Scripts for change data capture and delta record processing between newly arrived data and already existing data in HDFS.
- Created Hive tables and worked on them using Hive QL.
- Worked extensively with Flume for importing data from various webservers to HDFS.
- Populated HBase and Cassandra with huge amounts of data using Apache Kafka.
- Used Python for pattern matching in build logs to format errors and warnings.
- Used Informatica as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data on to HDFS.
- Installed Oozie workflow engine to run multiple Mapreduce jobs.
- Extensively used Oozie and Zookeeper to automate the flow of jobs and coordination in the cluster respectively.
- Involved in loading data from LINUX file system to HDFS.
- Recommendation engine for Portfolio and Research articles using Apache Spark andMongoDB.
- Experienced with batch processing of data sources using Apache Spark and Elastic search.
- Experienced in implementing Spark RDD transformations, actions to implement business analysis migrated HiveQL queries on structured into SparkQL to improve performance.
- Experienced in migrating the huge volume of data from EDW to IDW Environment.
- Wrote build and deployed scripts using Shell scripts and involved in performance analysis of the application and fixed problems/suggest solutions.
- Storing and loading the data from HDFS to Amazon AWS S3 and backing up the Namespace data into NFS Filers.
- Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.
- Worked with Linux systems and RDBMS database on a regular basis in order to ingest data using Sqoop.
- Used Pig as ETL tool to do transformations, event joins and some pre-aggregations before storing the data onto HDFS.
- Created many Java UDF and UDAFs in hive for functions that were not preexisting in Hive like the rank, Scum, etc.
Environment: Hadoop, Hive, MapReduce, Pig, Sqoop, Kafka, Cloudera, Spark, HBase, HDFS, Python, Scala, Solr, Zookeeper, Cassandra, MongoDB
Confidential
Java/J2EE Developer
Responsibilities:
- Involved in Analysis, Design and Implementation/translation of Business User requirements.
- Designed, Developed & implemented Web services on State street's cloud platform.
- Designed and developed various modules of the application with J2EE design architecture.
- Developed Reporting framework usingJAVAand J2EE which will generate daily, monthly and yearly reports and Prepared different framework design documents.
- Analyzed business requirements and existing software for High Level Design.
- Worked in an agile development process, monthly Confidential and daily Scrum.
- Used spring framework for building the application based on MVC design paradigm.
- Used Springs AOP to implement security, where cross cutting concerns were identified.
- Developed JSPs, Servlets and custom tags for creating user interfaces.
- CreatedJavaJ2EE application to View Data inMongoDB.
- Developed Business logic with the help of Spring and Data Access was implemented using Hibernate.
- Developed SQL queries and executed them by using JDBC Template provided by Spring.
- Developed HQL queries to retrieve data using Hibernate. The Data Manipulation operations were implemented using Hibernate Template provided by Spring
- Implemented OR Mapping with the tables in the Oracle database for one-to-one relation and many-to-one relation with the tables.
- Created applications inJava/Python to do the data migration from Oracle toMongoDB.
- Utilized Apache Tomcat server integrated with Eclipse for debugging and Unit testing
- Developed RESTFul services using Spring and Used JAXB API for XML parsing.
- Worked on Restful API and invoked web services (consumed) that are based on EJBs.
- Did Packaging and Deployment of builds through ANT script.
- Test Driven Development (TDD) approach was utilized and the test coverage was always maintained and validated using Clover and Cruise Control.
- Created LDAP services for user authentication and authorization.
- Web logic Application Server is being used as the Business Service Tool in middle tier.
- Worked with NoSQL database MongoDB and worked with it to perform many different operations
- Used Log4j for tracking the applications and Used Harvest as version control tool and Clear Quest for defect management.
- Consumed and created REST Web services for Quick Quote Details
- Involved in exposing, consuming and packaging Web services using Spring Framework
- Involved in the code review process and updating the best practices document.
Environment: Java1.6, Jersey REST, Web logic, Oracle 11, MongoDB, Spring MVC, IOC, Spring AOP, Hibernate, Scrum, NoSql, ANT, SVN, Jdeveloper, Putty.
Confidential
Java Developer
Responsibilities:
- Responsible for design and development of Web Application using Struts Framework.
- Written Action Classes, Form Bean Classes and configure the Application using Struts Configuration file.
- Did technical design to conform to STRUTS (MVC) framework.
- Wrote server side programs by using Servlets and JSP.
- Designed and developed the HTML front end screens and validated forms using JavaScript.
- Made use of Object Oriented concepts like Inheritance, polymorphism and Abstraction.
- Application and user level configurations have been maintained by using XML Files.
- Widely used HTML for web based design.
- Implemented MVC using Struts Framework.
- Utilized Servlets to handle various requests from the client browser and send responses.
- Created and implemented PL/SQL stored procedures, triggers.
- Designing and documenting of the stored procedures.
- Coding Test Classes using Junit for unit testing, Performed functional integration system and validation testing.
- Customized RESTful Web Service using Spring RESTful API, sending JSON format data packets between front-end and middle-tier controller.
- Used JDBC Prepared statements to call from Servlets for database access
- Implemented design patterns MVC, Session Facade for developing the application.
- Developed Message Driven Beans for asynchronous processing of alerts.
Environment: Java, J2EE, JSP, Servlets, HTML, DHTML, XML, JavaScript, Struts 2.2, Eclipse, Apache Tomcat, PL/SQL, Oracle9i.
