Big Data/hadoop Developer Resume
Phoenix, AZ
SUMMARY
- 7+years of extensive Professional IT experience, including 4+ years of Hadoop /Big data experience, 3 years of Java Experience, capable of processing large sets of structured, semi - structured and unstructured data and supporting systems application architecture.
- Expert in importing and exporting of data using Sqoop from HDFS to relational database system and vice-versa.
- Expert level of scripting using Pig scripts and Hive queries for processing and analyzing large volume of data.
- Experience with Oozie Workflow Engine in running workflow designing, job scheduling with actions that run Hadoop Map/Reduce and Pig jobs.
- Good Experience in developing and implementing big data solutions and data mining applications on Hadoopusing Hive, PIG, Hbase, Hue, Oozie workflows and designing and implementing Java, Python Map Reduce programs.
- Good knowledge onHadoopcluster administration, Monitoring and managingHadoopcluster using cloudera manager.
- Knowledge in installing, configuring, and usingHadoopecosystem components likeHadoopMap Reduce, HDFS, HBase, Oozie, Hive, Sqoop, Pig, Flume, Apache Spark, Zookeeper and Kafka.
- Experience in managing and reviewingHadooplog files.
- Hands on experience in converting Hive/SQL queries into Spark transformations using Scala.
- Good Experience with flume tool for data ingestion from various data producers (webservers) intoHadoop.
- Good knowledge in NOSQL databases HBASE, MongoDB.
- Sound Relational Database Concepts and extensively worked with ORACLE, MySQL, SQL Server.
- Good Experience with databases, writing complex queries and stored procedures using SQL and PL/SQL.
- Experience in using sequence file, RC file and Avro file formats.
- Good understanding of ClassicHadoopand Yarn architecture along with variousHadoopDemons such as Job Tracker, Task Tracker, Name Node, Data Node, Secondary Name Node, Resource Manager, Node Manager, Application Master and Containers.
- Very good experience with both MapReduce 1 (Job Tracker) and MapReduce 2 (YARN) setups.
- Expert in Java MapReduce Jobs, User Defined functions for Pig and Hive.
- Knowledge in handling messaging services using Apache Kafka.
- Familiarity on real time streaming data with Sparkfor fast large scale in memory Map Reduce.
- Experience in Business Intelligence tools such as Tableau, Datameer for visually analyzing the data.
- Experience in building, maintaining multipleHadoopclusters of different sizes and configuration and setting up the rack topology for large clusters.
- Developedmachine learning algorithms using Mahout for clustering and data mining.
- Involvement in all phases of SDLC from project proposal, planning, analysis, development, testing, deployment and support.
- Experience in developing and implementing web applications using Java, JSP, CSS, HTML, HTML5, XHTML and Java script, JSON, XML, JDBC.
- Experience in working in 24X7 Support and used to meet deadlines, adaptable to ever changing priorities.
- Proven ability to work with senior technical managers and staff to provide expert-level support for the installation, maintenance, upgrading, and administration of full-featured database management systems.
- Excellent interpersonal and communication skills, creative, research-minded, technically competent, result-oriented with problem solving as well and ability to work well with people and to maintain a good relation with the organization.
TECHNICAL SKILLS
Technology Tools: Big data andHadoop, Apache/Cloudera HDFS 1.X/2.X, MapReduce, YARN, Sqoop, Flume, Spark, Scala, Hive, Pig, Oozie, Zookeeper, Kafka
Operating Systems: MS Windows, Linux, Ubuntu, CentOS
Programming Languages: C, C++, Java, SQL, PL/SQL, Python, JavaScript,Unix Shell Scripting
Database: Oracle, MySQL, Microsoft Sql Server
NoSQL Databases: Hbase, MongoDB, Cassandra.
IDE Tools: Eclipse, Net Beans
Other Skills: Tableau, Datameer, HTML5, Java Script, JSON, CSS XML, Apache Tomcat
PROFESSIONAL EXPERIENCE
Confidential, Phoenix, AZ
Big Data/Hadoop Developer
Responsibilities:
- Installed and Configured ApacheHadoopclusters for application development andHadooptools like Hive, Pig, Oozie, Zookeeper, Hbase, Flume and Sqoop.
- Implemented multiple Map Reduce Jobs in java for data cleaning and pre - processing.
- Worked in a team with 40 node cluster and increase cluster by adding Nodes, the configuration for additional data nodes was done by Commissioning process inHadoop.
- Responsible for Cluster maintenance, adding and removing cluster nodes, Cluster Monitoring and Troubleshooting, manage and review data backups and log files.
- Responsible to manage data coming from different sources.
- Managed and scheduled Jobs on aHadoopcluster.
- Implemented a script to transmit information from Oracle to Hbase using Sqoop.
- Involved in defining job flows, managing and reviewing log files.
- Installed Oozie workflow engine to run multiple Map Reduce, HiveQL and Pig jobs.
- Participated in requirement gathering form the Experts and Business Partners and converting the requirements into technical specifications.
- Created Hive tables to store the processed results in a tabular format.
- Was done various compressions and file formats like snappy, Gzip, Avro, Sequence, text.
- Wrote complex Hive queries and UDFs in Java and Python.
- Created and exposed Hive views through Impalafor the business Users
- Involved in forecast based on the present results and insights derived from data analysis.
- Involved in collecting the data and identifying data patterns to build trained model using Machine Learning.
- Prepare Developer(Unit) Test cases and execute developertesting.
- Implemented test scripts to support test driven development and continuous integration.
- Developed and implemented some machine learning algorithmsusing Mahout for data mining for the data stored in HDFS
- Created and maintained Technical documentation for launchingHadoopClusters and for executing Hive queries and Pig Scripts.
- Worked on visualization tool tableau for visually analyzing the data.
Environment: Hadoop, HDFS, Pig, Hive, Map Reduce, Impala, Sqoop, Flume, Oozie, Big Data, java, Python, Mahout, Junit testing, Oracle, MySQL, Tableau, LINUX, Windows.
Confidential, Phoenix, AZ
Java/Hadoop Developer
Responsibilities:
- Worked on data migration from existing data sources toHadoopfile system.
- Interact with the Business Analysts to get the requirements and formulate them into system use cases.
- Understand customer business use cases and be able to translate them to analytical data applications and models to implement a solution.
- Created flow charts, sequence diagrams, schemas, data model of underlying system, pseudocode and class diagrams using Microsoft Vision.
- Ability to create Hive Avro Tables to read and write the data with the appropriate compressions.
- Created custom Database Encryption & Decryption UDF that could be plugged in while Ingesting data to External Hive Tables for maintaining security at table or column level.
- Worked on No - SqlHbase database for processing very large semi structured and structured tables by defining column families, region servers etc.
- ImplementedSparkSQL for faster testing and processing of data.
- Knowledge on handling Hive queries usingSparkSQL that integrate withSparkenvironment.
- Worked on migrating MapReduce programs intoSparktransformations usingSpark.
- ConfiguredSparkStreaming to receive real time data and store the stream data to HDFS.
- Developed map-reduce programs for different patterns of data onHadoopcluster.
- Developed java map reduce programs using core concepts like OOPS, Multithreading, Collections and IO. Compiled and built the application using MAVEN and used SVN as version control system.
- Created data ingestion plans for loading the data from external sources using Sqoop.
- Used Python for pattern matching in build logs to format errors and warnings
- Created flow charts, sequence diagrams, schemas, data model of underlying system, pseudo code and class diagrams using Microsoft Visio.
Environment: Hadoop, HDFS, Pig, Hive, Map Reduce, Sqoop, Oozie, Spark, Big Data, Hbase, java, Python,Unix Shell Scripting, Oracle 11g/10g, MySQL, MS Visio, LINUX, Windows.
Confidential, Chicago, IL
Java/Hadoop Developer
Responsibilities:
- Analyzed large data sets by running Hive queries and Pig scripts.
- Involved in creating Hive tables, and loading and analyzing data using hive queries.
- Developed Simple to complex Map Reduce Jobs using Hive and Pig
- Involved in runningHadoopjobs for processing millions of records of text data.
- Load and transform large sets of structured, semi structured and unstructured data.
- Responsible to manage data coming from different sources.
- Implemented multiple Map Reduce Jobs in java for data cleaning and pre - processing.
- Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
- Involved in loading data from Unix file system to HDFS.
- Monitor System health and logs and respond accordingly to any warning or failure conditions.
- Implemented the workflows using Apache Oozie framework to automate tasks.
- Worked with application teams to install operating system,Hadoopupdates, patches, version upgrades as required.
- Developed unit test cases forHadoopMapReduce jobs with JUnit.
- Developed multiple Map Reduce jobs in java for data cleaning and preprocessing.
- Involved in maintaining variousUnix Shell scripts.
- Assisted in exporting analyzed data to relational databases using Sqoop.
- Supported Map Reduce Programs those are running on the cluster.
- Created and maintained Technical documentation for launchingHadoopClusters and for executing Hive queries and Pig Scripts.
Environment: Hadoop, HDFS, Pig, Hive, MongoDB, Map Reduce, Sqoop, Oozie, Unix Shell Scripting, Big Data, java, Oracle 11g/10g, MySQL, LINUX, Windows.
Confidential
Java Developer
Responsibilities:
- Managed connectivity using JDBC for querying/inserting & data management including triggers and stored procedures.
- Developed UI using HTML, JavaScript and JSP developed Business Logic and Interfacing components using Business Objects, XML.
- Communicated with client to Analyze and Review of Business/Technical Requirements
- Designed and developed Co - Branding framework to change UI for different clients using same application.
- Used Struts, Spring (MVC) Framework to develop the application.
- Used Factory, DAO, Singleton and DTO and Value Object, Business Delegate design patterns.
- Designed and developed database layer using ORM technologies like Hibernate.
- Developed Web services client to consume with Vender Web services
- Generated Excel reports Using POI framework
- Implementing the business logic and creating dynamic web pages with JSP and using JavaScript to in corporate client side and server side validations and functionality.
- Preparation and Review of Unit Test Plans, Scripts and Results
- Used respective designed patterns to implement the reusable component of the application.
- Implemented various GUI screens using JSP, AJAX Frame work and JQuery.
- Worked with web methods for deploying and administering services.
- Used Hibernate to interact with database.
- Worked with requirement analysis team to gather software requirements for application development.
- Provided support and maintenance after deploying the web application.
- Resolved issues reported by the client.
Environment: Linux, Windows, Core Java, JSP, JavaScript Hibernate, Servlets, Spring MVC Framework, Hibernate, Oracle, JMS, Ajax, JQuery, XML, Log4j, Apache Tomcat and Eclipse.
