Hadoop Developer Resume
Pleasanton, CA
SUMMARY
- Hadoop Big Data/Java Engineer with 8 years of working knowledge in the areas of Java/Hadoop with interest to explore, learn and extract the best out of the modern technologies in the field of Big Data Engineering technologies.
- Expertise in HDFS, MapReduce, Spark, Hive, Impala, Pig, Sqoop, Hbase, Oozie, Flume, Kafka, Storm and various other ecosystem components
- Good understanding of cloud configuration in Amazon Web Services AWS.
- Highly interacted with Amazon Web Services S3 buckets to store the files and information in a secured and efficient manner to reduce the burden on the servers.
- Experience in installing, configuring and using Hadoop Ecosystem components.
- Working experience on designing and implementing complete end - to-end Hadoop Infrastructure including PIG, HIVE, Sqoop, Oozie, Flume and zookeeper.
- Experience in supporting analysts by administering and configuring HIVE.
- Experience in running Pig and Hive scripts
- Great interest in observing and learning the latest technologies in the Big Data world.
- Good Knowledge with NoSql Databases - Cassandra, Mongo DB and HBase. Experience in Big Data analysis using PIG and HIVE and understanding of SQOOP and Puppet
- Experience in analyzing data using HiveQL, PIG Latin, and custom Map Reduce programs in JAVA
- Very Good understanding and Working Knowledge of Object Oriented Programming (OOPS)
- Hands on experience in application development using Java, RDBMS, and Linux shell scripting.
- Hands on Experience in writing MapReduce programs using Scala.
- Hands on Experience in using IDE tools like Eclipse.
- Strong front-end UI development skills using scripting languages like JSP, HTML, JavaScript, JQuery and CSS.
- Responsible for ownership of complete software development lifecycle - which includes making requirement specifications, analysis design, implementation and testing - for several projects.
- Proficiency in SDLC methodologies like Agile, Waterfall models.
- Have flair to adapt to new software applications and products, self-starter, have excellent communication skills and good understanding of business work flow.
- Exceptional ability to quickly master new technology and capable of working in-group as well as independently with excellent communication skills.
- Ability to work under pressure with critical deadlines.
TECHNICAL SKILLS
Languages: C, C++, Java, Shell Scripting, Python, PHP
Big Data Technologies: Hadoop, MapReduce, HDFS, Hive, Pig, Sqoop, Flume, Zookeeper, SparkMongoDB
Databases: Oracle10g, MySQL and SQL Server, Microsoft SQL Server 2000, MS Access
DB Languages: SQL
Operating Systems: Linux, Windows XP, Server 2003, Server 2008
Development Tools &IDS Tools: Eclipse 3.3, ArcSight, Tripwire, Eclipse, MS Office Suite, Photoshop, Visual Studio.
Other programming Languages: HTML5,CSS 3,JAVASCRIPT,AJAX,JQUERY,.NET,Visual Studio 2010
Network protocols: TCP/IP, UDP, HTTP, DNS, DHCP, OSPF, RIP
Frameworks: MVC
PROFESSIONAL EXPERIENCE
Confidential, Pleasanton, CA
Hadoop Developer
Responsibilities:
- Worked on analyzing, writingHadoopMapReduce jobsusingJavaAPI,Pig and Hive.
- Responsible for building scalable distributed data solutions usingHadoop.
- Involved in loading data from edge node toHDFS using shell scripting.
- Worked oninstalling cluster, commissioning & decommissioning ofdatanode, namenodehigh availability, capacity planning, and slots configuration.
- CreatedHBasetables to store variable data formats of data coming from different portfolios.
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
- Used Sqoop job to import the data from RDBMS using Incremental Import.
- CustomizedAvrotools used in MapReduce, Pig and Hive for deserialization and to work withAvroingestion framework.
- Worked with using different kind of compression techniques to save data and optimize data transfer over network using Lzo, Snappy, etc.
- Analyze large and critical datasets using Cloudera, HDFS, Hbase, MapReduce, Hive, Hive UDF, Pig, Sqoop, Zookeeper, & Spark. customized Flume interceptors to encrypt and mask customer sensitive data as per requirement
- Continuous monitoring and managing the Hadoop cluster using Cloudera Manager.
- Worked with NoSQL database Hbase to create tables and store data.
- Developed custom aggregate functions usingSparkSQL and performed interactive querying.
- Used Pig to store the data into HBase.
- Creating Hive tables, dynamic partitions, buckets for sampling, and working on them using Hive QL
- Used Pig to parse the data and Store in Avro format.
- Stored the data in tabular formats using Hive tables and Hive SerDe's.
- Collecting and aggregating large amounts of log data using Apache Flume and staging data in HDFS for further analysis.
- Worked with NoSQL, databases like Hbase in creating Hbase tables to load large sets of semi structured data coming from various sources.
- Implemented test scripts to support test driven development and continuous integration.
- Worked on tuning the performancePig queries.
- Worked with application teams to install operating system,Hadoopupdates, patches, version upgrades as required.
- Involved in writing the shell scripts for exporting log files toHadoopcluster through automated process.
- Exported the analyzed data to the relational databases usingSqoopfor visualization and to generate reportsfor the BI team.
- Implemented MapReduce programs to handle semi/ unstructured data like XML, JSON, Avro data files and sequence files for log files.
- Worked on Storm and Kafka tools to stream Real time data. And also, performed operations on Statistical analysis.
- Responsible for cluster maintenance, adding and removing cluster nodes, cluster monitoring and troubleshooting, manage and reviewdata backups, manage and reviewHadooplog files.
- Analyzed large amounts of data sets to determine optimal way toaggregateandreporton it.
- Supported in setting upQA environmentand updating configurations for implementing scripts with Pig and Sqoop.
Environment: Hadoop, HDFS, Pig, Sqoop, Spark, MapReduce, Cloudera, Snappy, Zookeeper, NoSQL, HBase,, Storm, Kafka, Shell Scripting, Ubuntu, Linux Ubuntu.
Confidential, San Jose, California
Hadoop Developer
Responsibilities:
- Installed and configured Hadoop Map Reduce, HDFS, Developed multiple Map Reduce jobs in Java for data cleaning and preprocessing.
- Worked on data Importing and exporting into HDFS and Hive using Sqoop.
- Participated in development/implementation of Cloudera Hadoop environment.
- Developed UDFs using Scala and Spark for implementing Data enrichment technique for conducting more real time data analysis that is used for combining customer historical data with live customer behavioral data, for delivering targeted ads to customers
- Load and transform large sets of structured, semi structured and unstructured data.
- Experienced in managing and reviewing Hadoop log files.
- Worked on automation of delta feeds from, Teradata using Sqoop, also from FTP Servers to Hive.
- Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
- Developed MapReduce programs to cleanse the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis
- Used Sqoop to import the data from RDBMS to Hadoop Distributed File System (HDFS) and later analyzed the imported data using Hadoop Components
- Established custom MapReduce programs in order to analyze data and used Pig Latin to clean unwanted data
- Did various Performance tuning like using distributed cache for small datasets, Partition, Bucketing in hive and Map Side joins
- Involved in loading and transforming large sets of Structured, Semi-Structured and Unstructured data and analyzed them by running Hive queries and Pig scripts
- Worked on Storm and Kafka tools to stream Real time data. And also, performed operations on Statistical analysis.
- Participated in requirement gathering from the Experts and Business Partners and converting the requirements into technical specifications
- Implemented daily workflow for extraction, processing and analysis of data with Oozie.
- Involved in loading data from LINUX file system to HDFS.
- Actively participate in various internal meet / sessions with other project team members to understand the requirements and suggest suitable Hadoop Stack Technologies along with training sessions.
Environment: Cloudera HDFS, Hadoop, Map Reduce, Hive, Pig Latin, Java, SQL, Sqoop, Centos, NOSQL database, Storm,Scala, Kafka.
Confidential, Woodland Hills, CA
Hadoop Developer
Responsibilities:
- Involved with the application teams to installHadoopupdates, patches and version upgrades as required.
- Worked on Hadoopeco system components HDFS, MapReduce, Hive, Pig, Sqoop and HBase.
- Development experience in UNIX, LINUX and Cloud based virtual platforms.
- Involved in architecture design, development and implementation of Hadoop deployment, backup and recovery systems.
- Wrote MapReduce jobs to perform operations like copying data onHDFSand defining job flows onEC2 server, load and transform large sets of unstructured data.
- Worked on Amazon EMR to power approximately 20 separate batch scripts to processing the logs.
- CreatedHBasetables to store variable data formats of data coming from different portfolios.
- Involved in Configuring core-site.xml and mapred-site.xml according to the multi node cluster environment
- Wrote MapReduce jobs using Java API.
- Implemented Spark POC operations to mimic Hadoop ETL.
- Wrote MapReduce jobs to generate reports for the number of activities created on a particular day, during a dumped from the multiple sources and the output was written back to HDFS.
- Used Flume to collect, aggregate, and store the web log data from different sources like web servers, mobile and network devices and pushed to HDFS.
- Responsible for writingHiveQueries for analyzing terabytes of customer data from Hbase and put the results in output file.
- Wrote Pig Scripts to generate MapReduce jobs and performed ETL procedures on the data in HDFS.
- Processed HDFS data and created external tables using Hive, in order to analyze visitors per day, page views and most purchased products.
- Exported analyzed data to HDFS using Sqoop for generating reports.
- Used MapReduce and Sqoop to load, aggregate, store and analyze web log data from different web servers.
- Developed Hive queries for the analysts.
- Cluster co-ordi0nation services through Zookeeper.
- Experience in optimization of Map reduce algorithm using combiners and partitions to deliver the best results and worked on Application performance optimization for a HDFS/Cassandra cluster.
Environment: Java6, Mongo, Apache Web server, HTML, JDBC, NoSQL, meteor.js, Eclipse, UNIX, HTML, CSS, XML,JavaScript, JQuery, Oracle, SQL.
Confidential
JAVA Developer
Responsibilities:
- Involved in requirement, design and development phases of the application.
- Worked with DBA for the creation of new tables and new fields in the database.
- Developed the application using Struts Framework that leverages classical MVC architecture
- Used Multithreading in programming to improve overall performance.
- Created RESTful Web services for updating customer data from sent from external systems.
- Data was converted into JSON using JSP tags
- Developed the front end User Interface using HTML5, JavaScript, CSS3, JSON, JQuery and JS.
- Developed and refactored the web application based on client requirement in JS.
- Generated reports using Jasper which is friendly and efficient o use
- Implemented the application using the concrete principles laid down by several design patterns such as MVC, Business Delegate, Data Access Object, and Singleton.
- Developed integration services using SOA, ESB, Web Services, SOAP, and WSDL.
- Used Maven for building and deployment purpose
- Designed and developed Java Web Services to allow external clients to interact with the system.
- Developed web service components and WSDL documents using top-down approach with Apache Axis.
- Implemented Java Collections Frame work.
- Used CVS for version control across common source code used by developers.
- Used JavaScript for validations and integrating business server side components on the client side within the browser.
- Involved in the analysis, design, and development and testing phases of Software Development Life Cycle (SDLC).
- Designed and developed framework components, involved in designing MVC pattern using Struts and Spring framework.
Environment: Java, J2EE, Struts, Oracle 10g, JMS, Maven, DOM, Web service Consumer, SOAP UI, JBoss, JPA, Rationale Rose, Eclipse 3.1, Hibernate,MVC
Confidential
JAVA Developer
Responsibilities:
- Technical responsibilities included high level architecture and rapid development.
- Design architecture following J2EE MVC framework.
- Developed interfaces using HTML, JSP pages and Struts -Presentation View.
- Involved in designing & developing web-services using SOAP and WSDL.
- Developed and implemented Servlets running under JBoss.
- Used J2EE design Patterns for the Middle Tier development.
- Used J2EE design patterns and Data Access Object (DAO) for the business tier and integration Tier layer of the project.
- Created UML class diagrams that depict the code's design and its compliance with the functional requirements.
- Developed various EJBs for handling business logic and data manipulations from database.
- Designed and developed the UI using Struts view component, JSP, HTML, CSS and JavaScript.
- Implemented CMP entity beans for persistence of business logic implementation.
- Development of database interaction code to JDBC API making extensive use of SQL Query Statements and advanced prepared statement.
- Involved in writing Spring Configuration XML files that contains declarations and other dependent objects declaration.
- Inspection/Review of quality deliverables such as Design Documents.
- Involved in creation running of Test Cases for JUnit Testing.
- Experience in implementing Web Services using SOAP, REST and XML/HTTP technologies.
- Used Log4J to print the logging, debugging, warning, info on the server console.
- Wrote SQL Scripts, Stored procedures and SQL Loader to load reference data.
Environment: Java, J2EE, Spring, JSP, Hibernate, Java Script, CSS, JDBC, IntelliJ, LDAP, REST, Active Directory, SAML, Web Services, Microsoft SQL Server, HTML
