We provide IT Staff Augmentation Services!

Hadoop Developer Resume

4.00/5 (Submit Your Rating)

Irving, TX

PROFESSIONAL SUMMARY:

  • 10+years of IT experience in software design, analysis, and development on Core Java, Apache,MySQL, CordysBPM technologies and Apache Hadoop ecosystem. Strong Foundation from a product based company
  • Well versed in installation, configuration, supporting and managing of Big Data and underlying infrastructure of Hadoop Cluster.
  • Expert in creating PIG Latin Scripts and UDFs using JAVA for analysis of data efficiently.
  • Expert in creating Hive Queries and UDFs using Java for analysis of data efficiently.
  • Knowledge of HadoopGEN2 Federation, HighAvailability and YARN architecture.
  • Expert in using Sqoopinfetching data from different systems intoHDFS to analyze and exporting back.
  • Also used Hbase in accordance with PIG/Hive as and when required for real time low latency queries.
  • Hands on experience on operating system internals in multithreaded environment using Inter Process Communications and deep knowledge of UNIX operating system internals and working knowledge of Linux software distributions and High Performance Computing (HPC).
  • Experience with Oozie Workflow Engine in running workflow jobs with actions that run HadoopMR and Pig jobs.
  • Good Knowledge in using Hue web interface.
  • Excellent knowledge on HadoopEcosystem Architecture and components such as Hadoop Distributes File System (HDFS), MRv1, MRv2, Job Tracker, Task Tracker, Name Node, Data Node, Resource Manager, Node Manager and MapReduce programming.
  • Good experience in optimizing MapReduce algorithms using Mappers, Reducers, Combiners and Partitioners to deliver best results for the given dataset.
  • Used Flume to process real time processing data.
  • Good understanding of NoSQL Databases.
  • Worked in Windows, UNIX/Linux platform with different technologies such as SQL, PL/SQL, XML, HTML, CSS, Java Script, Core Java, Web Services etc.
  • Experience in using IDEs like Eclipse and NetBeans.
  • Extensive programming experience in developing web based applications using Core Java, J2EE, JSP and JDBC.
  • Experience in deploying applications in Web/Application Servers like Tomcat, WebLogic and Oracle Application Servers.
  • Strong knowledge ofSDLC and expertise in detailed design documentation.
  • Extensive experience with Waterfall and AgileScrum Methodologies.
  • Experience in development of logging standards and mechanism based on Log4J
  • Working knowledge of database such as Oracle 8i/9i/10g, Microsoft SQL Server and MySQL.
  • Experienced in creating Product Documentation & Presentations.
  • Strong expertise on MapReduce programming model with XML, JSON, CSV file formats.
  • Extensive experience worked on CVS and SVN for Source Controlling.
  • Ability to perform at a high level, meet deadlines, adaptable to ever changing priorities.
  • Understanding of Data warehouse and ETL tools.
  • Highly proficient in ObjectOrientedProgramming concepts.
  • Ability to blend technical expertise with strong Conceptual, Business and Analytical skills to provide quality solutions and result - oriented problem solving technique and leadership skills.
  • Strong hands on experience in writing CoreJavaenterprise class applications
  • Strong hands on experience in designing RDBMS schemas for enterprise class applications and implementing MySQLCircularMaster for HighAvailability
  • Comfortable experience in Python programming
  • Hands on experience in SOA development using Cordys Process Suite
  • Hands on experience in ApacheHTTPD, Proxy, CustomModules
  • Good Exposure to XML related technologies such as XSL, XSLT, XPath, DOM, and SAX and CordysNOM.
  • Experience on SOAP and REST for developing Web Services

TECHNICAL SKILLS

Big Data Technologies: Hadoop, Yarn, HDFS, Map Reduce, Sqoop, Hive, Pig, Flume, Zookeeper, Oozie, HBase, Spark

Languages: Core Java, C, C++, Python

SOA and Integration Technologies: Cordys BPMand Process Suite

OS: Windows, Linux, Redhat, CentOs

RDBMS: MySQL 5.x, Oracle and MS SQL Server

Database Scripting: SQL and PL/SQL

Application & Web Servers: Apache, IIS, Tomcat

PROFESSIONAL EXPERIENCE

Confidential

Hadoop Developer

Environment: Hadoop, MapReduce, HDFS, Yarn, Sqoop, Oozie, Pig, Hive, Hbase, Spark, Java, Eclipse, UNIX shell scripting, python, Hortonworks

Responsibilities:

  • Worked on analyzing, writing HadoopMapReduce jobs using JavaAPI, Pig and Hive.
  • Responsible for building scalable distributed data solutions using Hadoop.
  • Involved in loading data from edge node to HDFS using shell scripting.
  • Created HBase tables to store variable data formats of PII data coming from different portfolios.
  • Exported the analysed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
  • Worked with using different kind of compression techniques to save data and optimize data transfer over network using LZO, Snappy, Bzip etc.
  • Analyze large and critical datasets using Cloudera, HDFS, Hbase, MapReduce, Hive, HiveUDF, Pig, Sqoop, Zookeeper, &Spark.
  • Developed custom aggregate functions using SparkSQL and performed interactive querying.
  • Used Scoop to store the data into HBase and Hive.
  • Worked on installing cluster, commissioning & decommissioning of DataNode, NameNode high availability, capacity planning, and slots configuration.
  • Creating Hive tables, dynamic partitions, buckets for sampling, and working on them using HiveQL.
  • Used Pig to parse the data and Store in Avro format.
  • Stored the data in tabular formats using Hive tables and HiveSerDes.
  • Collecting and aggregating large amounts of log data using ApacheFlume and staging data in HDFS for further analysis.
  • Worked with NoSQL databases like Hbase for creating Hbase tables to load large sets of semi structured data coming from various sources.
  • Implemented a script to transmit information from Oracle to Hbase using Sqoop.
  • Implemented MapReduce programs to handle semi/unstructured data like XML, JSON, and sequence files for log files.
  • Fine-tunedPigqueries for better performance.
  • Involved in writing the shell scripts for exporting log files to Hadoop cluster through automated process.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports for the BI team.
  • Installed Oozie workflow engine to run multiple Hive and pig jobs.
  • Analyzed large amounts of data sets to determine optimal way to aggregate and reporton it.

Confidential, Irving, Tx

Hadoop Developer

Environment: Hadoop, MapReduce, HDFS, Yarn, Sqoop, Oozie, Pig, Hive, Java, Eclipse, UNIX shell scripting, python

Responsibilities:

  • DevelopedMapReducejobs in java to process the input data and generate output files in HDFS.
  • Worked with Business Analysts to gather all the requirements and documented all the use cases.
  • Importing and exporting data into HDFS and Hive using Sqoop.
  • Developed HiveQL for Hive
  • Designed workflow jobs in Oozie to combine a series of implementations.
  • Implemented coordinator workflows in Oozie to schedule various jobs.
  • Experienced in managing and reviewing Hadooplog files.
  • Configured Sqoop to map SQL types to appropriate Java classes.
  • Responsible to manage data coming from different sources.
  • Implemented Pig Scripts for basic data transformation tasks.
  • Developed UDFs in Pig and hive.
  • Worked with support and BI teams to generate reports from the output files generated from MapReduce jobs.

Confidential

Senior Software Engineer

Environment: Hadoop, HDFS, Sqoop, Zookeeper, Shell Scripting, Python, Java, MySQL, Ubuntu, Red Hat Linux.

Responsibilities:

  • Actively participated in the entire project life cycle
  • Importing and Exporting data inHDFS
  • Developed MapReduce jobs in java to process the input data
  • ShellScripts and Python scriptsare written to execute Hadoopsupporting tasks
  • Implemented Custom Input format classes to read the zip files
  • Designed RDBMS to handle customer requirements
  • Reports are automated as per Business use case
  • Integrated Fusion Charts in web display.
  • Mentoring the other resources
  • Prepared the Demos for acceptance testing

Senior Software Engineer

Confidential

Environment: RHEL 3.0, Apache HTTPD, MySQL, C, C++, Java, web-services, SOAP and Cordys ESB (Enterprise Service Bus).

Responsibilities:

  • ImplementedMySQLCircularMaster
  • Actively participated in the entire project life cycle such as analysis, design, planning and realization
  • NoSinglepoint of failure by design
  • Designed RDBMSsystem to meet the complex business use cases.
  • Developed Plug and Play modules on Apache HTTPD
  • ApplicationConnectors are written over CordysXQY layer
  • SingleUnifiedLogging mechanism implemented across the Confidential Modules
  • Mentored and handled other engineers to have them in the project realization
  • Suggested and Implemented architectural changes in the deployment design of the Server Cluster.
  • Providing support in case of urgent issues through WebEx and Teleconference.

Confidential

Senior Software Engineer

Environment: RHEL 3.0, Apache HTTPD, MySQL, C, C++, Java, web-services, SOAP and Cordys ESB (Enterprise Service Bus).

Responsibilities:

  • Explored the internals ofDomainController to get the LogOn information
  • Measures are taken to avoid SinglePoint of Failure
  • Actively participated in the entire project life cycle such as analysis, design, planning and realization
  • Handled entire realization, testing, bug fixing configuration management and documented the user and configuration guides for the project
  • Completely designed and developed the CustomApplicationConnector for this.
  • ApplicationConnector is written over Cordys XQY layer for better speed.
  • Developed all the Apache related modules for integration
  • Involved in maintenance and support for this project on various enhancement requirement
  • Mentored and handled other engineers to have them in the project realization
  • Architectural changes in the deployment design of the server cluster.
  • Maintenance of the servers and backtracking deployment issues.

Senior Software Engineer

Environment: RHEL 3.0, Apache HTTPD, MySQL, C, C++, Java, web-services, SOAP and Cordys ESB (Enterprise Service Bus).

Confidential

Responsibilities:

  • Actively participated in the entire project life cycle such as analysis, design, planning and realization
  • Integrated SOAP and XML for all CordysWS-Apps operations with MySQL Database transactions
  • Realized the entire “Custom Application Connectors” for various database operations using Cordys XQY layer
  • Designed RDBMSrelations to handle 15000transactions per second in a single instance.
  • Explored Apacheinternals to increase the throughput
  • Involved in optimization of various modules
  • Handled entire realization, testing, bug fixing configuration management and documented the user and configuration guides for the project
  • Maintain and upgrade software as well as advice on updates to OS on the servers, at our customer deployment who is an ISP with more than 60k users behind our Server.

Role: Software EngineerNov 2008 - Dec 2008

Project: Safe Search

Safe search is InFilter module developed to extend NetProtector capabilities. Most Search Engines offer a Safe Search option to evade visiting unacceptable websites. This option filters those web pages from appearing in the search results itself. Through this module NetProtector turns on the Safe Search option for most of the search engines and forbids the access at the initial stage itself.

Environment: RHEL 3.0, CentOS, C, Apache HTTPD.

Responsibilities:

  • Designed and developed this module
  • Handled entire realization, testing, bug fixing configuration management and documented the user and configuration guides for this module
  • Explored various search engine and their parameters usage

Role: Software EngineerMar 2008 - Sept 2008

Project: NetProtector Management Console

NetProtector provides a user friendly interface referred as NetProtector Management Console (NPMC). This acts as an administrative console used to configure, control, monitor, manage and track the Web page URLs and keywords that are graded under various lists in a single interface.

Environment: Cordys ESB, Java, JavaScript, HTML, Cordys Xforms, MySQL, SOAP & XML.

Customers: Confidential, Cordys India and Cordys Netherlands.

Responsibilities:

  • Developed the portal using Cordys Front End Framework.
  • Integrated SOAP and XML for all CordysWS-Apps operations with MySQL Database transactions

Client: Cordys R&D(I) Pvt Ltd, India(formerly Vanenburg)

Role: Software EngineerSep 2006 - Feb 2008

Project: NetProtector 2.0

NetProtector (aka FFI) is an internet-filter solution developed on Apache Proxy Server,integrated with intelligent Neural Network solutions and is exposed as web-service. NetProtector is an effective combination of static and dynamic filtering. Static filtering includes the database check-up and is achieved through Cordys ESB. Static filter itself is exposed as a web-service.Dynamic filtering is done by intelligent neural network solution. NetProtector is easily scalable and supports load-balancing by design

Environment: RHEL 3.0, Apache HTTPD, MySQL, C, C++, Java, web-services, SOAP and Cordys ESB (Enterprise Service Bus).

Confidential

Responsibilities:

  • Database lookups are done using Cordys Ws-Apps service containers.
  • Integrated SOAP and XML for all Cordys WS-Apps operations with MySQL Database transactions.
  • Configured and maintained the required Cordys Service Containers
  • Worked on the Custom modules of ApacheHTTPD, and ApacheProxy
  • Prepared JUNIT test cases and involved in performance testing using OpenSTA and SilkPerformer
  • Helped the documentation team in for the Technical documentation, user guides and FAQ for the end user and the technical team
  • Involved in all the support related activities of NetProtector

AssociateSoftware Engineer

Environment: Java, Java Swings, MySQL.

Responsibilities:

  • Analysis and knowledge gathering required for development of the application
  • Explored Java Swings
  • Coding and maintenance of the application.
  • Written JUNIT test cases and SQL queries

Engineer Trainee

Environment: Java,MySQL, Delphi.

Confidential

Responsibilities:

  • Analysis and knowledge gathering required for development of the application
  • Coding and maintenance of the app.
  • Testing the application for any design flaws as well as coding errors.

We'd love your feedback!