We provide IT Staff Augmentation Services!

Hadoop Developer Resume

2.00/5 (Submit Your Rating)

Chicago, IL

SUMMARY:

  • Around 5+ years of experience using BigData Ecosystems & Java.
  • Extensive experience in Apache Spark with Scala, Apache Solr, Python
  • Solid Mathematics, Probability and Statistics foundation and broad practical statistical and data mining techniques cultivated through various industry work and academic programs
  • Involved in the Software Development Life Cycle (SDLC)phaseswhich include Analysis, Design, Implementation, Testing and Maintenance.
  • Strong technical, administration, &mentoring knowledge in Linux and BigData/Hadoop technologies.
  • Have sound knowledge on In - Memory MEMSQL.
  • Experience with developing and maintaining applications written forAmazon Simple Storage,AWSElastic Map Reduce, andAWSCloudFormation.
  • Worked on migrating the on-premises applications toAWS
  • Hands on experience on major components in Hadoop Ecosystem like Hadoop Map Reduce, HDFS, HIVE, PIG, Pentaho, Hbase, Zookeeper, Sqoop, Oozie, Cassandra, Flume and Avro.
  • Experienced the deployment of Hadoop Cluster using Puppet tool
  • Used Kafka Streams to Configure Spark Streaming to get information and then store it in HDFS
  • Work experience with cloud infrastructure like Amazon Web Services (AWS).
  • Experience in importing and exporting the data using SQOOP from HDFS to Relational Database systems/mainframe and vice-versa
  • Experience in designing and developing applications in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.
  • Expertise in working with ETL Architects, Data Analysts and data modelers to translate business rules/requirements into conceptual, physical and logical dimensional models and worked with complex normalized and denormalized data models.
  • Installing, configuring and managing of Hadoop Clusters and Data Science tools.
  • Managing the Hadoop distribution with Cloudera Manager, Cloudera Navigator, Hue.
  • Setting up the High-Availability for Hadoop Clusters components and Edge nodes.
  • Strong experience in writing applications using python using different libraries like Pandas, NumPy, SciPy, Matpotlib etc
  • Experience in developing Shell scripts and Python Scripts for system management.
  • Well versed in using Software development methodologies like Rapid Application Development (RAD), Agile Methodology and Scrum software development processes.
  • Involved in converting Cassandra/Hive/SQL queries into Spark transformations using RDDs, and Scala
  • Analyzed the Cassandra/SQL scripts and designed the solution to implement using Scala
  • Experience with Object Oriented AnalysisandDesign (OOAD)methodologies.
  • UsedAWSservices like EC2 and S3 for small data sets processing and storage, Experienced in Maintaining theHadoopcluster onAWSEMR.
  • Decent skill in data warehousing and AWS migration with tools such AWS SCT, DMS, Data Pipeline.
  • Experience in installations of software, writing test cases, debugging, and testing of batch and online systems.
  • Supported data analysis projects using Elastic Map Reduce on the Amazon Web Services (AWS) cloud. Exporting and importing data into S3.
  • Experience in working onAWS, Flume to load the log data from multiple sources directly into HDFS & running Pig and Hive scripts.
  • Experience in Production, quality assurance (QA), SIT (System Integration testing) and user acceptance (UA) testing.
  • Expertise in J2EEtechnologies like JSP, Servlets, EJBs 2.0, JDBC, JNDI and AJAX.
  • Extensively worked on implementing SOA (Service Oriented Architecture)usingXMLWeb services (SOAP, WSDL, UDDI and XML Parsers).
  • Experience in Performance Tuning and Debugging of existing ETL processes.
  • Worked with XML parserslikeJAXP (SAX and DOM)and JAXB.
  • Expertise in applying Java Messaging Service (JMS)for reliable information exchange across Java applications.
  • Proficient with Core JAVA, AWT and also with the markup languages likeHTML 5.0,XHTML, DHTML, CSS, XML 1.1, XSL, XSLT, XPath, XQuery, Angular.js, Node.js
  • Expert in understanding the data and designing/Implementing the enterprise platforms like Hadoop Data lake and Huge Data warehouses.
  • Worked with version control systems like Subversion, Perforce, and GITfor providing common platform for all the developers.
  • Articulate in written and verbal communication along with strong interpersonal, analytical, and organizational skills.
  • Hands on experience with Microsoft Azure Cloud services, Storage Accounts and Virtual Networks.
  • Good at manage hosting plans for Azure Infrastructure, implementing and deploying workloads on Azure virtual machines (VMs).
  • Highly motivated team player with the ability to work independently and adapt quickly to new and emerging technologies.
  • Creatively communicate and present models to business customers and executives, utilizing a variety of formats and visualization methodologies.

TECHNICAL SKILLS:

Languages: C,C++,Java 6,Java 7, Java 8, Scala.

Big Data Skills: Map reduce, Hadoop, Spark, Kafka, Storm

Web Technologies: HTML5 JavaScript, Ajax, CSS, JQuery, XML,BootStrap.

Servers: WebSphere, Tomcat 6.x, MIIS (Microsoft Internet Information Server)

Case Tools and IDE: Eclipse, NetBeans, RAD, IntelliJ, Netezza.

Frameworks in Hadoop: Spark, Kafka, Storm

Databases: DB2, Oracle and MySQL Server

Version Tools: SVN, CVS, ClearCases

Web Services: SOAP, REST

WORK EXPERIENCE:

Confidential, Chicago, IL

Hadoop Developer

Responsibilities:

  • Migration of Oracle tables to the HDFS using SQOOP.
  • Designed and developed rich front end screens using JSF (Ice faces), JSP, Docker, CSS, HTML, Angular JS and JQuery.
  • Developed Managed beans and defined Navigation rules for the application using JSF.
  • Developed Angular JS 2.0 code and migrated pre-existing code to updated Angular JS 2.0 framework.Written a custom Sqoop class for the XMLTYPE datatype in the Oracle Databse
  • UsedJavaMessaging Services (JMS) for reliable and asynchronous exchange of important information such as payment status report to MQ Server using MQ Series.
  • Designed and implemented Apache Spark job which takes the Sequence File from th HDFS and migrate to the Hbase.
  • Motivated and assisted team of six memebers in reaching individual andd team goals for quality,productivity and revenue generation
  • Implemented Spark using Scala and Spark SQL for faster testing and processing of data
  • Used WebSphere Application Server Developer Tools for Eclipse (WDT) to createJavabatch projects based on theJavaBatch 1.0 standard (JSR 352) and submit them to a Liberty profile server
  • Involved and guide the team for the preparing the technical specification
  • Involved in Development,Build and Deployment Application
  • Implemented micro services in order to separate the tasks and not to have dependency on other Parallel ongoing tasks of same Application.
  • Developed various shellscripts and pythonscripts to automate Spark jobs and hive scripts.
  • Used Impala to read, write and query the Hadoop data in HDFS from HBase or Cassandra and configured Kafka to read and write messages from external programs.
  • Extending HIVE and PIG core functionality by using custom User Defined Function’s (UDF), User Defined Table-Generating Functions (UDTF) and User Defined Aggregating Functions (UDAF) for Hive and Pig using python.
  • Identifying opportunities to improve infrastructure that effectively and efficiently utilizes the Microsoft Azure Windows server 2008/2012/R2, Microsoft SQL Server, Microsoft Visual Studio, Windows PowerShell, Cloud infrastructure.
  • Deployed Azure IaaS virtual machines (VMs) and Cloud services (PaaS role instances) into secure VNets and subnets.
  • Developed Web service using Restful with Jersey, and implemented JAX-RS and also provided security-using SSL.
  • Creation Of HBase Tables and implemented Salting on the Hbase
  • Migration of tables from oracle to Hbase on the Tenant basis
  • Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
  • Design and develop ETL code using Informatica Mappings to load data from heterogeneous Source systems like flat files, XML’s, MS Access files, Oracle to target system Oracle under Stage, then to data warehouse and then to Data Mart tables for reporting.
  • Developed ETL with SCD’s, caches, complex joins with optimized SQL queries.
  • Written Programs in Spark using Scala and Python for Data quality check.
  • Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Scala.
  • Optimizing of existing algorithms in Hadoop using Spark Context, Spark-SQL, Data Frames and Pair RDD's.
  • Implemented ELK (Elastic Search, Log stash, Kibana) stack to collect and analyze the logs produced by the spark cluster.
  • Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark
  • One time migraion of 300 Billions of records using apache Spark BulkLoading
  • Involved in Delta Migration using Sqoop Incremental Updates
  • Using shell scripts to perform ETL process to call sql or pmcmd commands, pre-post ETL process like file validation, zipping, massaging and archiving the source and target files, and used UNIX scripting to manage the file systems.
  • Maintained, structured, and surveyed documents within the NoSQL MongoDB database; ensuring data integrity, correcting anomalies, and increasing the overall maintainability of the database.
  • Involved in architecturing the Data Pipe-Line for the Add/Update flow of real-time analysis flow for the IMS Application
  • Achitectured the whole design for the data migration process and also for the real time analysis.
  • Written Apache Spark Jobs using Scala API
  • Involved in administrating and configuring the mapr distribution
  • Implemented Mapr Streams (KAFKA 0.9 API) to the Spark Streaming using Java API.

Environment: Apache Spark, Spark Streaming, Spark SQL,, Hadoop Security, Mapr Streams, Mapr 5.1, Open-Shift, Scala, ETL Informatica, Java, Hbase, Eclipse, MVN, Sequence Files.

Confidential, Irving, TX

Hadoop Developer

Responsibilities:

  • Performed hands-on data manipulation, transformation, hypothesis testing and predictive modelingDeveloped robust set of codes that are tested, automated, structured and efficient
  • Defined service layer using EJB3.0 and also defined remote and local services.
  • Accessed remote and local EJB services from controller.
  • Developed application using JSP, Tag libraries, JSF and Struts (MVC) Framework.
  • Exposed web services to client developing WSDL also involved in developing web client for application interactions.
  • Evaluate, refine, and continuously improve the efficiency and accuracy of existing Predictive Models using Netezza.
  • Involved in developing several Data Pipelines using Apache Kafka.
  • Decent skills in data warehousing andAWSmigration with tools such asAWSSCT, DMS, Datapipeline
  • Developed Framework API for Tax calculations in Yoda using server-side components using J2EE and spring framework.
  • Involved in file movements between HDFS andAWSS3 and extensivelyworked with S3 bucket inAWS.
  • Designed, developed and implemented a messaging module usingJavaMessaging Service (JMS) point-to-point messaging and Message Driven Beans to listen to the messages in the queue for interactions with client ordering data.Collaborated on insights with other Data Scientists, Business Analysts, and Partners.
  • Imported data fromAWSS3 into Spark RDD, performed transformations and actions on RDDs.
  • Worked on importing metadata into Hive and migrating existing tables and applications to work on Hive andAWScloud.
  • Evaluate, refine, and continuously improve the efficiency and accuracy of existing Predictive Models.
  • Utilized various data analysis and data visualization tools to accomplish data analysis, report design and report delivery.
  • UsedJavaMessaging Services (JMS) for reliable and asynchronous exchange of important information such as payment status report to MQ Server using MQ Series.
  • Worked on POC’s with Apache Spark using Scala to implement spark in project.
  • Consumed the data from Kafka using Apache spark.
  • Worked with Apache Spark which provides fast and general engine for large data processing integrated with functional programming language Scala.
  • Experienced in Apache Spark for implementing advanced procedures like text analytics and processing using the in-memory computing capabilities written in Scala.
  • Responsible for Design and Development of ETL and Technical Design documents for Informatica mappings with the High level DFD’s, Process flow description, Source to Target row level transformation logic, performance tuning, and scheduling.
  • Importing and exporting the data from HDFS to RDBMS using Sqoop and Kafka.
  • Worked on creating Custom Azure Templates for quick deployments and advanced PowerShell scripting.
  • Contribute to the support forums (specific to Azure Networking, Azure Virtual Machines, Azure Active Directory, Azure Storage) for Microsoft Developers Network.
  • Populated HDFS and Cassandra with huge amounts of data using ApacheKafka.
  • Developed Scala and SQL code to extract data from various databases
  • Champion new innovative ideas around the Data Science and Advanced Analytics practices
  • Creatively communicated and presented models to business customers and executives, utilizing a variety of formats and visualization methodologies.
  • Participate in planning, implementation, and growth of our Amazon Web Services (AWS) foundational footprint.
  • Created stored procedures and packages in Oracle as a part of the pre and Post ETL process.
  • Designed, developed and implemented a messaging module usingJavaMessaging Service (JMS) point-to-point messaging and Message Driven Beans to listen to the messages in the queue for interactions with client ordering data.
  • Uploaded data to Hadoop Hive and combined new tables with existing databases.
  • Developed statistical models to forecast inventory and procurement cycles.8
  • Developed Python code to provide data analysis and generate complex data report.
  • Deployed the Cassandra cluster in cloud (Amazon AWS) environment with scalable nodes as per the business requirement.
  • Implemented the data backup strategies for the data in the Cassandra cluster.
  • Generated the data cubes using hive, Pig, JAVA Map-Reducing on provisioning Hadoop cluster in AWS.
  • Implemented the ETL design to dump the Map-Reduce data cubes to Cassandra cluster.
  • Imported the data from relational databases into HDFS using SQOOP.
  • Tune and optimize ETL jobs and SQL Queries for performance and throughput
  • Implemented POC for using APACHE IMPALA for data processing on top of HIVE.
  • Utilized Python Panda Frame to provide data analysis.
  • Worked on Hortonworks 2.3distribution
  • Utilized Python regular expressions operation (NLP) to analysis customer review.
  • Understanding of data storage and retrieval techniques, ETL, and databases, to include graph stores, relational databases, tuple stores, NOSQL, Hadoop, PIG, MySQL and Oracle databases

Environment: Apache Spark, Pyspark, Spark Streaming, Spark SQL, Scala, Apache Kafka, Apache Flume, Python Pandas, Cassandra, Hortonworks (HDP) 2.3, AWS, AKKA, Hive, PIG, ETL Informatica.

Confidential

Jr. Java/J2EE Developer

Responsibilities:

  • Developed User Interfaces module usingJSP,JavaScript, DHTML and form beansfor presentation layer.
  • Developed Servlets and Java Server Pages (JSP).
  • Developed PL/SQLqueries, and wrote stored procedures andJDBC routines to generate reports based on client requirements.
  • Enhancement of the System according to the customer requirements.
  • Involved in the customization of the available functionalities of the software for an NBFC (Non-BankingFinancialCompany).
  • Involved in putting proper review processes and documentation for functionality development.
  • Providing support and guidance for Production and Implementation Issues.
  • Used Java Script validation inJSP.
  • UsedHibernateframework to access the data from back-end SQL Server database.
  • Used AJAX (Asynchronous JavaScript and XML) to implement user friendly andefficient client interface.
  • UsedMDBfor consuming messages from JMS queue/topic.
  • Designed and developed Web Application usingStrutsFramework.
  • ANT to compile and generate EAR, WAR, and JAR files.
  • Created tes1t case scenarios for Functional Testing and wrote Unit test cases with JUnit.
  • Responsible for Integration, unit testing, systemtesting and stress testing for all the phases of project.

Environment: Java, J2EE, JSP 1.2, Performance Tuning,Spring1.2,Hibernate 2.0, JSF1.2,EJB 1.2, IBM WebSphere6.0, Servlets, JDBC, XML, XSLT, DOM, CSS, HTML, DHTML, SQL, Java Script, Log4J, ANT1.6, WSAD 6.0, Oracle 9i, Windows 2000.

We'd love your feedback!