Big Data / Hadoop Developer Resume
Houston, TX
SUMMARY
- 8 years of experience in Software Development Life Cycle (SDLC) and Software Engineering including Requirement Gathering, Analyzing, Designing, Implementing, Testing, Support and Maintenance.
- Possesses 4+ years of comprehensive experience as a Hadoop, BigData & Analytics Developer.
- Have experience with Hadoop Ecosystem including HDFS, MapReduce, PIG, HIVE, HBase and Spark.
- Have experience in working with Hadoop in stand - alone, pseudo and distributed modes.
- Experience in importing and exporting data from different RDBMS like MySql, Oracle and SQL Server into HDFS and Hive using Sqoop.
- Experience in developing custom Map-Reduce programs using Apache Hadoop to perform Data Transformation and analysis as per requirement.
- Strong competency in HIVE Schema design, Data imports and Analysis.
- Hands-on experience in writing Pig Latin scripts, working with grunt shell and job scheduling with Oozie.
- Experience in Design & Development, tuning and maintenance of NoSQL databases.
- Experience in working with NoSQL databases like HBase.
- Strong understanding of Data Modeling and experience with Data Cleansing, Data Profiling and Data analysis.
- Excellent experience in ETL analysis, designing, developing, testing and implementing ETL processes including performance tuning and query optimizing of databases.
- Excellent experience in extracting source data from Sequential files, XML files, Excel files, transforming and loading it into the target data warehouse.
- Expertise in Java/J2EE technologies such as Core Java, Struts, Hibernate, JDBC, JSP, JSTL, HTML, JavaScript, JSON
- Have knowledge of Python programming. Proficiency in programming with different IDE's like Eclipse, NetBeans and Canopy.
- Experience IaaS managing Amazon Web Services (AWS) infrastructure with automation and configuration management tools such as Chef, Ansible, Puppet, boto.
- Have hands on experience on various DB platforms like Oracle, MySQL, DB2 and MS SQL Server.
- Experience in deploying applications in heterogeneous application servers TOMCAT and WebLogic.
- Good understanding of service oriented architecture (SOA) and web services like XML, XSD, XSDL, SOAP.
- Good Knowledge about scalable, secure cloud architecture based on Amazon Web Services (leveraging AWS cloud services: EC2, Cloud Formation, VPC, S3, etc.
- Experience in object oriented analysis and design (OOAD), unified modeling language (UML) and Agile Methodologies.
- Strong communication skills with professional attitude and can take the pressures to drive with enthusiasm to support with full potential.
TECHNICAL SKILLS
Hadoop Ecosystem: HDFS, MapReduce, Hive, Pig, Zookeeper, Sqoop, Oozie, Impala, Flume, Spark, Kafka (beginner).
Hadoop Distribution: Cloudera, Apache
NO SQL Databases: HBase, Cassandra
Languages: C, Java, Scala, Python, SQL, PL/SQL, Pig Latin, HiveQL, Unix, Java Script, Shell Scripting, NiFi(beginner).
Java & J2EE Technologies: Core Java, Servlets, Hibernate, Spring, Struts, JMS, EJB.
Operating Systems: Windows Variants, DOS,UNIX, LINUX
Build Tools: Jenkins, Maven, ANT.
Database: Oracle 11g/10g/9i, MySQL
IDE Tools: Eclipse, Net Beans, SQL Developer, MS Visual Studio
Version Control: Git
Software Tools: MS Office Suite(Word, Excel, Project),MS Visio, Rational Rose, Apache Maven
Web Technologies: HTML, CSS, XML, JSP
CloudComputing Tools: Amazon AWS.
Application Servers: Web Logic, Web Sphere, JBoss, Tomcat.
Development Methodologies: Agile/Scrum, Waterfall.
PROFESSIONAL EXPERIENCE
Confidential, Houston, TX
Big data / Hadoop Developer
Responsibilities:
- Used Sqoop to transfer data between RDBMS and HDFS.
- Involved in collecting and aggregating large amounts of streaming data into HDFS using Flume and defined channel selectors to multiplex data into different sinks.
- Implemented complex map reduce programs to perform map side joins using distributed cache. I have a good exposure to Oracle Golden gate for data migration.
- Managing Amazon Web Services (AWS) infrastructure with automation and configuration management tools such as Chef, Ansible, Puppet, or custom-built .designing cloud-hosted solutions, specific AWS product suite experience.
- Migrated services from a managed hosting environment to AWS including: service design, network layout, data migration, automation, monitoring, deployments and cutover, documentation, overall plan, cost analysis, and timeline.
- Implemented multiple Map Reduce Jobs in java for data cleansing and pre-processing.
- Worked with the team to increase cluster from 32 nodes to 46 nodes, the configuration for additional data nodes was done by Commissioning process in Hadoop.
- Designed MarkLogic solutions to provide highly available, fully scalable high speed ingestion of published oriented data. Solution allowed for both transactional processing and warehouse reporting against the same system.
- Designed and implemented custom writable, custom input formats, custom partitions and custom comparators in Mapreduce.
- Thoroughly tested Mapreduce programs using MRUnit and Junit testing frameworks. Responsible for troubleshooting issues in the execution of Mapreduce jobs by inspecting and reviewing log files
- Converted existing SQL queries into Hive QL queries. Configured, deployed and maintained a single node storm cluster in DEV environment
- Implemented UDFs, UDAFs, UDTFs in java for hive to process the data that can’t be performed using Hive inbuilt functions.
- Effectively used Oozie to develop automatic workflows of Sqoop, Mapreduce and Hive jobs.
- Wrote the Storm topology with HDFS Bolt and Hive Bolts as destinations.
- Experience in writing Storm topology development, maintenance and bug fixes.Gathered the business requirements from the Business Partners and Subject Matter Experts.
- Utilized Agile Scrum Methodology to help manage and organize a team of 4 developers with regular code review sessions.
- I was actively involved in Creating and managing process on Golden Gate (replicat, extract, pump). Also I have done Advanced troubleshooting in golden gate and processes.
- Experienced and implemented best practices successfully in the new environment for Data Ingestion using Beam-Storm.Loaded and transformed large sets of structured, semi structured and unstructured data in various formats like text, zip, XML and JSON.
- Refined the Website clickstream data from Omniture logs and moved it into Hive.
- Written multiple MapReduce programs to power data for extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV & other compressed file formats.
- Defined job flows and developed simple to complex Map Reduce jobs as per the requirement.
- Optimized Map/Reduce Jobs to use HDFS efficiently by using various compression mechanisms.
- Developed PIG UDFs for manipulating the data according to Business Requirements and also worked on developing custom PIG Loaders.
- Worked on developing ETL processes (Data Stage & Talend open Studio) to load data from multiple data sources to HDFS using FLUME and SQOOP, and performed structural modifications using Map Reduce, HIVE.
- Responsible for creating Hive tables based on business requirements.
- Used Spark streaming to receive real time data from the Kafka and store the stream data to HDFS using Scala and databases such as HBase, and MongoDB.
- I have Administrated Golden Gate sever in a heterogeneous and Data warehouse environments. Configured the adhoc Golden gate setup for restore recovery the data as per business requirement Written RDDs in Scala to execute in Spark to see the performance benefit against Map Reduce.
- Worked on regular expression related text-processing using the in-memory computing capabilities of Spark using Scala.
- Implemented Partitioning, Dynamic Partitions and Buckets in HIVE for efficient data access.
- Involved in NoSQL database design, integration and implementation.
- Loaded data into NoSQL database Hbase.
- Knowledge on handling Hive queries using Spark SQL that integrate Spark environment.
- Also, explored Spark MLIB library to do POC on recommendation engines.
Environment: Hadoop, CDH4, Map Reduce, HDFS, Pig, Hive, Impala, Oozie, Java, spark, Kafka, Flume, Storm, Knox, Linux,Talend, Scala, Maven, Java Scripting, Oracle 11g/10g, SVN.
Confidential, Westlake, TX
Big data / Hadoop Developer
Responsibilities:
- Involved in building a multi-node Hadoop Cluster
- Imported data into HDFS using Sqoop.
- Experience in retrieving data from databases like MYSQL and Oracle into HDFS using Sqoop and ingesting them into HBase.
- Developed Hive Queries to analyze the data in HDFS to identify issues and behavioral patterns.Worked on shell scripting to automate jobs.
- Used Pig Latin to analyze datasets and perform transformation according to business requirements.
- Responsible for developing, support and maintenance for the ETL (Extract, Transform and Load) processes using Talend Integration Suite.
- Experience in using Sqoop to import the data on to Cassandra tables from different relational databases and also importing data from various sources to the Cassandra cluster using Java API’s.
- Worked on implementing Flume to import streaming data logs and aggregating the data to HDFS through Flume.
- Implemented MapReduce programs to perform joins using secondary sorting and distributed cache.
- Generated daily and weekly Status Reports to the team manager and participated in weekly status meeting with Team members, Business analysts and Development team.
Environment: Hadoop, Map Reduce, HDFS, Hbase, Hive, MYSQL, Oracle, Sqoop, Pig,Flume, UNIX, Java, Java Script, Maven, Eclipse.
Confidential, Austin, TX
Java Developer
Responsibilities:
- Designed and Created test plan, test scenarios and test cases for Data warehouse and ETL testing.
- Responsible to translate business requirements into quality assurance test cases
- Handled the entire project from planning to the delivery for modules of WebApps, Credit Bureau, Customer Mapping, Payments module, ETL filter changes.
- Developed the project framework and handled the entire project till closure.
- Done Six Sigma Green belt certification on “Reducing Cycle time for data validation in Data Consolidation Project” and devised new process flow and received appreciation from AIC (Associate In charge), Delivery Manager and Reporting Manager
- Prepared Test strategies and Test exit report for all the modules of data consolidation project.
- Review of test scenarios, test cases and Data warehouse Test Results.
- Developed test scripts using SQL queries to validate data.
- Prepared Regression Test Plans, Requirements Traceability Metrics (RTM), positive and negative test scenarios, detailed oriented Test Scripts, Test Kickoff documents, Test Scorecard for test progress status, Test Results, Release Check list, Lessons Learned documents and Regression Test Suite for future use.
- Responsible in testing Initial and daily loads of ETL jobs.
- Interacted with design team to decide on the various dimensions and facts to test the application.
- Planned ahead of time to test the mapping parameters and variables by discussing with BA’s.
- Extensively usedHP Quality Centerto track defects and managed them.
Confidential
Java Developer
Responsibilities:
- Actively participated in requirements gathering, analysis and design and testing phases.
- Responsible for use case diagrams, class diagrams and sequence diagrams using Rational Rose in the Design phase.
- Involved in Analysis, design and coding on J2EE Environment.
- Connectivity with Databases MySQL and Oracle.
- Involved in writing the database integration code.
- Web application support and maintenance using Java/J2EE(Struts 1.2), Oracle, MS SQL server, MySQL.
- Used the JDBC for data retrieval from the database for various inquiries.
- Involved in writing database connection classes for interacting with Oracle database.
- Worked on Presentation Layer using Struts Tiles, JSPs, and Servlets.
- Development of Parser classes to parse the data received from the Front tier to pass it to back end.
- Development of Servlets in web application.
- Used Spring Core for dependency injection/Inversion of Control (IoC).
- Responsible to write complex SQL and HQL queries to retrieve data from the Oracle database.
- Worked with J2EE and core java concept like Oops, GUI, Networking in java
- Created quality working J2EE code to design, schedule and cost to implement use cases.
- Development of web-pages and Applets.
- Involved in complete development of "Agile Development Methodology" and tested the application in each iteration.
- Prepared the design document based on requirements and sending project status report on weekly basis.
Environment: Java, J2EE, JSP, JDBC 3.0, Servlets 3.0, SQL 2000, MySQL 5.1, Oracle 10g, Apache Tomcat 6.0
Confidential
Java Developer
Responsibilities:
- Implementation of “Password Expiry Notification” for the application users.
- Written ETL programs using PLSQL.
- Developed “Electronic Invoicing program” which sends flat files to Oracle AR system.
- Implementation of “Generic Dog watch Mailing” Package which is responsible for sending Output, Log and Error files of all the data Extraction loads through E-Mail.
- Developed FTP UNIX shell which sends the files from local server to Remote server and acknowledges through E-Mail.
- Fixed few critical bugs which led to application stability.
- Done “Special Character” T&M using Dynamic SQL.
- Generation of Adhoc Reports using Complex SQL Queries.
- Handling the issues within SLA’s - involves writing PLSQL Scripts to fix the discrepancy in data, Using Unix Commands to debug the Log, Error files etc.
- Worked on Critical RCA’s- Provided Workaround and fixed them permanently (PLSQL, Pro*C) and played a prominent role in Stabilizing the application.
- Developed a Social Health Recommender System where in the users can find forums, discussions and solutions to their health related issues.
- Integrated with the developer tools from Facebook, the social networking site, so that users can join in groups and discussion forums of their choice, giving them more option for refining their search.
- The search-related data from the required sites are gathered and this data is all stored and XML is used for retrieval of data.
Environment: Java, ETL, PLSQL, Unix, Oracle, FTP, Dynamic SQL, SQL, XML.
