Hadoop Developer Resume
Miami, FL
PROFESSIONAL SUMMARY:
- 8+ Years of working experience in IT Development, Enhancements, Production Support, Administration and Development including 4 years of Big Data Ecosystem related technologies.
- Over all two years of hands on experience using Spark framework wif Scala.
- 3+ years of Java programming experience in developing web - based applications and Client- Server technologies.
- Good Knowledge onHadoopCluster architecture and monitoring teh cluster
- Excellent understanding / noledge ofHadooparchitecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node and MapReduce programming paradigm.
- Strong Knowledge and development experience inHadoopand Big Data Ecosystem including MapReduce, HDFS, Hive, Pig, Spark, Cloudera Navigator, Mahout, HBase, Zookeeper, Kafka, Strom, Sqoop, Flume, Oozie and Impala.
- Extensive noledge in programming wif Resilient Distributed Datasets (RDDs).
- Strong experience as a JavaDeveloperin Web/intranet, client/server technologies using Java, J2EE, Servlets, JSP, JSF, EJB, JDBC, SQL, Hibernate framework and Spring frame work.
- Experience in NoSQL Column-Oriented Databases like HBase and its Integration wifHadoop cluster.
- Experience in Amazon AWS services such as EMR, EC2, S3, CloudFormation, RedShift whichprovides fast and efficient processing of Big Data.
- Extensive experience in data ingestion, big data storage planning, complex transformations, data integration, analysis for Pharmaceutical, Healthcare and Retail sectors.
- Experience wif scripting languages (SQL, Scala, Java, Pig, Bash/Python) to manipulate data.
- Procedural noledge in cleansing and analyzing data using HiveQL, Pig Latin, and custom MapReduce programs in Java.
- Worked wif various formats of files like delimited text files, click stream log files, Apache log files, Avro files, JSON files, XML Files.
- Familiar wif data architecture including data ingestion pipeline design,Hadoopinformation architecture, data modeling and data mining, machine learning and advanced data processing. Experience optimizing ETL workflows.
- Expertise wif using Oracle, MySQL, DB2 databases and writing highly complex SQL queries.
- Experienced in using Integrated Development environments like Eclipse, NetBeans, IntelliJ, Spring Tool Suite.
- Hands on experience in application development using Java, RDBMS, and Linux shell scripting.
- Developed simple to complex MapReduce jobs using Java language.
- Excellent understanding and noledge of NOSQL databases like MongoDB, HBase, andCassandra.
- Experience in Data warehousing, Data Extraction, Transformation and loading (ETL) data from various sources like Oracle, Teradata, DB2, Microsoft Excel and Flat files into Data Warehouse and Data Marts using Informatica Power Center.
- Excellent working noledge of popular frameworks like Struts, Hibernate, and Spring MVC.
- Experience in Agile Engineering practices.
- Good experience working wif Hortonworks Distribution and Cloudera Distribution.
- Involved in upgrading existing MongoDB instances from version 2.4 to version 2.6 by upgrading thesecurity roles and implementing newer features.
- Experienced in providing technical solutions to teh business on applications dat are developed onHadoopand Its eco-systems. Experience in cloud platforms like AWS, AZURE.
- Handled importing teh data from various data sources and performed transformation using Java, Hive, Pig, Yarn, HBase, Sqoop, Oozie, Flume, Windows Azure, Zookeeper.
- Knowledge of data warehousing and ETL tools like Informatica, Talend and Pentaho.
- Analyzed large amount of data wif formats including XML, Json and Relational files from different Data Sources.
- Experienced in different application servers like JBoss/Tomcat, WebLogic and IBM WebSphere.
- Worked wif efficient storage formats like PARQUET, AVRO and ORC integrated them wifHadoop and teh ecosystem (Hive, Impala, and spark). Also used compressions like Snappy and Z lib.
- Building, publishing customized interactive reports and dashboards, report scheduling usingTableau server.
- Created Hive external tables, uploading Data through HDFS into Hive External Tables.
- Experience in extending HIVE and PIG core functionality by using custom UDF’s and UDAF’s.
- Good experience working on analysis tool like Tableau for regression analysis, pie charts, and bar graphs.
- Good understanding of Software Development Life Cycle (SDLC) and sound noledge of project implementation methodologies including Waterfall and Agile.
- Ability to work in high-pressure environments delivering to and managing stakeholder expectations.
- Application of structured methods to: Project Scoping and Planning, risks, issues, schedules and deliverables.
- Strong analytical and problem-solving skills.
- Excellent interpersonal and communication skills, creative, research-minded, technically competent and result-oriented wif problem solving and leadership skills
TECHNICAL SKILLS
Big Data Echosystem: HDFS, Sqoop, Flume, Hive, Pig, MapReduce, YARN, Oozie, Kafka, Spark, Impala, Storm, Hue, Zookeeper, Parquet, MongoDB, Java, SQL.
Hadoop Distributions: Hortonworks, Cloudera, Amazon, IBM Big Insight
Databases & NOSQL Databases: Oracle, MYSQL, Microsoft SQL Server, HBase and Cassandra, MongoDB.
RDBMS: Teradata, Oracle Pl/SQL, MS SQL Server, MySQL and DB2
Operating Systems: Linux, UNIX, Windows.
Development Methodologies: Agile/Scrum, Waterfall.
Frameworks: Struts, spring and Hibernate
IDE's: Eclipse, Net Beans, GitHub, Jenkins, Maven, IntelliJ, Ambari.
Programming Languages: C, C++, JSE, XML, JSP/Servlets, Spring, HTML, JavaScript, JQuery, Web services, Python, Scala, PL/SQL & Shell Scripting.
ETL Tools: Ab initio, Informatica Power center and Pentaho
Reporting tools: Tableau
PROFESSIONAL EXPERIENCE
Confidential, Miami, FL
Hadoop developer
Responsibilities:
- Creating end to end Spark applications using Scala to perform various data cleansing, validation, transformation and summarization activities on user behavioral data.
- Developed custom Input Adaptor utilizing teh HDFS File system API to ingest click stream log files from FTP server to HDFS.
- Developed end-to-end data pipeline using FTP Adaptor, Spark, Hive and Impala.
- Used Scala to write code for all Spark use cases.
- Implemented design patterns in Scala for teh application.
- Implemented Spark using Scala utilized Spark SQL heavily for faster development, and processing of data.
- Exploring wif Spark for improving teh performance and optimization of teh existing algorithms inHadoopusing Spark Context, Spark-SQL, Data Frame, Pair RDD's, YARN.
- Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Java and Scala.
- Used Scala collection framework to store and process teh complex consumer information.
- Implemented a prototype to perform Real time streaming teh data using Spark Streaming wif Kafka.
- Handled importing other enterprise data from different data sources into HDFS using Sqoop and performing transformations using Hive, Map Reduce and tan loading data into HBase tables.
- Exported teh analyzed data to teh relational databases using Sqoop, to further visualize and generate reports for teh BI team.
- Collecting and aggregating large amounts of log data using Flume and staging data in HDFS for further analysis
- Analyzed teh data by performing Hive queries (Hive QL) and running Pig scripts (Pig Latin) to study customer behavior.
- Used Hive to analyze teh partitioned and bucketed data and compute various metrics for reporting.
- Created components like Hive UDFs for missing functionality in HIVE for analytics.
- Worked on various performance optimizations like using distributed cache for small datasets, Partition, Bucketing in Hive and Map Side joins.
- Created validate and maintain scripts to load data using Sqoop manually.
- Created Oozie workflows and coordinators to automate Sqoop jobs weekly and monthly.
- Installed and configured ApacheHadoop, Hive and Pig environment on AWS.
- Implemented POC Spark Cluster on AWS.
- Uploaded and processed more TEMPthan 30 terabytes of data from various structured and unstructured sources into HDFS (AWS cloud) using Sqoop and Flume.
- Tibco Jasper Soft studio was used for teh i-report analysis using AWS cloud.
- Used Reporting tools like
- to connect wif Hive for generating daily reports of data.
- Created reports for teh BI team using Sqoop to export data into HDFS and Hive.
- Used Oozie and Oozie coordinators to deploy end-to-end data processing pipelines and scheduling teh workflows.
- Continuous monitoring and managing theHadoopcluster.
- Used JUnit framework to perform Unit testing of teh application.
- Developed interactive shell scripts for scheduling various data cleansing and data loading process.
- Performed data validation on teh data ingested using Spark by building a custom model to filter all teh invalid data and cleanse teh data.
- Experience wif data wrangling and creating workable datasets.
Environment: HDFS, Pig, Hive, Sqoop, Flume, Spark, MapReduce, Scala, Tableau, Oozie, Oracle 11g, YARN, UNIX Shell Scripting, Agile Methodology
Confidential, Mansfield, MA
Hadoop Developer
RESPONSIBILITIES:
- Developed MapReduce jobs in Java API to parse teh raw data and store teh refined data.
- Developed custom Pig & Hive UDFs in Java for transforming teh data as per teh requirement for analysis.
- Supported Map Reduce Programs dat are running on teh cluster.
- Worked on Tableau for exposing data for further analysis and for generating transforming filesfrom different analytical formats to text files.
- Integrated teh Hive warehouse wif HBase.
- Cluster monitoring, maintenance and troubleshooting.
- Created staging tables and ingested data as dynamic partitions in Hive.
- Imported data using Sqoop from teh MySQL tables.
- Involved in loading data from UNIX file system to HDFS.
- Developed Spark scripts using Python as per teh requirement.
- Used Spark API over ClouderaHadoopYARN to perform analytics on data in Hive.
- Exported teh analyzed data to teh relational databases using Sqoop for visualization and to generate reports for teh BI team.
- Implemented Oozie workflow for automating teh complete process.
- Managing and scheduling jobs on aHadoopcluster
- Written multiple MapReduce programs in Java for data extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV and other compressed file formats.
- Worked wif Avro Data Serialization system to work wif JSON data formats.
- Performed data validation on teh data ingested using MapReduce by building a custom model to filter all teh invalid data and cleansing teh data.
- Created HBase tables of large data sets of structured and unstructured data.
- Used Flume to collect, aggregate, and store teh log data from different web servers.
- Worked collaboratively wif all levels of business stakeholders to architect, implement and test Big Data based analytical solution from disparate sources.
- Responsible to manage data coming from different sources.
- Managed and reviewedHadooplog files.
- Mentored analyst team in writing required Hive Queries.
Environment: OEM, CDH 5.0.6, HDFS, MapReduce, HBase, Tableaus, Java, Python Hive, Flume, Pig, Spark, SQL and Sqoop.
Confidential, Flint, MI
Hadoop Developer
Responsibilities:
- Installed and configuredHadoopMapReduce, HDFS, developed multiple MapReduce jobs in Java for data cleaning and preprocessing.
- Involved in Installing, ConfiguringHadoopEco System, and Cloudera Manager using CDH3 Distribution.
- Good understanding and related experience wifHadoopstack - internals, Hive, Pig and MapReduce.
- Wrote MapReduce jobs in python to discover trends in data usage by users.
- Involved in defining job flows.
- Involved in creating Hive tables, loading wif data and writing hive queries which will run internally in map reduce way.
- Developed Pig Latin scripts to extract teh data from teh web server output files to load into HDFS.
- Involved in managing and reviewingHadooplog files.
- Implemented HBase Co-processors to notify Support team when inserting data into HBase Tables.
- Involved in runningHadoopstreaming jobs to process terabytes of text data.
- Load and transform large sets of structured, semi structured and unstructured data.
- Responsible to manage data coming from different sources.
- Supported Map Reduce Programs those are running on teh cluster.
- Involved in loading data from UNIX file system to HDFS.
- Installed and configured Hive and written Hive QL scripts.
- Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
- Experienced wif working on Avro Data files using Avro Serialization system.
- Solved small file problem using Sequence files processing in Map Reduce.
- Monitor System health and logs and respond accordingly to any warning or failure conditions.
- Performed cluster co-ordination through Zookeeper.
- Used Clear case for version control.
- Used JUnit for unit testing and Continuum for integration testing.
Environment:Hadoop, MapReduce, HDFS, Hive, PIG, Linux/Ubuntu, Java (jdk1.6), Datamining,Hadoop distribution of Cloudera, HBase, Map R, UNIX Shell Scripting, Clear case, JUnit.
Confidential, Madison, WI
Big Data Developer
Responsibilities:
- Worked in Loading and transforming large sets of structured, semi structured and unstructured data.
- Experienced in managing and reviewingHadooplog files.
- Imported Data from Different Relational Data Sources like RDBMS, Teradata to HDFS using Sqoop.
- Worked on writing transformer/mapping Map-Reduce pipelines using Apache Crunch and Java.
- Imported Bulk Data into Cassandra file system Using Thrift API.
- Involved in creating Hive Tables, loading data &writing Hive queries to invoke & run MapReduce jobs.
- Perform analysis on Time Series Data in Cassandra using Java API.
- Designed and implemented Incremental Imports into Hive tables
- Involved in collecting, aggregating and moving data from servers to HDFS using Apache Flume.
- Performed Hive jobs to parse logs and structure them in tables to get effective querying on teh log data.
- Implemented teh workflows using Apache Oozie framework to automate tasks.
- Worked wif Avro Data Serialization system to work wif JSON data formats.
- Worked on different file formats like Sequence files, XML files and Map files using MapReduce programs.
- Involved in Unit testing and delivering Unit test plans &results documentation using Junit and MR-Unit.
- Exported data from HDFS to RDBMS using Sqoop for report generation and data visualization purpose.
- Developed scripts and automated data management from end to end and sync up between all teh clusters.
- Created and maintained Technical documentation for launchingHadoopCluster.
Environment:Hadoop, HDFS, MapReduce, Hive, Oozie, Sqoop, Pig, MySQL, Java, Rest API, Maven, MR Unit, Junit
Confidential
Java Developer
Responsibilities:
- Responsible for developing and modifying teh existing service layer based on teh business requirements.
- Involved in designing & developing web-services using SOAP and WSDL.
- Created tables, stored procedures in SQL for data manipulation and retrieval, Database Modification using SQL, PL/SQL, Stored procedures, triggers, Views in Oracle 9i.
- Involved in integration testing teh Business Logic layer and Data Access layer.
- Designed user-interface and checking validations using JavaScript.
- Involved in design of JSP's and Servlets for navigation among teh modules.
- Designed cascading style sheets and XML also performed client-side validations wif java script using Business Objects, XML, and JDBC.
- Developed various EJBs for handling business logic and data manipulations from database.
- Integrated JSF wif JSP and used JSF Custom Tag Libraries to display teh value of variables defined in configuration files.
- Developed UI using HTML, JavaScript, and JSP, and developed Business Logic and Interfacing components.
- Involved in JUnit testing of teh application using JUnit framework
- Used Maven build to wrap around Ant build scripts
- SVN Tortoise is used for version control of code and project documents.
Environment: Eclipse, JQuery, JSP, Servlets, JSF, JDBC, HTML, JUnit, JavaScript, XML, SQL, Maven, Web Services, UML, WebLogic Workshop and SVN.
