Hadoop Developer Resume
Irving, TX
PROFESSIONAL SUMMARY:
- Around 7 years of experience in IT and 4 years hands on experience in Hadoop ecosystem’s implementation, maintenance, ETL and Big Data analysis operations.
- Excellent knowledge in understanding Hadoop architecture and its components.
- Experience in using Hadoop ecosystem components such as HDFS, MapReduce, PIG, Hive, Sqoop, Spark, Kafka, and Flume.
- Full - scale knowledge of Hadoop components such as HDFS, Job Tracker, Name Node, Data Node.
- Good experience in Map Reduce Programming, Pig Scripting and analyzing data using HiveQL, Pig Latin , and HBase .
- Developed enterprise applications using Scala.
- Experience in designing and developing POCs in Spark using Scala to compare the performance of Spark with Hive and SQL/Oracle.
- Experience in managing and monitoring Hadoop cluster using Cloudera Manager.
- Experience in writing custom UDF’s which extends Hive and Pig core functionalities.
- Experience in working with different NOSQL databases like HBase .
- Handled different file formats like Parquet, Avro files, RC files using different SerDes in Hive.
- Experience in importing and exporting data using Sqoop from HDFS/Hive/HBase to Relational Database Systems.
- Experience creating real-time data streaming solutions using Apache Spark core, Spark SQL, Kafka, Spark Streaming and Apache Storm.
- Converted Map Reduce Applications to Spark.
- Performed Data Ingestion from multiple disparate sources and systems using Kafka.
- Transformed big data requirements into Hadoop-driven technologies working along with BI team.
- Experience in job workflow schedulers and monitoring applications such as Oozie and Zookeeper.
- Experience in dumping shared data into HDFS from MySQL by writing shell scripts.
- Good knowledge in the collection of log files from different sources using Flume and Kafka.
- Experience in Core Java, Java Virtual Machines and multi-thread processing.
- Proficiency in working with databases like Oracle, MySQL.
- Extensive experience in writing stored procedures and functions using SQL and PL/SQL.
TECHNICAL SKILLS:
Hadoop/Big Data: HDFS, MapReduce, HBase, Pig, Hive, Sqoop, Spark, Zookeeper, Impala, Oozie, Cassandra, MongoDB
Hadoop Distributions: Cloudera, Hortonworks
Programming Languages: C, C++, SQL, PL/SQL, HiveQL, Java, Python
Java/J2EE: JDBC, Java Script, JSP, Servlets, AngularJS
Operating Systems: Windows, Linux and Unix
Teradata, Oracle, MS: SQL Server, MySQL, MS-Access
NoSQL Databases: HBase, Cassandra, MongoDB
Methodologies: Agile, UML, Design Patterns (Core Java)
Education: Bachelor s Degree.
PROFESSIONAL EXPERIENCE:
Hadoop Developer
Confidential, Irving, TX
Responsibilities:
- Worked on a live 24 node cluster working on HDP (3.1.0).
- Worked with Sqoop jobs with incremental load to populate HAWQ External tables to Internal table.
- Importing and exporting data jobs, to perform operations like copying data from HDFS and to HDFS using Sqoop.
- Worked with Spark core, Spark Streaming and SQL modules of Spark.
- Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Scala.
- Worked with Pig, HBase, NoSQL database HBase and Sqoop, for analyzing the Hadoop cluster as well as big data.
- Very good understanding of Partitions, bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance.
- Creating Hive tables and working on them for data analysis in order to meet the business requirements.
- Developed a data pipeline using Spark and Hive to ingest, transform and analyzing data.
- Hands on experience with pivotal HAWQ. Created external and internal tables using HAWQ.
- Worked on data cleansing in order to populate into hive external table and internal tables.
- Experience in using Sequence files, RCFile, AVRO and HAR file formats.
- Hands on Experience writing PIG Scripts to Tokenized sensitive information using PROTEGRITY.
- Supporting and building the Data Science team projects on to Hadoop.
- Used FLUME to dump the application server logs into HDFS.
- Automating backups by shell for Linux to transfer data in S3 bucket.
- Experience in working with NoSQL database HBASE in getting real time data analytics.
- Hands on experience working as production support Engineer.
- Worked on RCA documentation.
- Automated incremental loads to load data into production cluster.
- Ingested the data from various file system to HDFS using Unix command line utilities.
- Hands on experience in moving data from one cluster to another cluster using DISTCP.
- Experience in reviewing Hadoop log files to detect failures.
- Worked on EPIC user stories and delivered on time.
- Worked on data ingestion part for malicious intent model. Automated daily incremental jobs that can run on daily basis.
- Hands on experience in Agile and scrum methodologies.
Environment: MapReduce, HDFS, Hive, HBase, Sqoop, Pig, Flume, Oracle 11/10g, DB2, Teradata, MySQL, HAWQ, PL/SQL, Java, Linux, Shell Scripting, SQL Developer, HP ALM.
Hadoop Developer
Confidential, Irving, TX
Responsibilities:
- Analyzing the requirement to setup a cluster.
- Installed and configured Hadoop, MapReduce, HDFS, developed multiple MapReduce jobs in Java.
- Developed Map Reduce programs in Java for parsing the raw data and populating Staging Tables.
- Developed Spark scripts by using Scala shell commands as per the requirement.
- Used Spark API over Cloudera Hadoop YARN to perform analytics on data in Hive.
- Developed Scala scripts, UDFFs using both Data frames/SQL and RDD/MapReduce in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop.
- Importing and exporting data into HDFS and Hive using Sqoop.
- Experienced in analyzing data with Hive and Pig.
- Writing PIG scripts to process the data.
- Developed PIG Latin scripts to extract the data from the web server output files to load into HDFS.
- Integrating bulk data into Cassandra file system using MapReduce programs.
- Involved in HBase setup and storing data into HBase, which will be used for further analysis.
- Installed and configured Hive and written Hive UDFs.
- Involved in creating Hive tables, loading with data and writing Hive queries using the HiveQL which will run internally in the map-reduce way.
- Extracted the data from MySQL into HDFS using Sqoop.
- Developed Scala and SQL code to extract data from various databases Champion new innovative ideas around the Data Science and Advanced Analytics Practices Creatively communicated and presented models to business customers and executives, utilizing a variety of formats and visualization methodologies.
- Performed advanced procedures like text analytics and processing, using the in-memory computing capabilities of Spark using Scala.
- Used HiveQL to analyze the partitioned and bucketed data and compute various metrics for reporting.
Environment: Java, Hadoop, MapReduce, HDFS, Hive, Sqoop, HBase, Pig, Oozie, Kerberos, Linux, Scala, Shell Scripting, Oracle 12c.
Hadoop Developer
Confidential, Cedar Rapids, IA
Responsibilities:
- Installation and Configuration of Hadoop Cluster.
- Working with Cloudera Support Team to Fine Tune Cluster.
- Working Closely with SA Team to make sure all hardware and software is properly setup for Optimum usage of resources.
- Plugin allows Hadoop MapReduce programs, HBase, Pig and Hive to work unmodified and access files directly.
- The plugin also provided data locality for Hadoop across host nodes and virtual machines
- Wrote data ingesters and map reduce program.
- Developed MapReduce jobs to analyze data and provide heuristics reports
- Good experience in writing data ingesters and complex MapReduce jobs in java for data cleaning and preprocessing and fine tuning them as per data set.
- Extensive data validation using Hive and also written Hive UDF.
- Involved in creating Hive tables loading with data and writing Hive queries which will run internally in map reduce.
- Moved data from HDFS to Cassandra using MapReduce and Bulk Output Format class.
- Experienced with different scripting language like Python and shell scripts.
- Lots of scripting (python and shell) to provision and spin up virtualized Hadoop clusters.
- Adding, Decommissioning and rebalancing node.
- Worked on HBase Java API to populate operational HBase table with Key value.
- Developed multiple MapReduce jobs in java for data cleaning and pre-processing.
- Developing and running MapReduce jobs on YARN and Hadoop clusters to produce daily and monthly reports as per user's need.
- Applying Patches and Perform Version Upgrades.
- Incident Management, Problem Management and Change Management.
- Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and Extracted the data from MySQL into HDFS using Sqoop.
- Automating and scheduling the Sqoop jobs in a timely manner using Unix Shell Scripts.
- Schedule MapReduce Jobs - FIFO and FAIR share.
- Installation and Configuration of other Open Source Software like Pig, Hive, HBASE, Flume and Sqoop.
- Integration with RDBMS using swoop and JDBC Connector.
- Working with Dev Team to tune Job Knowledge of Writing Hive Jobs.
- Continuously monitored and managed the Hadoop Cluster using Cloudera Manager.
Environment: Windows, UNIX, Java, Apache HDFS Map Reduce, Avro, Storm, Cloudera Pig Hive HBase Flume Sqoop, Cassandra, NOSQL
Hadoop Developer
Confidential, Columbus, OH
Responsibilities:
- Involved in gathering and analyzing business requirements and designing Hadoop stack as per the requirements.
- Worked with Sqoop import and export functionalities to handles large data set transfer between Oracle database and HDFS.
- Developed Unix shell scripts to load large number of files into HDFS from Linux File system.
- Moved data from Relational Database using Sqoop into Hive dynamic partition tables and external tables.
- Wrote Flume configuration files for importing streaming log data into HBase with Flume.
- Responsible for performing extensive data validation using Hive.
- Developed and executed shell scripts to automate the jobs.
- Wrote complex Hive queries and UDFs.
- Extensively using Pig for cleansing and pre-process the data for analysis.
- Performed validation and standardization of raw data from XML and JSON files with Pig and MapReduce.
- Implemented MapReduce programs to classified data organizations into different classified based on different type of records.
- Installed Oozie workflow engine to run multiple Hive and Pig jobs.
- Wrote MapReduce jobs for data processing and the result is stored in HBase for BI reporting.
- Worked on Text files, Sequence files, Map side joins, bucketing, partitioning for Hive performance enhancement and storage improvement.
- Implemented complex MapReduce programs to perform joins on Map side using Distributed Cache in Java.
Environment: Hadoop, MapReduce, HDFS, Hive, Sqoop, Flume, HBase, Pig, Java, SQL, UNIX, Shell Scripting, Oracle.
Java Developer
Confidential
Responsibilities:
- Develop the complete website for the company from the scratch and deploy the same
- Involved in requirements gathering.
- Designed and developed user interface using HTML, CSS and JavaScript.
- Designed HTML screens with JSP for the front-end.
- Involved in Database Design by creating Data Flow Diagram (Process Model) and ER Diagram (Data Model).
- Designed, Created and maintained database using MySQL
- Made JDBC calls from the Servlets to the Database to store the user details
- JavaScript was used for client-side validation.
- Servlets are used as the controllers and Entity/Session Beans for Business logic purpose.
- Used Eclipse for project building
- Participated in User review meetings and used Test Director to periodically log the development issues, production problems and bugs.
- Used WebLogic to deploy applications on local and development environments of the application.
- Debugged and fixed the errors
- Implemented and supported the project through development, Unit testing phase into production environment.
- Involved in documenting the application.
- Designed HTML screens with JSP for the front-end.
- Made JDBC calls from the Servlets to the Database.
- Involved in designing stored procedures to extract and calculate billing information connecting to oracle.
- Formatting the results from the Database as HTML reports to the client.
- JavaScript was used for client-side validation.
- Servlets are used as the controllers and Entity/Session Beans for Business logic purpose.
- Used WebLogic to deploy applications on local and development environments of the application.
- Used Eclipse for building the application.
- Participated in User review meetings and used Test Director to periodically log the development issues, production problems and bugs.
- Implemented and supported the project through development, Unit testing phase into production environment.
- Used PVCS Version manager for source control and PVCS Tracker for change control management
- Implemented Test First unit testing framework driven using Junit.
Environment: Java, JSP, Servlets, JDBC, Java Script, HTML, CSS, WebLogic, Eclipse and Test Director
Java Developer
Confidential
Responsibilities:
- Involved in the process Design, Coding and Testing phases of the software development cycle.
- Developed Servlets, JSP pages, Beans, JavaScript.
- Involved in developing a module for the transformation of files across the remote systems using JSP and servlets.
- JSP’s had HTML tags and JSP tags that were used as to develop the view of the module and servlets acted as backend java classes that were used to connect to Database retrieve files from there and send it across the systems or store the receiving files from other systems.
- Development of Various Reports using Java and SQL queries
- Participation in Design, Issue Resolution, and Sustainment meetings.
Environment: Java, Servlets, JSP, Oracle, Windows NT and UNIX, Tomcat
