Big Data/hadoop Developer Resume
Houston, TX
SUMMARY
- 9+Years of professional experience in IT which includes 4 years of comprehensive experience in working with Apache Hadoop Eco - components, Spark streaming and Amazon Web services(AWS).
- Having Good Working Expertise on handling Terabytes of structured and unstructured data on significantly big Cluster Environment.
- Good understanding of HDFS Designs, Daemons, HDFS High Availability (HA).
- Good understanding and working experience on Hadoop Distributions like Cloudera and Horton works.
- Good Knowledge in creating event-processing data pipelines usingflume, KafkaandStorm.
- Expertise in data transformation & analysis using SPARK, PIG, HIVE
- UsedApache Solrto Query Indexed Data for Analytics for boosting search catalyst operations.
- ImplementedMLibfunctions for training and building fully loaded classifiers models usingSpark streaming, Spark SQLandMachine Learning APIs.
- Worked on real time data integration using Kafka - Storm data pipeline, Spark streaming andHBase.
- Worked on ingesting, reconciling, compacting and purging base table and incremental table data using Hive andHBaseand job scheduling throughOozie.
- Hands on experience with AWS components likeVPC, EC2, EBS, RedShift, CFT.
- Worked on all major components of Hadoop Eco-components such asHDFS, HIVE, PIG, Oozie, Sqoop, Map Reduce and YAR NonCloudera, MapR and Hortonworks distributions.
- Worked on setting upAWS EMR, EC2 clustersandMulti-Node Hadoop Clusterinside developer environment.
- Developed scripts and batch jobs to monitor and schedule variousSpark jobs.
- Worked on ingesting, reconciling, compacting and purging base table and incremental table data usingHiveandHBaseand job scheduling through Oozie.
- Worked on importing and exporting data from different databases likeOracle, Teradata,MySQL, and IBMDB2intoHDFSandHiveusingSqoop.
- Worked on collecting stream data into HDFS usingKafka, Flume and Flink.
- WrittenSpark SQL, Cassandra QL, Hive QL and PIG Latin queriesfor data analysis to meet the business requirements.
- Created base and incremented tables with partitioning, bucketing of table and creating UDF’s in Hive using optimization and tuning.
- Worked on NoSQL databases such asHBase, MongoDB and Cassandra.
- Experience with Oozie Workflow Engine to automate and parallelize Hadoop Map/Reduce, Hive and PIG MR Frameworks.
- Implemented automatic workflows and job scheduling usingOozie, Zookeeper and Ambari
- Good working experience on Hadoop Cluster architecture and monitoring the cluster. In-depth understanding of Data Structure and Algorithms.
- Build and configured ApacheTEZonHiveandPIGto achieve better responsive time while runningMRJobs.
- Extending Hive and Pig core functionality by writing custom UDFs, UDTF and UDAFs.
- Implemented Proofs of Concept on Hadoop stack and different big data analytic tools, migration from different databases to Hadoop
- Experienced in working with spark eco system using Spark SQL and Scala queries on different formats like Text file, Avro Parquet files.
- Worked on Software Development Life Cycle (SDLC)
- Experienced in creatingPL/SQL Stored Procedures, Functions, Cursors against Oracle (10g, 11g)
- Key participant in all phases of software development life cycle with Analysis, Design, Development, Integration, Implementation, Debugging, and Testing of Software Applications in client server environment, Object Oriented Technology and Web based applications.
TECHNICAL SKILLS
BIGDATA Technologies: Apache Spark, ApacheSolr, HDFS, MapReduce, Pig, Sqoop, Hive, HBase, Kafka, Flume, OOZIE, YARN
AWS Components: VPC, EC2, EBS, RedShift, CFT, EMR, Cloudwatch
Azure Components: Azure Data factory, Databricks and Azure SQL
Programming Languages: Scala, Python, R and Java
Java Technologies: STRUTS, J2SE, JDBC, Java Script, HTML, XML
Servers: Web Logic and Tomcat
Java IDEs: Lucidworks Fusion, NetBeans, Eclipse.
Databases: Oracle10g, PL/SQL, MYSQL, HBase, MongoDB, NOSQL, Cassandra
Operating Systems: Windows7, Windows XP, Unix and Linux, CentOS, Solaris
PROFESSIONAL EXPERIENCE
Confidential, Houston TX
Big Data/Hadoop Developer
Responsibilities:
- Involved in complete project life cycle starting from design discussion to production deployment.
- Installed Hadoop, Map Reduce, HDFS, AWS and developed multiple MapReduce jobs in PIG and Hive for data cleaning and pre-processing.
- Developed a job server (REST API, Spring boot, ORACLE DB) and job shell for job submission, job profile storage, job data (HDFS) query/monitoring.
- Implemented solutions for ingesting data from various sources and processing the Data utilizing Big Data Technologies such as Hive, Pig, Sqoop, Hbase, Map reduce, etc.
- Design and develop a daily process to do incremental import of raw data from DB2 into Hive tables using Sqoop.
- Involved in debugging Map Reduce job using MR Unit framework and optimizing Map Reduce.
- Extensively used Hive/HQL or Hive queries to query data in Hive Tables and loaded data into Hive tables.
- Developed data pipeline using Flume, Sqoop, Pig and MapReduce to ingest data into HDFS for analysis.
- Used Oozie and Zookeeper for workflow scheduling and monitoring.
- Effectively used Sqoop to transfer data from databases (SQL, Oracle) to HDFS, Hive.
- Integrated Apache Storm with Kafka to perform web analytics. Uploaded click stream data from Kafka to HDFS, Hbase and Hive by integrating with Storm.
- Designed Hive external tables using shared meta-store instead of derby with dynamic partitioning &buckets.
- Worked on Big Data Integration &Analytics based on Hadoop, SOLR, Spark, Kafka, Storm and web Methods.
- Worked on migrating MapReduce programs into Spark transformations using Spark and Scala.
- Design & implement ETL process using Talend to load data from Worked extensively with Sqoop for importing and exporting the data from HDFS to Relational Database systems/mainframe and vice-versa. Loading data into HDFS.
- Developed PIG Latin scripts to extract the data from the web server output files to load into HDFS.
- Created concurrent access for hive tables with shared/exclusive locks enabled by implementing Zookeeper in cluster.
- Strongly recommended to bring in Elastic Search and was responsible for installing, configuring and administration.
- Implemented using SCALA and SQL for faster testing and processing of data. Real time streaming the data using with KAFKA.
- Used OOZIE Operational Services for batch processing and scheduling workflows dynamically. worked on creating End-End data pipeline orchestration using Oozie.
- Populated HDFS and Cassandra with massive amounts of data using Apache Kafka.
- Involved in design and developed Kafka and Storm based data with the infrastructure team.
- Worked on major components in Hadoop Ecosystem including Hive, PIG, HBase, HBase-Hive Integration, Scala, Sqoop and Flume.
- Developed Hive Scripts, Pig scripts, Unix Shell scripts, programming for all ETL loading processes and converting the files into parquet in the Hadoop File System.
- Worked with Oozie and Zookeeper to manage job workflow and job coordination in the cluster.
Environment: Hadoop, Hive, Impala, Oracle, Spark, Pig, Sqoop, Oozie, Map Reduce, SQL
Confidential, Plano TX
Big data/Hadoop Developer
Responsibilities:
- Worked on Streamline analytics and data consolidation projects on the product Lucidworks Fusion.
- Integrated Kafka, Spark, Scala and HBase for streamline analytics for creating a predictive model and implemented Machine Learning protocol.
- Developed Scala scripts, UDFFs using both Data frames in Spark for Data Aggregation, queries and writing data back into OLTP system through Sqoop.
- Used Solr/Lucene for indexing and querying the JSON formatted data.
- Handled cloud operations inside Rackspace for persistence logic.
- Monitored OOTB requests with ATG, Akamai and TIBCO.
- Used REST services for handling unfinished jobs, knowing the status and creating a dataset inside a URL.
- Worked on ingesting, reconciling, compacting and purging base table and incremental table data using Hive and HBaseand job scheduling through Oozie.
- Designed and implemented streaming data on UI with Scala.js
- Utilized DevOps principle components to ensure operational excellence before deploying in production.
- Operating the cluster on AWS by using EC2, Akka, EMR, S3 and cloudwatch.
- Transported data to HBase using Flume.
- Used Java UDFs for performance tuning in Hive and Pig by manually driving the MR part.
- Used Java APIs such as machine learning library functions, graph algorithms for training and predicting the linear model in spark streaming.
- Have implemented unit testing in Java for pig and hive applications.
- Developed workflow in Oozie to automate the tasks of loading the data into HDFS and pre-processing with Pig.
- Involved in source system analysis, data analysis, and data modeling to ETL (Extract, Transform and Load).
- Written Spark programs to model data for extraction, transformation and aggregation from multiple file formats including XML, JSON, CSV& other compressed file formats.
- Worked on loading the RDBMS data onto Hadoop/HDFS, using sqoop.
- Developing the Pig scripts / UDFs to manipulate/transform the loaded data.
- Creating Hive tables, loading data into Hive using UDFs, partioning and bucketing principles.
- Migration Hive data into Hbase.
- Involved in integrations among Pig, Hive and Hbase.
- Instrumental in debugging. Created Hive External tables to store the processed data from Map Reduce
Environment: Hadoop0.20, HDFS, Hive, Pig, Sqoop, Java, Cloudera Distribution for Hadoop3(cdh3u5), Linux.
Confidential - ADA, MI
Spark/Hadoop Developer
Responsibilities:
- Developed multiple Spark jobs in PySpark for data cleaning and Pre-processing.
- Analyzed large data sets by running Hive queries and Pig scripts.
- Involved in creating Hive tables and loading and analyzing data using Hive queries.
- Developed simple/complex MapReduce jobs using Hive and Pig.
- Migrated existing MapReduce programs to Spark using Scala and Python
- Loaded and transformed large sets of structured, semi structured and unstructured data.
- Involved in running Hadoop jobs for processing millions of records of text data.
- Worked with application teams to install Operating Systems, Hadoop updates, patches, and version upgrades as required.
- Used APIs such as machine learning functions, graph algorithms for training and predicting the linear model in spark streaming.
- Wrote the PIG UDF in for converting Date format and time stamp formats from the unstructured files to required date formats and processed the same.
- Created 30 buckets for each Hive table based on clustering by client Id for better performance (optimization) while updating the tables.
- Written the Apache PIG scripts to process the HDFS data.
- Created Hive tables to store the processed results in a tabular format.
- Developed the Sqoop scripts in order to make the interaction between Pig and SQL Database.
- Writing the script files for processing data and loading to HDFS
- Writing CLI commands using HDFS.
- Developed the UNIX shell scripts for creating the reports from Hive data.
- Completely involved in the requirement analysis phase.
- Moved all log/text files generated by various products into HDFS location
- Created External Hive Table on top of parsed data
Environment: CDH4, MapReduce, HDFS, Hive, Pig, Sqoop, Linux, XML, PL/SQL, SQL connector
Confidential, Blue Ash, Oh
Hadoop Developer
Responsibilities:
- Importing and exporting data in HDFS and Hive using Sqoop
- Extracted files from MongoDB through Sqoop and placed in HDFS and processed
- Written Hive UDFs to extract data from staging tables
- Analyzed the web log data using the HiveQL and process through Flume
- Responsible for loading the customer's data and event logs from Oracle database, MySQL,Teradata into HDFS using Sqoop
- Implementing MapReduce programs to analyze large datasets in warehouse for business intelligence
- End-to-end performance tuning of Hadoop clusters and Hadoop MapReduce routines against very large data sets.
- Responsible for building scalable distributed data solutions using Hadoop.
- Handled importing of data from various data sources, performed transformations using Hive, MapReduce.
- Experience in optimization of MapReduce algorithm using combiners and partitions to deliver the best results and worked on Application performance optimization.
- Written the Spouts and Bolts after collecting the real stream customer data from Kafka broker to process and store into HBASE.
- Installed and configured Hive, Pig, Sqoop, Flume and Oozie on the Hadoop cluster.
- Developed simple to complex Map/Reduce jobs using Java, and scripts using Hive and Pig.
- Analyzed the data by performing Hive queries (HiveQL) and running Pig scripts (Pig Latin) for data ingestion
- Experienced on loading and transforming of large sets of structured and semi structured data.
- Managing and Reviewing Hadoop Log Files, deploy and Maintaining Hadoop Cluster.
- Involved in creating Hive tables, loading with data and writing Hive queries.
- Created data-models for customer data using the Cassandra Query Language.
- Ran many performance tests using the Cassandra-stress tool to measure and improve the read, Queried and analyzed data from Datastax Cassandra for quick searching, sorting and grouping.
Environment: CDH4, MapReduce, HDFS, Hive, Pig, Sqoop, Linux, XML, PL/SQL, SQL connector
Confidential
Hadoop Developer
Responsibilities:
- Responsible for loading the customer's data and event logs from Oracle database, Teradata into HDFS using Sqoop
- End-to-end performance tuning of Hadoop clusters and Hadoop MapReduce routines against very large data sets.
- Developed the Pig UDF'S to pre-process the data for analysis.
- Loading data from LINUX file system to HDFS.
- Importing and exporting data into HDFS and Hive using Sqoop and Flume.
- Proficient in using Cloudera Manager, an end-to-end tool to manage Hadoop operations.
- Wrote MapReduce jobs to generate reports for the number of activities created on a day, during a dumped from the multiple sources and the output was written back to HDFS
- Installed and configured Hadoop HDFS, MapReduce, Pig, Hive, and Sqoop.
- Wrote Pig Scripts to generate MapReduce jobs and performed ETL procedures on the data in HDFS.
- Prepared TEZ build from the source code and run the HIVE Query Jobs using TEZ execution engine rather using MR jobs for better performance
- Participate in requirements gathering and designing, development, testing and analysis phase of the project in documenting the business requirements by conducting workshops/meetings with various business users.
- Participated in client calls to gather and analyze the requirement.
- Importing and exporting data into HDFS from database and vice versa using Sqoop.
- Involved in Writing the Hive queries and Pig script files for processing data and loading to HDFS internally run in map reduce way.
- Wrote Hive queries and Pig script for data analysis to meet the Business requirements.
- Highly knowledge on Hadoop Administrator has extensive abilities of building, configuring and administration of large data clusters in big data environments using Apache distribution.
- Having knowledge on Hive and Hbase integration, pig and Hbase Integration.
- Used Spark-Streaming APIs to perform necessary transformations and actions on the fly for building the common learner data model which gets the data from Kafka in near real time and Persists into Cassandra
Environment: Hadoop, MapReduce, HDFS, Hive, Flume, Sqoop, Cloudera, Oozie, Data Stage, Java Cron Jobs, UNIX Scripts, Spark
