Hadoop Developer Resume
4.00/5 (Submit Your Rating)
SUMMARY
- Over 3 years of experience in Design, Development, Implementation, and Maintenance of Hadoop and SQL Server applications.
- Expertise in using Hadoop (HDFS, Map Reduce) and its Ecosystems such as, Pig, Hive, Sqoop, Oozie, Zookeeper, HBase and Flume for data storage, processing and analysis.
- Experience in writing and developing custom UDF’s for Pig and Hive, also troubleshooting issues in Map Reduce, Pig, and Hive.
- Experience in using Spark Standalone Cluster Manager and YARN, creating a Spark Context, configuring properties, building a Spark application, RDDs, Pair RDDs, RDD operations (Transformations and Actions), RDD Partitions, RDD Persistence, Initializing Streaming Context, D - Streams, Data frames and SQL operations, check pointing, Caching, deploying and monitoring applications, and passing functions to Spark using Scala.
- Worked on Spark-SQL, Loading and saving data using the Parquet, JSON, and relational databases.
- Used Flume for streaming server logs into HDFS and for website log aggregation.
- Good knowledge on monitoring and workflow scheduling tools such as Oozie and Ganglia.
- Excellent working knowledge on configuring Hadoop Single node and Multi node Clusters.
- Working Knowledge on NOSQL DB’s such as Mongo DB, and HBase.
- Good knowledge on Amazon Web Service components using EC2 for computing and S3 for storage.
- Experience in Core Java, and Web Services.
- Good hands-on knowledge of Configuration Management and Deployment tools such as Puppet, Ansible, and IBM Urban code deploy.
- Proficient in scripting, Git and Git work flows using Bit bucket.
- Experience in developing Continuous Integration/ Continuous Delivery pipelines using Team city.
- Good understanding of container Orchestration using Docker, Kubernetes and Open shift.
- Working knowledge of Job scheduling using Autosys.
- Experience in TSQL DDL/DML, Creating Tables, Views, Triggers, Stored Procedures, clustered, non-clustered indexes, User Defined Functions and complex SQL queries.
- Excellent communication and interpersonal skills. Versatile team player with proven problem-solving skills.
TECHNICAL SKILLS
- Cloudera CDH
- Apache Hadoop (MRv1
- MRv2)
- Hive
- Pig
- Sqoop
- HBase
- Oozie
- Flume
- Phoenix
- Zookeeper
- Spark
- Mongo DB
- Cassandra
- Unix/Linux
- Python
- Java
- SQL Server 2000/2005/2008
- SSIS
- MS Excel
- Toad
- SQL Analyzer
- ER win
- Windows.
PROFESSIONAL EXPERIENCE
Hadoop Developer
Confidential
Responsibilities:
- Moving Product Database data from MS SQL Server to BDPaaS (Big Data Platform as a Service) using Sqoop.
- Creating Hive External tables as per requirement and defined with appropriate partitions, loading and transforming the structured data using Hive.
- Creating a Rules engine and Hive External tables as per the requirements.
- Data export, aggregation and loading Incremental Data into HDFS using Sqoop.
- Integrating Hive with Aqua Data Studio and Advanced Query Tool using MapR-ODBC.
- Building the Spark source code with Maven and Deploying Spark on a cluster.
- Developed Spark applications in Eclipse and Scala IDE with Maven and SBT.
- Loaded data from the local file system, HDFS, relational databases and Apache Cassandra.
- Creating Hive Context, inferring schema using case classes, Loading and saving data using the Parquet, JSON, relational databases, and arbitrary sources
- Worked on spark streaming using Kafka with Gobblin. Used Spark JSON, Log Analysis, and Xml processing.
- Resolving Spark-Hive Integration related Issues and Kafka Integration with Spark and Hadoop.
- Setting up New Tenant Volumes for Dev, Stage, and Prod and creating security groups.
- Setting up security based on UNIX based POSIX groups and SQL Server Authentication based.
- Worked on Team Foundation Server, and Rally source control systems.
Environment: MAPR, Spark, Scala, HDFS, Hive, Kafka, Gobblin, Sqoop, Oozie, Java.
Hadoop Data Analyst
Confidential
Responsibilities:
- Creating managed and external tables using Hive with required partitions and buckets. Moved Hive tables from one database to another. Used hive for fetching JSON data and process.
- Creating Hive UDFs for missing functionality for analytics.
- Created Hive queries to set performance tuning parameters to increase overall performance.
- Worked on Avro to JSON, JSON to Avro, Avro to Meta, Avro to Schema conversions, and merging all Avro files into one Avro file, and used hive to create a table from Avro file.
- Worked on Spark RDD Transformations and Actions for Spark applications.
- Building the Spark source code with Maven and Deploying Spark on a cluster.
- Developing Spark applications with Maven and SBT.
- Worked on Spark-Sql, Loading and saving data using the Parquet, JSON, and relational databases.
Environment: CDH, Spark, Scala, HDFS, YARN, Hive, Pig, Flume, Sqoop, Oracle, Java.
Big Data/ Hadoop Developer
Confidential
Responsibilities:
- Written Map Reduce jobs in java for data cleaning and preprocessing.
- Involved in loading data from Linux local file system to HDFS, defining job flows, managing and reviewing log files.
- Created components such as Hive UDFs for missing functionality in Hive for analytics.
- Created Hive queries to set performance tuning parameters to increase overall performance.
- Migrated the data to MySQL from Hive using Sqoop.
Environment: Cloudera CDH, HDFS, MR, Hive, Sqoop, Java, MySQL.
