We provide IT Staff Augmentation Services!

Sr Hadoop Developer Resume

3.00/5 (Submit Your Rating)

Halifax, NS

SUMMARY

  • Seeking a position where I can leverage my broad knowledge of Big Data Eco Systems experience in ingestion, storage, querying, processing and analysis of Big Data and software development to plan the analysis, architecture and development of software solutions to answer business challenges.Creative Web Developer/Web Designer/Graphic Designer with over 5+ years of professional experience.
  • Technically accomplished professional with 5+ years of total experience in Software Development and Requirement Analysis in Agile work environment and 5 years of Big Data Eco Systems In - depth knowledge and hands-on experience in dealing with Apache Hadoop components like HDFS, MapReduce, HiveQL, HBase, Pig, Hive, Sqoop, Oozie, Cassandra, Flume, and Spark.
  • Very good understanding/knowledge of Hadoop Architecture and various components such as HDFS, JobTracker, TaskTracker, NameNode, DataNode, Secondary Namenode and MapReduce concepts
  • Extensively worked on MRV1 and MRV2 Hadoop architectures.
  • Hands on experience in writing MapReduce programs, Pig & Hive scripts.
  • Designing and creating Hive external tables using shared meta-store instead of derby with partitioning, dynamic partitioning and buckets.
  • Extending Hive and Pig core functionality by writing custom UDFs.
  • Experience in importing and exporting data using Sqoop from Relational Database Systems to HDFS and vice-versa.
  • Knowledge on RabbitMQ.
  • Knowledge on Hortonworks Distribution.
  • Loaded streaming log data from various webservers into HDFS using Flume.
  • Experience in building Pig scripts to extract, transform and load data onto HDFS for processing.
  • Excellent knowledge of data mapping, extract, transform and load from different data source.
  • Experience in writing HiveQL queries to store processed data into Hive tables for analysis.
  • Excellent understanding and knowledge of NOSQL databases like HBase and Cassandra.
  • Expertise in database design, creation and management of schemas, writing Stored Procedures, Functions, DDL, DML, SQL queries & Modeling.
  • 4 Years of IT experience in ETL Architecture, Development, enhancement, maintenance, Production support, Data Modeling, Data profiling, Reporting including Business requirement, system requirement gathering.
  • 2 years of hands-on experience in shell scripting.
  • Knowledge on cloud services Amazon web services (AWS) and Azure.
  • Knowledge on Elastic MapReduce(EMR)
  • Proficient in using RDMS concepts with Oracle, SQL Server and MySQL.
  • Experience in Object Oriented Analysis, Design (OOAD) and development of software using UML Methodology, good knowledge of J2EE design patterns and Core Java design patterns.
  • Very good experience in complete project life cycle (design, development, testing and implementation) of Client Server and Web applications.
  • Hands on experience in application development using Java, RDBMS, and Linux shell scripting.
  • Worked extensively with CDH3, CDH4.
  • Skilled in leadership, self-motivated and ability to work in a team effectively
  • Possess excellent communication and analytical skills along with a can-do attitude.
  • Strong work ethics with desire to succeed and make significant contributions to the organization.

TECHNICAL SKILLS

Hadoop / Big Data Tools: HIVE, SQOOP,Spark MapReduce, HDFS,HBase Kafka, Flume, Storm, Zookeeper, Oozie

Programming Languages & Scripting Languages: C, C#, Java, Scala,R. Unix Shell Scripting, Perl.

Web/ Application Servers & Web Technologies: Apache Tomcat, WebLogic, WebSphere HTML, CSS, XML, JavaScript, jQuery, AngularJS, Node JS, AJAX, SOAP

Development Tools & Frameworks: Eclipse, Net Beans, SBT, SVN, Git, ANT, Maven, Jenkins, Bamboo, SOAP UI, QC, WebDriver, Jira, Bugzilla, SQL Developer, Splunk,Talend, Informatica. Struts, Hibernate and Spring MVC

Java/J2EE Technologies& Methodologies: Servlets, JSP, JSF, JDBC,Web services (SOAP, Restful). Agile/Scrum, UML, and Waterfall

Databases & Operating Systems: Oracle 12c, MySQL, MS - SQL Server, SQL Server, PostgreSQL Windows XP/Vista/7/8,10, UNIX, LINUX.

PROFESSIONAL EXPERIENCE

Sr Hadoop Developer

Confidential - Halifax, NS

Responsibilities:

  • Evaluated Spark's performance vs Impala on transactional data.
  • Used Spark transformations and aggregations using Python and Scala to perform min, max and average on transactional data.
  • Experienced in migrating data from HiveQL to SparkSQL using Scala.
  • Knowledge in using Spark Data-frames to load data in Spark Data-frames.
  • Knowledge on handling Hive queries using Spark SQL that integrate with Spark environment.
  • Used java to develop Restful API for database Utility Project.
  • Responsible for performing extensive data validation using Hive.
  • Designed a data model in Cassandra (POC) for storing server performance data.
  • Implemented a Data service as a rest API project to retrieve server utilization data from this Cassandra Table.
  • Implemented Python script to call the Cassandra Rest API, performed transformations and loaded the data into Hive.
  • Designed data model to ingest transactional data with and without URIs into Cassandra.
  • Implemented shell script to call python script to perform min, max and average on utilization data of 1000s hosts and compared the performance on various levels of summarization.
  • Involved in creating Oozie workflow and Coordinator jobs for Hive jobs to kick off the jobs on time for data availability.
  • Generated reports from this hive table for visualization purpose.
  • Migrated HiveQL to SparkSQL to validate Spark's performance with Hive's.
  • Implemented Proof of concept for Dynamo DB, Redshift and EMR
  • Proactively researched on Microsoft Azure.
  • Presented Demo on Microsoft Azure, an overview of cloud computing with Azure.

Environment: Hadoop, Azure, AWS, HDFS, Hive, Hue, Oozie, Java, Linux, Cassandra, Python, Open TSDB, Scala

Hadoop Engineer

Confidential

Responsibilities:

  • My responsibility involves in setting up the Cloudera Hadoop cluster for various line of Business.
  • Commissioning and Decommissioning of datanodes depending upon the incoming data and the storage space required.
  • Maintaining the cluster securely using Kerberos and making the cluster upend running all the time also troubleshooting if any problem persists.
  • Setting up permissions to allow the cluster used only by certain users.
  • Configured various property files like core-site.xml, hdfs-site.xml, mapred-site.xml based upon the job requirement
  • Exclusively worked on Hive UDF with Partition tables.
  • Developed mapreduce code using Java in order to get rid of certain security issues in the data.
  • Synchronizing both the unstructured and structured data using Pig and Hive.
  • Importing the complete data from RDBMS to HDFS cluster using Sqoop.
  • Creating external tables and moving the data onto the tables from managed tables.
  • Performing the subqueries in Hive.
  • Partitioning and Bucketing the imported data using HiveQL.
  • Partitioning dynamically using dynamic-partition insert feature.
  • Moving this partitioned data onto the different tables as per as business requirements.
  • Invoking an external UDF/UDAF/UDTF python script from Hive using Hadoop Streaming approach which is supported by Ganglia.
  • Setting up the work schedule using oozie.
  • Identifying the errors in the logs and rescheduling/resuming the job.
  • Worked with HUE for launching job and for data visualization.
  • After the transformation of data is done, this transformed data is then moved to Spark cluster where the data is set to go live on to the application using Spark streaming and kafka.
  • Created RDD's in Spark Scala and Python.
  • Created topics for kafka and helped in producing the messing feeds and making those are subscribed by the consumers.
  • Extracting data from data warehouse (TeraData) on to the Spark RDD's.
  • Experience on Spark with Scala/Python.
  • Worked on Stateful Transformations in Spark Streaming.
  • Worked on Batch processing and Real-time data processing on Spark Streaming using Lambda architecture.
  • Good hands-on experience on Loading data onto Hive from Spark RDD's.
  • Worked on Spark SQL UDF's and Hive UDF's.
  • Worked with Spark accumulators and broadcast variables.
  • Extensively worked on the cloud era Navigator for auditing purpose.
  • Worked on Falcon for replicating the DR solution.
  • Successfully build the Multi-tent Hadoop cluster and made sure that resource are effectively utilized among the different tenets.
  • Worked on HBASE security.
  • Used Impala for multi user quires as Impala is faster than Hive on Tez and Spark SQL.

Environment: HDFS, Hive, Pig, Sqoop, Oozie, MapReduce, Spark, Kafka, Zookeeper, Python, Java, HBase, Sentry, Falcon, Yarn.

Hadoop Developer

Confidential

Responsibilities:

  • Extracted data from relational databases such as SQL Server and MySql by using sqoop and loaded the data it to Hive and combined new tables with existing databases.
  • Developed code to pre-process large sets of various types of file formats such as Text, Avro, Sequence files,XML, JSON and Parquet using spark.
  • Configured big data workflows to run on the top of Hadoop which comprises of heterogeneous jobs like Hive,Sqoop and MapReduce.
  • Loaded various formats of structured and unstructured data from Linux file system to HDFS.
  • Created POC for Flume implementation for pushing batches of data to HDFS and HBase for real time processing of the files.
  • Parsing XML data into structured format and loading into HDFS.
  • Scheduled various ETL process and Hive scripts by developing wrapper scripts on tidal scheduler.
  • Utilized Tableau to visualize the analyzed data and validate reports.
  • Involved in reviewing both functional and non-functional aspects of the business model.
  • Championed to communicate and present the models to business customers and executives, using the same. configuration management, backup, and disaster recovery systems and procedures.
  • Experience using Talend for ETL tools and also extensive knowledge on Netezza.
  • Installed and configured Flume, Hive, Pig, Sqoop and Oozie on the Hadoop cluster, involved in Analyzing system failures, identifying root causes, and recommended course of actions.
  • Documented the systems processes and procedures for future references, responsible to manage data coming from different sources.

Environment: Hadoop, HDFS, Map Reduce, Flume, Pig, Sqoop, Hive, Pig, Sqoop, Oozie, Ganglia, HBase, Shell Scripting

Hadoop Developer

Confidential

Responsibilities:

  • Responsible for building scalable distributed data solutions using Hadoop.
  • Optimized Hive queries and used Hive on top of Spark engine.
  • Developed Kafka producer and consumers, Cassandra clients and Spark along with components on HDFS, Hive.
  • Populated HDFS and HBase with huge amounts of data using Apache Kafka.
  • Used Kafka to ingest data into Spark engine.
  • Hands on experience in Spark and Spark Streaming creating RDD's, applying operations - Transformation and Actions.
  • Developed Spark applications using Scala for easy Hadoop transitions.
  • Managing and scheduling Spark Jobs on a Hadoop Cluster using Oozie.
  • Experienced data pipelines using Kafka and Akka for handling large terabytes of data.
  • Involved in integration of Hadoop cluster with Spark engine to perform BATCH and GRAPHX operations.
  • Worked on indexes, scalability and query language supporting using Cassandra.
  • Created Sqoop scripts for importing data from different data sources to Hive and Cassandra.
  • Used HUE for running Hive queries and created partitions according to day using Hive to improve performance.
  • Implemented Apache Nifi flow topologies to perform cleansing operations before moving data into HDFS.
  • Developed a Data flow to pull the data from the REST API using Apache Nifi with context configuration enabled.
  • Experience working on Talend ETL for performing data migration and data synchronization processes on the data warehouse.
  • Installed and Configured Hadoop cluster using Amazon Web Services (AWS) for POC purposes.
  • Worked on Amazon AWS concepts like EMR and EC2 web services for fast and efficient processing of Big Data.
  • Responsible for maintaining and expanding AWS (Cloud Services) infrastructure using AWS.
  • Developed the batch scripts to fetch the data from AWS S3 storage and do required transformations in Scala using Spark framework.
  • Written Python, Shell scripts for various deployments and automation process.
  • Extracted files from MongoDB through Sqoop and placed in HDFS and processed.
  • Developed data pipeline using Flume, Sqoop, Pig and Java MapReduce to ingest behavioral data into HDFS for analysis.
  • Visualized the analytical results using Tableau visualization tool.
  • Developed R scripts to implement predictive analysis graphs in tableau.
  • Moved Relational Database data using Sqoop into Hive Dynamic partition tables using staging tables.

Environment: Apache Spark, Kafka, Map Reduce, Cassandra, YARN, Sqoop, Oozie, HDFS, Hive, Pig, Java, Hadoopdistribution of Cloudera 5.4/5.5, Linux, XML, Eclipse, MySQL.

We'd love your feedback!