We provide IT Staff Augmentation Services!

Hadoop/spark Developer Resume

3.00/5 (Submit Your Rating)

VA

SUMMARY

  • Hadoop Developer with around 8+ years of experience in Information Technology & 5+ years in Hadoop Ecosystem and 4+ years in Java, J2EE
  • Expertise in Hadoop Ecosystem components HDFS, MapReduce, Hive, Pig, Sqoop, Hbase and Flume for Data Analytics.
  • Has hands on experience on fetching the live stream data from DB2 to Hbase table using Spark Streaming and Apache Kafka.
  • Capable of processing large sets of structured, semi - structured and unstructured data sets.
  • Experience in job workflow scheduling and monitoring tools like Oozie and Zookeeper.
  • Expertise in writing MapReduce Jobs in Java for processing large sets of structured, semi-structured and unstructured data sets and store them in HDFS.
  • Experience in developing Custom UDFs for datasets in Pig and Hive.
  • Proficient in designing and querying the NoSQL databases like HBase.
  • Experience in designing and developing tables in HBase and storing aggregated data's from Hive Table.
  • Knowledge on integrating different eco-systems like HBase - Hive, HBase - Pig
  • Experience on streaming data using Apache Flume.
  • Good Knowledge in Apache Spark and SparkSQL.
  • Skilled on migrating the data from different databases to Hadoop HDFS and Hive using Sqoop.
  • Deep Knowledge in the core concepts of MapReduce Framework and Hadoop ecosystem
  • Analyzed large structured datasets using Hive's data warehousing infrastructure
  • Extensive knowledge of creating manages tables and external tables in Hive Eco system.
  • Worked extensively in design and development of business process using SQOOP, PIG, HIVE, HBASE
  • Knowledge on Spark framework for batch and real time data processing.
  • Knowledge on Scala Programming Language.
  • Good knowledge in Software Development Life Cycle (SDLC) and Software Testing Life Cycle (STLC).
  • Excellent communication and inter-personal skills detail oriented, analytical, time bound, responsible team player and ability to coordinate in a team environment and possesses high degree of self-motivation and a quick learner.

TECHNICAL SKILLS

Big data/ Hadoop Ecosystem: HDFS, MapReduce, HIVE, PIG, HBase, Sqoop, Flume, Oozie, Spark, Kafka, Impala.

Java / J2EE Technologies: Core Java, Servlets, JSP, JDBC, XML, REST, SOAP, WSDL

Programming Languages: Java, Scala, Python, PySpark, SQL, PL/SQL, Linux shell scripts.

NoSQL Databases: MongoDB, Cassandra, HBase

Database: Oracle 11g/10g, DB2, MS-SQL Server, MySQL, Teradata.

Web Technologies: HTML, XML, JDBC, JSP, CSS, JavaScript, SOAP

Tools: Used: Eclipse, Putty, Winscp, NetBeans, QC, QlikView

Operating System: Ubuntu (Linux), Win 95/98/2000/XP, Mac OS, Red Hat

Methodologies: Agile/Scrum, Rational Unified Process and Waterfall

Distributed platforms: Hortonworks, Cloudera, MapR

PROFESSIONAL EXPERIENCE

Hadoop/Spark Developer

Confidential, VA

Responsibilities:

  • Experienced in implementing and working on the python code using shell scripting.
  • Responsible for developing data pipeline Apache NIFI and Spark/Scala to extract the data from vendor and store in HDFS and REDSHIFT.
  • Worked on integrating Python with Web development tools for developing Web Services in Python using XML, JSON.
  • Optimizing the poorly written Spark/Scala jobs by monitoring the YARN UI to use more parallelism.
  • Working on python's UNITTEST and PYTEST frameworks.
  • Used Python/ HTML / CSS to help the team implement dozens of new features in a massively scaled Google App Engine web application.
  • Installed and configured Hadoop MapReduce, HDFS, HIVE, PIG, SQOOP, Flume, Oozie on the Hadoop cluster are installed and configured.
  • Strong Experience with Python automation in automating Rest API and UI automation using Selenium web driver using Python.
  • Worked in Spark streaming to get ongoing information from the Kafka and store the stream information to HDFS.
  • Developed and Configured Kafka brokers to pipeline server logs data into Spark Streaming.
  • Completed data extraction, aggregation and analysis in HDFS by using PySpark and store the data needed to Hive.
  • Worked on MySQL database to retrieve information from storage using Python.
  • Having good knowledge on an Apache Cassandra for storing the data in a Cluster.
  • Built an Ingestion Framework dat would ingest the files from SFTP to HDFS using Apache NIFI and ingest financial data into HDFS.
  • Developed a fully automated continuous integration system using Git, Jenkins.
  • Performed job functions using Spark API's in Scala for real time analysis and for fast querying purposes.
  • Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Python and Scala.
  • Designed and developed scalable Azure APIs using Flask web framework in Python and Integrated with Azure API Management, Logical Apps and other Azure services.
  • Loaded all datasets into Hive from Source CSV files using Spark and Cassandra from Source CSV files using Spark/PySpark.
  • Conducted systems design, feasibility on recommend cost-TEMPeffective cloud solutions such as Amazon Web Services (AWS), Microsoft Azure and Rackspace.
  • Worked on Micro services for Continuous Delivery environment using Docker and Jenkins.
  • Worked on the MySQL migration project to make the system completely independent of the database being used.
  • Perform data analysis on NoSQL databases such as HBase and Cassandra.
  • Providing a responsive, AJAX-test driven design using JavaScript libraries such as JavaScript, jQuery, AngularJS and Bootstrap.js - Using Subversion for version control.
  • Written test cases using PyUnit test framework and Selenium Automation testing for better manipulation of test scripts.

Environment: Python, Mapreduce, Spark, Hadoop, HBase, Scala, Kafka, Hive, Pig, Sqoop, RBAC, ACL, Kafka, Pandas, Docker, ReactJS, Google APIs, SOAP, REST, Azure APIs, Shell Scripting, Selenium, AWS.

Data Engineer

Confidential - Tempe, AZ

Responsibilities:

  • Create Tables, Stored Procedures and Views as per requirements.
  • Written python scripts to extract data from different data sources like Salesforce, Google analytics Big Query, Amazon Redshift and Life plus
  • Constructed ETL interfaces using SSIS to aggregate, cleanse and migratedataacross enterprise-wideMDM, transactional database, andDataWarehousing systems using stageddataprocessing techniques, patterns and best practices.
  • Data sources are extracted, transformed and loaded to generate CSV data files with Python programming and SQL queries.
  • Created Jobs and scheduled them using SQL server agent.
  • Automated many routine tasks using SSIS.
  • UsedAWS Lambdato perform data validation, filtering, sorting, or other transformations for every data change in a DynamoDB table and load the transformed data to another data store.
  • Troubleshoot the SQL Server jobs failed and provide detailed analysis for the failure.
  • Involved in requirement gathering and data analysis and Interacted with Business users to understand the reporting requirements.
  • DevelopedSSRSreports by migrating data from DB2.
  • Utilized Spark SQL API in PySpark to extract and load data and perform SQL queries.
  • Worked extensively on AWS Components such as Elastic Map Reduce (EMR), Elastic Compute Cloud (EC2), Simple Storage Service (S3)
  • Worked on developing pyspark script to encrypting the raw data by using Hashing algorithms concepts on client specified columns.
  • Developed solutions using Alteryx tool to provide the data for the dashboard in formats dat include JSON, csv, excel, etc. Experience in working with business intelligence and data warehouse software, including SSAS/SSRS/SSIS, Business Objects, Amazon Redshift, Azure Data Warehouse and Teradata.
  • Utilized Apache Spark with Python to develop and execute Big Data Analytics and Machine learning applications, executed machine Learning use cases under Spark ML and Mllib.
  • Identifying and reducing the errors in an ERP,CRMor SCM system
  • Involved in logical and physical designs and transforming logical models into physical implementations.
  • Worked closely with the BI Data Engineers to identify necessary data, understand the schema and ETL, as well as build the meta-data needed to satisfy business requirements.
  • Worked on creating DDL, DML scripts for the data models.
  • Developed Spark/Scala, Python for regular expression (regex) project in the Hadoop/Hive environment with Linux/Windows for BigData resources.
  • StreamedAWSlog group intoLambdafunction to create service now incident.
  • Worked and learned a great deal from Amazon WebServices (AWS) Cloud services likeEC2, S3, EMR.
  • As part of Data Lake team, involved in ingesting 207 source systems which include databases (DB2, MySQL, oracle), flat files, mainframe files, xml files into the Data Lake Hadoop environment which are later explored by reporting tools.
  • Strong Experience in implementing Data warehouse solutions inAmazon web services (AWS)Redshift; Worked on various projects to migrate data from on premise databases to AWS Redshift, RDS and S3.
  • Responsible for setting up a MemSql cluster on AWS EC2 Instance
  • Translatingdatamodels and Business rules of client'sCRMand transactional source systems like Salesforce, SAP HANA, DB2, Microsoft Azure, Teradata containing marketingdataelements required for Digital Campaigns.
  • Performed extensive data analysis on fraudulent data using complex SQL queries and discovered patterns for anti-money laundering activities.
  • Prepared Test plan, test case according to the Source to target mapping document.
  • Created new complex & simple reports and visualizations and supported existing reports.
  • Created interactive dashboards and visualizations of claims reports, Fraud analysis and improved statistical data using Tableau.
  • Source system analysis for required metrics & dimensions anddatarefresh statistics from sources like Oracle, DB2, AS400, SQL server consisting ofCRM, Sales orders, Service Applications.
  • Mapped out the MasterDataManagement roadmap, from the initial RegistryMDMStyle phase to align the sourcedatamodels, DE DuplicateCRMleads and contacts from different source systems holistically in a top-down manner.
  • Advanced the data analysis and reporting skills using SQL server Data Tool 2012 (SSRS) and Tableau Desktop.

Environment: Spark, Scala, Python, Kafka, Mapreduce, Pig, Pyspark, AWS, Hive, Hbase, Cloudera, Cassandra, GCP, Azure.

Hadoop/ Spark Developer

Confidential - Brooklyn, OH

Responsibilities:

  • Extracted files from MySQL, Oracle, and Teradata through Sqoop and placed in HDFS storage Distribution and processed.
  • Worked on migrating MapReduce programs into Spark transformations using Spark and Scala, initially done using python.
  • Experienced in querying data using Spark SQL on top of Spark engine for faster data sets processing.
  • Collected and aggregated large amounts of web log data from different sources such as webservers, mobile and network devices using Apache Kafka and stored the data into HDFS for analysis.
  • Developed multiple Kafka Producers and Consumers from scratch implementing as per organization's requirements.
  • Responsible for creating, modifying topics (Kafka Queues) as and when required with varying configurations involving replication factors, partitions and TTL.
  • Worked on Ad hoc queries, Indexing, Replication, Load balancing, Aggregation in MongoDB.
  • Processed the Web server logs by developing Multi-hop flume agents by using Avro Sink and loaded into MongoDB for further analysis, Extracted files from MongoDB through Flume.
  • Expert knowledge on MongoDB NoSQL data modelling, tuning, disaster recovery backup used for distributed storage and processing using CRUD.
  • Planning, designing and creating data-wise rules for stream and streaming to Data-lake. Validating the data using Databricks. Converted structured data to Avro schema for streaming purpose.
  • Worked with Terraform replication tasks, for intermediate storing of data from PostgreSQL to S3 buckets. And capturing the CDC events and updating the buckets.
  • Worked on large sets of structured and semi-structured historical data.
  • Involved in working with Sqoop to import the data from RDBMS to Hive.
  • Created Hive tables to load the Data and stored as ORC files for processing.
  • Implemented Hive Partitioning and bucketing for further classification of data.
  • Worked on Performance and Tuning optimization of Hive.
  • Involved in cleansing and transforming the data.
  • Used spark SQL to perform sort, join and filter the data.
  • Copied the ORC files to amazon S3 buckets using Sqoop for further processing in amazon EMR.
  • Wrote custom UDF in Spark SQL using Scala.
  • Performed data Aggregation operations using Spark SQL queries.
  • Copied output data back to Hive from Amazon S3 buckets using Sqoop after getting the output desired by the business.
  • Setup Kafka to subscribe to topics(sensors) and load data directly to the Hive table.
  • Automated filter and join operations to join new data with the respective Hive tables using Oozie workflows on a daily basis.
  • Used Oozie and Oozie coordinators to deploy end to end data processing pipelines and scheduling workflows.

Environment: Hadoop, Amazon S3, EMR, Redshift, HDFS, Hive, Impala, Spark, Scala, Python, Pig, Sqoop, Oozie, GIT, Oracle, DB2, MySQL, UNIX Shell Scripting, JDBC.

Hadoop Developer

Confidential

Responsibilities:

  • Developed multiple MapReduce jobs in java for data cleaning and preprocessing.
  • Written MapReduce code to process and parsing the data from various sources and storing parsed data into HDFS.
  • Worked with Hive scripts to extract, transform and load the data.
  • Worked on moving all log files generated from various sources to HDFS for further processing.
  • Developed workflows using custom MapReduce, Pig, Hive and Sqoop.
  • Implemented Flume to collect the data from various sources and is loaded in to HDFS.
  • Written Hive UDF to sort Structure fields and return complex data type.
  • Responsible for loading data from UNIX file system to HDFS.
  • Used Sqoop to import from different database sources and file systems to HDFS and vice versa.
  • Developed suit of Unit Test Cases for Mapper, Reducer and Driver classes using MR Testing library.
  • Developed workflow in TWS to automate tasks of loading data into HDFS and preprocessing with PIG.
  • Modeled Hive partitions extensively for data separation and faster data processing and followed Pig and Hive best practices for tuning.
  • Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports.
  • Coordinated with Move IT Team to place the reports in secured Location from HDFS Path.
  • Build and release software baselines, code merges, branch creation in Subversion/GIT and interfaced between development and infrastructure.

Environment: Hadoop, YARN (Map Reduce -2), Oracle, HDFS, HIVE, PIG, Hue, Sqoop, Scala, TWS, Flume, Cloudera, GIT, Git Hub.

Java Developer

Confidential

Responsibilities:

  • Worked with the business community to define business requirements and analyze the possible technical solutions.
  • Requirement gathering, Business Process flow, Business Process Modeling and Business Analysis.
  • Extensively used UML and Rational Rose for designing to develop various use cases, class diagrams and sequence diagrams.
  • Used JavaScript for client-side validations, and AJAX to create interactive front-end GUI.
  • Developed application using Spring MVC architecture.
  • Developed custom tags for table utility component
  • Used various Java, J2EE APIs including JDBC, XML, Servlets, and JSP.
  • Designed and implemented the UI using Java, HTML, JSP and JavaScript.
  • Designed and developed web pages using Servlets and JSPs and also used XML/XSL/XSLT as repository.
  • Involved in Java application testing and maintenance in development and production.
  • Involved in developing the customer form data tables. Maintaining the customer support and customer data from database tables in MySQL database.
  • Involved in mentoring specific projects in application of the new SDLC based on the Agile Unified Process, especially from the project management, requirements and architecture perspectives.
  • Designed and developed Views, Model and Controller components implementing MVC Framework.

Environment: JDK 1.3, J2EE, JDBC, Servlets, JSP, XML, XSL, CSS, HTML, DHTML, JavaScript, UML, Eclipse 3.0, Tomcat 4.1, MySQL.

Jr. Java developer

Confidential

Responsibilities:

  • Implemented the project according to the Software Development Life Cycle (SDLC).
  • Implemented JDBC for mapping an object-oriented domain model to a traditional relational database.
  • Created Stored Procedures to manipulate the database and to apply the business logic according to the user’s specifications.
  • Developed the Generic Classes, which includes the frequently used functionality, so dat it can be reusable.
  • Exception Management mechanism using Exception Handling Application Blocks to handle the exceptions.
  • Designed and developed user interfaces using JSP, Java script and HTML.
  • Involved in Database design and developing SQL Queries, stored procedures on MySQL.
  • Used CVS for maintaining the Source Code.
  • Logging was done through log4j.

Environment: JAVA, Java Script, HTML, JDBC Drivers, Soap Web Services, UNIX, Shell scripting, SQL Server.

We'd love your feedback!