We provide IT Staff Augmentation Services!

Big Data Developer Resume

0/5 (Submit Your Rating)

Warren Township, NJ

SUMMARY

  • 5 years of overall IT experience and 3+ Years of comprehensive experience as Big DataDeveloper Expertise in writingHadoopJobs for analyzing data using Hive, Pig, Spark, Python, Scala and Oozie.
  • Good knowledge ofHadoopArchitecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node and MapReduce concepts.
  • Experience in working with MapReduce programs using Hadoopfor working with Big Data.
  • Build a continuous ETL pipeline by using Kafka, Spark Streaming and HDFS.
  • Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice - versa.
  • Working experience on designing and implementing complete end-to-end Hadoop Infrastructure including PIG, HIVE, Sqoop, Oozie, Flume and zookeeper.
  • Experience loading data to hive partitions and creating Buckets in Hive.
  • Experience in providing support to data analyst in running Pig and Hive queries.
  • Developed Map Reduce programs to perform analysis.
  • Performed Importing and exporting data into HDFS and Hive using Sqoop.
  • Experience in writing shell scripts to dump the Shared data from MySQL servers to HDFS.
  • Experience in designing both time driven and data driven automated workflows using Oozie.
  • Experience in setting up InfiniBand network and build Hadoop cluster to improve the map reduce performance.
  • Experience in performance tuning the Hadoop cluster by gathering and analyzing the existing infrastructure.
  • Experience with reporting tool Tableau.
  • Extensively used ETL methodology for supporting Data Extraction, transformations and loading processing.
  • Ability to design and develop Extract Transform Load (ETL) packages in Oracle for manipulating data as per the requirement of the project.
  • Strong debugging and problem-solving skills with excellent understanding of system development methodologies, techniques and tools.
  • Worked in complete Software Development Life Cycle (analysis, design, development, testing, implementation and support) in different application domain involving different technologies varying from object-oriented technology to Internet programming on Windows NT, Linux and UNIX platforms.)
  • Strong with RDBMS concepts and worked on Oracle 8i/9i/10g.
  • Involved in writing shell scripts for Unix OS for application deployments to production region.
  • Created Components and services to consume/provide REST API’S.
  • Extensive database designing and programming skills using SQL Server 2008/2012.
  • Experience creating Data Access Layer usingEntity Framework Codefirst approach.
  • Experience using Source Code Version Control Systems like MS Visual Source Safe (VSS) and Team Foundation Server (TFS).
  • Good hands on creation of the HLD and LLD documents based on client requirements.
  • Experience in analytical and technical skills and developing systems using different software methodologies like Agile and Waterfall.

TECHNICAL SKILLS

Hadoop Eco System: Hadoop, MapReduce, HDFS, Hive, RedShift, Pig, SqoopOozie, Spark, Flume, ZooKeeper.

Hadoop Architectures: MR1 and MR2 (or) YARN.

Hadoop Distributions: Horton Works and Cloudera.

Languages: HQL, PIG LATIN, SQL, Python, Scala, Shell Scripting.

Tools: SQL Developer, SQOOP and Putty, SOAP, Fiddler, GitHub.

Cloud Platforms: AWS and Azure

Virtualization: VMware, Virtual Box, Docker.

Monitoring: Ganglia, Nagios.

Databases: HBASE, Cassandra, MYSQL Oracle, Microsoft Access.

Databases: SQL Server, Microsoft Access

Web Technologies: HTML, XHTML, XML, CSS

Version Control Tools: Visual Source Safe (VSS), Team Foundation Server (TFS).

Development Tools: MS Visual Studio, SQL Server Management Studio

Operating Systems: Microsoft Windows 8/7, Windows Server 2008 R2/2012

PROFESSIONAL EXPERIENCE

Big Data Developer

Confidential, Warren Township, NJ

Responsibilities:

  • Involved in Big data requirement analysis, develop and design solutions for ETL and Business Intelligence platforms.
  • Experience in writing Spark Core, Spark SQL, Hive/HQL scripts to extract and load data in to Hive Data Warehouse.
  • Used Sqoop extensively to import & export data to and from RDBMS systems like (Oracle, MySQL, Sql server) in to HDFS, HIVE data warehouse.
  • Experience in interacting with Business Analysts and Data Modelers and defined Source and Target Mapping documents and documenting best practices.
  • Good understanding of Hadoop1.x (MapReduce) & Hadoop 2.x (Yarn) programming models.
  • Experience with Oozie Workflow Engine in running workflow jobs with actions that run Hadoop Map/Reduce and Hive.
  • Have done SCALA POC's on this project to implement in the other project.
  • Experience in NoSQL Column-Oriented Databases like HBase and its Integration with Hadoop cluster.
  • Have done Batch process by using apache spark with Scala programing.
  • Worked with efficient storage formats like PARQUET, AVRO and ORC integrated them with Hadoop and the ecosystem (Hive and spark). Also used compressions like Snappy and Zlib.
  • Apache Spark DataFrames/RDD's were used to apply business transformations and utilized HiveContext objects to perform read/write operations.
  • Developed multiple POCs usingPySparkand deployed on the Yarn cluster, compared the performance ofSpark, with Hive and SQL.
  • Performed Hadoop day-to-day operations (HDFS, Map-Reduce, Hbase and Hive) including operation, deployment and debugging of job issues.
  • Developed sematic layer on top of Hive to facilitate the analytics team to generate ad-hoc reports.
  • Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting.
  • Apply Partitioning and buckets on the table and writing queries for retrieving data from cluster, presently working on this project with Spark for fast processing.
  • Used Hive Queries in Spark-HQL for analysis and processing the data.
  • Used Kafka Streams to Configure Spark streaming to get information and then store it in HDFS.
  • Experienced in managing and reviewing the Hadoop log files.
  • Used Hive as ETL tool to do Transformations, even joins and some pre-aggregations before storing the data onto HDFS.
  • Involved in creating Hive Tables, loading data and writing Hive queries.
  • Worked on implementing the Master Data Management strategies for the new analytics platform.
  • Defining and creating the Unified Data Platform for all the Enterprise needs.
  • Exported data from HDFS environment into RDBMS using Sqoop for report generation and Visualization purpose.
  • Involved in Unit testing and delivered Unit test plans and results documents

Environment: Azure Data Lake, HDFS, Java Map Reduce, Yarn, Hive, Sqoop, Oozie, Shell ScriptingAzure SQL, Elastic Search, Spark, Hive, Pig, SAP HANA, Oracle.

Big Data Developer

Confidential, NY

Responsibilities:

  • Involved in extracting large sets of structured, semi structured and unstructured data.
  • Imported data from RDBMS (MySQL, Oracle) to HDFS and vice versa using Sqoop.
  • Developed jobs to move inbound files to HDFS file location based on monthly, weekly, daily and hourly partitioning.
  • Implemented a centralized Data Lake in Hadoop with data from various sources.
  • Created Hive External tables and loaded the data in to tables and query data using HQL.
  • Created Hive external tables and views, on the data imported into the HDFS.
  • Developed and implemented Hive scripts for transformations such as evaluation, filtering and aggregation.
  • Implemented ORC data format for Apache Hive computations to handle the custom business requirements.
  • Worked with data in multiple file formats including Avro, Parquet, and Sequence files, ORC and Text / CSV.
  • Worked on partitioning Hive tables and running the scripts in parallel to reduce run-time of the scripts.
  • Prepared ETL mapping Documents for every mapping and Data Migration document for smooth transfer of project from development to testing environment and then to production environment.
  • Developed User Defined Functions (UDF) in Java if required for hive queries.
  • Used Oozie Operational Services for batch processing and scheduling workflows dynamically.
  • Worked on creating End-End data pipeline orchestration using Oozie.
  • Developed bash scripts to automate process of Extraction, Transformation and Loading.
  • Stored MapReduce program output in Amazon S3 and developed a script to move the data to RedShift for generating a dashboard using QlikView.
  • Very good experience in managing the Hadoop cluster using Ambari.

Environment: Apache Hadoop 1.0.1, HDFS, Hive, Pig, Oozie, Eclipse, Sqoop, Hbase, Amazon S3, RedshiftQlikView.

Big Data Developer

Confidential

Responsibilities:

  • Experienced on loading and transforming of large sets of structured, semi structured and unstructured data from HDFS through Sqoop and placed in Hbase for further processing.
  • Involved in creating Hive Tables, loading data and writing Hive queries.
  • Analyzed the web log data using the HiveQL to extract number of unique visitors per day, page views, visit duration, most purchased product on website.
  • Integrated Oozie with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as Pig, Hive, and Sqoop) as well as system specific jobs (such as Java programs and shell scripts).
  • Converted Data from Cassandra and cleaned data based on the business requirement.
  • Converting ETL operations to Hadoop system using Pig Latin operations, transformations and functions
  • Collected and aggregated large amounts of web log data from different sources such as webservers, mobile and network devices using Apache Flume and stored the data into HDFS for analysis.
  • Experienced in managing and reviewing the Hadoop log files.
  • Exported data from HDFS environment into RDBMS using Sqoop for report generation and visualization purpose.
  • Deploying data from various sources into HDFS and building reports using Tableau.

Environment: Apache Hadoop 1.0.1, HDFS, HBase, Hive, Pig, Oozie, Eclipse, Sqoop, Hbase, Tableau.

Big data Developer

Confidential

Responsibilities:

  • Extracted files from MySQL, Oracle, and Teradata through Sqoop and placed in HDFS and processed.
  • Load and transform large sets of structured, semi structured and unstructured data.
  • Responsible to manage data coming from different sources.
  • Assisted in exporting analyzed data to relational databases using Sqoop.
  • Experience in using Apache Flume for collecting, aggregating and moving large amounts of data.
  • Analyzed large data sets by running Hive queries and Pig scripts
  • Worked with the Data Science team to gather requirements for various data mining projects
  • Involved in creating Hive tables, and loading and analyzing data using hive queries
  • Migrated ETL processes from Oracle, MSQL to Hive to test the easy data manipulation
  • Developed Simple to complex MapReduce Jobs using Hive and Pig
  • Involved in creating Hive tables, loading with data and writing hive queries which will run internally in MapReduce way
  • Involved in running Hadoop jobs for processing millions of records of text data
  • Worked with application teams to install operating system, Hadoop updates, patches, version upgrades as required
  • Developed multiple MapReduce jobs in java for data cleaning and preprocessing
  • Involved in loading data from LINUX file system to HDFS
  • Responsible for managing data from multiple sources
  • Created and maintained Technical documentation for launching HADOOP Clusters and for executing Hive queries and Pig Scripts
  • Develop Shell scripts for automate routine tasks.
  • Used Oozie and Zookeeper operational services for coordinating cluster and scheduling workflows.

Environment: Hadoop, HDFS, Pig, Hive, MapReduce, Sqoop, Flume, ETL tools LINUX, and Big Data

ETL/SQL Developer

Confidential

Responsibilities:

  • Responsible for requirements analysis, application design, coding, testing, maintenance and support.
  • Interacted with users, did requirements analysis and object-oriented design.
  • Compiled, programmed, built and debugged applications.
  • Created Stored Procedures, functions, Data base triggers, Packages and SQL Scripts based on requirements.
  • Developed Procedures for efficient error handling process by capturing errors into user managed tables.
  • Created complex SQL queries using views, sub queries, correlated sub queries.
  • Prepared, developed, and documented Technical analysis & test cases.
  • Developed UNIX shells/scripts to support and maintain the implementation.
  • Created Shell Scripts for invoking SQLscripts and scheduled them using crontab.
  • Defect management involving discussion with Business, Process Analysts, and team.
  • Defect Tracking and Prepare Test Summary Reports
  • Co-ordinate in delivery and implementation of new release and enhancements.
  • Developed application by using agile methodologies by following agile ceremonies
  • Experience in partitioning tables to improve performance and scalability.
  • Experience in developing analytical functions like Rank, Dense Rank, Row Number and Partition Over.
  • Tuned database SQL statements and procedures by examining explain plans, monitoring run times and system statistics. Applied database hints, fine-tuned indexes and rewrote code to improve performance.
  • Developed core PL/SQL Packages accounted for error handling and logging routines.
  • Developed reports using queries as per client request as a part of production support.

Environment: Oracle PL/SQL(MS Visual Studio, SQL Developer), Unix Shell Scripts.

We'd love your feedback!