We provide IT Staff Augmentation Services!

Lead Big Data Consultant Resume

3.00/5 (Submit Your Rating)

PROFESSIONALSUMMARY

  • Around 12 years of professional IT experience including 4 years in Big data ecosystems, experience in design, ingestion, storage, querying, processing and analysis of Big Data.
  • Big Data and Social Analytics certification from MIT using Python
  • Experience in Hadoop Architecture and its eco system like HDFS, Spark, MapReduce, Hive, Pig, HBase and Sqoop for data extraction, storage and analysis
  • Involved in Data Fabric platform designed to support Enterprise Data Management needs
  • Involved in Data Enrichment process through integration of multiple systems and applying business rules
  • Experience in analyzing data using PIG Latin, HIVEQL and custom MapReduce programs in JAVA
  • Experience in Apache Camel framework using Spring Boot and Java 8
  • Experience in Designing, Developing and implementing connectivity products that allow efficient exchange of data between core database engine and Hadoop ecosystem
  • Strong Experience in working using Spark core and Spark SQL with Hive support to load, transform and storage teh data
  • Experienced in performing in memory data processing for batch, real time, and advanced analytics using Apache Spark
  • Worked on NoSQL databases including HBase and MongoDB
  • Experience in developing big data processing applications using Hadoop map reduce.
  • Experience in Partitioning, bucketing and perform different types of joins on Hive tables and implementing with different serdes using Regular Expressions
  • Experience in ELK (Elastic Search, Log Stash and Kibana) store teh data in ES, process npm log files using Log Stash and visualize teh data using Kibana
  • Knowledge on Data Analytics, Predictive Analysis, Statistical modelling and Machine Learning concepts
  • Works with insights team for forecasting and predictive Analysis using Statistics concepts like RMSE, Time Series Analysis
  • Experience in use of pyspark, Pandas and NumPy and tools like Jupyter and Anaconda
  • Extensively worked on database applications using Oracle, SQL Server 2008/2005 and My SQL

TECHNICAL SKILLS

Big Data Technologies: Hadoop, Spark, Scala, Hive, Python, Map Reduce, HDFS, HBase, Pig, Sqoop, Impala, Cloudera HUE, ElasticSearch, Log Stash and Kibana

Java/J2EE: Java, J2EE, JavaScript, Servlets, JDBC, Struts, Java Beans, JMS, EJB

Frameworks: MEAN - Mongo, Express, Angular and NodeJS, Spring MVC.

Databases: MongoDB, SQL Server 2008/2005, Oracle 9i/10g/11g, My SQL

Languages: C, SQL, PL/SQL, Shell Scripting, JAVA, J2EE, Scala, Python

Web Services / Technologies: REST, SOAP, MVC, JavaScript, XML, HTML, JSON and Maven

Tools: Eclipse, Intellij IDEA, WebStorm, RoboMongo, Docker, npm, GIT

Build Tools: Maven, SBT

Hadoop Distributions: Cloudera and MapR

PROFESSIONAL EXPERIENCE

Confidential

Lead Big Data Consultant

Roles and Responsibilities:

  • Part of COMET Team to calculate Risk Weighted Assets as per Basel III norms
  • Design teh project flow from reading source files and reference data relational data from star system, enrich teh data using rule engine, web service calls to source system and export data with 9 quarters projections to source system
  • Used Apache Camel for setting up process workflow using Spring Boot Framework
  • Camel Context looks continuously for source files trade level data from Axiom and stress factor data from propl systems and redirect to different routes.
  • Reading source files, applying file validations, creating Hive tables, reading reference tables from Hazelcast instance, Pull rules from GIT repository as per teh business logic.
  • Cloudera and ecosystem is being used for teh entire project and update accordingly.
  • Sqoop scripts to read reference data from source STAR system and refer multiple times using Hazelcast instance
  • Involved in multiple phases of teh project from design, development, FT, UAT and release to production
  • Build multiple COMET UI reports using Angular.js where business can review teh projections RWAs for future 9 quarters based on teh projected ratings and stress factor
  • Build Apache Spark programs and apply rule based engine to enrich teh source data
  • Calculate Exposure RW percentage and calculate projected RWAs for teh next 9 quarters and send teh data back to upstream systems for IHC reporting
  • Using Camel through WEB Service continuously sending status messages to upstream system while exporting projected data
  • Involved in developing HiveQL scripts to apply teh transformation logics of source data and reference data
  • Writing Enriched data back to HDFS and exporting to RDBS to make it available to teh real-time reports.
  • Used Tortoise SVN for code repository and GIT for rules
  • Used Parquet files format in all data layers to improve teh performance.
  • Building teh application modules and deploying to FT Environment for teh FT testing team.

Environment: Cloudera CDH5, Hadoop, Map Reduce, Spark, HDFS, Hive, SQOOP, Apache Camel, Angular.js, Impala, Spring, Java 8, Unix, Shell scripting

Confidential, BaskingRidge, NJ

Sr. BigData Developer

Roles and Responsibilities:

  • Part of uHUB Command Center Product configurable for anomaly detection and alerting based system consumable for executives to see data and insights
  • Partner with Insight team to explore advanced analysis and visualization techniques
  • Support data sources like ORS, CDB and Myuhc from data lake including design, compliance and implementation
  • Supports cross organization analytics and segregation of analytics UHG organizations (UHC, Optum, OptumRx)
  • MapR 5.2 and its ecosystem is being used for teh entire project and update accordingly.
  • Implemented distressed constituents, repeat caller, multichannel switching and call data routing and reopens stats using Spark, Hive, Elastic Search and Log Stash
  • Associated with insights team in predictive analysis like predicting number of open issues and teh number of issues to each department and office
  • Involved in developing HiveQL scripts to apply teh transformation logics to generate teh uHUB shared views as per data mapping rules and business requirements.
  • Migration files in JavaScript to process teh query files with dependency using npm, Agenda, Mongo and NodeJS
  • Log Stash and Kibana for query monitoring dashboards for batch file processing
  • Implemented ECIBuilder trigger engine in Spark using Scala.
  • Spark Engine to trigger Hive queries for each consumer and generates an output file for teh Business to consume
  • Implemented Spark SQL using python and Scala to process ORS call data and Multichannel data for performance improvement
  • Configured spark configurations (dynamic memory allocation, shuffle partitions etc...) in spark-defaults conf file.
  • Involved in data versioning to create Hive views to get updated records and lineage to trace back teh source of each record.
  • Used ORC files format in all data layers to improve teh performance.
  • Pig Latin to process multichannel switching data and push teh dataset into Tableau for a dashboard to visualize teh members call data
  • Implemented business logic by writing Pig UDFs in Java used various UDFs from Piggybanks and other sources
  • Created and modified UDF and UDAF's for Hive whenever necessary and developed Hive queries to analyze data and generate results
  • Appreciation from Insights team and Product owners for fixing performance problem and logic given by insights

Environment: MapR 5.2, Apache Hadoop, HDFS, Spark, Hive, Pig Latin, HBase, Docker, npm, Agenda, JavaScript, Python, Elastic Search, Log Stash, Kibana Unix, Shell scripting, RoboMongo, Mongo DB, Jenkins, WebStorm

Confidential, Westlake, TX

Sr. Hadoop Developer

Roles and Responsibilities:

  • Working on Mortgage Risk Analytical Solutions Project at Confidential involves data importing, loading, analyzing, transformations and storage in HDFS and teh transformed data will be down streamed to teh analytics team to perform their analysis and generate reports for business insight.
  • Cloudera 5.5.1 and its ecosystem is being used for teh entire project and update accordingly.
  • Using sqoop to import historical data from RDBMS like DB2, sqlserver etc in data lake as text files into hdfs
  • Worked in Ingestion team to extract data files of different sources systems from landing zone to a centralized Data storage called data lake
  • Implemented pre-ingestion checks like file count check, Duplicate file checks and Entity count check as configured in source partner profile
  • Pig is used for post ingest checks validation of teh source data files matches teh control files
  • Created Historical view tables in HBase and snapshot tables in hive as part of Data Enrichment process
  • Involved in developing HiveQL scripts to apply teh transformation logics to generate teh required tables in HUB layer as per data rules and business requirements.
  • Involving in developing teh Spark SQL application and properties files to run teh spark job to ingest source data from datalake to edge nodes and create Hive views
  • Involved in data versioning to create Hive views to get updated records and lineage to trace back teh source of each record.
  • Implemented business logic by writing Pig UDFs in Java and used various UDFs from Piggybanks

Environment: Hadoop, Hdfs, Spark, Hive, Impala, Sqoop, Java, Shell scripting, Yarn, Oozie, pig, Cloudera 5.5.1

Confidential, Irving TX

Hadoop Developer

Roles and Responsibilities:

  • Worked on analyzingHadoop clusterand different big data analytic tools includingPigHBasedatabase andSqoop
  • Wrote extensive MapReduce jobs in Java to train teh cluster and developed Java MapReduce programs for teh analysis of sample log files stored in cluster.
  • Performance tuning and troubleshooting of MapReduce jobs by analyzing and reviewing Hadoop log files
  • Developed pig scripts for analyzing large data sets in teh HDFS.
  • Knowledge on handling Hive queries using Spark SQL that integrates with Spark environment.
  • Responsible for creating Hive tables, loading teh structured data resulted from MapReduce jobs into teh tables and writing hive queries to further analyze teh logs to identify issues and behavioral patterns.
  • Performed streaming of data into Flume transferred to HDFS for analysis.
  • Responsible for performing extensive data validation using Sqoop jobs, Pig and Hive scripts were created for data ingestion from relational databases to compare with historical data
  • Created HBase tables to load large sets of structured, semi-structured and unstructured data coming from UNIX, NoSQL and a variety of portfolios.
  • Involved in creating Oozie workflow and Coordinator jobs to kick off teh jobs on time for data availability.

Environment: Apache Hadoop, HDFS, Pig, Hive, Sqoop, NoSQL, Map Reduce, HBase, Shell Scripting, Java, Flume

Confidential,

Hadoop Developer

Roles and Responsibilities:

  • Design and Implementation of existing ETL process from SSIS to Hadoop
  • Developed Map Reduce programs as part of functional requirements
  • Implemented Oozie workflows for Map Reduce, Hive and Sqoop actions
  • Worked with different file formats and compression techniques to determine standards
  • Developed hive queries, Pig Scripts and UDFS to analyze/transform teh data in HDFS as per teh Business logic
  • Participation in Performance tuning in database side, transformations, and jobs level.
  • Responsible for cluster monitoring and troubleshooting, manage and review data backups, manage and review Hadoop log files.
  • Provided daily support to teh user team as well as technical team for system wide tasks including monitoring, alerting and problem resolution
  • Involved in testing, bug fixing and documenting teh work for teh project at each and every phase

Environment: Hadoop, HDFS, Hive, Apache Pig, Sqoop, Java, Shell Scripting, Ubuntu, SSIS

Confidential

Senior Java Developer

Roles and Responsibilities:

  • Worked with business analyst in understanding business requirements, design and development of teh project.
  • Implemented teh Struts frame work with MVC architecture.
  • Created new JSP's for teh front end using HTML, Java Script, Jquery and Ajax.
  • Developing JSP pages and configuring teh module in teh application.
  • Developed teh presentation layer using JSP, HTML, CSS and client side validations using JavaScript.
  • Involved in designing, creating, reviewing Technical Design Documents.
  • Applied J2EE design patterns like Business Delegate, DAO and Singleton.
  • Deployed and tested teh application using Tomcat web server.
  • Writing SQL queries to fetch teh business data using Oracle as database.

Environment: Java, JSP, JavaScript, Servlets, Struts, Hibernate, REST, EJB, JSF, JSP, Tomcat, Eclipse, SQL, Oracle.

Confidential

Java Developer

Roles and Responsibilities:

  • Involved in various phases of Software Development Life Cycle (SDLC) of teh application like Requirement gathering, Design, Analysis and Code development.
  • Extensively worked on user interface for few modules using HTML, JSP's, and JavaScript.
  • Generated Business Logic using servlets, Session beans and deployed them on Web logic server.
  • Created complex SQL queries and stored procedures.
  • Used Hibernate ORM framework with spring framework for data persistence and transaction management.
  • Analyzed teh banking and existing system requirements and validated them to suit J2EE architecture.
  • Designed teh process flow between front-end and server side components
  • Used JavaScript for client-side validation and Struts Validator Framework for form validations.
  • Implemented Java/J2EE Design patterns like Business Delegate and Data Transfer Object (DTO), DAO.

Environment: JAVA, JSF Framework, Chordiant, Eclipse, JavaScript, SQL, XML, JSTL, Facelets, JDBC, AJAX, Web Sphere, HTML 5, CSS, JUnit, Maven, Servlets

We'd love your feedback!