PROFESSIONAL SUMMARY:

• A dynamic and goal oriented DATA ENGINEER with 6+ years of extensive experience specializing in development of Enterprise application using various technologies.
• Leadership and managerial role in improving common software process and initiation new business solutions.
• Expertise in using JAVA, BIG DATA, HTML, DHTML, CSS, D3JS, JSON, Font awesome, JavaScript and Bootstrap, Neo4j and Keylines.
• In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node, and MapReduce concepts and experience in working with MapReduce programs using Apache Hadoop for working with Big Data to analyze large data sets efficiently.
• Extensive knowledge in all phases of Software Development Lifecycle (SDLC) and well versed in using Software development methodologies like Agile, Scrum and Test Driven Development(TDD).
• Excellent understanding of Object Oriented Programming Concepts and Design.
• Good knowledge on Big Data Hadoop.
• Good knowledge on HDFS, PIG, Hive, Sqoop, Flume, Oozie, NIFI,UNIX commands, base.
• Good knowledge with ETL and Query big data tools like Pig Latin and Hive QL.
• Good Knowledge in big data ingestion tools like Flume and Sqoop.
• Good knowledge on NoSQL database experience with HBase.
• Strong knowledge of Pig and Hive's analytical functions, extending Hive and Pig core functionality by writing custom UDFs.
• Knowledge in designing, sizing and configuring Hadoop environments. worked with application teams to install operating system, Hadoop updates, patches and version upgrades as required.
• Knowledge with managing and reviewing Hadoop log files.
• Having strong interpersonal, analytical and organizational skills.
• Good understanding of Business workflow, Business logic and Business methods for further implementation of user requirement in a distributed application environment.
• A quick learner, Self - motivated with ability to work in team/independently and adapt quickly to new and emerging technologies.

PROFESSIONAL EXPERIENCE:

Confidential, NJ

Sr. Data Engineer

Responsibilities:

• Lead adhoc team, where several requests from users are analyzed and data is provided.
• Worked with various adhoc request, using SQL, and automated the scripts.
• Monitored daily jobs running in cluster and validated the tables and identified nulls.
• Responsible for application and platform integration design and planning .
• Developed SQOOP scripts to migrate data from Oracle to Big data Environment
• Created Sqoop job with incremental load to populate Hive External tables.
• Responsible for coding Java Batch, Restful Service , Map Reduce program, hive query’s , testing, debugging, Peer code review, troubleshooting and maintain status report.
• Experience in manipulating/analyzing large datasets and finding patterns and insights within structured and unstructured data.
• Created Oozie workflows for scheduling.
• Maintained effective communication with clients, business and partners as needed.
• Worked with various POCs related to graph databases(Neo4j) and BI solution to improve the current analytics platform.
• Closely worked with architects to improve the applications and initiated team in implementing.
• Designed and developed PIG data transformation scripts to work against unstructured data from various data points and created a base line.
• Interacted with Business Team, Business Requirement Gaps Analysis, Customer Experience and Impact Analysis of application.
• Participated in all phases of the IT Project Delivery Framework, including review of complex business and functional requirements. (Kick Off meeting, Change Control, etc.).
• Started using apache NiFi to copy the data from local file system to HDP.
• Aligned execution priorities - project delivery, employee goals, and establish service levels with business and functional owners to drive service quality through quality control
• Provide direct oversight of development teams, adhering to development standards and SLAs, and escalating risks and issues to the Engineering Delivery Head.
• Communicate with business users regularly and understand their requests and provide the required data.
• Organize daily status calls and weekly meeting with the team.

Environment: Data Analysis, MySQL, HBase, Hive, Impala, Flume, NIFI, Agile, Neo4j, KEYLINES, Cypher, Shell Scripting.

Confidential, CA

Data Engineer

Responsibilities:

• Have been part of Risk team in Confidential .
• Develop Map reduce using JAVA.
• Worked with various Hadoop technologies like FLUME, MapReduce, HIVE, HBase, Data Ingestion etc.
• Used Pig to perform data transformations, event joins, filter and some pre-aggregations before storing the data onto HDFS.
• Experience in troubleshooting in MapReduce jobs by reviewing log files.
• Analyzed large amount of data with formats including XML, Json and Relational files from different data Sources.
• Experience running spark SQL queries in order to avoid complex algorithms for querying structured data.
• Developed data pipeline using Kafka, Sqoop, Hive and Java map reduce to ingest customer behavioral data into HDFS for analysis.
• Developed MapReduce programs for pre-processing and cleansing the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.
• Implement Partitioning, Dynamic Partitions, Buckets in HIVE. Responsible to manage data coming from different sources.
• Import the data from different sources like HDFS/HBase into Graph DB.
• Worked as a UI developer for NEO4J to develop Tools used by Digital Review Team( DRT ) and Monitor review team( MRT ) using underscoreJS, HTML, Ajax, JQuery, CSS, Bootstrap.
• Working with Keylines to implement link analysis which is used for real time decision.
• Good knowledge in using apache NiFi to automate the data movement between different Hadoop systems.
• Worked with Cypher and SQL queries to fetch test data
• Implemented AJAX for cross-browser compatibility.
• Followed agile methodology and involved in daily SCRUM meetings, sprint planning, showcases and retrospective.
• Worked with HBASE and Data ingestion for the project BDP and age variables.
• Worked on data visualization (Dashboards) for the Risk Analysis and management (RAM) for the payments in Confidential
• Involved in analyzing the requirements and participated in discussions with business analysts

Environment: MySQL, Base, Hive, Impala, Flume, Agile, HTML, CSS, JQUERY, AJAX, BOOSTRAP, JAVASCRIP, KEYLINES, Cypher.

Confidential, TX

Data Engineer

Responsibilities:

• Followed agile methodology and involved in daily SCRUM meetings, sprint planning, showcases and retrospective.
• Involved in analyzing the requirements and participated in discussions with business analysts.
• Involved in creating Technical Design documents and Integration Solution Design documents.
• Primary responsibilities include building scalable distributed data solutions using Hadoop ecosystem.
• Worked on analyzing Hadoop cluster and different big data analytic tools including MapReduce, Hive.
• Involved in loading data from LINUX file system, servers, Java web services using Kafka Producers, partitions.
• Analyze existing DAT/IDW Framework and provide suggestions for real time/batch data injection into Hadoop Data Lake and batch data load through hdfs/sqoop scripts into Hadoop through hive.
• Importing and exporting data into HDFS and HIVE using Sqoop.
• Used Sqoop to transfer data between databases (Oracle & Teradata) and HDFS and used Flume to stream the log data from servers.
• Developed MapReduce programs for pre-processing and cleansing the data in HDFS obtained from heterogeneous data sources to make it suitable for ingestion into Hive schema for analysis.
• Experienced in managing and reviewing Hadoop log files.
• Desired exposure to NoSQL databases, data flow design, Hadoop, map-reduce
• Developed Map Reduce programs in Java for parsing the raw data and populating staging tables.
• Suggested an appropriate solution for the assigned modules and problems, also doing their High level and Low level design.
• Implement the workflows using Apache Oozie framework to automate tasks.
• Use SQL queries and other tools to perform data analysis and profiling.
• Actively involved in the enhancement, maintenance and design for the application.
• Developed data access components using JDBC, DAOs, and Beans for data manipulation.
• Wrote embedded SQL and Stored procedures for data manipulation and interacting with the Oracle database.
• Designed and developed firm's responsive single-page (SPA) investor web portal; fit for desktops, tablets and mobile, using: Bootstrap , RESTful API and SQL.
• Validated the Queries and Stored Procedures.
• Used GIT repository as Version Control.
• Used Maven as build and configuration tool.
• Involved in unit, system, integration and regression testing and reviewed the code.

Environment: Java, Java Script, XML, HTML, CSS, JSP, JDBC, Eclipse, Maven.

Confidential, TX

Program Analyst

Responsibilities:

• Implemented the architecture of the modules according to the J2EE standards (Presentation-tier, Business-tier and Persistence-tier).
• Involved in writing refined SQL queries for extracting the records of attackers .
• Used Agile methodology named SCRUM for all the work performed.
• Extensively used Spring Inversion of Control and Transaction Management feature.
• Extensive experience in designing and developing front-end/user-interface (UI) of web applications using HTML 4.0/5.0, CSS3, JavaScript, JQuery, Bootstrap, AJAX.
• Worked with Java Script events and functions.
• Implemented Ajax/JQuery for Asynchronous Retrieval of data.
• Updated CSS layout to reflect the new changes as per the addition of new components.
• Used SoapUI for Web service’s testing.
• Used Log4j for logging errors
• Performed extensive test driven development-using JUnit for unit testing.
• Used Maven for build code.
• Deployed the application on WebSphere application server.
• Worked with the MKS version control software.

Environment: Java, J2EE, DB2, Java Script, XML,, Eclipse, AJAX/JQuery, JUnit, Maven, MKS, SoapUI