SUMMARY

• 9+ years of overall experience as Big Data and Java specialist
• Design customized pyspark internal framework to generate the report where complex data are gathered from various sources
• Develop customized pipeline for storing data in Hadoop environment from various source using "Streamset"
• Develop pyspark tool to apply transformation as per business requirements i.e. retrieve data from raw zone to Hive table and store processed output into the Hadoop environment for next - level processing
• Design and develop Data Quality framework using Spark framework to check data quality level before data loading into Hadoop environment.
• Expertise in using various BigData technologies such as MapReduce, Pig, Hive, Zookeeper, Sqoop, Oozie, Spark, Scala, NiFi
• Proficient in writing Hadoop jobs for analyzing data using Map Reduce, Hive and Pig
• Developed Kafka consumer to retrieve information from producers. And, Converting retrieved data into multiple files.
• Developed Java Mapper and Reducer programs for complex business requirements
• Developed customized spark internal framework for applying business logic
• Create control-M job for scheduling the work that execute jobs at specific time.
• Optimized Hive tables using techniques such as partitions and bucketing to provide better performance with HiveQL queries
• Responsible for Code review and moving Code from local feature branch to development.
• After unit testing, deploy component to test server and Production server
• Developed shell wrapper script to execute common jobs on cloud
• Good knowledge in SQL database - Oracle and MySQL
• Proficient in job workflow scheduling and monitoring tools like Oozie, Zookeeper and NiFi
• Experienced in developing applications using Java/J2EE technologies like Servlet, JSP, Hibernate, Spring, JDBC, JNDI, JMS, RESTful Web Services etc
• Expertise in developing applications using Spring and Hibernate (Object/Relational mapping framework)
• Proficient in writing build scripts using Ant & Maven
• Experience in Requirements gathering, Design & Development
• Experience in writing database objects like Stored Procedures, Triggers, SQL, PL/SQL packages and Cursors for Oracle
• Knowledge of various design patterns
• Experienced in using SVN, Clearcase for version control
• Proficient in unit testing the application using JUnit, MRUnit and logging the application using Log4j
• Experienced in Waterfall & Agile development methodology
• Strong communication and analytical skills with very good experience in programming & problem solving

TECHNICAL SKILLS

Hadoop/Big Data Technologies: Spark, HDFS, MapReduce, Pig, Hive, Sqoop, NiFi, Oozie, Zookeeper, YARN, Scala, Impala, Kafka, EMR Cluster, Cloudera, Control-M, Hbase, pyspark

Programming Languages: C, C++, Java, Python, Scala, FTL, SQL, Shell Scripting,

J2EE Technologies: Servlet, JSP, JDBC, JNDI, EJB

Frameworks : Struts, Spring, Hibernate, OFBiz

Web Technologies : HTML JavaScript, Ajax, CSS, XML, XSD, JSON, jQuery

Databases : Oracle, MySQL,DB2, SQL Server

Application/Web Servers : Apache Tomcat, JBoss

Web Services: RESTful Web Services.

IDE: Eclipse,IntelliJ, NetBeans

Version Control Systems : SVN, JIRA, Clear Case, Bitbucket

Testing/Logging tools : JUnit, EasyMock, PowerMock, log4j

Operating Systems : Windows, Unix, Linux

PROFESSIONAL EXPERIENCE

Confidential

Senior Software Engineer

Responsibilities:

• Design customized pyspark internal framework to retrieve source complex data and apply customized business logic and generate on-demand stakeholder reports
• Design and developed the java tool for converting fixed width file to structured data.
• Develop Streamset pipeline that would gather data from various source and stored in Hadoop environment
• Developing pyspark tool that can apply customized business transformation requirements to the necessary tables and store transformed data in Hive for future usage
• Design and develop the internal pyspark framework to provide the statistical report for the claims domain data.
• Design and develop Data Quality framework using Spark framework to check data quality level before data loading into Hadoop environment.
• Develop shell common script for creating Hive table and start / stop the pipeline, customized Data quality check for source data and calling customized Spark tool with make HBASE entry for storing meta data information.
• Develop control-M job as per the requirement to schedule and execute custom jobs. Build and manage job flows, Passing parameter to jobs, Creating job dependencies and alert as per application and business requirements.
• Use GIT Bit Bucket as the version control. After Unit Testing, integrate code (change-set) in the development branch. After tester and validates new change-set, it is deployed to SIT. Again integration testing is performed and based on satisfactory test results, code is finally deployed in the production.
• Collaborate and work with Business, Data Governance and System integration testing team to address code defect, resolve and fix bugs in a timely fashion from QA to Production

Environment: Java, Python, Spark, Pyspark, Hbase, Streamset, Kafka, hive, Impala, HUE, Maven, Intellij, Pycharm, Shell Script, Bitbucket,Log4j,Control M, Cloudera, pyTest

Confidential

Software Engineer

Responsibilities:

• Developed Kafka consumer to retrieve all types of loan information from producers. And, Converting retrieved data into multiple files as per requirement and stored into hive table as parquet file format.
• Creating python script for retrieving the data using sqoop and store the data into hive tables.
• Developing customized spark internal framework for applying business logic to hive table and exposed to the end user.
• Developed shell script to run spark application on cloud.
• Developed python script to download the source file from RESTful web service.
• Developing the Java-based tool for generating "hql" files.
• Developing Java-based application for retrieving data from varies source and stored in the MySQL database.

Environment: Java, Python, Spark, Spark SQL, Kafka, hive, Impala, MySQL, HUE, Maven, Intellij, Shell Script, GIT, Control-M, Cloudera

Confidential

Software Engineer

Responsibilities:

• To Ingest the data from S3 to HDFS in EMR using Hive External table and partition it appropriately for ease of access from spark
• Develop Spark - Scala program to perform all logical operations of huge volume (80GB) of data
• Setup the EMR Cluster Environment.
• Partition the output data on Weekending date using hive query
• Used Spark API over Amenzon AWS(EMR) to perform analytics on data in Hive.
• Loaded the data into Spark RDD and do in memory data Computation to generate the Output response.
• Implemented Partitioning, Dynamic Partitions, Buckets in HIVE.
• Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself.
• Involved in creating Hive tables, and loading and analyzing data using hive queries
• Shell Scripting to coordinate the entire process
• Involved in scheduling NIFI workflow engine to run scala and hive jobs.
• Tuning of Hive Jobs by controlling the number of mapper, reducer launched and spark Jobs

Environment: EMR Cluster, Spark, Scala, Hive, Shell Scripting, SBT,NIFI,Log4j,Eclipse,Putty

Confidential

Software Engineer

Responsibilities:

• Installed and configured Hadoop MapReduce, HDFS, developed multiple MapReduce jobs in Java for data cleaning and pre-processing
• Handled importing of data from various data sources, performed transformations using Hive, MapReduce, loaded data into HDFS and extracted the data from MySQL into HDFS using Sqoop
• The logs and semi structured content that are stored on HDFS were pre-processed and the processed data is imported into Hive warehouse which enabled business analysts to write Hive queries
• Built reusable Hive UDF libraries for business requirements which enabled users to use these UDF's in Hive querying
• Developed suit of Unit Test Cases for Mapper, Reducer and Driver classes using MR Testing library
• Getting the data from different system and integrate the different type data and sending the corresponding data to different type of system using the Web Service

Environment: Hadoop, Hive, MapReduce, HDFS, Pig Hive, Oozie, Sqoop, Java, Oracle, UNIX Shell Scripting, Eclipse, Maven, SVN

Confidential

Software Engineer

Responsibilities:

• Developed utility application to calculate days remaining and expiry calculation for ticketing for all types of reservations
• SPOC for all customer interactions related to business needs, requirements gathering
• Integrated with external application for sending mail and SMS for support all type of communication.
• Developed RESTful web service for providing all customer booking details and all other details to varies booking sites and customer front end
• Receiving Business Event from Sonic messenger and related information and storing in database .
• Developed the Spring batch service for getting all ticketing information and storing in the database in regular interval
• Writing Oracle Store Procedure for deleting outdated records from the database
• Expert in writing Hibernate Query Language (HQL) and Tuning the hibernate queries for better performance
• Developed the Unit Test cases using Power Mock and Easy Mock for all java classes.

Environment: Java, J2EETechnologies, Spring, Hibernate, Oracle, RESTful web Service, XML, JUnit, Eclipse, SVN, Maven, Ajax, JSON, Tomcat

Confidential

Software Engineer

Responsibilities:

• Developed the reports for clients such as Inventory report, different types of Order Report, Customer Report, Discount Products Report
• From the customer log information collect the needed information and store it in the database for creating report.
• Retrieved the information from varies source and developed the report for up-sell/cross-sell recommendations.
• Import all customer Products to the database to display the site.
• Writing JUnit Test cases using Power Mock and Easy Mock.

Environment: Java, J2EE Technologies, Spring, Hibernate, MySQL, JUnit, Eclipse, SVN, Maven, Ajax, Json, Tomcat

Confidential

Module Lead

Responsibilities:

• Lead development team with peak size of 4
• SPOC for all customer interactions related to business needs, requirements gathering.
• Integrated with external payment gateway for all Debit and credit card transactions.
• Developed end-user interface including new features such as loyalty point accumulation, redemption, delivery slot Management, cart Management, Order Tracking.
• Writing JUnit Test cases using Power Mock and Easy Mock.
• Developed one Page checkout for end user
• Developed background process for discount system for products
• Developed the inventory system for all types of products
• Developed the reports for clients such as Inventory report, different types of Order Report, Customer Report, Discount Products Report
• Writing logic for Importing thousands of client Products to the database
• Managing admin side such as order management, customer relationship management, delivery slot management, Product Management, Inventory Management

Environment: Java, J2EE Technologies, Spring, Hibernate, MySQL, JUnit, Eclipse, SVN, Maven, RESTFUL Web Services, Ajax, JSON, Tomcat

Confidential

Software Engineer

Responsibilities:

• Integrated various 3rd party applications such as eBay to OfBiz framework
• Developing end-user Interface in Ecommerce site
• Good working knowledge of various components within framework such as Accounting components, Order Components
• Working on Creating Account Module GL Account
• Working on Separation of Entity Engine and Application Using RMI
• Developed the reports for clients such as Inventory report, different types of Order Report, Customer Report, Discount Products Report
• Writing JUnit Test cases using Power Mock and Easy Mock
• Working on Accounts payable, Accounts receivable and General Ledger, Expense Management, Payment Managements
• Working on Customer Management, Marketing Management in Customer Relationship Management
• Working on Order Management for NicheSuite ERP

Environment: Java, J2EE Technologies, OfBiz, FTL, Groovy, XML, JQuery, MySQL, SVN, JUnit, Eclipse, RESTful Web Service, Tomcat

Confidential

Software Engineer

Responsibilities:

• Developed UI for inventory management and order Management using JSP 2.0
• Developed the background Process for Inventory Management
• Developed different campaign pattern for market updates to customers
• Prepared the report for inventory, Order and also customer management
• Performed Unit Testing and System Testing

Environment: Java, J2EE Technologies, JSP, Spring, Eclipse, Ajax, Ant, XML, Tomcat, Oracle, SVN, JUnit