SUMMARY

• Results oriented seasoned emerging technology guru and visionary architect specialized in Big Data/cloud/JAVA/SOA/EAI
• Over Twenty years of experience in Information Technology with proven success in Big Data Analytics, Enterprise Application Integration, Service - oriented architecture, Business Process Management, performance improvement, and system implementation.
• Proven project experience in connecting dots to solve challenging complex business problems with ambiguity and deliver awarding winning enterprise solutions to adaptively align IT capabilities with ever changing business demands/priorities
• Unique blend of business and technical skill sets to pragmatically innovate and pioneer cutting edge products and services
• Played key role as architect in developing next generation big data applications using Hadoop ecosystem (CDH, MapReduce, Pig, Hive, Sqoop, Oozie, Spark, Scala, Spark SQL, Impala) in place of existing mainframe based fraud and risk detection application.
• Designed efficient process to ingestTB of Line of Credit (LOC), and Mortgage servicing feeds into HDFS with Parquet format and Compression.
• Developed Pig jobs for data cleaning and pre-processing, created hive tables enabling business users analyze data to discover hidden fraud and risk trends.
• Hands on experience moving large amount of data from RDBMS to Hadoop/Hive and vice-versa.
• Architect and developed Spark data frames in appropriate file format to provision data for stake holders using Scala/Spark SQL
• Hands on experience with Open Source Apache Spark Core and Spark SQL.
• Designed and developed data pipelines built on Apache Spark, and Hadoop that can perform at scale.
• Developing predictive analytic solutions usingApache Spark, Spark SQL/HiveQL.
• Experienced writingSpark programsto load, parse, refine and store sensor data intoHadoopand process aggregate data for visualizations (High charts: Outlier, data distribution, Correlation/comparison).
• Developed Hive jobs that aggregate customer data to create visualizations in Business Intelligence tool Tableau.
• Implemented Hortonworks, Cloudera Hadoop stack.

TECHNICAL SKILLS

• Hadoop
• JAVA
• Tableau
• Pig
• Hibernate
• Business Process
• Spark
• Python
• Oozie
• Web Services
• Spring
• Message Oriented Middleware
• R Language
• Hive
• Service Oriented Architecture
• Business Rules
• Eclipse
• Visualizations/Presentations

PROFESSIONAL EXPERIENCE

Confidential

Lead Developer and Architect

Responsibilities:

• Configured nightly Oozie jobs to load Line of Credit (LOC), and Mortgage servicing text feeds into HDFS.
• Involved in gathering the requirements, designing, development and testing
• Built a platform for data collection and visual ETL with MapReduce parallelize multi source feeding with compression
• Involved in discussions and guiding other region teams on Big data platform, data model and strategy
• Completely involved in the requirement analysis phase.
• Analyzing the requirement to setup a clusters
• Setting up jobs to delete Hadoop logs/local old job files/cluster temp files.
• Developed and scheduled Map-Reduce jobs to clean and convert unstructured LOC text feeds into AVRO format.
• Supported and Monitored Map Reduce Programs running on the cluster
• Migrating the needed data from various sources into HDFS using Sqoop and importing various formats of flat files in to HDFS.
• Created Hive jobs that aggregate customer data to create visualizations in Business Intelligence tool Tableau.
• Designed and developed Pig ETL scripts to process data in a Nightly batch
• Created Pig Macros to improve reusability of code and modularizing the code
• Developed Hive scripts for end user / analyst requirements for ad-hoc analysis
• Very good understanding of Partitions, Bucketing concepts in Hive and designed both Managed and External tables in Hive for optimized performance
• Solved performance issues in Hive and Pig scripts with understanding of Joins, Group and aggregation and how does it translate to MapReduce jobs
• Worked in tuning Hive and Pig scripts to improve performance
• Good experience in writing MapReduce programs in Java on MRv2 / YARN environment
• Good working knowledge of using Sqoop in performing incremental imports from Oracle to HDFS.
• Experience in using Sequence files, AVRO and HAR file formats.
• Good experience in working with compressed files and related formats.
• Developed Oozie workflow for scheduling and orchestrating the ETL process
• Hands on experience withHBaseand its architecture
• Performed data analysis with HBase using Hive external tables to HBase
• Configured HadoopMapReduce, HDFS, Developed multiple MapReduce jobs in java for data cleaning and preprocessing
• Written MapReduce code to process and parsing the data from various sources and storing parsed data into HBase and Hive using HBase-Hive Integration
• Worked on moving all log files generated from various sources to HDFS for further processing
• Developed workflows using custom MapReduce, Pig, Hive and Sqoop
• Developing predictive analytic product for using Apache Spark, SQL/HiveQL, JavaScript, and High Charts.
• Writing Spark programs to load, parse, refined and store sensor data into Hadoop and also process analyzed and aggregate data for visualizations.
• Developing data pipeline programs with Spark Scala APIs, data aggregations with Hive, and formatting data (json) for visualization, and generating. E.g. High charts: Outlier, data distribution, Correlation/comparison
• Creating various views for HBASE tables and also utilizing the performance of Hive on top of HBASE.
• Developed the Apache Storm, Kafka, and HDFS integration project to do a real time data analyses.
• Designed and developed the Apache Storm topologies for Inbound and outbound data for real time ETL to find the latest trends and keywords.
• Developed Map Reduce program for parsing and loading into HDFS information
• Built reusable Hive UDF libraries for business requirements which enabled users to use these UDF's in Hive Querying.
• Written Hive UDF to sort Structure fields and return complex data type
• Responsible for loading data from UNIX file system to HDFS
• Cluster co-ordination services through Zookeeper
• Used Maven extensively for building jar files of MapReduce programs and deployed to Cluster
• Modelled Hive partitions extensively for data separation and faster data processing and followed Pig and Hive best practices for tuning
• Reengineered a mainframe Mortgage database application to an Internet based client/server modern multi user database system
• Researched feasible approaches of direct web access to the distributed relational databases explore Java client technology
• Initiated an applicability study of a pattern driven rapid development approach widgets for feature rich GUI templates
• Developed Consumer Credit loan application automatic process project.
• Developed a J2EE compliant framework using Weblogic 7.0, WebSphere MQ Workflow 3.4.2, DB2 UDP 7.2 FP5 and ILOG Jrule 4.5.5.
• Developed MQ Work Java UPES using Weblogic MDB to handle consumer application requests.
• Coordinated with Business side for Rule requirement, BOM (XML Schema) construction and updating.
• Extended BR Studio functionality to XML binding by Converted XML string to JAXB objects for JRule execution and BOM registration. Used Axis 1.1 Web Services functionality.
• Responsible for application design using FileNet and FN work Place. Mentored team with workflow technologies and best practices using WFMC (Workflow Management Coalition) specifications. Also involved in administering, coding, and support.