PROFESSIONAL PROFILE SUMMARY:

• Result Oriented Professional building on 14+ years of progressive experience in Software Development includes application architect, administration, design and development along with 4+ years in Big data/ Hadoop ecosystem such as Hive, Impala, Pig, Flume, Oozie, Sqoop, Zookeeper, Hbase, Spark, Kafka, MapReduce and Solr 7.3 in distributed platform.
• Understanding to identify the viability of a business problem for a big data solution. Defining a logical architecture of the layers and components of a big data solution like data capacity planning and node forecasting. Selecting the right products to implement a big data solution.
• Experience in developing POC on various aspects of Hadoop work apart from creating strategic vision.
• Familiar with data architecture including data ingestion pipeline design, Hadoop information architecture, data modeling and data mining, machine learning and advanced data processing. Experience optimizing ETL workflows.
• Working knowledge as Software Developer in Java/J2EE and related technologies such as JSP, JDBC, JavaBeans
• Designed and implemented Stream processing pipeline workflow for user which update user s data nearly real - time
• Experience in Data Analysis, Data Cleansing, Data Validation and Verification, Data Conversion, Data Migrations and Data Mining
• Working knowledge on Sentry & Kerberos for security based on LDAP authentication and authorization.
• Experience in design and development applications using Microsoft .Net technologies & SQL Server
• Experience in utilizing SQL integration Services (SSIS), SQL Reporting services (SSRS), SQL Management studio and SQL tools
• Hands-on experience on Project Management & Leading activities like estimation, planning, scope definition, resource administration, process compliance.

TECHNOLOGY/PLATFORM/TOOLS:

Big Data Platform: Hortonworks (HDP 2.2)/ BigInsight 4.2/Cloudera (5.7) / AWS (S3, EMR, EC2)

Cloud Infrastructure: MS Azure, Amazon Web Services (AWS)

Apache Hadoop Yarn 2.0: HDFS, HBase, Pig, Hive, Sqoop, Kafka, Zookeeper, Oozie

Real Time Data Streaming: Apex, Malhar, Spark (Java/Scala), Kafka

Source Control: GitHub, VSS, TFS

Databases and NoSQL: MSSQL Server 2014, Oracle 11g (PL/SQL), IBM BigSQL and MySQL 5.6, MongoDB

Data Visualization Tools: Tableau and DataTorrent

Development Methodologies: Agile/Scrum and Waterfall

Development Tool: Eclipse, Scala IDE, Visual Studio, SQuirreL, Toad

Programming Languages: Java, .Net ASP.Net, C#, Python, Scala

Scripting Languages: JavaScript, JSP, XML, Python and Bash/Shell Scripting

STRENGTHS:

• Hands on design and development experiences on various Hadoop projects and technologies
• Collaborates well across technology groups
• Requirement Analysis, PoCs and Documentation such as HLD, LLD, Test plan and Test cases
• Excellent oral and written communication skills
• Project Management, Project Leadership & Team building
• Customer and Vendor relationship management
• Offshore - Onshore team handling and coordination
• Experience on SEI-CMM Level 5 and ISO 9001 certified processes and framework
• Experience on Microsoft technologies such as .Net, WCF, WPF

PROFESSIONAL EXPERIENCE:

Hadoop Architect /Big Data Consultant

Confidential, Pleasanton, CA

Responsibilities:

• Design and develop Spark Structured Streaming data pipe - line under Scala project & modular approach
• Continuous streaming to cater oracle json files, processing and output to hive tables through spark data-frames based on customized input schemas
• Dynamic Hive tables creation with parquet file formats from spark
• Handle SparkSession as singleton pattern instance creation to use across the application
• Spark streaming checkpoint implementation to handle and recover any point of job-failure
• Integration of Log4j based on spark/conf log4j template
• Integration of external type-safe application configuration to cater dev/prod specific configurable items
• Written Flume agent to move json files from source to HDFS location
• Design and development on Hadoop distribution Cloudera 5.8 with Hive 0.13
• Design and Architect Health-care financial data pipe-line for processing and business analytics
• Implement change-data capture mechanism in Hive through HUE for cost and space benefit analysis
• Written various shell/bash-scripts to integrate and dynamic calling of hive-QL
• Integrate log-rotation concept with bash-scripts for dynamic logging configuration
• Written oozie workflow solution for calling bash scripts
• Optimize Hive queries for performance gain.
• Written Python scripts to convert the xls2csv, xlsb2csv and csv2csv as customized solutions conversion tool

Hadoop Architect / Spark Consultant

Confidential, Sunnyvale, CA

Responsibilities:

• Design and Architect support for Confidential - maps data pipe-line for data processing and update for multiple locations
• Defining the standards for context passing for different teams to optimize resource allocations on given cluster
• Optimizing the Jenkins jobs based on priorities and order to follow job executions
• Spark and Hive performance optimizations running for different jobs 1400 node cluster.
• Monitoring and optimization cluster performance for Cloudera distribution
• Written python scripts to clean-up the cluster space example Spark history server for history logs
• Support for granting HDFS spaces to different groups based on hdfs quotas
• Written Spark jobs to handle streaming data and written shell/bash-scripts
• Hadoop distribution - Cloudera 5.7 with Spark 2.0, Hive 0.14, Cassandra.

Hadoop Architect /Big Data Spark Consultant

Confidential, Alpharetta, GA

Responsibilities:

• Design and Architect financial domain data - pipeline to handle near real-time data for streaming and batch processing with Spark (1.6) RDD, SparkSQL & Streaming, Kafka (0.10)
• Hands-on experience to configure MS Azure cloud infrastructure with IBM BigInsight 4.2 Hortonworks 2.2 Hadoop distribution with IBM BigSQL (MPP) and SQL-Server as cold and warm storages.
• Fraud-Detection Analytics Work-Flow ETL design for Data Cleansing, Integration & Transformation using RapidMiner Data Analytics & Modeling tool
• Hands-on experience to write Java based Kafka and Spark streaming programs.
• Performance benchmarking and load testing on BigSQL to handle multiple client data loads.
• Shared administration responsibility for Hadoop, Spark and Kafka installation & configuration on MS Azure
• Exported analyzed data to the relational databases using Sqoop for visualization & Report generation for Tableau
• Configuration and setup Zeppelin for quick data validation and testing.
• Predictive Analytics for indicator generations and scoring model computation through RapidMiner

Hadoop Architect /Developer

Confidential, Melno Park, CA

Responsibilities:

• Designed and build data pipeline which stream data from client apps using web - sockets to server and from there Kafka Consumer which consumes that data and write to HDFS data store. From HDFS store different spark jobs are reading this data using Spark-SQL and processing this data in stream and batches jobs
• Experience designing AWS data pipeline and configuring AWS cluster for Spark
• Written trivial Spark programs with Scala to read data using Spark-SQL
• Shared responsibility for administration of Hadoop, Hive and Pig and configuring Spark cluster on AWS
• Managed, reviewed and interpreted Hadoop log files. Involved with the application teams to install Hadoop updates, patches and version upgrades as required
• Data Ingestion: Involved in importing and exporting data (SQL Server, Oracle, csv and text file) from local/external file system and RDBMS to HDFS. Load log data into HDFS using Flume
• Capacity Planning to setup Hadoop Cluster on AWS-EC2 for data
• Setup local cluster with Hortonworks distribution for pilot run
• Developed MapReduce programs to parse the semi structured data, populate staging tables
• Used Sqoop to dump data from relational database into HDFS for processing and exporting data
• Used Pig (Pig Latin scripts) and Hive in the analysis of data
• Worked on Sequence and ORC files, bucketing, partitioning for Hive performance and storage improvement
• Used Oozie scheduler to automate the pipeline workflow and orchestrate the map reduce jobs that Extract the data on a timely manner

Technical Manager/ Architect

Confidential, Reading, PA

Responsibilities:

• Developing parser and loader MR application to retrieve data from HDFS and store to HBase and Hive
• Built ETL workflow to process sales and marketing data on hive tables
• Importing the data from the MySql and Oracle into the HDFS using Sqoop
• Experienced on loading and transforming of large sets of structured and unstructured data
• Developed PIG Latin scripts to extract the data from the web server output files to load into HDFS
• Created Hive Internal and External tables and loaded the data in to tables and query data using HQL
• Used Hue for UI based PIG script execution, Oozie scheduling and creating tables in Hive
• Deployment of applications using AWS EC2
• Written Map Reduce java programs to analyze the log data for large - scale data sets
• Participated in building CDH4 test cluster for implementing Kerberos authentication. Installing Cloudera manager and Hue

Project Lead

Confidential

Responsibilities:

• Application framework designing and development including database designing.
• Designed and developed data pipeline using SQL Server 2010, SSIS, SSAS and related technologies.
• Configuration Management, Code Review, Build Deployment Management and Releases.
• Developed user interfaces using JSP, HTML, XML and JavaScript.
• Used JavaScript for client side validation and created external style sheets using CSS.
• Web application development using J2EE: JSP, Servlets, JDBC, Hibernate, JUnit and Apache Log4J, Web Services, Message Queue (MQ).
• Involved in the preparing the Unit test plan documents
• Support to setup process implementation for ISO 9001, ISMS 27001 process documentations.
• Implemented security in modules as Identity, Authentication, and Authorization

Lead Analyst

Confidential

Responsibilities:

• Application designing and development including database modeling for airline domain product.
• Component design and development for various GDS systems and data handling.
• Export or Import data from other data sources like flat files using Import/Export of DTS
• Requirement Analysis, design, development and implementations.
• Involved in Extraction, Transformation and Loading (ETL process) from Source to target systems using SSIS
• Developed an API to write XML documents from a database. Utilized XML and XSL Transformation for dynamic web - content and database connectivity.
• Used ASP.Net web application in LINQ to SQL for database connectivity.
• Application developed under .Net framework with MVC, JavaScript, XML
• Developed Stored Procedures and Complex Views using Joins for robust and fast retrieval of data in SQL Server.
• Experienced in designing report layouts in SSRS and deployed cubes using SSAS
• Worked with source control tool such as Team Foundation Server (TFS) to manage code and DB changes
• Coordinated with technical team for production deployment of software applications for maintenance

Sr. Software Engineer

Confidential

Responsibilities:

• Application framework designing support and development including database designing.
• Designed solution using N - tier Architecture model and design patterns mainly Abstract Factory and Singleton
• Assisted in creation of ETL processes for data transformation sources from SQL database and Legacy systems
• Created entire project in Subversion and created Ant build script for compiling and building the application
• Created Use case, Sequence diagrams, functional specifications and User Interface diagrams using Star UML
• Implemented Model-View-Control (MVC) software architecture in web applications to view the html.
• Used List, Trees, Toolbars, Menus and Context Menus for navigating between pages in WPF.
• Developed Windows based GUI using WPF, Expression blend & binding using one way to source data binding.
• Configured Windows Communication Foundation (WCF) service to authenticate clients with Windows credentials
• Documentation designs like HLD, LLD and Presentations.

Software Engineer

Confidential

Responsibilities:

• Involved in Analyzing User Requirements, Application Design, Code Development and Deployment
• Designed User Interface and Implemented the Application Logic under Microsoft .NET Framework 1.0 using ASP.NET and VB.NET to use .NET features that powered with Common Language Runtime. Served as Application Management, Build Management and Admin Management modules
• Produced User Controls for Common Header's in the ASP.NET web pages
• Used Microsoft Visio 2003 to create Use Case, ER - Diagrams and Class Diagrams
• Crystal reports on web platform from databases to generate reports based on viewer customization
• Created Web Services used for the application as well as for some other departments to Reusable components
• Module Testing, Writing Test-case s, Implementation and Client Training