SUMMARY

• 16 Years of extensive IT experience in Business System Analysis, Design, Development, Testing and Implementation of software applications which includes 6 plus years of experience in Azure Cloud and Bigdata Technologies. Roles performed - Technical Lead, Sr Developer, Business System Analyst , Application lead, Junior Bigdata Architect
• Analyzing a full range of business concerns, needs, and requirements to identify and recommended processes, products, and solutions to meet goals and objectives.
• Good experience in Azure Cloud services such as Data Factory (ADF), Databricks, Azure datalake, Logic apps, Event hub and Devops.
• Good experience in working with different Hadoop ecosystem components such as HDFS, Hive, Sqoop, Scala, Spark, Yarn, Kafka, Zookeeper, HIVE, MapReduce, HBase, Oozie, Scala and Flume.
• Very good knowledge on usage of various big data ingestion techniques using Sqoop, Flume, Kafka, Native HDFS java API.
• Experienced in handling large datasets using Partitions, PySpark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins Transformations using Python/ Scala.
• Design and Development of Scala code/PySpark Jobs for Data Ingestion into Hadoop Telecom Datawarehouse
• Good experience in developing workflows using Streamsets and Azure Data Factory.
• Extensive experience in NoSQL and SQL development.
• Experience in writing shell scripts to dump the Sharded data from MySQL servers to HDFS Oozie.
• Performed Importing and exporting data into HDFS and Hive using Sqoop and Kafka.
• Responsible for complete SDLC management using different methodologies like Agile Incremental, Waterfall, etc.
• Experience in configuring the Zookeeper to coordinate the servers in clusters and to maintain the data consistency.
• Experience in designing both time driven and data driven automated workflows using Oozie.
• Excellent written and verbal communication skills in fast-paced, demanding and team-oriented settings, independent and self-motivating.
• Good knowledge in Software development life cycle. Provided end- to-end system solutions for various business functions in Insurance and Banking domain projects.
• Good experience and understanding of Enterprise Data warehouse (EDW) architecture and possess End to End knowledge of EDW functioning
• Experience in writing Test Plans, Test Strategies and Test Cases for various Bigdata projects implementation.
• Evaluated and implemented emerging technologies and providing Training to team members .

TECHNICAL SKILLS

• Azure Datalake, Databricks, Azure Data Factory, Eventhub, Streamsets, Deveops, Hadoop, MapReduce, HDFS, HBase, Zookeeper, Hive, Sqoop, Oozie, Flume, Kafka, Octoparse, Beautiful Soup, Linux, Ranger, Knox, Ambari, Hue, Spark,
• C, C++, JAVA.
• MR Unit, SOAP UI, JIRA, Remedy.
• ANT, TOAD, Rally, SPLUNK, Maven, Github, SVN, Jenkins, Nexus. WinSCP, Putty.
• Oracle 11g/10g, DB2, MySQL, MS Access.
• Apache Tomcat, Web Sphere
• Windows 10, UNIX, Linux
• Agile and Waterfall.
• Power Exchange, DB2, Teradata V2R6, Oracle 9i, Flat Files, XML/XSL files,
• Autosys, Cognos 8.2, SQL, PL/SQL, Windows XP, UNIX, Erwin
• PL/1, COBOL, JCL, CICS, SQL, AS, MARK IV, VBA, Easytrieve, DTCN, ENDEVOR, CHANGEMAN, QMF, XPEDITOR, XICS, FILE-AID, QUICK EDIT, Telon, CMR, MoveIT, MQ Series, INFORMATICA 7.0.

PROFESSIONAL EXPERIENCE

Confidential

Sr Big data Consultant

Responsibilities:

• Provided Azure technical expertise including strategic design and architectural mentorship, assessments, POCs, etc., in support of the Revenue Management Legod Project.
• Maintained few data products for PAS smooth.
• Migrated Revenue Management system from Teradata to Azure Databricks and ADLS using PySpark.
• Ingestion of the Data from different source by creating Data Factory pipelines and Logic Apps WorkFlows.
• Job tracking and Monitoring of Revenue Management platform and handling production issues and hot fixes for PSS.
• Handle and converted XML data to Parquet using XSD.
• Extracted streaming data from Eventhub and processed with minimal latency using PySpark.
• Automated reports to compare the data from Teradata and ADLS for Legod program.
• Supported production E2E process, data validation and bug fixings for PSS project.
• Converted and handled data from JSON to parquet by providing dynamic schema.
• Created workflows using streamsets for ingestion, alerts, data migration.
• Running unix batch scripts, Cron jobs, ADLS data migration and triggering logicapps using streamsets.

Confidential

Hadoop Consultant

Responsibilities:

• Participated in the initiation and planning of projects - including facilitating the definition of objectives, scope, and deliverable. Led the project on monthly, quarterly and annual reporting implementation in capital markets using Agile methodology. Analyzed, captured and documented requirements from business stockholders
• Designed and developed big data analytic models to read huge files from remote server and process using spark and Scala.
• Conduct data reconciliation and data mining for wholesale portfolio. Facilitate and participate in problem-solving discussions with business and IT teams to help to find solution to existing complex data issues. Identified data discrepancies for more than 750M
• Analyse existing Data Ingestion Design Patterns and provide inputs on Performance improvement to Application Teams
• Working with Data Scientists to understand data requirements and guide developers to work with data providers to create data pipelines for analytics
• Worked on tuning the performance of spark SQL and hive queries.
• Prepared technical design documents, detailed design documents.
• Importing and exporting the data using Sqoop from HDFS to Relational Database systems and vice-versa.
• Create issues in JIRA in order to prioritize, delegate, and track data discrepancy issues
• Participate in creating and validating business requirements, business process modelling, data mapping. Develop test plans, perform User/Business acceptance testing (UAT/BAT)
• Automated the workflow using shell scripts
• Installed Oozie workflow engine to run multiple Hive and spark jobs.
• Played role in project estimation and management activities.
• Conducted and participated in walkthroughs and reviews to validate quality and obtain approval for systems and business requirements.

Confidential

Hadoop Consultant/ Architect

Responsibilities:

• Developed big data analytic models for customer fraud transaction pattern detection models using Hive from customer transaction data. It also involved transaction sequence analysis with gaps and no gaps, network analysis between common customers for the top fraud patterns.
• Developed customer transaction event path tree extraction model using Hive from customer transaction data.
• Developed the spark scripts by using spark-scale shell commands
• Developed Scala scripts, UDFFs using both Data frames/SQL and RDD/MapReduce in Spark 1.6 for Data Aggregation, queries and writing data back into OLTP system through Sqoop.
• Worked on migrating Map Reduce programs into Spark transformations using Spark and Scala.
• Worked on tuning the performance of spark SQL and hive queries.
• Prepared technical design documents, detailed design documents.
• Process large data sets utilizing our Hadoop cluster.
• Designing NoSQL schemas in Hbase.
• Extensive data validation using HIVE.
• Importing and exporting the data using Sqoop from HDFS to Relational Database systems and vice-versa.
• Automated the workflow using shell scripts
• Installed Oozie workflow engine to run multiple Hive and Pig jobs.
• Assigning the work to offshore team and ensuring the work is delivered on time.
• Co-ordinating with other programmers in the team to ensure that all the modules complement each other well.
• Played role in project estimation and management activities.