SUMMARY:

• Passionate about Information Technology with experience in Big data /Hadoop ecosystem, Apache SPARK, ETL technologies and data engineering. Urge to learn new technologies and implement them to relevant project wherever applicable.
• 7+ years of programming, data transformation, building data pipeline, ETL experience using vast array of technologies and framework in Banking domain.
• Expert in Requirements Gathering, designing, development, implementation and testing of Multi - tired, Distributed Applications and Web Based Applications using Big data and JAVA/J2EE technologies.
• Strong experience in various phases of Software Development Life Cycle (SDLC) as requirement gathering, modeling, analysis, architecture design, development, testing and implementation.
• Strong experience on designing Big data pipelines such as Data Ingestion, Data Processing (Transformations, enrichment and aggregations) and Reporting.
• Strong experience in developing jobs using Apache Spark.
• Extensive knowledge in programming with Resilient Distributed Datasets (RDDs).
• Profound experience in implementing real time data streaming solutions using Spark Streaming, Kafka.
• Strong experience in submitting Spark applications in different clusters such as Spark Standalone and Hadoop Yarn.
• Experience in tuning and improving the performance of spark jobs by exploring various options.
• Strong experience in working with Core Hadoop components like HDFS, Yarn and Map Reduce.
• Experience in launching spark applications by using the Kerberos authentication.
• Strong knowledge in developing Spark application using Java.
• Good understanding and knowledge of NoSQL databases like HBase.
• Big data/Hadoop experience and capable of processing large sets of structured, semi-structured and unstructured data and supporting systems application architecture.
• Able to assess business rules, collaborate with stakeholders and perform source-to-target data mapping, design and review.
• Experience in importing and exporting data from web-services and publishing them to Kafka topics.
• Able to apply appropriate transformation rules using Apache SPARK and storing the desired data to on premises storage, HDFS.
• Proficient in developing Apache HIVE databases and tables and loading data into them using HIVE, Spark HIVE API.
• Capable in applying partitioning/clustering on HIVE tables based on the data reading requirements and frequencies to improve performance.
• Experienced in writing Java and SQL based transformations using Apache Spark.

TECHNICAL SKILLS:

Operating Systems: Windows, UNIX, Linux.

Tools: Apache Hadoop Eco System Components, Apache Spark, Kafka, HIVE, HDFS, Spring Boot, Micro Services, JPA, Maven, Teradata SQL, ETL tools Ab Initio, Informatica, Talend, MS Visio, Eclipse IDE, IntelliJ, Microsoft Office, HP ALM 11.5, Control M

Software / Languages: Java, Scala, Visual Basic for Apps, Teradata SQL, PL SQL, JUNIT and C.

Bug Reporting Tools: HP ALM 11.5, Atlassian JIRA

Functional Knowledge: Banking

Databases: Apache HIVE, Terada, Oracle, DB2, Cassandra, Mongo DB.

PROFESSIONAL EXPERIENCE:

Confidential

Java and Hadoop Developer

Responsibilities:

• Responsible for requirements gathering and technology identification for CTRM-DerivConnect product.
• Provide product design and design analysis for CTRM-DerivConnect NextGen - Big Data and Analysis platform and Microservices to ensure all the functional requirements are addressed.
• Development of software for CTRM-DerivConnect product based on the requirements by following the approved software development standards.
• This includes building the pipelines to pull data from stock exchanges, trading platforms etc to Hadoop/No SQL databases using Flume/Kafka, and cleansing it.
• Completing the analytics using Data Science models to get the predictions on Market. Using Technologies Java, Spark, Hadoop(HDFS, Map Reduce), Hive, HQL, HBase.
• Processing and transforming data to be available in different formats(Structured for SQL like users, Json for REST Apis, semi/un structured for Data scientists and analysts)
• Use DevOps tools like Jenkins, Maven, GIT, Perforce, BitBucket, Nexus for continuous integration and delivery.
• This includes code reviews, passing on the developed features to different phases like Development, Integration, QA, UAT, Pre-prod and PROD environments. And automate these processes for efficient and increase productivity.
• Responsible for documenting the user stories, design documents, architecture and flow diagrams.
• Develop test scripts for Regression tests.
• Bug and vulnerability fixing during the testing phase. Support and Maintenance after the release during the warranty period.
• Participate in daily/weekly scrums, Sprint pre-planning/planning/review/retrospective meetings with Project Managers, Business Analysts, Architects etc.
• Training and onboarding newly recruited people to get into the work environment.

Environment: Spark, Scala, Java, Spring Boot, Bitbucket, Jira

Confidential, Jersey City.

Data Engineer

Roles and Responsibilities:

• Hands on experience with Apache Kafka using the open source as well as confluent’s schema registry platforms.
• Proficient in publishing different formats of data on Kafka Producer and also extracting information from Kafka topics with appropriate consumers
• Experienced with spark structured streaming for data ingestion, analysis and spitting out to data sinks.
• Capable in extracting API exposed data and publishing into Kafka topic for further processing.
• Load and transform large sets of structured, semi structured and unstructured data using Hadoop/Big data concepts.
• Responsible for creating Hive tables, loading data and writing hive queries (HQLs).
• Handled importing data from various data sources, performed transformations using Hive, SPARK, and loaded data into HDFS and AWS S3.
• Good knowledge of Bitbucket, GIT and Jenkins build tools.
• Good knowledge of Hadoop eco system components like HDFS, Sqoop, Flume, Hive.
• Working knowledge of memory-based processing engine Apache SPARK Core and Spark SQL APIs.
• Proficient in language Java, built Micro Services for collecting audit data with Spring boot.
• Worked with file types like AVRO, Sequence Files, Parquet, JSON file formats.

Environment: Spark, Java, GAIA Cloud (Private cloud based on Cloud Foundry), Spring Boot, Bitbucket, Jira

Confidential, Columbus Ohio

ETL Developer/Data Engineer

Roles and Responsibilities:

• Identifying the repetitive processes and creating the common components/Process to accelerate the development effort.
• Developed common components like CAIP READ, CAIP WRITE, CDC TYPE1 AND TYPE2, TDQ.
• Hands on experience with Ab Initio, Unix shell scripting, Teradata
• Built Big Data common components for various business use cases to handle huge data for Talend in spark.
• Worked on Read component and Write component for reading and writing files on Hadoop file system in Talend Spark.
• Worked on Pre-processor component to perform data check based on the business requirements in Talent Spark.

Environment: Ab Initio, Hive, Talend, Spark, Java, Shell Scripting (KSH), Control M, Jira, Oracle, Jenkins.

Confidential, Hyderabad

Data Engineer & ETL Developer

Roles and Responsibilities:

• Involved in requirement gathering and performed ETL Process.
• Understand the ETL mappings provided in the business and requirement documents.
• Build ETL design document specifying the traceability of the business rules to the component wise.
• Create the mappings, transformations and Workflows as per requirement of business to implement the logic.
• Developed ETL scripts (SSIS jobs) to extract data from flat files (with different types of delimiters), XML files and RDBMS tables.
• Developed ETL scripts to apply complex transformations to the extracted data and load them to desired destinations (RDBMS or flat files).
• Validated the complex transformations and flow of the data between various input and output sources and destinations.
• Performed SQL validation to verify the data extracts and record counts in the database tables by writing complex SQL minus/except queries.
• Identified the requirements and arranged peer reviews, walkthroughs and sign off meetings.
• Tested the ETL mappings from Source to Target.

Environment: Apache Spark/ Ab Initio, Teradata SQL/Stored Procedures/Unix, Java for Automation.

Confidential, Hyderabad

Data Engineer & ETL Developer

Roles and Responsibilities:

• ETL mappings from Source to Target.
• Loading the data into ESSBASE Multi dimension Cubes.
• Used Control M for scheduling the jobs
• Logging and tracking the defects in JIRA bug tracking tool.

Environment: Ab Initio, Teradata SQL/Stored Procedures/Unix, Java for Automation