SUMMARY

• Around 8+ Years of extensive IT experience in Analysis, Database Design, Development, Implementation, Debugging and Testing of ETL Solutions with Big Data, Talend, Abinitio for Banking domain.
• Proficient in data Extraction, Transforming and Loading (ETL) using BI tools.
• Extracting data from one or many data systems, applying a series of rules or functions on the extracted data to derive it in the end target and finally loading the data in the end target or destination.
• 4+ Years of expertise in Hadoop Eco system which includes HDFS, MapReduce, Hive, PIG, Sqoop, Java, Scala Spark.
• 4+ Years of ETL development experience in Talend 6.2 Real Time Bigdata Enterprise Edition
• 3+ Years exclusive ETL development experience in populating data marts and generating reports with Ab Initio (GDE, Co>operating System).
• Designed and developed many Real time application in Talend with Spark and Kafka.
• Created a Data Ingestion Engine for business analysts to import historic data from RDBMS to Hadoop in configurable iterations.
• Expertise with SCALA for the batch processing and spark streaming data.
• Experience in developed the Pig UDF's and Hive UDF's to pre - process the data for analysis.
• Developed analytical components using Scala, Spark, Spark Stream.
• Proficient in developing real time pipelines using kafka connect,kafka stream,streamsets and other real time processing components.
• Developed heavy load Spark Batch processing on top of Hadoop for massive parallel computing.
• Scheduling the Hue jobs via Oozie workflows for BAU purpose.
• Developed frameworks to extract and load data to AWS and process the data in the cloud.
• Designed and developed Report Generator in talend which extract data from different sources and generate reports on top of it and share them in PDF or Excel via mail or FTP to stake holders.
• Experienced in applying business transformations on Terabyte size of datasets in Hadoop .
• Implemented encryption and decryption of sensitive business data in Hadoop to restrict the protected data visibility to specific access level.
• Worked in the Data Stax Cassandra for lightning speed writing and reading for online gaming payment transactions coupled with Solr and built behaviour analysis on top of it with Spark which helps to determine user behaviour.
• Experienced in MongoDB for geospatial projects to acquire the near real stations
• Experience in reading multiple types of data sources and flat files like XML, EBCDIC, ASCII, JSON, Spreadsheets etc.
• Experienced in developing advanced analytics dashboards, datasheets using Tableau desktop. Connecting one or more data sources to a report to generate data insights.
• Expertise in gathering of information pertaining to data requirements and transformations in order to initialize a solid implementable and repeatable process.
• Experienced in working in Agile/Scrum, Waterfall, Iterative and software development life cycle (SDLC) methodology environments.
• Experienced in creating Views, stored procedures and optimized queries in SQL SERVER, MySQL 5.0, Teradata 12, DB2 .
• Expertise in applying complex business rules in Unix environment with Advanced Shell scripting
• Captivated proper error handing techniques like logging, checkpoints, and event handler to handle errors in the flow.
• Good understanding of dimensional modelling, Relational Database management systems
• Experienced in handling jobs using Control-M and Cron scheduler

TECHNICAL SKILLS

BI/Reporting Tools: Tableau 9

RDBMS: Oracle 8i/9i/10g/11g, SQL Server 2000/2005, DB2, Teradata 12

NoSQL and Hadoop: Cassandra, Hive, Mongo DB and CDH 5.7,Spark 1.6 (HDFS,Hadoop,Hive,Sqoop,Pig,Oozie)

Database Tools: SQL Developer, TOAD

Operating Systems: Windows, UNIX

Programming: Java, PL/SQL, SQL,Scala

Data ware housing: Talend 6.2, Ab Initio 3.0

Tools: Control M, Autosys

Real Time Ingestion Tools: Stream Sets, Attunity, Kafka Connect

PROFESSIONAL EXPERIENCE

Confidential, Pleasanton, CA

Senior Big Data Developer/Spark/ETL

Responsibilities:

• Worked with business stakeholders, application developers, production teams and across functional units to identify business needs and discuss solution options.
• Develop code that matches the prototype and specification as necessary, portable to other environments.
• Creating and updating the data lake for the delta and full refresh data from the stores every 4 hours
• Setting up the environment in the Azure to host the data in cloud.
• Created packages and built generic ETL jobs.
• Designed and Developed Real time Data Ingestion frameworks to fetch data from Kafka to Hadoop.
• Optimized existing long running high data load jobs to perform better with more production loads.

Technologies Used: Talend, HDFS, Spark core API’s (Sparksql, Streaming), Teradata, Linux, HBase, Azure.

Confidential, Warren, NJ

Senior Big Data Developer/Spark/ETL

Responsibilities:

• Real time data processing with CDC replication using Attunity and Streamsets tools.
• Designed frameworks to connect AWS instances to store and access data.
• Created native Spark frameworks to solve business use cases which were taking longer time in map reduce process.
• Developed Scala applications for spark streaming and SPARK for ongoing transactions of customers.
• Developed spark based Hive Ad-Hoc queries filters the data to increase the efficiency of the process execution by using functions like joins, group by and so on.
• Used Spark RDD for faster Data sharing.
• Developed streaming jobs to ingest large amount of data in Hadoop.
• Developing Hue/Oozie Work flows for the BAU and adhoc process.
• Created Data Profiling framework to monitor the data quality and create a report with the detailed analysis with custom functions.
• Working on Dockers and containers on the Cloud
• Active member in the solution engineering team to provide solutions to production issues of the Talend Frameworks.
• Created Business-Crucial packages and functions to support efficient data storage and manipulation.
• Created Spark based Talend Bigdata Integration jobs to do lighting speed analytics over the spark cluster
• Worked on performance tuning of existing jobs in Talend.

Technologies Used: Talend, Hadoop, Hive, Eclipse, Scala, Java, BT, Spark, Hue, Kafka, Streamsets, Maven, MongoDb, Spark, Kafka, Attunity, Kafka Connect, AWS, Java, Unix scripting

Confidential, CA

Senior Big Data Talend Developer

Responsibilities:

• Lead a team of ETL Developers to plan, design, and implement Data Warehousing for the end client
• Designed many scratch ETL development which involves components in Hadoop (HDFS, MapReduce, Hive, Flume, Sqoop and HUE)
• Created many complex business mappings with Talend BI tool with multiple sources
• Worked on User behaviour analysis to improve the business via big data system
• Worked on securing the Hadoop data system via encryption and decryption of sensitive data.
• Created many Talend applications which applies business rules on RDBS and migrate data to hadoop.
• Worked on reading multiple data formats on HDFS using Scala.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs and Scala.
• Developed multiple applications in Scala and deployed on the Yarn cluster
• Data analysis through Pig, Map Reduce, Hive.
• Analyzed large data sets by running Hive queries and Pig scripts.
• Worked with the Data Science team to gather requirements for various data mining projects
• Worked on performance tuning of existing jobs in Talend.
• Worked on Spark and Cassandra for the User behaviour analysis and lightning speed execution
• Worked on PIG Scripts to process the large set of records and created UDFs
• Created process logging and new monitoring tools, integrity reports and mapping tools
• Used Talend efficiently to manipulate context variables in various level (Wrapper, Job,Subjob)
• Extensive experience in database using SQL SERVER, MYSQL, Hive queries in both source and target.
• Called Unix scripts and Unix files inside Talend jobs to execute in different environments
• Designed many data warehouses to store all information from OLTP to Staging and Staging to Enterprise Data warehouse to do better analysis.
• Designed and developed monthly, quarterly, yearly reports and send them via Mail/FTP in PDF or Excel.
• Rigorously tested and debugged the Stored Procedures to test the validity of the data after the insert, update or delete.
• Extensive knowledge on Cloudera Manager and other monitoring tools.
• Contributed to help Quality Analysts understand the design and development of the ETL logic.
• Extensive usage of Transform component to match data from multiple tables to retrieve data.
• Produced Unit Test results to validate the mapping development and data consistency.
• Worked with SQL Developer to access database and do the analysis on data validity.
• Worked on JIRA to track and fix bugs in the code by analyzing the logs in Talend and Unix scripts.
• Worked on Groovy to generate reports in Business Format.

Technologies Used: Hadoop, Hive, Eclipse, Scala, Java, SBT, Spark, Hue, Kafka, Talend, MongoDb, Cassandra, Spark, DB2, Oracle, Java, Unix scripting, Tableau and Groovy

Confidential, MO

Talend ETL Developer

Responsibilities:

• Collected and analysed the user requirements and the existing application and designed logical and physical data models for ING.
• Designed the Data Mart Structure and build the Dimension Tables and Fact Tables and relative keys
• Worked with many mappings and jobs using Abinito ETL tool to design logic and conversions
• Extracted data from various sources like RDBMS, .CSV, Excel and Text file sources, alongside from various other servers, creating a centralized data warehouse structure.
• Scheduled ETL Tasks to re-process the cubes after updating the data mart
• Worked on migrations and development of existing solutions to Hadoop
• Responsible for allpre-ETLtasks upon which the Data Warehouse depends, including managing and collection of various existing data sources
• Processing massive data in Talend ETL before passing to reporting area.
• Analysed data anomalies to provided solution at right time
• Created packages and built generic ETL jobs.
• Responsible forPerformance-tuningof Talend jobs
• Created Unix Shell scripts (wrapper scripts) for CTRL-M scheduling
• Worked onMulti file systemswith extensive parallel processing
• Extensively worked onTeradata and database components
• Used Ab Initio EME (Enterprise Meta>Environment)to promote graphs form Development to test (QA) and then to the production environment
• Prepared HLD and Technical design documents for multiple projects
• Worked under Agile Practice in 3 weeks deliverables
• Conducted design approach meetings with team
• Involved in Teradata upgrade plan meetings and co-ordinated with all the teams for the data catch up
• DesignedTest Plansand wrote Test Cases to ensure all the functional requirements are met
• Responsible for testingthe graph(Unit testing)for Data validationsand preparing the test reports
• Documentedmapping documentand unit test result documents
• Reviewed test cases prepared by QA team and involved in system and integration testing
• Prepared CTRL-M visio and spreadsheet and raised CO’s to schedule jobs in CTRL-M
• Provided KT to support teams for new applications migrated to the production

Technologies Used: Talend, Abinitio, AbInitio EME, Hadoop, Hive, Java, Unix scripting, Teradata, Ctrl-M.

Confidential

ETL Developer

Responsibilities:

• Designed and developed ETL packages to import data from various sources such as Excel, Flat Files, ODBC and OLE DB sources.
• Created detailed level design for developing Talend Packages and organized review meetings with business for approval
• Created Stored Procedures, Triggers, Functions, client specific Views for the Application.
• Optimized SQL queries for efficiency and provided queries and procedures for the front-end application.
• Created ad-hoc queries and reports as requested and provide on-going analytical support for these requests.
• Converted existing reporting tools to a standardized reporting infrastructure using SQL and BI products.
• Generated reports with dynamic grouping and dynamic sorting as well as running values.
• Worked on complex data loading (implemented the batch data cleansing and data loading).
• Creation/ Maintenance of Indexes for fast and efficient reporting process.
• Configured Server for sending automatic mails to the respective people when a process failure or success.
• Involved in development and design of OLAP cubes, fact tables, dimensional tables
• Created HLD and DLD for new development and enhancement projects
• Worked on performance improvement projects and tuned Ab initio graphs to increase the performance
• Created test cases and performed unit testing for the Ab Initio graphs.
• Reviewed the code and maintained the quality of deliverables
• Handled production support team, resolved data and code issues on time
• Created ad hoc SQL queries and Ab initio graphs to do the data analysis for Business team
• Actively involved in all Business meetings and provided solutions whenever required

Technologies Used: Abinitio, Ab Initio EME, Unix scripting, MySQL, Oracle, Ctrl-M

Confidential

ETL Developer

Responsibilities:

• Analyzed Functional Requirements Interacted with End user and Source Programmers for business understandings
• Developed ETL applications to replicate the SQL,PL/SQL code and implemented in production
• Created new tables, written stored procedures for Application Developers and some user defined functions.
• Worked with several Control Flow Tasks, Data Flow Tasks and Transformations to achieve business requirements
• Designed many packages which involve components in ETL tools to achieve the desired result.
• Rigorously tested and debugged the Stored Procedures to test the validity of the data after the insert, update or delete
• Designed and developed monthly, quarterly and yearly reports and send them via Mail/FTP in PDF or Excel
• Created mapping documents based on the requirements from Business
• Designed and developed several reports using the data from complex Loads and heterogeneous sources
• Generated adhoc reports for immediate management consumption.
• Identified, tested and resolved database performance issues (monitoring and tuning) to ensure data optimization.
• Prepared detailed design document for the developers to develop the adhoc reports
• Worked on Multi file systems to implement the data parallelism
• Developed Generic graphs for the error validations
• Used EME for Version Control
• Performed unit testing, system testing and followed all the deployment process
• Worked in Production support.

Technologies Used: Abinitio, Unix scripting, MySQL, Oracle, DB2, Ctrl-M