Big Data Bi Engineer Resume
Charlotte, NC
SUMMARY
- Over 15 + years of experience in IT working with Banking, Financial and Pharmacy Clients.
- 5+ years of experience working on Big Data Analytics Technologies and successfully implementing Big data business intelligence and Data Warehouse initiatives.
- Expert experience in developing Tableau dashboards with Hadoop Hive, Impala, Redshift.
- Has Expert experience in understanding Business Requirements and implementing the requirements through well design and development process.
- 3 + years’ experience in end to end design of a big data lake with reports/dashboards in Tableau for Confidential Virtual Assistant ERICA (It is similar to Alexa, Siri, Google Assistant).
- Has 5 + years in Tableau development and self service tableau server maintenance .
- Has expertise in migrating legacy mainframe Application to newer product migrations. I have successfully helped team to migrate to FICO DM9 application from mainframe.
- Has expert knowledge in successfully migrating from one bank source application to another.I have successfully worked in Confidential & T merge acquisions of Susquehanna and Penn Bank and handled migrating RMS application from source to target bank.
- Has Experience in Latest Tableau versions 2019.x and 2020.x.
- Proficiency in importing and exporting data using Sqoop, ETL Tools from Relational database systems to HDFS and vice versa.
- Expert in designing hive database system.
- Hands - on experience in Unix/Python scripting.
- Has experience in bringing data from datalake using Spark SQL .
- Proficiency in BigData tools Hadoop, Spark,Sqoop and hive related technologies.
- Strong knowledge of Hadoop Architecture and Daemons such asHDFS,JOB Tracker,Task Tracker,Name Node,Data NodeandMap Reduceconcepts.
- Good experience in writing Spark applications using Python.
- Worked on Apache Spark by analyzing text analytics needed for predictive modeling.
- Has the ability to understand and design an end to end big data analytics requirement by using data from a traditional data warehouse and from a Hadoop big data system.
- Big Data Analytics experience using Microsoft Business Intelligence suite, Hadoop Cloudera and Tableau.
- Strong Data Warehousing experience in designing data marts needed for the business requirements.
- Ability to write complex SQLs needed for ETL jobs and analyzing data, and is proficient and worked with databases like SQL Server, DB2, Excel sheets, Flat Files and XML files.
- Worked on AWS S3 for pulling data from S3 and loading to AWS Redshift.
- Have excellent understanding of Scala language and its implementation with Spark.
TECHNICAL SKILLS
Big Data/Hadoop Stack: Hadoop, Hive, Sqoop, Flume,Apache Spark
Database: SQL Server, Oracle, Teradata
Programming Languages: Python,Scala, Unix/Linux Shell Scripting
ETL Tools: SSIS
Cloud: AWS
Operating Systems: Windows, UNIX/Linux, UBUNTU, Mac OS 10x
Hadoop Distributions: Hortonworks HDP, Cloudera
Visualization: Tableau 10.2, Tableau 2019.2.4
PROFESSIONAL EXPERIENCE
Confidential, Charlotte, NC
Big Data BI Engineer
Responsibilities:
- Developing the new reporting tables in hive /impala based on the business requirements for Chatbot Conversation reporting.
- Lead team to design tableau dashboards as per Business User requirements.
- Redesigned the existing Insights data tables to include new insights. This involved designing of new tables and creating a daily load process similar to a data ware house in hive.
- Involved in creating CONTACT US Deflection Reporting to evaluate the usage of ERICA in directly handling Contact US Center Calls.
- Handled end to end development of ERICA Weekly Update dashboard involving different metrics.
- Created Executive dashboards with bar charts, line graphs, dual axis charts.
- Has expert knowledge in implementing parameters,filters, hierarchies,sets.
- Worked on migrating all the reports from tableau 10.2 server to new tableau 2019.2.4 server without impacting business.
- Designing the new Big Data Analytics system with existing OLTP database and generating Visualizations with Tableau needed by business team.
- Creating Ad Hoc reports from big data tables needed for the business users.
- Analyzing the OLTP Teradata warehouse databases to get the needed data for business requirements.
- Improving the performance of existing tableau reports that are slow.
- Designing and developing the databases and reports for various functionalities of ERICA Virtual Assistant features like Insights reporting, Help Suggestions reporting, Intent Reporting.
- Handled end to end UAT testing of Hadoop files from ICH platform to Hive tables in UAT environment for all the projects.
- Handled end to end table loading of intent mapping loads.
- Designed and implemented a typical daily flow of data in hive tables for Insight reporting.
- Analysed various existing hive tables and re designed by partitioning those tables for better efficiency.
- Created daily Hive load jobs for daily reporting ETL.
- Daily Handled entire support of hive ETL reporting jobs working with various prod support team for any issues.
- Wrote Hive/Spark queries for LOB to provide a consolidated view of data for analysis and adhoc querying.
Environment: Cloudera, Tableau, Flat files, UNIX scripting, Python, Teradata, Hive, Impala, pyspark.
Confidential, Charlotte, NC
Python Data Analyst
Responsibilities:
- Fetching the data available in the data source location for processing through Python.
- Developing the data processing and parsing rules required for each data source.
- Developing a process in python to make the parsed file ready for the database loading.
- Creating the database tables needed for each data source.
- Loading the clean data file into oracle database using python.
- Validating the load process and making sure that source data is loaded without any missing data.
- Interacting with the NLP team on the data availability that is needed for the data models and ensuring that data is ready for further use.
Environment: Anaconda Python, Flat files, Oracle database.
Confidential, Charlotte, NC
ETL Big Data BI Engineer
Responsibilities:
- Developing the new reporting tables in hive /impala based on the business requirements for Chatbot Conversation reporting.
- Developing the new reports/dashboards in Tableau needed by business users.
- Designing the new Big Data Analytics system with existing OLTP database and generating Visualizations with Tableau needed by business team.
- Creating Ad Hoc reports from big data tables needed for the business users.
- Analyzing the OLTP Teradata warehouse databases to get the needed data for business requirements.
- Improving the performance of existing tableau reports that are slow.
- Performing text analytics to understand the occurrence of Intents in the conversation history data with machine learning.
- Designing and developing the databases and reports for various functionalities of ERICA Virtual Assistant features like Insights reporting, Help Suggestions reporting, Intent Reporting.
Environment: Cloudera, Tableau, Flat files, UNIX scripting, Python, Teradata, Hive, Impala,pyspark.
Confidential, Wilson, NC
Data Analytics Consultant
Responsibilities:
- Designing the new ETL architecture as part of DM9 application Migration.
- Designing the new reporting database to generate Tableau dashboards needed by business users.
- Designing the new Big Data Analytics system with existing OLTP database and generating Visualizations with Tableau needed by business team.
- Analysis on the Retail loans, Commercial loans and Mortgage loans applications and their various feeds to the DataMart.
- Multiple mapping documents are implemented for various feeds of Retail Loans, Commercial Loans and Mortgage Loans etc.
- Detailed design of the various Ad Hoc fields that are needed for the DataMart that is not available in the original source systems.
- Actively involved in the transformation process of the source application feeds to destination DataMart.
- Involved in the various test region setups for the ETL environment and code migration activities from one testing region to another.
- Work with Business owners and Business Analysts to understand the Business Requirements
- Work with Technical Lead and Solution Engineer to come up with technical design documents
- Work with Data Architects and SMEs to come up with the data mapping documents and the data transformation
- Developed ETL mappings, transformations using SSIS.
- Helped end users to design test cases and worked through multiple UAT cycles.
Environment: SSIS, Tableau, Flat files, T-SQL, UNIX scripting, Python, Hortonworks HDP
Confidential, Charlotte, NC
Technical Data Analyst
Responsibilities:
- Being part of the initial requirement gathering and project kick off discussions.
- Creating reports with MicroStrategy.
- Created Multidimensional cubes and Tabular models with SSAS.
- Guide and Mentor for Team.
- Interacting with the clients to understand the business and mapping rules of all the interfaces.
- Involved in creating Low level and High level design documents.
- Creating the big data architecture for the existing OLTP database.
- Worked on SQOOP to import data from RDBMS to Hadoop system.
- Worked on hive to extract data from Hadoop and traditional systems.
- Designing ETL framework for establishment of batches.
- Responsible for performance tuning of the ETL process and upgrading the ETL best practices document.
- Developed batch jobs to maintain day to day updates and a process to check if source and target are in sync or not.
- Investigate data to identify errors and validate the contents of fields in a data file.
- Created shell scripts for scheduling Informatica Workflows.
- Working on SQLs to create needed data from datasets.
- Design Autosys scheduling scripts to schedule Informatica Workflows and run then through Autosys scheduler.
- Provided post implementation support.
Environment: Informatica,SQL Server, Cloudera Hadoop.
Confidential
Technical Data Analyst
Responsibilities:
- Preparing of Analysis and design documents.
- Developing the Informatica Workflows and apply data transformations rules using the Informatica Powercenter 8.6 according to requirements and load the data to the File.
- Extensively used Informatica Client Tools like Informatica Designer, Informatica Workflow Manager, Informatica Workflow Monitor to develop Informatica Mappings and Workflows.
- Used Informatica Workflow Monitor to see the logs and monitor the data flow and fix the jobs.
- Involved in logging error messages and fixed the errors.
- Parsing high-level design specifications to simple ETL coding and mapping standards.
- Developed mappings using different types of transformation like Sorter, Filter, Aggregator, Router, Lookup, Update Strategy, Joiner and Source Qualifier.
- Developed, executed and documented Unit testing, System Integration testing of workflows.
- Extensively used Normalizer (for COBOL source files), Router, Lookup, Aggregator, Expression and Update Strategy Transformations.
- Translation of Business processes into Informatica mappings for building Data marts.
- Created and Documented ETL Test Plans, Test Cases, Expected Results, Assumptions and Validations.
- Extensively worked with the Debugger for handling the data errors in the mapping designer
- Involved in the Migration process from Development, Test and Production Environments.
- Used SQL tools like TOAD to run SQL queries and validate the data.
- Wrote stored procedures and triggers in Oracle 10g for managing consistency and referential integrity across data mart.
- Debugging any failed sessions using breakpoints, testing of stored procedures and functions, testing of Informatica sessions, batches and the target Data.
- Identified bottlenecks in sources, targets, mappings and sessions and tuned them to improve performance.
Environment: Informatica 8.6.0, DB2, Oracle 10g,Mainframe Cobol files, Z/OS.JCL
Confidential
Data Analyst
Responsibilities:
- Created Dimension Tables and Fact Tables based on the warehouse design.
- Involved in creating Logical and Physical design of databases using Erwin.
- Participated in build of the Data Warehouse, which includes the Design of Data mart using Star Schema.
- Created ETL detail design document and ETL standards document.
- Created Repository using Informatica Power Center - Repository Manager.
- Extracted data from flat files and oracle database, applied business logic to load them in the central oracle database.
- Involved in Extraction, Transformation and Loading (ETL) Process.
- Worked on SQL queries and tuned them to improve performance.
- Extensively used Shell scripts to automate the Pre-Session and Post-Sessions processes.
- Performed data manipulation using basic functions and Informatica Transformations.
- Designed & developed reports, Drilldown reports using Microstrategy Reporting module, designed reports with Slice and Dice & Drill down operations.
Environment: IBM DB2, XML files, Flat Files,SQL Server.
Confidential
Business Analyst
Responsibilities:
- Working to automate repetitive tasks and processes in order to maximize performance.
- Collating business requirements, then translating these into functional specifications and detailed test plans.
- Providing information to operations management teams to enable them to look at different angles.
- Maintaining system protocols by writing and updating procedures.
- Participating in meetings, workshops, and technical meetings.
- Establishing and maintaining communication channels with senior managers, key team members and stakeholders.
- Write up and maintain complete and accurate business process documentation.
- Conducting research into project-related issues and products.
- Document business requirements and also define its scope and objectives.
Confidential
Business Analyst
Responsibilities:
- Understanding of the business requirements and enhancing the existing data warehouse architecture design for a better performance.
- Worked with Informatica Designer to create, Load, Update mappings using different transformations to move data to different data marts in Data Warehouse.
- Identified and tracked the slowly changing dimensions, heterogeneous Sources and determined the hierarchies in dimensions.
- Created users and user groups with appropriate privileges and permissions, folders and folder permissions in Repository manager.
