Enterprise Data Architecture - Hadoop & Spark Developer Resume
Torrance, CA
SUMMARY:
- Over 12 years of IT Consulting experience that includes Data Engineering, Data Modelling, Data Analysis, Data Analytics & BI, Software Development, Project Delivery and Operations for key enterprise programs in the Retail, Insurance & Automotive industries
- Expert in designing and implementing enterprise data solutions including batch / real - time processing
- Experience in working with large operational databases / large tables / Transactional Data
- Proficient in writing PL/SQL and T-SQL scripts for the database objects like tables, views, cursors, procedures, functions, packages, database triggers, indexes, sequences and flashbacks
- Experience in working with ETL tools like DTS, SSIS, Informatica
- Proficient in performing Data analysis with the RDBMS systems and exposure to Data Analysis in Big Data (Hadoop) environment using Sqoop, Hive, Impala, Spark - pyspark
- Exposure to real time data streaming & processing using Flume, Kafka and Spark Streaming
- Exposure to working with Mongo DB - No SQL database
- Proficient in Performance tuning of ETL processes using Explain Plan, SQL Profiler, AWS Reports
- Experience with generating reports & dashboards using Power BI, Tableau, Salesforce, Hyperion Brio & BO
- Proficient in Marketing Analytics
- Successful management of projects & programs ensuring quality & timely delivery
- Proficient in client & vendor relationship management
TECHNICAL SKILLS:
Statistical Programming: R, Python
Desktop / Web Technologies: VB, ASP, C#.Net, ASP.Net
Data Warehouse/ ETL Processing: PL-SQL, T-SQL, DTS, SSIS, Informatica, SF Data Loader, Shell Scripts
Data Visualization / BI / Reporting: Power BI, Tableau, Hyperion - BRIO, Business Objects
Big Data Framework / Querying Tools: Hadoop, Spark, Hive, Pig, Impala
Data Pipes: Flume, Kafka
NO SQL Data bases: MongoDB
Development Tools: R-Studio, Sublime Text, HUE, Microsoft SQL Server, Toad, Microsoft Visual Studio
Configuration Management Tools: GitHub, Bit Bucket, VSS
Project Management / Agile Tools: Microsoft Project, JIRA, Confluence, SharePoint
CRM: Salesforce
PROFESSIONAL EXPERIENCE:
Confidential, Torrance, CA
Enterprise Data Architecture - Hadoop & Spark Developer
Responsibilities:
- Loaded Structured Data from MySQL Database into HDFS and Hive and Impala Databases using Sqoop
- Loaded data from files (text, CSV, Sequence, Avro, Parquet) in to HDFS
- Performed Data Analysis and Data Transformation on the data stored in HDFS and in Hive Database by creating RDDs and Data Frames in Spark - Python shell
- Performed Data Analysis using Hive QL
- Loaded transformed data into HDFS / Hive Database from RDDs
- Performed real time data analysis on Twitter feed data using Spark - Streaming
- Implemented POCs on Flume real time Data streaming
Environment: Cloudera Distributed Hadoop Cluster, HDFS, Sqoop, Hive, Spark, Python, Spark Streaming, Flume
Confidential, Torrance, CA
Data Analytics & Standards Consultant
Responsibilities:
- Data Analytics
- Design and capture PMO Operations KPIs in SQL Server 2005
- Generate Reports and Dashboards (Operational dashboards, Sourcing dashboards, Project Compliance Dashboards, Project Specific Dashboards) in Power BI by connecting to multiple data sources
- 1. RDBMS (SQL Server / Oracle)
- 2. OData feeds (SharePoint, PWA)
- 3. GitHub
- 4. JIRA
- Evaluated Tableau and Generated POC reports using Tableau Public
- Process Adoption and Standardization
- Meet with Project teams periodically to ensure PMO processes / Standards / Guidelines are adhered
- Tools: Power BI, Tableau, Sql Server 2005, SharePoint 2013, Bit Bucket, Microsoft Project Online, JIRA
Confidential, Louisville, KY
Data Architect - Project Lead
Responsibilities:
- Meet with Business / Product teams and understand the project ask
- Identify the required Data munging and Data transformations
- Provide Data Integration oversight - review and tweak the design & data models using tools like Toad / Erwin
- Conduct Estimation workshops / WBS sessions
- Resource Loading and Project Planning using Microsoft Project
- Budget tracking - Track and report Project Financial data
- Tools: Erwin, Toad, SharePoint 2010, Microsoft Project
Confidential, Framingham, MA
Data Engineer / Data Analyst / Data Architect / Project Lead (Consultant)
Responsibilities:
- Lead the architecture, design and delivery of multiple Data products / solutions in the Customer Loyalty, Customer Hub, Telesales, Campaign Management, Retail Analytics and Point of Sales domain areas
- Designed database models to store huge volume of Sales / transactional data
- Developed Oracle stored procedures, Packages, functions, sequences, indexes, triggers
- Designed and Developed ETL Scripts / packages using PL/SQL scripts, Shell Scripts, Oracle Stored Procedures / Packages, SSIS, DTS and Informatica
- Designed and Developed reports using Hyperion Brio and Business Objects
- Monitored and improved performance of ETL scripts using Explain Plan, SQL Profiler, AWS reports
- Designed and migrated ETL jobs scheduling from Crontab to Tidal
- Gathered Statistics and Analyzed Tables and Indexes for Performance Tuning
- Consumed a SOAP based real time Java web services from PL/SQL using UTL HTTP package for real time data updates
- Proposed process improvements - Automation / Migration / Performance Tuning / Dedupes / Redundant processes
- Did Feasibility analysis for migrating Data from Oracle DB to Mongo DB
- Worked with QA team for Load Testing and Performance Testing
Environment: / Tools: Unix / Linux, Oracle 10g, Sql Server 2008, Mongo DB, Shell Scripting, SSIS, DTS, Informatica, SalesForce Data Loader, Salesforce, VB/ASP, ASP.Net, Hyperion Brio, Business Objects, Crystal reports, JIRA
Confidential
Tech Lead / Senior Software Engineer
Responsibilities:
- Gathered the Business requirements from offshore by interacting with the client
- Estimated the effort by reaching out to the Technical leads and published the project schedule
- Lead the design and development of Oracle PL/SQL stored procedures and packages for customer ranking
- Worked with Software Quality Assurance Group for the code Review and System Testing
- Worked with the Confidential Business users for the user acceptance testing (UAT) and adhered to change management process for the Produce release
- Construction management System - This Project was for Toll Brothers a leading home building company in US. This Project aimed at providing a web based application and a PDA based (hand held) app for the Project managers for tracking all the tasks associated with building a lot
- Assisted requirements gathering from off shore
- Created Design Documents and System Test Plans
- Designed the ASP.Net web application and the .Net 2.0 compact Framework smart client application using Rational Rose
- Developed the proof of concept for all the possible ways of data synchronization from the handheld with the central database using ASP.Net, ADO.Net and C#.Net
- Developed a proof of concept for the smart client application - Data synchronization module using the .Net compact framework 2.0
- Involved in analysis of the requirements and created design documents (High level and Detailed)
- Developed the windows based application using C#.Net and used Web Services to get authenticated to connect to the UK government secure site. Used XML Serialization methods to process the downloaded forms and involved in unit testing & system testing and fixed all the test issues
