SUMMARY:

• Over 8+ Years of IT experience and currently working as Data Engineer for both SQL and Big Data wif the halp of Hadoop Eco System across internal and cloud based platforms.
• Experience wif all stages of the SDLC and Agile Development model right from the requirement gathering to Deployment and production support.
• Good exposure to Python programming.
• In - depth noledge of. Snowflake Database and Table structures.
• Hands - on experience in Azure Cloud Services Azure Synapse Analytics, SQL Azure, Data Factory, Application Insights, Azure Monitoring, Key Vault, Azure Data Lake.
• Excellent experience on AWS cloud services (Amazon Redshift and Data Pipeline).
• Skill experience in installing, configuring and using Apache Hadoop ecosystems such as Map Reduce, Hive, Pig, Sqoop, Spark, Kafka and Oozie.
• Experience in Google Cloud components, Google container builders and GCP client libraries and cloud SDK s
• Experience in Dimensional Data Modeling experience using Data modeling, Relational Data modeling, ER/Studio, Erwin, and Sybase Power Designer.
• Expert in writing SQL queries and optimizing the queries in Oracle, SQL Server and Teradata.
• Working experience in PL/SQL, SQL*Plus, Stored Procedures, Functions & Packages.
• Excellent technical and analytical skills wif clear understanding of design goals and development for OLTP and dimension modeling for OLAP.
• Depth noledge in Data Warehousing concepts wif emphasis on ETL
• Excellent Programming skills at a higher level of abstraction using Scala and Python.
• Good working experience on Spark (spark streaming, spark SQL) andScala.
• Experience in Data transformation, Data Mapping from source to target database schemas, Data Cleansing procedures.
• Data importing and exporting by using Sqoop from HDFS to Relational Database Systems and vice-versa.
• Experience in transferring the data using Informatica tool from AWS to AWSRedshift
• Excellent noledge and extensively using NOSQL databases (HBase).
• Pro

PROFESSIONAL EXPERIENCE:

Confidential

Sr. Data Engineer

Responsibilities:

• Working as a Sr. Data Engineer to extract historical and real - time data by using Hadoop, Map Reduce and HDFS. Worked in SCRUM (Agile) development environment wif tight schedules. Extensively Involved in Big data pipeline development and maintenance. Involved in provisioning, installing, configuring, monitoring, and maintaining HDFS, Sqoop, Spark, Kafka and Hive. Worked wif, Data architect, Data governance, Data quality, data lineage to design various models and processes. Ensured ETL succeeded and loaded data successfully in SnowflakeDB.
• Designed and implemented database solutions in Azure Data Lake, Azure Data Factory, Azure Synapse Analytics Created Talend Mappings to populate the data into dimensions and fact tables Developed a data pipeline using Kafka to store data into HDFS. Involved in import the data from different sources like HDFS into Spark RDD and Loading files to HDFS. Implemented drill down functionality by building hierarchies in Power BI visualizations. Independently coded new programs and design Tables to load and test the program TEMPeffectively for the given POC's using Big Data/Hadoop.
• Developed Python scripts to clean the raw data. Involved in developing PySpark to transform data from one Data Lake to other Data Lake. Designed and developed continuous deployment pipelines (CI/CD) in Azure DevOps. Involved in using Azure Data Factory to create Pipelines to transfer data across systems. Developed various ETL process for complete end to end Data Integration. Integrated visualizations into a Spark application using DataBricks. Developed and designed continuous integration pipeline and integrated using Git and Jenkins. TEMPEffectively worked in Azure Synapse Analytics, Azure Data Lake and Azure Data Bricks. Exported the analyzed data to the relational databases using Sqoop for visualization and to generate reports by our BI team.
• Created modules using Spark streaming in data into Data Lake using Spark. Created a Data Warehouse Model wif de-normalized attributes at all places. Developed PySpark script to merge static and dynamic files and cleanse the data. Installed and configured Hadoop MapReduce and HDFS. Automated various data extraction, transformation, and loading tasks wif Python Worked on catapulting data from teradata to snowflake to consume on DataBricks.
• Involved in designing Azure Data factory/Data Lake solutions. Designed and maintained GIT Repositories, views and access control strategies. Worked on complex SNOW SQL and Python Queries in Snowflake. Configured Input & Output bindings of Azure Function wif Azure Cosmos DB collection to read and write data from the container whenever the function executes. Involved in performance tuning and monitoring of T-SQL blocks. Designed and implemented ETL pipelines between from SnowflakeDB to the Data Warehouse using Apache Airflow. Worked wif Azure Blob Storage usage and handling files. Implemented Data Archiving

Confidential

Sr. Data Engineer

Responsibilities:

• Worked as a Data Engineer designed and deployed scalable, highly available, and fault tolerant systems on Azure. Involved wif all the phases of Software Development Life Cycle (SDLC) methodologies throughout the project life cycle. Designed and implemented TEMPeffective Analytics solutions and models wif Snowflake. Worked wif Azure BLOB and Data lake storage and loading data into Azure SQL Synapse analytics (DW). Created data pipeline of gathering, cleaning and optimizing data using Hive, Spark.
• Extensively developed complete end to end Big - data processing in Hadoop eco system. Worked on reading multiple data formats on HDFS using python. Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Blob storage, Azure SQL Data warehouse, write-back tool and backwards. Created context variables and groups to run Talend jobs against different environments. Created Power BI reports and dashboards on OLAP and relational db. Evaluated and improved application performance wif Spark. Tested the ETL process for both before data validation and after data validation process.
• Worked in setting up the repos to various projects in SVN Analyzed the SQL scripts and designed the solution to implement using PySpark Worked wif NoSQL databases like HBase in creating tables to load large sets of semi structured data coming from source systems. Created post commit and pre-push hooks using Python in GIT repos. Involved wif Azure Data Factory, Azure Data Lake, Azure Databricks, Azure Data Warehouses. Data sources are extracted, transformed, and loaded to generate CSV data files wif Python programming.
• Performed File system management and monitoring on Hadooplogfiles. Designed and Developed Data Collectors and Parsers by using Python. Created Oozie workflows to manage the execution of the crunch jobs and vertica pipelines. Extensively worked on creating role playing dimensions Fact. Developed scripts for build, deployment, maintenance and related tasks using Jenkins and Python. Worked on Azure on services like Azure Data Factory, Azure Synapse Tuned and Troubleshooted Snowflake for performance and optimize utilization.
• Created Hive tables on top of HBase using Storage Handler for TEMPeffective OLAP analysis. Uploaded data to Hadoop Hive and combined new tables wif existing databases. Worked wif Services and Portal teams on various occasion for data issues in OLTP system. Migrated databases to cloud platform SQL Azure and as well the performance tuning. Built real time pipeline for streaming data using Kafka and Spark Streaming. Designed and built a Data Discovery Platform for a large system integrator using Azure HdInsight components. Worked on Continuous Integration System me.e.Jenkins. Tracked the work by Azureboards, Repos, pipelines, test plans and Artifacts Used Azure data factory and data Catalog to ingest and maintain data s

Confidential

Data Engineer

Responsibilities:

• As a Data Engineer worked wif the analysis teams and management teams and supported them based on their requirements. Followed the Agile methodology to implement the application. Implemented the Big Data solution using Hadoop, hive to pull/load the data into the HDFS system.
• Developed and deployed the outcome using spark and Scala code in Hadoop cluster running on GCP. Designed Data Marts by following Star Schema and Snowflake Schema Methodology, using industry leading Data modelingtools like Erwin. Build data ingestion from various source systems to Hadoop or GCP using Sqoop, Spark Streaming etc Extracted data using Sqoop Import query from multiple databases and ingest into Hive tables. Used apache airflow in GCP composer environment to build data pipelines Installed and configured HDFS, PIG, HIVE, Hadoop and MapReduce. Imported data from RDBMS to HDFS and Hive using Sqoop on regular basis.
• Designed logical and physical database structure to facilitate analysis of data from both operational and customer perspectives. Involved in debugging and Tuning the PL/SQL code, tuning queries, optimization for the SQL database. Extracted files from MongoDB through Sqoop and placed in HDFS and processed. Worked on POC to check various cloud offerings including Google Cloud Platform (GCP). Developed Source to Target Matrix wif ETL transformationlogic for ETL team. Involved in migration of data from existing RDBMS (oracle) to Hadoop using Sqoop for processing data. Worked on google cloud platform (GCP) services like compute engine, cloud load balancing, cloud storage, cloud SQL, stack driver monitoring and cloud deployment manager.
• Handled performance requirements for databases in OLTP and OLAP models. Installed and configured Hadoop and responsible for maintaining cluster and managing and reviewing Hadoop log files. Used Hive to analyze the partitioned and bucketed data and compute various metrics for reporting on the dashboard. Used GCP Console, monitor dataproc cluster and jobs. Developed predictive analytic product by using Apache Spark,SQL.
• Connected Tableau server to publish dashboard to a central location. Created HiveExternaltables to stage data and tan move the data from Staging to main tables Created SSISpackages to migrate data from heterogeneous sources such as MS Excel. Involved in Oozie and workflow scheduler to manage hadoopjobs wif control flows. Written T - SQL queries, created dynamic Stored Procedures by using input values

Environment: Erwin9.8, Big Data, GCP, MongoDB, Hive, HDFS, Agile, Pig, ETL, SQL, PL/SQL, Oozie, Oracle, Sqoop, OLAP, OLTP, Spark, Tableau, SSIS, MS Excel, T-SQL

Confidential

Data Modeler/ Data Engineer

Responsibilities:

• Worked as a Data Modeler/Data Engineer to generate Data Models using Erwin and developed relational database system. Designed of Redshift Data model, Redshift Performance improvements/analysis Performed the Data Mapping, Data design (Data Modeling) to integrate the data across the multiple databases in to EDW.
• Conducted data modeling JAD sessions and communicated data - related standards. Created scripts for importing data into HDFS/Hive using Sqoop. Used ER/Studio and the many too many relationships between the entities are resolved using associate tables. Defined cubes, dimensions, user hierarchies, aggregations in an OLAP environment. Developed normalizedLogical and Physical database models for designing an OLTP application. Developed SQL scripts for creating tables, Sequences,
• Triggers, views and materialized views. Designed and Developed PL/SQL procedures, functions and packages to create Summary tables. Created Teradataobjects like Tables and Views. Designed both 3NF Data models and dimensional Data models using Star and Snowflakeschemas. Created Rich dashboards using TableauDesktop and prepared user stories to create compelling dashboards to deliver actionable insights. Developed Oozieworkflow jobs to execute hive, and Sqoop. Connected to
• Amazon Redshift through Tableau to extract live data for real time analysis. Designed and Developed Oracle databaseTables, Views, Indexes wif proper privileges and Maintained and updated the database by deleting and removing old data. Created mapping tables to find out the missing attributes for the ETL process. Designed and developed the conceptual tan logical and physical data models to meet the needs of reporting. Implemented a proof of concept deploying this product in Amazon Web
• ServicesAWS. Involved in Normalization/De-normalizationtechniques for optimum performance in relational and dimensional database environments. Developed various T-SQL stored procedures, triggers, views and adding/changing tables for data load, transformation and extraction. Worked wif reversed engineer Data Model from Databaseinstance and Scripts. Worked closely wif the SSRS Developers to explain the complex data transformation using Logic.

Environment: ER/Studio, OLAP, OLTP, SQL, PL/SQL, HDFS, Hive, Sqoop, Oracle, Teradata, ETL, AWS, T-SQL, SSRS.

Confidential

Data Analyst/Data Modeler

Responsibilities:

• Worked on Data Analysis, Data Modeling and Data Profiling identifying Data Sets, Source Data, Source Data Definitions and Data Formats. Created physical and logical data models using Power Designer. Conducted JAD sessions wif management, vendors, users and other stakeholders for open and pending issues to develop specifications.
• Created customized report using OLAP Tools such as Crystal Report for business use. Used Python library Beautiful Soup for web scrapping to extract data for building graphs. Involved in extensive data analysis on Teradata, and Oracle Systems Querying and Writing in SQL. Designed and developed a business intelligence dashboard using Tableau Desktop, allowing executive. and cross businesses processes. Designed the dimensionalData Model of the data warehouse. Extensively involved in writing
• PL/SQL, stored procedures,functions and packages. Interacted wif business users to understand the business requirements. Enforced referential integrity in the OLTP data model for consistent relationship between tables and efficient database design. Worked wif data compliance teams, Data governance team to maintain data models, Metadata, DataDictionaries define source fields and its definitions. Implemented Source - to-Target mapping wif the necessary transformation logic for the ETL team to build
• Used Python scripts to update the content in database and manipulate files Created SSIS Packages using varsious Transformations. Used Star/Snowflake schemas in the data warehouse architecture. Developed dashboards using Tableau Desktop. Created partitions and indexes for the tables in the data mart. Developed database triggers and stored procedures using T-SQL cursors and tables. Worked on Teradata and its utilities - tpump, fastloadthroughInformatica. Loaded multi format data from various sources like flat-file, Excel, MS Access and performing file system operation. Written simple and advanced SQL queries and scripts to create standard and ad hoc reports for senior managers

Environment: Power Designer, OLAP, BI, Python, SQL, PL/SQL, OLTP, SSIS, Tableau, Oracle, MS Access, Informatica, Teradata, T-SQL