SUMMARY

• Data Engineer with 8+ experience in conceptual/logical data modeling skills and strong emphasis on coding with analytical programming using Python and SQL
• Good experience on deployment process of Azure data bricks notebooks, SQL scripts, Azure Data Factory code using Azure Devops and GIT HUB platforms with the
• Sound knowledge on the architecture of Azure Data and Analytics services in combination with designing and building data pipelines, data streams, and system integrations
• Designed and implemented end - to-end data pipelines to extract, cleanse, process and analyze huge amounts of behavioral data and log data.
• Ability to architect solutions that meet business and IT needs, create Data Platform roadmaps, and enable the Data Platform to scale to support additional use cases in Azure Data Lake, Azure Data bricks, Azure Data Factory
• Strong Knowledge on designing & building the data model for Snowflake cloud data warehouse.
• Experience in developing production-ready spark applications using Spark RDD API’s, Data frames, Spark-SQL and Spark-Streaming API.
• Worked on data processing and transformations and actions in spark by using Python (Pyspark) language.
• Strong experience in using Spark Streaming, Spark SQL, and other components of spark like accumulators, Broadcast variables, different levels of caching and optimization techniques for spark jobs
• Used hive extensively to perform various data analytics required by business teams.
• Solid experience in working with various data formats like Parquet, Orc, Avro, Json etc.,
• Performed onIntegrated data from around 40 different sources and developed data pipelines to load the data into Elasticsearch and Salesforce for the use in targeted e-mail marketing (Stone branch, AWS, S3, Hive, Elasticsearch, Redshift, Python, Spark, Salesforce), to bring dozens of millions of dollars of additional revenue.
• Experience automating end-to-end data pipelines with strong resilience and recoverability.
• Worked on Spark Streaming and Structured Spark streaming along with Kafka for real-time data processing.
• Good level of experience with Oracle PL/SQL, SQL Server, and shell scripting.
• Experienced in using waterfall, Agile and Scrum models of software development process framework.
• Created various reports and dashboards in Power BI with different types of sources like ADLS Gen2, SharePoint path, Postgres SQL, SQL server, salesforce objects, Power BI shared datasets and SSAS cubes
• Having good exposure towards visual like Table, Bar, Card, Funnel, Line, Gauge, Doughnut, Waterfall, Slicers, Matrix visualizations.
• Worked on DAX expressions like filters, Aggregate, Mathematical, Time Intelligence Functions etc.
• Created New Calculated Columns and Measures using DAX Expression like filters, Aggregate, Mathematical, Time Intelligence Functions etc.
• Implemented datasets in Power BI based on mapping sheet by following star schema modeling.
• Flexible in working with selection and bookmark panes for hiding and displaying visuals according to the required selections.
• Worked on Schedule refresh in Power BI Service based on timings provided for all types of source datasets
• Implemented Role Based Security as part of security in Power Bi for users who are going to utilize the reports.
• Experience in Using multiple join statements to retrieve data from multiple tables
• Created different logics for implementing transformation of data using split columns, conditional columns, append and merge queries in Power Query editor.
• Created various reports and dashboards in Power BI with different types of sources like ADLS Gen2, SharePoint path, Postgres SQL, SQL server, salesforce objects, Power BI shared datasets and SSAS cubes
• Having good exposure towards visual like Table, Bar, Card, Funnel, Line, Gauge, Doughnut, Waterfall, Slicers, Matrix visualizations.
• Active team player with excellent interpersonal skills, a keen learner with self-commitment& innovation.
• Superior communication skills, strong decision making and organizational skills along with outstanding analytical and problem-solving skills to undertake challenging jobs.

TECHNICAL SKILLS

Cloud Technologies: Azure Data Bricks, Synapse, Azure Data factory, Azure key vault

Programming Language: Python

Databases: Oracle, SQL Server

Deployment Platform: GITHUB

NoSQL Databases: HBase, Cassandra, MongoDB, DynamoDB

PROFESSIONAL EXPERIENCE

Confidential, Norfolk Virginia

Data Engineer/ Power BI Developer

Responsibilities:

• Implemented Azure Data Factory pipeline to extract data from oracle on premises to Azure Data Lake Gen 2 incrementally.
• Implemented data vault model by analyzing source tables which will be processed to Synapse
• Created Data Mapping sheet for better understating of data which will be utilized while implementing code for data modeling
• Writing complex SQL scripts, views, and Stored procedures for transforming data based on the business requirements.
• Hands-on Experience with different types of activities in Azure data Factory like LOOK UP, Filter, Get Metadata, Execute pipeline, switch, and notebooks.
• Hands-on Experience on Azure Data Bricks platform for implementing code in Spark SQL and Py-spark
• Worked on Synapse for implementing different types of complex SQL Views after transforming data from parquet files to Data bricks data.
• Design and Develop ETL integration patterns using Python on Spark.
• Analyzed the SQL scripts and designed the solution to implement using PySpark.
• Developed highly optimized Spark applications to perform various data cleansing, validation, transformation, and summarization activities according to the requirement
• Created datasets and linked services for both cloud and on premises resources like oracle, ADLS Gen 2, SQL server
• Optimization of spark SQL code by reducing join conditions and sub queries which is used for good performance of execution.
• Created Kimball model for data in synapse by implementing code according to table data transformation
• Migrating implemented SSIS packages to cloud platform by using Azure data factory and Azure Data bricks.
• Created different types of Dimensional and Fact Tables by following business rules and verifying legacy data warehousing.
• Used different types of window function and cte’s in SQL script development.
• Using GITHUB configuration for deploying ADF code and Data Bricks notebooks from one environment to another with the help of CI CD pipelines which are available.
• Worked with business process managers and be a subject matter expert for transforming vast amounts of data and creating business intelligence reports in Power BI.
• After Data Processing created calculated columns and measures to display the results in the form of visuals for end user
• Configured data connections (Import and Direct query) for all types of sources which are required for report development.
• Created incremental refresh for required fact tables in Power BI Desktop by passing end date as parameter values
• Published reports in Power Bi Service.
• Done schedule refresh for the reports.
• Transform data by using merge queries/ append queries in Edit Queries section on Power Bi Desktop.
• Creatingbookmarks for report to view in current scenario.
• Gateway connections implementation in Power BI Service for allowing data refresh in on premises
• Implemented DAX expressions for MTD and YTD based on slicer selection.
• Using bookmark and selection pane for hiding and displaying visuals based on selection.
• Configuration of RLS (Row Level Security) for specified users with the help of specific user selection in Power BI service.
• Worked on different DAX functions like EOMONTH, DATEDIFF, CALCULATE, COUNT, SUM, FILTER, ALL etc.
• Implemented Data sets using data modeling by connecting all the tables to satisfy all required conditions.
• Worked on all types of filters like Page level, Visual level, report level and drill through filters.
• Used all the functionalities of visual in Power BI like drill up, drill down, conditional formatting, tool tips
• Displaying images in all visuals using web URL from source tables.
• Implementation of sorting techniques in visual display of reports.
• Configuration of Power BI service account after publishing reports
• Published developed reports to Power BI service and implemented gateways to allow data refresh based on schedule timings.
• Worked with testing team for data quality
• Followed sprint wise tasks by following JIIRA as tracking tool.

Confidential, Dallas, TX

Data Engineer/Power BI Developer

Responsibilities:

• Loading history and incremental data from oracle on premises system to ADLS Gen 2 storage account by creating pipelines in Azure data factory which is called as Raw Layer
• Worked on POC’s related to Data Processing which was initially given by the business.
• Created pipeline for migrating parquet files generated by the above pipeline to the form of delta files using UPSERT in Data Flows of Azure data factory which is called as Refined Layer
• Created SQL scripts and stored procedures to insert and update data in Audit table which is holding all the pipeline details and load status
• Using functionalities of data flows in Azure Data Factory for processing and transforming data based on the mapping sheet provided by Data Modeler
• Transformed data using JOINS, LOOKUP, WINDOW FUNCTION, EXISTS, Conditional Stream and branch in data flows.
• After completing Refined Layer developed several Data Flows based on business requirements which will load data using different load strategies like SCD 1 and SCD2.This final layer is called as EDW layer
• Created triggers for the pipelines by creating trigger file such that pipelines will trigger in sequential order
• After EDW layer started involving in several Power BI report creation to get clear visuals which will be utilized by end user
• Created schedule and event-based triggers for automatic trigger of pipelines
• Created SQL queries to extract data from oracle based on business requirements
• Created partition query pipelines to load huge amount of data by executing pipeline in sequence one partition after other
• Helped whole team in getting ETL architecture flow to be settled in a quite good manner
• Created views on top of delta tables which are utilized as source for extracting data from delta files which in the form of parquet
• Implemented Data flows and Azure Data Factory which are dynamically parameterized without any manual execution.
• Helped the testing team for generating test scripts which is very complex and involved in creating the test scenarios/cases.
• After development pushed all the changes from feature branch to master branch using Azure Devops by validating and comparing the changes made.
• Created secrets in key vaults which will be used in Azure data Factory linked services for connection purpose.
• Created mail notification trigger using logic apps in Azure data factory by sending parameters for the required fields.
• Building datasets in Power BI based on the data mapping sheet and creating relationships according to that model.
• Developing visual reports, KPI scorecards, and dashboards using Power BI desktop.
• Connecting data sources, importing data, and transforming data for Business intelligence.
• Analytical thinking for translating data into informative reports and visuals.
• Capable of implementing row-level security on data along with an understanding of application security layer models in Power BI.
• Implemented several DAX queries based on selection and Date time intelligence functions and also created several measures using complex DAX expressions for getting desired outputs.
• Designed methodology and project documentaries for Report delivery to end users.
• Developed tabular and multidimensional models that are compatible with data warehouse standards in Power BI
• Developed reports in Power BI using clear visual which will be published in power bi service.

Confidential, Bentonville, AR

Power BI Developer

Responsibilities:

• Created various reports and dashboards in Power BI with different types of sources like ADLS Gen2, SharePoint path, Postgres SQL, SQL server, salesforce objects, Power BI shared datasets and SSAS cubes
• Having good exposure towards visual like Table, Bar, Card, Funnel, Line, Gauge, Doughnut, Waterfall, Slicers, Matrix visualizations.
• Created New Calculated Columns and Measures using DAX Expression like filters, Aggregate, Mathematical, Time Intelligence Functions etc.
• Implemented datasets in Power BI based on mapping sheet by following star schema modeling.
• Flexible in working with selection and bookmark panes for hiding and displaying visuals according to the required selections.
• Worked on Schedule refresh in Power BI Service based on timings provided for all types of source datasets
• Implemented Role Based Security as part of security in Power Bi for users who are going to utilize the reports.
• Experience in Using multiple join statements to retrieve data from multiple tables
• Created different logics for implementing transformation of data using split columns, conditional columns, append and merge queries in Power Query editor.
• Created star schema Data model in Power BI by following data mapping sheet provided by the business
• Implemented dynamic RLS configuration to restrict the data for the provided users by using RLS configuration table in the data model.
• Worked on custom visual summary table by changing JSON code view in the visual for displaying the desired output.
• Worked on different file formats like Text, Sequence files, Avro, Parquet, JSON, XML files and Flat files
• Creating pipelines using different activities (filter, copy, stored procedure, web activity etc..,) in azure data factory for moving data from files to tables in Azure SQL DB.
• Worked on Microsoft Azure cloud to provide IaaS support to client.

Confidential, New York

Data Engineer

Responsibilities:

• Implemented SSIS packages for extracting data from oracle and transforming data and finally loading to SQL server.
• Configuration of SSIS while implementing packages and building solutions for complex logics involved in it.
• Created views and cte’s for implementing transformations like additional columns, case statements and aggregate functions in SQL
• Orchestration of ETL pipelines and understanding available work loads which will be migrated using SSIS
• Created Indexes on the tables for the faster data retrieval to improve the query performance.
• Participated in Performance Tuning of SQL queries using Explain Plan to improve the performance of the application.
• Worked extensively on exception handling for handling errors using system defined exceptions and user defined exceptions.
• Worked on creating new reports, enhancements, break fixing reports using oracle report builder.
• Created database objects tables, views, triggers, public Synonyms according to the requirements.
• Involved in validating the objects participating in replication of database by writing test scripts.
• Created Datastage jobs based on the HLD and mapping document.
• Used various processing stages in the IBM Datastage tool like join, merge, lookup, remove duplicates, filter to implement the business logic.
• Extracting, cleansing, transforming, integrating and loading data into Data warehouse using Data stage Designer.
• Created Business requirement documents and worked on production support of apex applications
• Used version control tool Tortoise to keep the latest version of the code developed at one place, which helps multiple users to access and make changes to the collection of source code files.
• Worked on tickets to enhance oracle apex database application
• Wrote procedures and functions for after submit page processing, validations and computations on apex applications.

Confidential

Data Engineer

Responsibilities:

• Actively involved in admin activities during the migration process.
• Involved in Testing after the migration process.
• Worked on developing reports exclusively based on the end user requirement.
• Interacted with End user community to understand the business requirements and in identifying data sources.
• Studied and modified the existing PL/SQL code to relate the source and target mappings.
• Migrated and Converted PLSQL code to Datastage code for already existing sources.
• Designed, developed new ETL code for new sources
• Helped in preparing the mapping document for source to target.
• Designed and developed ETL processes using Datastage designer to load data from Oracle, MS SQL, Flat Files (Fixed Width) to staging database and from staging to the target Data Warehouse database.
• Used DataStage stages namely Hash file, Sequential file, Transformer, Aggregate, Sort, Datasets, Join, Lookup, Change Capture, Funnel, Peek, Row Generator stages in accomplishing the ETL Coding.
• Developed job sequencer with proper job dependencies, job control stages, triggers.
• Used Quality Stage to ensure consistency, removing data anomalies and spelling errors of the source information before being delivered for further processing.
• Excessively used DS Director for monitoring Job logs to resolve issues.
• Involved in performance tuning and optimization of DataStage mappings using features like Pipeline and Partition Parallelism and data/index cache to manage very large volume of data.
• Documented ETL test plans, test cases, test scripts, and validations based on design specifications for unit testing, system testing, functional testing, prepared test data for testing, error handling and analysis.
• Used Orsyp/DS Director job scheduler for automating the monthly regular run of DW cycle in both production and UAT environments.
• Verified the Cognos Report by extracting data from the Staging Database using PL/SQL queries
• Wrote Configuration files for Performance in production environment.
• Participated in weekly status meetings.
• Worked extensively on SQL, PLSQL. Created packages for Cognos reports, Adhoc reports.
• Worked on historical data fixes.
• Extensively worked on Geo coding the addresses to geo code and validate the address using ARCGIS geocoding tool.