Azure Data Engineer Resume
Dallas, TX
SUMMARY
- 8 years of experience in MS SQL Server 2018/2016/2012/2008 Oracle/Sybase databases and business intelligence technologies SSIS/SSRS/SSAS/Tableau/Power BI/Azure cloud
- Experience in building data pipelines using Azure Data factory, Azure databricks and loading data to Azure data Lake, Azure SQL Database, Azure SQL Data warehouse and controlling and granting database access.
- Experience in Designing and Implementation of Relational Database Model for business requirements using Normalization forms.
- Experienced in creating Tables, Views, Stored procedures, Functions, Clustered, Non - clustered indexes as per the business and design requirements using T-SQL.
- Good Confidential writing complex SQL queries involving multiple tables' joins, common table expressions (CTEs), user defined functions, indexed views, cursors and DDL/DML triggers for data consistency and data manipulation.
- Azure Data Factory (ADF), Integration Run Time (IR), File System Data Ingestion, Relational Data Ingestion.
- Experience in Developing Spark applications using Spark - SQL, Pyspark and Delta Lake in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
- Experienced and Implemented SAN TR migrations like Host based and Array based migrations.
- Developed OLAP cubes using SQL Server Analysis Services and calculated the metrics dat were revealed in the functional analysis.
- Hands on Experience in performing Host based online SAN migrations.
- Experience with MS SQL Server Integration Services (SSIS), T-SQL skills, stored procedures, triggers.
- Respond rapidly to system maintenance needs, including on evenings and weekends.
- Generate incident reports, change reports, turnovers Summary report weekly basis.
- Proficient in making DAX queries in Power BI desktop.
- Experience Confidential Transforming and validating data using SSIS Transformations like Conditional Split, Lookup, Merge Join and Sort and Derived Column for unstructured and redundant data.
- Good Confidential improving Performance using Table Indexes and Indexed Views by implementing Schema Bindings.
TECHNICAL SKILLS
Azure Cloud: Data Factory, Databricks, Azure Analysis Service, Synapse, Data Lake, Logic App, Function App
Big Data Tech: HDInsight, Databricks, Cosmos DB, Hadoop, Spark, Hive, Sqoop
Relational Databases: Microsoft SQL Server, Azure SQL, MySQL
Business Intelligence tools: Power BI, QlikView, Tableau
Languages: T-SQL, USQL, Python, MDX, DAX, PySpark
Development Tools: Visual Studio, SQL Server Management Studio
ALMS: Azure DevOps, VSTS, Jira,SNOW
Methodology: Agile, Waterfall, Prototype
PROFESSIONAL EXPERIENCE
Confidential, DALLAS, TX
AZURE DATA ENGINEER
RESPONSIBILITIES:
- Understand requirements, build codes, and guide other developers during development activities to develop high standard stable codes within the limits of Confidential and clients’ processes, standards, and guidelines.
- Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics. Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.
- Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL synapse, write-back tool and backwards.
- Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) dat processes the data using the SQL Activity.
- Worked on transformations to transform the data required by analytics team for visualization and business decisions.
- Developed and Deployed Stored Procedures on Azure Synapse Analytics (SQL DW).
- Performed ETL operations in Azure Databricks by connecting to different relational database source systems using JDBC connectors.
- Good understanding Hadoop distributed file system architecture
- Authored Azure Data Factory pipeline to manage a regular process of data movement as part of a wider enterprise analytical solution.
- Performed data migration from on-premises server to an Azure Data Lake Gen2 storage.
- Designed, created, loaded and queries data warehouse schema and tables
- Involved in migrating the client data warehouse architecture from on-premises into Azure cloud.
- Creating storage accounts which involved with end-to-end environment for running jobs.
- Implement Azure Data Factory operations and deployment into Azure for moving data from on-premises into cloud.
- Designed and developed a new solution to process the NRT data by using Azure stream analytics, Azure Event Hub and Service Bus Queue.
- Expert in using Databricks with Azure Data Factory (ADF) to compute large volumes of data.
- Performed ETL operations in Azure Databricks by connecting to different relational database source systems using jdbc connectors.
- Ensure overall system stability, reliability, and manageability of Azure Synapse Analytics deployments.
- Built visual reports, dashboards and KPI scorecards using power BI desktop.
- Proficient in making DAX quires in Power BI desktop.
- Experience in implementing row level security on data using Power BI.
- Used Logic App to take decisional actions based on the workflow.
- Developed Databricks ETL pipelines using notebooks, Spark Dataframes, SPARK SQL and python scripting.
- Expertise in writing complex DAX functions inPower BI and Power Pivot.
- Installed and configured Enterprise gateway and Personal gateway in Power bi service. Scheduled Automatic refresh and scheduling refresh in power bi service.
- Wrote calculated columns, Measures queries in Power Bi desktop to show good data analysis technique
- Develop Informatica mappings to be implemented based on client requirements and for the analytics team.
- Transferred data outside the synapse workspace using the PySpark connector
- Involved in functional testing and regression testing
- Worked on transformations to transform the data required by analytics team for visualization and business decisions.
- Implemented the role play dimension in tabular model by using DAX functions
- Monitoring end to end integration using Azure monitor.
Environment: SQL Database, Azure data factory, Azure data lake storage, Azure synapse analytics, Azure synapse workspace, Synapse SQL pool, Power BI, Python, Data masking, Azure data bricks, Azure SQL Data warehouse, Azure stream analytics.
Confidential, DALLAS, TX
AZURE DATA ENGINEER
RESPONSIBILITIES:
- Has extensive experience in creating pipeline jobs, schedule triggers using Azure data factory.
- Review plan and provide feedback on gaps, timeline and execution feasibility etc. as required in the project
- Participate in KT sessions conducted by customer/ other business teams and provide feedback on requirements
- Involved in migrating the client data warehouse architecture from on-premises into Azure cloud.
- Create pipelines in ADF using linked services to extract, transform and load data from multiple sources like Azure SQL, Blob storage and Azure SQL Data warehouse.
- Creating storage accounts which involved with end to end environment for running jobs.
- Implement Azure Data Factory operations and deployment into Azure for moving data from on-premise into cloud.
- Designed and developed a new solution to process the NRT data by using Azure stream analytics, Azure Event Hub and Service Bus Queue.
- Worked on migration of data from On-prem SQL server to Cloud databases(Azure Synapse Analytics (DW) & Azure SQL DB).
- Hands - on experience in Azure Cloud Services (PaaS & IaaS), Azure Synapse Analytics, SQL Azure, Data Factory, Azure Analysis services, Application Insights, Azure Monitoring, Key Vault, Azure Data Lake .
- Design data auditing and data masking for security purpose.
- Monitoring end to end integration using Azure monitor.
- Implementation of data movements from on-premises to cloud in Azure.
- Develop batch processing solutions by using Data Factory and Azure Data bricks
- Implement Azure Data bricks clusters, notebooks, jobs and auto scaling.
- Created Linked service to land the data from SFTP location to Azure Data Lake.
- Design for data auditing and data masking
- Preparing ETL test strategy, designs and test plans to execute test cases for ETL and BI systems.
- Interacting with business users and understanding their requirements.
- Good understanding of data warehouse concepts.
- Analyzed data where it lives by Mounting Azure Data Lake and Blob to Databricks.
- Strong skills in visualization toolsPower BI,Confidential Excel - formulas
- Developed Python scripts to do file validations in Databricks and automated the process using ADF.
- Good exposure and understanding of Hadoop Ecosystem
- Used Cosmos DB for storing catalog data and for event sourcing in order processing pipelines.
- Proficient in SQL and other relational databases.
- Good exposure to Microsoft Power BI.
- Good understanding and working noledge of Python language
- Calculate the monthly, quarterly, and yearly running totals by using DAX time intelligence functions
- Hands on experience in Azure Dev Ops CI/CD process.
Environment: SQL Database, Azure data factory, Azure data lake storage, Azure synapse analytics, Synapse sql pool, Power BI, Python, Data masking, Azure data bricks, Azure SQL Data warehouse, Azure Cosmos DB, Azure stream analytics.
Confidential, IRVING TX
AZURE DATA ENGINEER
RESPONSIBILITIES:
- Develop, design data models, data structures and ETL jobs for dataacquisition and manipulation purposes.
- Develop deep understanding of the data sources, implement datastandards, maintain data quality and master data management.
- Created Linked service to land the data from Caesars SFTP location toAzure Datalake.
- Expert in developing JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) dat process the data.
- Used Azure Data Factory extensively for ingesting data from disparate source systems.
- Designed complex data intensive reports inPower BIutilizing various graph features such as gauge, funnel.
- Developed an automated process in Azure cloud which can ingest data daily from web service and load in to Azure SQL DB.
- Analyze the existing reports and create mapping document of each element used in the report.
- Perform end to end system integration testing
- Led automation of Azure deployments utilizing PowerShell.
- Creating ETL test scenarios and test cases and plans to execute test cases.
- Design for data encryption for data Confidential rest and in transit
- Design relational and non-relational data stores on Azure.
- Scheduling the sessions to extract, transform and load data into warehouse database on Business requirements using scheduling tool.
- Migrate data from traditional database systems to Azure databases
- Developed Streaming pipelines using Azure Event Hubs and Stream Analytics to analyze data for dealer efficiency and open table counts for data coming in from IOT enabled poker and other pit tables.
- Used Logic App to take decisional actions based on the workflow.
- Create Pipelines for copying data over from the Netezza csv files to Azure SQL.
- Developed custom alerts using Azure Data Factory, SQLDB and Logic App.
- Used Python and Shell scripts to Automate Teradata ELT and Admin activities.
- Performed Application level DBA activities creating tables, indexes, and monitored and tuned Teradata BETQ scripts using Teradata Visual Explain utility.
- Performance tuning, monitoring, UNIX shell scripting, and physical and logical database design.
- Developed UNIX scripts to automate different tasks involved as part of loading process.
- Worked on Tableau software for the reporting needs.
- Automated jobs using Scheduled, Event based, Tumbling window triggers in ADF.
- Worked on creating few Tableau dashboard reports, Heat map charts and supported numerous dashboards, pie charts and heat map charts dat were built on Teradata database.
- Implement Copy activity, Custom Azure Data Factory Pipeline Activities.
- Review and write sql scripts to verify data from source systems to target.
- Design and implement database solutions in Azure SQL Data Warehouse, Azure SQL.
- Collaborate with application architects on infrastructure as a service (IaaS) applications to Platform as a Service (PaaS).
Environment: Azure Data Factory, Tableau, Shell Scripting, Teradata, python scripting, Azure data lake storage, Blob storage, Azure SQL Database, Power BI, Azure SQL Data warehouse, Azure Cosmos DB
Confidential, Irving, TX
UNIX/LINUX SYSTEM ADMINISTRATOR
RESPONSIBILITIES:
- Day to Day administration tasks - adding and removing users, file, share permissions, trust management, replication management, server backups and server setups on AIX, LINUX. Server health monitoring and system tuning as per vendor recommendations.
- Performing Online host-based migrations, pre and post validations in weekdays off hours.
- Participate in a team responsible for the design, implementation, and automate deployments of systems, network, and storage hardware and software.
- Upgradation of WebLogic servers in development, testing and production environment and applying patch and service packs.
- Responsible for preparing weekend work order and participating in weekend Data migrations.
- Schedule cron jobs for disk utilization, CPU utilization, backup of the data & operating system and other applications as per the requirement using shell scripting.
- Created disk groups, volumes, volume groups and RAIDs using Veritas Volume Manager.
- Checking disk size and capacity and also warranty of the storage frames.
- WebLogic Administration, Monitoring and Troubleshooting using Admin Console.
- Installed and Configured WebLogic, APACHE and TOMCAT Servers with JAVA Application tools on Red Hat and AIX server environment.
- Respond rapidly to system maintenance needs, including on evenings and weekends.
- Generate incident reports, change reports, turnovers Summary report weekly basis.
- Simultaneously work on different projects and problem tasks.
- Monitoring WebLogic server health and security.
- Work with the team to develop system administration operations documentation.
- Implement file sharing on the network by configuring NFS on the system to share essential resources.
- Troubleshooting, diagnosing, and resolving hardware, software, and other network and system problems.
- Experienced on Veritas volume manager, VXVM data migrations.
Environment: Linux 7.x/6.x/5.x, Oracle 10g, SUN Servers, Solaris, HP-UX, ESX, SUN Fires, Linux, AIX, Jump Start, Shell Script, VM ware, Autosys, Netapp, Veritas Volume Manager (VVM), LDAP, Confidential Storage SAN, Veritas Cluster Server (VCS), VMware servers, jboss, Websphere.
Confidential
SERVER ADMINISTRATOR
Responsibilities:
- Checking disk size and capacity and also warranty of the storage frames.
- Working with Change management to raise new change requests in Service now.
- Administration and configuring of centralized autantication using LDAP, NIS and NIS+
- Responsible for managing of Linux, AIX and Solaris Operating System.
- Responsible for preparing weekend work order and participating in weekend Data migrations.
- Schedule cron jobs for disk utilization, CPU utilization, backup of the data & operating system and other applications as per the requirement using shell scripting.
- Working with Change management to raise new change requests in Service now.
- Working on chasing acceptances and Approvals for change tasks for weekend activities.
- Performing Online host based migrations, pre and post validations in weekdays off hours.
- Implement file sharing on the network by configuring NFS on the system to share essential resources
- Troubleshooting, diagnosing and resolving hardware, software, and other network and system problems.
- Responsible for managing of Linux, AIX and Solaris Operating System.
- Administration of Solaris Zones and LODMS.
- Experience on physical servers migrates to virtualization using Branded Zones.
- Administration and configuring of DNS and DHCP on Linux and Solaris systems.
- Administering of Linux Virtualization using VMWareESXi.
- Scheduling cron jobs for disk utilization, CPU utilization, backup of the data & operating system and other applications as per the requirement using shell scripting.
- Configured volume groups and logical volumes, extended logical volumes for file system growth needs using Logical Volume Manager (LVM) commands.
Environment: Red hat Linux 4/5, Solaris 9/8, HP-UX, AIX 5.1/5.3L, OS 2200, Sun Fire, Sun Enterprise Servers, Ultra Enterprise 2/1 Servers, SPARC server 1000, SPARC server 20 Enterprise Servers, SPARC server HP 9000 K, L, N class server, IBM RS/6000, pSeries servers.