We provide IT Staff Augmentation Services!

Data Engineer Resume

SUMMARY

  • Microsoft Certified and AWS Certified Professional with Experience in database design, Programming, Development and Implementation of Client - Server Applications & Database systems using MS SQL Server 2014/2012, Microsoft Dynamics CRM, MongoDB and Expertise in moving data into S3 file storage.
  • Experience in using Microsoft BI studio products like SSIS, SSRS and Big Data tools like Apache Hadoop, MongoDB for implementation of ETL methodology in data extraction, transformation and loading.
  • Strong understanding of BI application design and development principles.
  • Extensive working experience in:
  • Writing Complex Transact SQL (T-SQL) Queries, Sub queries, Co-related sub queries, Dynamic SQL queries etc.
  • Programming in SQL Server Stored Procedures, Triggers, User-defined Functions and Views, Cursors.
  • Expertise in performing data processing tasks using Python libraries Pandas, NumPy.
  • Design and maintain efficient, reusable and reliable C# code.
  • Logical & Physical Data modeling, data architecture designs etc.
  • Export/Import data from Hive Warehouse to RDBMS (Oracle, SQL Server) and vice versa.
  • Involved in Backing up databases and refreshing Data across environments
  • Creating data load and Upgrade scripts for data migrations, data formatting and data validations
  • Implementing all high availability techniques like clustering, replication.
  • Experienced in whole cycle of DTS/SQL server integration services (SSIS 2012) Packages (Developing, Deploying, Scheduling, Troubleshooting and monitoring) for performing Data transfers and ETL Purposes across different servers.
  • Experience in providing Logging, Error handling by using Event Handler, and Custom Logging for SSIS Packages.
  • Expertise in data migration from SQL on Perm to Cloud AWS.
  • Involved in complete release cycle and deploying new features/code using TFS and Source control to Production Environment.
  • Expertise in data processing and performing ETL tasks using Python.
  • Strong experience with Core Python and NumPy, Panda and Flask.
  • Experience in working on databases like MS SQL Server, MongoDB database.
  • Experience in Scala/spark to build pipeline for load data into to s3 files system
  • Expertise in generating reports using SQL Server Reporting Services, Crystal Reports, and MS Excel spreadsheets and Power Pivot.
  • Responsible for building and supporting a Hadoop-based ecosystem designed for enterprise-wide analysis of structured, semi-structured and unstructured data.
  • Experience working on Experian Pandora tool to generate complete report regarding Table data.
  • Experience in working on JIRA, GitHub and TFS tools.
  • Experience with Agile software development practices.
  • Good team player, Excellent interpersonal and communication skills combined with self-motivation, initiative and the ability to think outside the box.

TECHNICAL SKILLS

Languages: T-SQL, PL/SQL, NoSQL, C#, Python, Java, Spark, HiveQL

Database Platform’s: MS SQL Server 2012, 2014, MongoDBv3.2, Apache Hive, Oracle, AWS, Microsoft Dynamics CRM 365

ETL / BI Tools: SQL Server Integration Services (SSIS), SSRS, Business Intelligence Development Studio (BIDS), Kingsway Soft SSIS Integration Tool, Tableau, Ambari 2.6., DB Visualizer, Visual Studio.

Packages: MS Office, Microsoft Office Suit, Microsoft Performance Point Server 2007

Data Quality Tools: Experian Pandora, SQL Server Profiling, Data Profiling using SSIS.

Management Tools and Repositories: Team Foundation Server (TFS), GitHub, SharePoint, JIRA.

PROFESSIONAL EXPERIENCE

Confidential

Data Engineer

Responsibilities:

  • Design, develop and implement database systems based on requirements.
  • Contribute heavily to building a system to gather business intelligence using ETL methodology.
  • Works closely with project manager to develop work plan for Data Warehouse projects and keep the manager aware of any issues.
  • Key Contributor in working on Oracle, Hive, Hadoop, Spark, Python, Snowflake .
  • Creating data models to reduce system complexities and hence increase efficiency & reduce cost.
  • Wrote and executed various Oracle database queries using Python Oracle connector
  • Provide support to Software Development Life cycle.
  • Migrating the existing data from Oracle to Hadoop and Perform ETL Operations on it.
  • Perform root cause analysis on all processes and resolve production issues and validate all data and perform routine tests on databases
  • Designed and developed ETL workflow using Oozie and automated them using Autosys.
  • Responsible for loading unstructured and semi-structured data into Hadoop by creating static and dynamic partitions.
  • Designed and implemented Sqoop incremental import and Export data from Hive Warehouse to RDBMS (Oracle, SQL Server) and vice versa.
  • Expertise in performing data processing tasks using Python libraries Pandas, NumPy.

Environment: MS SQL Server Integration Services (SSIS), Oracle 11g, Hadoop, Snowflake, Python, Visual C#, Hive, Ambari v2.6.

Confidential

Sr. ETL Developer

Responsibilities:

  • Used SSIS to create ETL Packages to validate, extract, transform and load data to data warehouse databases, data mart databases
  • Extracting the data from Mainframe system and transforming it into different Stages (Implemented required Business Logic) and loaded into the target Database and then to Microsoft Dynamics CRM using SSIS (SQL Server 2012).
  • Experience working on Correct Address tool which validates the address that coming from mainframe before getting loaded into Data warehouse using C#.
  • Experience in migration of data among multiple databases.
  • Design, code, test and document all new or modified BI systems, applications and Programs.
  • Deployed the created SSIS packages in Development and Testing environments.
  • Involve in Complete release cycle starting from Backup and Restore of Databases and deploy new features/code using Team Foundation Server and Red gate’s Source control to sync code from Dev to Test and from UAT to Production Environment.
  • Implemented Table analysis on shared Point and created Validation stored Procedures on each column based on analysis.
  • Creating and Modifying T-SQL Stored Procedures/Triggers for validating the integrity.
  • Used Power BI Power Pivot to develop data analysis prototype and used Power View and Power Map to visualize reports.
  • Hands-on experience to design and code in various technologies such as Python and SQL
  • Worked on Experian Pandora tool to generate complete report regarding Table data.
  • Expertise in Python developed an ETL for transferring files from server to server.
  • Using Python scheduled hourly jobs.

Environment: MS SQL Server Management Studio 2012, Experian Pandora, MS SQL Server Integration Services (SSIS), Team Foundation Server.MS SQL Server Reporting Services (SSRS), Microsoft Dynamics CRM, Visual C#, SharePoint, TFS.

Confidential

SQL Server/ Data Engineer

Responsibilities:

  • Involved in planning and Development of data migration and performance tuning to achieve cost efficiency and effectiveness of the system.
  • Extensively worked with SSIS ETL Tool in creating and designing mappings using various transformations tasks like OLEDB Command, Conditional Split, Lookup, Aggregator, Multicast and Derived Column.
  • Utilized business rules for data profiling, data analysing and cleansing.
  • Created ETL procedures to transfer data from legacy sources to staging area and from staging area to data warehouse
  • Expertise in data migration from SQL on Perm to Cloud AWS.
  • Built Data sources, ad-hoc and Tableau reports and scheduled daily Extracts.
  • Used Pandas API to put the data as time series and tabular format for east timestamp data manipulation and retrieval.
  • Build Data source for Tableau Reports based on Requirement.
  • Attended Practical sessions on NoSQL, Hive, Scala and Spark.
  • Worked on MongoDB database concepts such as locking, transactions, indexes, Sharding, replication, schema design, creating and modifying documents.
  • Experience in working on job tracking tool such as JIRA and code repository system to maintain and review code like GitHub.
  • Responsible for building and supporting a Hadoop-based ecosystem designed for enterprise-wide analysis of structured, semi-structured and unstructured data.
  • Experience in Scala/spark to build pipeline for load data into to s3 files system
  • Hands on experience in moving data between hive, s3 and MongoDB.
  • Used Various Big Hadoop analytical Tools like presto, Looker.

Environment: SQL Server 2012 Enterprise Edition, SQL, T-SQL, MS Access, Business Intelligent Integration Services (SSIS 2012/2014), Business Intelligence Development Studio (BIDS), JIRA, GitHub, Hadoop, MongoDB(NoSQL), AWS,Python.

Hire Now