SUMMARY

• Overall 7 years of IT experience in focusing on Data warehousing, Data modeling, Data integration, Data Migration, ETL process and Business Intelligence.
• Expertise in designing and developing scalable Big Data solutions, data warehouse models on large - scale distributed data, performing wide range of analytics to measure service performance.
• Designed and developed secure and scalable ETL Big data pipelines on the Hadoop ecosystem for diverse use cases.
• Developed solutions using Spark SQL, Spark streaming, and Kafka to process web feeds and server logs.
• Worked on Hive and Pig for data analysis as well as hands-on experience on NoSQL databases such as HBase, MongoDB, Cassandra, and DynamoDB.
• Experience developing large-scale batch and real-time data pipelines with data processing frameworks like Apache Storm, Flink, Spark and Kafka on the AWS.
• Development experience using cloud technologies like AWS EC2, S3, EMR, VPC, LAMBDA, EBS, Redshift, Glue, etc.
• Design, build, test and maintain large-scale batch and real-time data pipelines.
• Hands on experience in Hadoop ecosystem including Spark, Kafka, HBase, Hive, Pig, Impala, Sqoop, Oozie, Flume, Storm.
• Experience developing reports and dashboards using Visualization/Reporting tools like Power BI, Tableau and Spotfire.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDD and Pyspark concepts.
• Experience using workflow management and scheduler tools like Apache Airflow, oozie, Autosys etc.
• Hands on experience in creating data pipeline using Kafka, flume and Storm for Security Fraud and Compliance Violations use cases.
• Worked on standardization of data, Data Analysis & Profiling, Data cleansing, validating data using Business Rules, Parsing, Improving the data quality by eliminating the redundant data, Standardization and merging.
• Expertise in troubleshooting, debugging, performance tuning and optimization of slow running ETL/ELT jobsusing push down optimization, Partitioning techniques to manage large volume of data.
• Strong hands-on experience using Teradata utilities - BTEQ, Fast Load, Multiload, Fast Export, Tpump and Unix Shell scripting.
• Experience to Data Modeling using Dimensional Data Modeling techniques like Star Schema and Snowflake Modeling.
• Possess in-depth knowledge of Database Concepts, Design of algorithms, SDLC, OLAP, OLTP, Data marts and Data Lake.
• Experience in all stages of Software Development Lifecycle (SDLC) - Agile, Scrum, Waterfall methodologies, right from Requirement analysis to development, testing and deployment.
• Actively learning implementation of Data science concepts like Machine learning, Deep learning, NLP using Python and Anaconda Juptyer Notebooks.

TECHNICAL SKILLS

Operating Systems: UNIX, MAC OS X, Windows

Programming Languages: Python, .Net, C, SQL, PLSQL, HTML and XML

Databases: Oracle, SQL Server,Teradata, PostgreSQl, MySQL,MongoDB, Cassandra, HBase, Neo4j, Redshift, DynamoDB

Big Data Eco System: Hadoop, MapReduce, Pig, Hive, Sqoop, Zookeeper, Yarn, Spark, Storm, Impala, Flume, Kafka, HBase, PySpark, NIFI, Airflow

ETL Tools: Informatica Power Center, Informatica IDQ, SSIS

Reporting Tools: Power BI, Tableau

Version controls: TFS, Git, SVN.

PROFESSIONAL EXPERIENCE

Data Engineer

Confidential, Irving, TX

Responsibilities:

• Design, build, test, and maintain large-scale batch and real-time data pipelines.
• Supported clients in building scalable solutions in the Hadoop ecosystem for diverse use cases.
• Optimizing existing ETL pipelines to improve reliability, performance, adding data quality checks, alerting, and improving SLA landings times for Tier-0 pipelines.
• Developed ETL jobs using Sqoop to import/export data from RDBMS to HDFS and vice versa.
• Created Flume configure file to collect, aggregate, and store the web-log data IN HDFS.
• Worked on AWS EC2 instance and S3 bucket to store the data in the cloud.
• Creating S3 buckets also managing policies for S3 buckets and have Utilized S3 bucket and Glacier for storage and backup onAWS.
• Was responsible for creating on-demand tables on S3 files using Lambda Functions and AWS Glue using Python and PySpark.
• Joined various tables in Cassandra usingSpark and Scalaand ran analytics on top of them.
• Implemented a generic ETL framework withhigh availabilityfor bringing related data for Hadoop from various sources using spark.
• Participated in various upgrades and troubleshooting activities across the enterprise.
• Knowledge in performancetroubleshooting and tuningHadoop clusters.
• Implemented ApacheDrillon Hadoop to join data from SQL and No SQL databases and store it in Hadoop.
• Implemented Spark using Scala and utilizing Data frames and Spark SQL API for faster processing of data.
• Brought data from various sources into Hadoop usingKafka and applied spark streaming for real-time data transforming.
• Experience with building data pipelines in python/Pyspark/HiveSQL/Presto/BigQuery and building python DAG in Apache Airflow.
• Implemented Partitioning, Dynamic Partitions, and Buckets inHIVEfor efficient data access.
• Build data pipelines in airflow in GCP for ETL-related jobs using different airflow operators.
• ImplementedCompositeserver for thedata virtualizationneeds and created multiple views for restricted data access using a REST API.
• Implemented Spark using Scala and Spark SQL for faster testing and processing of data.
• Used AWS Glue for the data transformation, validate and data cleansing.
• Used python Boto 3 to configure the services AWS glue, EC2, S3
• ImplementedApache Sentryto restrict access to the hive tables on a group level.
• EmployedAVROformat for the entire data ingestion for faster operation and less space utilization.
• Worked inan Agile environment,and used a Rally tool to maintain the user stories and tasks.
• Implemented test scripts to support test-driven development and continuous integration.
• Configured the Oozie workflows to manage independent jobs and automate shell, Pig, Hive, and Sqoop jobs.
• Developed Airflow Workflow to schedule batch and real-time data from source to target.
• Involved in implementing the Land Process of loading the customer/product Data Set intoInformatica MDM from various source systems.
• Involved in error handling, debugging, and troubleshooting sessions using the Session logs, Debugger, and Workflow Monitor.
• Used Scala sbt to develop Scala coded spark projects and executed using spark-submit
• Used AWS glue catalog with crawler to get the data from S3 and perform sql query operations

Environment: HDFS, Hive, Sqoop, Oozie, MapReduce, Spark, Zookeeper, Python, Hbase, Flume, GCP, AWS, S3, EC2, Informatica Power Center, IDQ.

Data Engineer

Confidential, Houston, Texas

Responsibilities:

• Analyze, design, and build modern data solutions using Azure PaaS service to support visualization of data. Understand the current Production state of the application and determine the impact of new implementation on existing business processes.
• Automated resulting scripts and workflow using Apache Airflow and shell scripting to ensure daily execution in production.
• Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.
• Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in In Azure Databricks.
• Implemented Proof of concepts for SOAP & REST APIs
• REST APIs to retrieve analytics data from different data feeds
• Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool, and backward.
• Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
• Designed and developed an entire module called CDC (change data capture) in python and deployed in AWS GLUE using PySpark library and python
• Responsible for estimating the cluster size, monitoring, and troubleshooting of the Spark data bricks cluster.
• Expertized in implementing Spark using Scala and Spark SQL for faster testing and processing of data responsible to manage data from different sources.
• Experienced in performance tuning of Spark Applications for setting right Batch Interval time, the correct level of Parallelism, and memory tuning.
• Install and configure Apache Airflow for S3 bucket and Snowflake data warehouse and created dags to run the Airflow.
• Experience in moving data between GCP and Azure using Azure Data Factory.
• Hands-on experience in developing SQL Scripts for automation purposes.
• Created Build and Release for multiple projects (modules) in the production environment using Visual Studio Team Services (VSTS).
• WritingMapReduce code using pythonin order to get rid of certain security issues in the data.
• Data transformation had done by using Spark cluster where the data was set to go live on to the application usingSpark StreamingandKafka.
• Created RDD’s inSpark technology, Extracting data fromthe data warehouse(Teradata) onto the SparkRDD’s
• Experience onSpark with Scala/Python and Working onStateful TransformationsinSpark Streaming.
• Worked onBatch processingandReal-time data processingonSpark Streaming using Lambda architecture.
• Wrote and executed several complex SQL queries in AWS glue for ETL operations in Spark data frame using SparkSQL.
• Good hands-on experience onloading dataonto Hive from Spark RDD’s.
• Worked onSpark SQL UDF’s and Hive UDF’s.
• Extensively used ETL processes to load data from various source systems such as Oracle, Teradata and Flat Files, XML files into target system Oracle by applying business logic on transformation mapping for inserting and updating records when loaded.
• Developed Unix Shell Scripts for automation of ETL process.
• Involved in doing error handling, debugging, and troubleshooting sessions using the Session logs, Debugger, and Workflow Monitor.

Environment: SQL, Hive, Sqoop, Oozie, Map Reduce, Spark, PySpark, Zookeeper, Python, Scala, GCP, Azure, Informatica Power Center, IDQ.

ETL Developer

Confidential, New York, NY

Responsibilities:

• Performed data modeling (conceptual, logical, physical) using CA Erwin Data Modeler 9.0.
• Designed and developed complex ETL mappings making use of transformations like Source Qualifier, Joiner, Update Strategy, Lookup, Sorter, Expression, Router, Filter, Aggregator, XML parser, and Sequence Generator.
• Worked with the Business Intelligence and Strategy Group in Data Profiling, Data Quality, and BI Development.
• Performed Peer Code reviews and tested deliverables against user stories and acceptance tests.
• Collaborated with Agile customers to assign business value to technical stories.
• Examined and analyzed large sets of data, identifying patterns, data manipulation patterns, and performed data profiling using Informatica analyst.
• Performed data profiling, data discovery and created rulesto deliver quality information using standardization, validation, enrichment, de-duplication, and consolidation transformations in InformaticaData Quality (IDQ).
• Working with business users and business analysts for requirements gathering and business analysis.
• Designing and customizing data models for Data warehouse supporting data from multiple sources in real-time.
• Extracted data from different Flat files, MS Excel, MS Access, and transformed the data based on user requirement using Informatica Power Center and loaded data into the target, by scheduling the sessions.
• Worked on Informatica Power Center 8.6 tool - Source Analyzer, Data Warehousing designer, Mapping & Mapplet Designer, and Transformation Designer.
• Developed Informatica mappings and in the tuning of mappings for better performance.
• Created reusable mapplets and Transformations starting the concurrent batch process in server and did backup, recovery, and tuning of sessions.
• Created sequential batches and concurrent batches for sessions.
• Developed PL/SQL procedures/packages to kick off the SQL Loader control files/procedures to load the data into Oracle.
• Worked with Memory cache for static and dynamic cache for the better throughput of sessions containing Rank, Lookup, Joiner, Sorter, and Aggregator transformations.
• Created SSIS Packages to extract data from Excel Files, MS Access files using Pivot Transformation, Fuzzy Lookup, Derived Columns, Condition Split, Term extraction, Aggregate, Execute SQL Task, Data Flow Task, and Execute Package Task.
• Export cleaned data from Excel Spreadsheets, Text files, MS Access, and CSV files to data warehouse
• Written SQL Scripts and PL/SQL Scripts to extract data from Database and for Testing Purposes.
• Executing test scripts to verify actual results against expected results by using Power Connect for source (DB2) validation and Oracle for target validations.
• Stored reformatted data from relational, flat file, XML files using Informatica.
• Implemented data cleansing for files using Informatica and Trillium.
• Done automation of file provisioning process using UNIX, Informatica Mappings and Oracle utilities.
• Developed adhoc reports using calculated fields, groups, sets, and hierarchies inPower BI.
• Worked on performance tuning and optimizing slow-runningETL jobsusing push-down optimization and Partitioning techniques to manage a large volume of data.
• Extensively used Teradata utilities such as Multi Load, Fast load, Export, and BTEQ to import/export data and load target tables.
• Worked with heterogeneous Sources and Targets and performed full and Incremental loads.
• Involved in unit testing, system testing, and functional testing of the ETL and reporting jobs
• Coordination within the team and completing the project within the stipulated time.
• Worked in an Agileprocess and used AgileScrumMethodology for the project planning, implementing, developing, and executing.
• Attended daily stand-up meetings and performed sprint reviews with the team members.
• Provide estimates to the required level of accuracy for stories and tasks.

Environment: Informatica PowerCenter, IDQ, Teradata, Oracle, SQL server, BTEQ, Informatica Analyst, Teradata Utility tools, Unix, Linux, Shell scripting, ETL.

ETL Developer

Confidential

Responsibilities:

• Involved in the full development cycle of Planning, Analysis, Design, Development, Testing and Implementation.
• Extensively worked on SQL joins, subqueries, tracing, indexing strategies and query tuning and performance tuning.
• Developed PL/SQL Stored Procedures, functions and triggers as per the design and requirements of the project.
• Develop Extraction, Transformation and Loading (ETL) pipelines from various heterogeneous sources like Oracle, Teradata, PostgreSQL, DB2, Microsoft SQL Server, Flat files, XML, JSON and CSV files.
• Extensively used transformations like lookup, aggregator, router, sorter, joiner, normalizer, SQL transformation, and update strategy transformations.
• Developed complex Type 1 and Type 2 SCD workflows to perform batch and historical loads.
• Worked withInformaticaCloud to create Source/Target connections, monitor, synchronize the data in the data warehouse.
• Developed, modified, and optimized complex queries for data retrieval and modification. Optimized and Tuned SQL Queries by running Explain plans.
• Developed UNIX Shell scripts to automate repetitive database processes and maintained shell scripts for data conversion.
• Handled high-priority incident management tickets.
• Worked extensively in the database migration between applications and Informatica upgrade.
• Coordination with different teams during the month-end special jobs and during testing process.

Environment: Informatica PowerCenter, IDQ, MDM, Oracle, SQL server, PostgreSQL, DB2, PL/SQL, Teradata, Unix, Linux, Shell scripting, ETL.