We provide IT Staff Augmentation Services!

Sr. Data Engineer Resume

3.00/5 (Submit Your Rating)

Dallas, TX

SUMMARY

  • Around 10 years of IT experience in software analysis, design, development, testing and implementation of Big Data, Hadoop, SQL, NoSQL technologies.
  • Experience in working with Developer Toolkits like Eclipse IDE, Maven.
  • A very good understanding of job workflow scheduling and monitoring tools like Oozie and Control - M.
  • Experience in the Software Development Life Cycle (SDLC) phases which include Analysis, Design, Implementation, Testing and Maintenance.
  • Strong technical, administration and mentoring knowledge in Linux and Big data/Hadoop technologies.
  • Extensively used Azure Data Lake and worked with different file formats csv, avro, parquet, orc and delta.
  • Documented business, process and data flow documentationon multipledatasystems
  • Designed Data models onmultipledatabase systems, customer support, troubleshooting and modifying databases to meet the customer’s requirements.
  • Written Terraform scripts to automate AWS services which include CloudFront distribution, RDS, EC2 and S3 Bucket
  • Implemented Lamda architectures using Azure Data platform capabilities like Azure Data Lake, Azure Data Factory, HDInsight, Azure SQL Server, Azure ML and Power BI.
  • Expertise in Snowflake to create and Maintain Tables and views.
  • Hands-on experience with multiple ML/AIAlgorithms such asNeural networks,Linear and Logistic Regression,Support Vector Machine,Decision Tree,Naive Bayes using R and Python libraries (Pandas,NumPy,Matplotlib,Whoosh, Pyspark) to perform predictive Analytics on various datasets
  • Designed and developed Employee’s Training database system that exported data to an on-demand excel spreadsheet, this new system improves processing time from 30 days to 30 minutes a month.
  • Working knowledge on Azure cloud IaaS and PaaS Services, Azure SQL, Azure storage and Azure Services.
  • Good working experience in using Spark SQL to manipulate Data Frames in Python.
  • Good knowledge in NoSQL databases including Cassandra and MongoDB.
  • Expertise on Datawarehouse’s like Azure Synapse and Snowflake.
  • Worked and learned a great deal from AWS Cloud services like EC2, S3, EBS, RDS and VPC.
  • Extensive knowledge of Teradata utilities (BTEQ, Fastload, Fast Export, Multiload Update/Insert/Delete/Upset)
  • Good experience in Hive partitioning, bucketing and performing different types of joins on Hive tables and implementing Hive SerDe like JSON and Avro.
  • Experience with build tool ANT, Maven and continuous integrations like Jenkins.
  • Working experience in Development, Production and QA Environments.
  • Experience in NoSQL Column-Oriented Databases like HBase and its Integration with Hadoop cluster.
  • Knowledge on implementing Big Data in Amazon Elastic MapReduce (Amazon EMR) for processing, managing Hadoop framework dynamically scalable Amazon EC2.
  • In depth knowledge of Hadoop Architecture and various components such as HDFS, Job Tracker, Task Tracker, Name Node, Data Node andMapReduceconcepts.
  • Develop ETL mappings for various Sources (.TXT, .CSV) and also load the data from these sources into relational tables with Talend Enterprise Edition.
  • Experience working with Azure BLOB andData lakestorage and loading data intoAzure SQL Synapse analytics (DW).
  • Experience with Amazon Web Services, AWS command line interface, and AWS data pipeline.
  • Experience in writing SQL, PL/SQL queries, Stored Procedures for accessing and managing databases such as Oracle, AnsiSQL, PostgreSQL, MySQL, Amazon Aurora, Azure Synapse and IBM DB2.
  • Execute faster MapReduce functions using Spark RDD for parallel processing or referencing a dataset in HDFS, HBase and other data sources
  • Experience in DevelopingSparkapplications usingSpark - SQLinDatabricksfor data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
  • Developed Power BI reports & effective dashboards after gathering and translating end user requirements
  • Used various sources to pull data into Power BI such as SQL Server, Excel, Oracle, SQL Azure etc.
  • Created Power BI Visualizations and Dashboards as per the requirements
  • Experience on Scripting languages likeBash, Unix Shell scriptingand knowledge onPython
  • Good working knowledge on analysis tool like Tableau for regression analysis, pie charts, and bar graphs.
  • Expertise in developing Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
  • Experience with Apache Spark ecosystem using Spark-SQL, Data Frames, RDD's and knowledge on Spark MLlib.
  • Hands on experience with Big Data Ecosystems including Hadoop, MapReduce, Pig, Hive, Impala, Sqoop, Flume, NIFI, Oozie, MongoDB, Zookeeper, Kafka, Maven, Spark, Scala, HBase, Cassandra.
  • Good Knowledge in usingNiFito automate the data movement between different Hadoop systems.
  • Designed and implemented customNiFiprocessors that reacted, processed for the data pipeline
  • Experience in installation, configuration and deployment of Big Data solutions.
  • Good usage of Apache Hadoop along the enterprise version of Cloudera and Hortonworks.
  • Good Knowledge on MAPR distribution & Amazon's EMR.
  • Extensive experience in developing stored procedures, functions, Views and Triggers, Complex queries using SQL Server, TSQL and Oracle PL/SQL
  • Experience with Data Cleansing, Data Profiling and Data analysis. UNIX Shell Scripting, SQL and PL/SQL coding.
  • Database / ETL Performance Tuning: Broad Experience in Database Development including effective use of Database objects, SQL Trace, Explain Plan, Different types of Optimizers, Hints, Indexes, Table Partitions, Sub Partitions, Materialized Views, Global Temporary tables, Autonomous Transitions, Bulk Binds, Capabilities of using Oracle Built-in Functions. Performance Tuning of Informatica Mapping and workflow.
  • Superior communication skills, strong decision making and organizational skills along with outstanding analytical and problem-solving skills to undertake challenging jobs. Able to work well independently and also in a team by helping to troubleshoot technology and business-related problems.

TECHNICAL SKILLS

Hadoop/Big Data Technologies: Hadoop 3.0, HDFS, MapReduce, HBase 1.4, Apache Pig, Hive 2.3, Sqoop 1.4, Apache Impala 2.1, Oozie 4.3, Yarn, NIFI, Apache Flume 1.8, Kafka 1.1, Zookeeper

Cloud Platform: Amazon AWS, EC2, EC3, Aurora, MS Azure, Azure SQL Database, Azure SQL Data Warehouse, Azure Analysis Services, HDInsight, Azure Data Lake, Data Factory

Hadoop Distributions: Cloudera, Hortonworks, MapR

Programming Language: Java, Scala, Python 3.6, SQL, PL/SQL, Shell Scripting, Storm 1.0

Databases: Oracle 12c/11g, SQL, PostgreSQL

Operating Systems: Linux, Unix, Windows 10/8/7

IDE and Tools: Eclipse 4.7, NetBeans 8.2, Intellij, Maven

NoSQL Databases: HBase 1.4, Cassandra 3.11, MongoDB, Accumulo

Web/Application Server: Apache Tomcat 9.0.7, WSDL’s

SDLC Methodologies: Agile, Waterfall

Version Control: GIT, SVN, CVS

Other Tools: Visual Studio 2010, Business Intelligence Studio 2008, Attunity, Power BI, SQL Server Integration Services (SSIS) 2005/2008, ANSI SQL, SQL Server Reporting Services (SSRS) 2008, SQL Server 2008 R2

PROFESSIONAL EXPERIENCE

Confidential - Dallas, TX

Sr. Data Engineer

Responsibilities:

  • Configured Apache Airflow for AWS S3 bucket and created Dags to run on Airflow
  • Prepared scripts to automate the ingestion process using Pyspark and Scala as needed through various sources such as API, AWS S3, Teradata and Redshift.
  • Created multiple scripts to automate ETL/ ELT process using Pyspark from multiple sources
  • Developed Pyspark scripts utilizing SQL and RDD in spark for data analysis and storing back into S3
  • Developed Pyspark code to load from stg to hub implementing the business logic.
  • Developed code in Spark SQL for implementing Business logic with python as programming language.
  • Worked on Sequence files, Map side joins, bucketing, partitioning for hive performance enhancement and storage improvement.
  • Compiled and executed programs as necessary using Apache Spark in Scala to perform ETL jobs with ingested data.
  • Automated Data Dashboards with the stack through Terraform Scripts.
  • Developed file cleaners using Python libraries and made it clean.
  • Utilized Python Libraries like Boto3, NumPy for AWS.
  • Used Amazon EMR for Spark jobs and test locally using Jenkins.
  • Data Extraction, aggregations and consolidation of Adobe data within AWS Glue using PySpark.
  • Created external tables with partitions using Hive, AWS Athena and Redshift.
  • Developed Pyspark code for AWS Glue jobs and for EMR.

Environment: Agile, Hadoop 3.0, Pig 0.17, Maven, HBase, Sqoop, AWS, Kubernetes, Dynamo DB, Kinesis, Hive 2.3, HDFS, NoSQL, Pyspark, Impala, YARN, intellij, PL/SQL, Nifi, Snowflake, XML, JSON, Avro, Spark Kafka, Tableau, PostgreSQL, MySQL, Apache Flume 2.3

Confidential - Pleasanton, CA

AWS Big Data Engineer

Responsibilities:

  • Exported necessary spark Jars to run on Microsoft Azure Cloud.
  • Created Nifi flows to make the process data to reach downstream systems.
  • Extract Transform and Load data from Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, Confidential -SQL, Spark SQL and U-SQL Azure Data Lake Analytics.
  • Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in InAzure Databricks.
  • Created Talend jobs to populate the data into dimensions and fact tables.
  • Used dplyr and plyr for data wrangling and data analysis using R.
  • Architected and designed the data flow for the collapse of 4 legacy data warehouses into an ADLS
  • Designed the ELT strategy into an Analytics Sandbox and the Business Intelligence Facility (EDW)
  • Imported, exported data from various databases ORACLE, and MYSQL into HDFS using Talend.
  • Created Context Variables and Groups to run Talend jobs against different environments like Dev, Test and prod.
  • Loaded and transformed large sets of semi structured data likes XML, JSON, Avro, Parquet.
  • Integrated Custom Visuals based on business requirements usingPower BI desktop.
  • Instituted a process of continuous iterative exploration and investigation of business metrics to identify patterns and key performance indicators for data driven strategic decisions using AI, ML, and data mining techniques (predictive modeling, natural language processing).
  • Developed complex SQL queries using stored procedures, common table expressions (CTEs), temporary table to supportPower BIand SSRS reports uning ANSI SQL standards.
  • Performed creative interactive report applications using R shiny package.
  • Created Talend jobs to copy the files from one server to another and utilized Talend FTP components.
  • Worked with Terraform Templates to automate the Azure Iaas virtual machines using terraform modules and deployed virtual machine scale sets in production environment.
  • Performed data visualization using GGplot ly in R.
  • Written Templates for Azure Infrastructure as code using Terraform to build staging and production environments.
  • Integrated Azure Log Analytics with Azure VMs for monitoring the log files, store them and track metrics and used Terraform as a tool, Managed different infrastructure Cloud resources.
  • Good in technical assistance from design and development standpoint. Designs, tracks and coordinates project activities to meet project deliverable
  • Build and Deploy Data Pipelines onAzureto enableAI & MLcapabilities.
  • Used SSIS variables to capture start time, end time and failure of a package and execute the package dynamically
  • Designed SSIS packages to transfer the data from DB2, flat files and excel documents to the Staging Area in SQL Server 2012 and 2014.
  • Developed complex calculated measures usingData Analysis Expression language (DAX).
  • Used python modules of urllib, urllib2, Requests for web crawling. Experience using all these ML techniques: clustering, regression, classification, graphical models.
  • EmbeddedPower BIreports on SharePoint portal page and also managed access of reports and data for individual users using Roles.
  • Excellent knowledge of all the phases of Software Development Life Cycle (SDLC).
  • Experienced on Migrating SQL database toAzure data Lake, Azure data lake Analytics,Azure SQL Database, Data BricksandAzure SQL Data warehouseand Controlling and granting database access and Migrating On premise databases toAzure Data lake storeusing Azure Data factory.
  • Developed spark applications in Python (PySpark) on distributed environment to load huge number of CSV files with different schema in to Hive ORC tables.
  • Experienced in DevelopingSparkapplications usingSpark - SQLinDatabricksfor data extraction, transformation and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
  • Good understanding ofSpark Architectureincluding Spark Core, Spark SQL, Data Frames, Spark Streaming, Driver Node, Worker Node, Stages, Executors and Tasks.
  • Good understanding ofBig Data Hadoopand Yarn architecture along with various Hadoop Demons such as Job Tracker, Task Tracker, Name Node, Data Node, Resource/Cluster Manager, andKafka(distributed stream-processing). creating required Hive tables, data loading and writing hive queries.
  • Created Databricks notebooks using ANSI SQL, Python and automated notebooks using jobs.
  • Encoded and decoded json objects using PySpark to create and modify the dataframes in Apache Spark
  • Involved in converting Hive/SQL queries into Spark transformations using Spark RDD and Pyspark concepts.
  • Involved in data migration from On-Prem to azure cloud.
  • Created Partitions, Buckets based on State to further process using Bucket based Hive joins.
  • Successfully redesigned and migrated complex applications from Access to SharePoint and SQL Server 2008 by evaluating database structures and data movements.
  • Provided 24X7 production support to ensure timely and error free delivery of reports.
  • Worked on importing and exporting data from Snowflake, Oracle and DB2 into HIVE using Apache NiFi for Data analysis and transformations
  • Worked with different file formats like Json, AVRO and parquet and compression techniques like snappy. Nifi ecosystem is used
  • Used Time Intelligence Functions like Next Quarter, opening balance and START OF QUARTER to perform time analysis on the Patients Information.
  • Used NiFi data Pipeline to process large sets of data and configured Lookup’s for Data Validation and Integrity.
  • Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool and backwards.
  • Connected Azure SQL to Power BI and Built Power BI reports using SQL Azure as data source
  • Developed complex SQL queries using stored procedures, common table expressions (CTEs), temporary table to supportPower BIand SSRS reports.
  • Developed complex calculated measures usingData Analysis Expression language (DAX).
  • EmbeddedPower BIreports on SharePoint portal page and also managed access of reports and data for individual users using Roles.
  • Successfully redesigned and migrated complex applications from Access to SharePoint and SQL Server 2008 by evaluating database structures and data movements.
  • Developed and maintained multiple Power BI dashboards/reports and content packs
  • Prepared necessary pipelines to trigger containers with Azure Data Factory
  • Experienced in performance tuning of Spark Applications for setting right Batch Interval time, correct level of Parallelism and memory tuning.
  • Developed JSON Scripts for deploying the Pipeline in Azure Data Factory (ADF) that process the data using the Sql Activity.
  • Involved in monitoring, Scheduling and Managing the work flows Confidential Apache Airflow and
  • Managed the storage in Azure Data Lake storage (ADLS) by delivering the processed data using Highly optimized storage techniques.

Environment: Hadoop 2.6.0, Sqoop-1.4.6, Oozie-4.10, Hive-1.1, Impala-2.6, HBASE1.2 Apache Spark 2.2.1, Cassandra 3.1, RDBMS, Snowflake, HDFS, XML, JSON, Parquet, ANSI SQL, Avro, Apache Airflow, Azure Data Factory, Azure Synapse and Apache Nifi 1.6

Confidential - Eden Prairie, MN

Big Data Developer

Responsibilities:

  • Worked on Two projects parallelly, one is Taleo API Development and the other one is Salesforce REST API Dev and Prod Support
  • My preeminent and productive contributions to the project was recognized with a "DIAMOND" Award.
  • Integrating with multiple data sources likeTaleo SOAP API’S, WSDL’s and Salesforce RESTFUL API’stoextract the data.
  • Python 2.7 is used in the main script to Ingest tables from Taleo SOAP API’S.
  • Implemented Server-to-Server(S2S) Authentication based on token-based system using Java to access Remote REST API
  • Applied required spark transformations using PySpark and appropriate frameworks
  • Delivered the processed data to the down stream systems
  • Data validation between source (Oracle DB) and destination systems (MapR and HDFS).
  • Established a health check of the overall Hadoop platform using Ambari, in-house scripts, Attunity Enterprise Manager dashboards etc.
  • Exported necessary Spark Jars to run in the Hadoop clusters.
  • Created Stream setsflows to make the process data to reach downstream systems.
  • Loaded and transformedlarge sets of semi structured data.
  • Wrote Hive UDFs in Java and Python
  • Developed bash scripts to bring theTlog filesfromftp serverand then processing it to load into hive tables.
  • Oozie workflows are used to view the DAG and logs for Hive, Python and Spark jobs scheduled in both Dev and Prod
  • Used PostgreSQL as a database to run SQL as well as JSON queries.
  • Developed various automated scripts forDI (Data Ingestion) and DL (Data Loading)usingpython & java map reduce.
  • Workedwith different file formats likeJson,AVROandParquetand compression techniques likesnappy.
  • PerformedImport and Exportof data intoHDFS and HiveusingStreamsetsand managed data within the environment.
  • Worked on UDF tables where TXML files are converted to XML files using Java and XML parsers.
  • Createdrequired Hive tables,data loadingand writinghive queries.
  • Scheduled different PySparkjobs using Apache Airflow using Cron time format.
  • All the bash scripts are scheduled using Resource Manager Scheduler.
  • Developed the CDC pipeline for Data Warehouse, Using Attunity tool.
  • CreatedPartitions, Bucketsbased on State to further process using Bucket based Hivejoins.
  • UsedStreamsetsdata Pipeline to process large sets of data and configured Lookup’s for Data Validation and Integrity.
  • Prepared necessary pipelines to trigger containers withMapr
  • Monitored,Scheduled and Managedt heworkflowsat Apache Airflow, Oozie scheduler using Hueand
  • Managed the storage in HiveMetastore and Maprby delivering the processed data using Highly optimized storage techniques.

Environment: Hadoop 3.0, Pig 0.17, Hive 2.3, HBase, Sqoop, Maven, PostgreSQL, Python, Java, MapR, HDFS, ANSI SQL, Agile, Pyspark, Apache Kafka 2.0.0, Spark, Nifi, intellij, NoSQL, Cassandra 3.11, Java, MongoDB, Flume, XML, JSON, Elastic search, AWS, Kubernetes

Confidential - Irving, TX

Big Data/Nifi Developer

Responsibilities:

  • Developed ETL workflows using NiFI to load data into Hive and HDFS.
  • Worked on Migrating jobs from NiFi development to Pre-PROD and Production cluster.
  • Scheduled different Spark jobs using NiFi.
  • Worked on Azure Data Platform components - Azure Data Lake, Data Factory, Data Management Gateway, Azure Storage Options, DocumentDB, Data Lake Analytics, Stream Analytics, EventHubs, Azure SQL.
  • Created DataFrames from RDDs using reflection and programmatic inference of schema over RDD.
  • Good knowledge on handling medical data and building systems that conform to the compliance set by HIPAA
  • Responsible for handling large and real-time datasets using Partitions, Spark’s in-memory capabilities, Broadcast variables, Joins, and other Transformations Confidential the time of data Ingestion.
  • Developed and designed data integration and migration solutions in Azure.
  • Designed solution for various system components using Microsoft Azure.
  • Designed and Implemented Spark Jobs to be deployed and run on existing Active clusters.
  • Developed multiple Scala user defined functions (UDF’s) to fit specific analytical requirements.
  • Designed and Developed custom data flow using Apache Nifi to fully automate the ETL process by taking various worst-case scenarios into account.
  • Developed NiFiworkflow to pick up the multiple retail files from ftp location and move those to HDFS on a daily basis.
  • Used NiFi data Pipeline to process large sets of data and configured Lookup’s for Data Validation and Integrity.
  • CreatedNififlows totriggersparkjobs and used put email processors to get notifications if there are any failures.
  • Worked with different file formats like Json, AVRO and parquet and compression techniques like snappy. Nifi ecosystem is used.
  • SupportedMapReduce Programsthat are running on the cluster.
  • Involved in creatingHivetables, loading them with data, and writingHivequeries which ran internally inMapReduce.
  • Worked on a Spark Job to Convert fixed width files and CSV files data to Avro, parquet and .xml formats by applying transformations on the data based on business requirements.
  • Involved in writing code with Scala which has support for functional programming.
  • Developed Impala scripts for end user / analyst requirements for adhoc analysis.
  • Responsible for writing bash scripts to automate various small but time-consuming tasks such as user creation, importing multiple tables from RDBMS to HDFS using sqoop, setting up a development environment for new users.
  • Responsible for converting complex SQL queries from Teradata BTEQ scripts to SPARK SQL to leverage the distributed nature of the underlying spark RDDs

Environment: Hadoop 3.0, Scala 2.12, Spark, SQL, Hive 2.3, Scala, Cassandra 3.11, Oozie, Apache Nifi, Azure, Oracle 12c, RDBMS, HDFS, XML, JSON, Parquet, Avro, Spark Kafka, Snowflake, Sqoop

Confidential - Washington, DC

Big Data Developer

Responsibilities:

  • Developed Big Data solutions focused on pattern matching and predictive modeling
  • Involved in Agile methodologies, daily scrum meetings, spring planning.
  • Performed multiple MapReduce jobs in Pig and Hive for data cleaning and pre-processing.
  • Worked on MongoDB, HBase databases which differ from classic relational databases
  • Involved in converting HiveQL into Spark transformations using Spark RDD and through Scala programming.
  • Worked on 2 predictive AI models which were commercialized and deployed. The models would predict 4 hours and 1 hour respectively in advance and take corrective action before the degradation.
  • Integrated Kafka-Spark streaming for high efficiency throughput and reliability
  • Worked on Apache Flume for collecting and aggregating huge amounts of log data and stored it on HDFS for doing further analysis.
  • Worked in tuning Hive & Pig to improve performance and solved performance issues in both scripts
  • Developed Nifi flows dealing with various kinds of data formats such as XML, JSON, Avro.
  • Experience with container-based deployments using Docker, working with Docker images, Docker Hub and Docker-registries and Kubernetes.
  • Implemented use ofreal-time datausingStorm and Sparkas well asSpark ML.
  • Created Infrastructure in a Coded manner (Infrastructure as Code) usingTerraform.
  • Built Jenkins jobs to create AWS infrastructure from GitHub repos containingTerraformcode.
  • Managed different infrastructure resources, like physical machines,VMsand evenDockercontainers usingTerraform.
  • Built and managed a large deployment of RedHat Linux instances systems withChefAutomation and
  • Implemented automated local user provisioning VMs created inOpenstack and AWScloud through Chef Recipes.
  • Experienced Cloud and DevOps engineer in Automating, Configuring and deploying instances on AWS, AZURE Cloud environments and Data centers.
  • Involved working with aPaaSsolution such asRedHat Openshift.
  • Design of Cloud architectures for customers looking to migrate or develop newPaaS, IaaS, or hybrid solutions utilizing Microsoft Azure or Amazon Web Services (AWS).
  • Provisioned RedHat Enterprise Linux Server using PXE Boot according to requirements.
  • Design roles and groups using Azure Identity and Access Management (IAM)
  • Migrated SQLServer 2008 database to Windows AzureSQL Database
  • Configured and Managed in Hadoop Clusters for Spark streaming jobs using Microsoft Azure and along withSQL.
  • AWS and Open Stack to provision new machines for clients.S3Storage Services onAWS.
  • Hands on experience in installing and administrating CI tools like Hudson/Jenkins, Team city, Bamboo and ALM tools like Anthill Pro and TFS.
  • Performed integration of Code Quality Analysis Techniques like Find Bugs, PMD, Cobertura with CI tools.
  • Proficient with container systems likeDockerand container orchestration likeEC2 Container Service,Kubernetes, worked withTerraform.
  • ManagedDocker orchestration and Docker containerizationusingKubernetes.
  • UsedKubernetesto orchestrate the deployment, scaling and management of Docker Containers.
  • Worked on Proof of concept with Spark with Scala and Kafka.
  • Worked on visualizing the aggregated datasets in Tableau.
  • Worked on importing data from HDFS to MYSQL database and vice-versa using SQOOP.
  • Implemented MapReduce jobs in HIVE by querying the available data.
  • Configured Hive meta store using Amazon Aurora which is a MySQL and PostgreSQL compatible relational database that stores the metadata for Hive tables.
  • Design and Develop ETL Processes in AWS Glue to migrate Campaign data from external sources like S3, ORC/Parquet/Text Files into AWS Redshift.
  • Use of Docker and Kubernetes to manage micro services for development of continuous integration and continuous delivery.
  • Extensively used Maven, SVN as a code repository and Version One for managing day agile project development process and to keep track of the issues and blockers.
  • Created external tables with partitions using Hive, AWS Athena and Redshift.
  • Built ETL pipelines end to end from AWS S3 buckets to Key, Value storage like DynamoDB, and Snowflake Datawarehouse for analytical queries and specifically for cloud data.
  • Migrated an existing on-premises application to AWS. Used AWS services like EC2 Kinesis, S3 and Dynamo DB for small data sets processing and storage, Experienced in Maintaining the Hadoop cluster on AWS EMR.
  • Optimize the Pyspark jobs to run on Kubernetes Cluster for faster data processing
  • Implemented Maven Build Scripts for building the application.
  • Imported data from AWS S3 into Spark RDD, Performed transformations and actions on RDD's.
  • Implemented Elastic Search on Hive data warehouse platform.
  • Experience in Server infrastructure development on Gateway, ELB, Auto Scaling, Dynamo DB, Elastic search.
  • Worked with ELASTIC MAPREDUCE and setup Hadoop environment in AWS EC2 Instances.
  • Developed multiple MapReduce jobs to perform data cleaning and preprocessing.
  • Performance tuning of Hive queries, MapReduce programs for different applications.
  • Proactively involved in ongoing maintenance, support and improvements on Hadoop cluster.
  • Developed Spark code using Scala and Spark-SQL/Streaming for faster testing and processing of data.
  • Handled importing of data from various data sources, performed transformations using Hive, PIG, and loaded data into HDFS.
  • Involved in identifying job dependencies to design workflow for Oozie & YARN resource management.
  • Worked on data ingestion using Sqoop from HDFS to Relational Database Systems and vice-versa. Maintaining and troubleshooting
  • Exploring with Spark to improve the performance and optimization of the existing algorithms in Hadoop using Spark context, Spark-SQL, Data Frame, pair RDD's.
  • Created Hive Tables, loaded claims data from Oracle using Sqoop and loaded the processed data into target database.
  • Developed microservice on boarding tools leveraging Python and Jenkins allowing for easy creation and maintenance of build jobs and Kubernetes deploy and services.
  • Worked on deployment automation of all the micro services to pull image from the private docker registry and deploy to docker swarm cluster using Ansible.
  • Building/Maintaining Docker container clusters managed by Kubernetes Linux, Bash, GIT, Docker, on GCP
  • Worked with OpenShift platform in managing Docker containers and Kubernetes Clusters.
  • Worked on infrastructure with Docker containerization and maintained Docker Images and containers.
  • Implemented and managed Dynatrace on business social critical desktop and mobile applications.
  • Involved in PL/SQL query optimization to reduce the overall run time of stored procedures.
  • Worked on NoSQL support enterprise production and loading data into HBase using Impala and Sqoop.
  • Worked on analyzing Hadoop cluster and different big data analytic tools including Pig, HBase database and Sqoop.

Environment: Agile, Hadoop 3.0, Pig 0.17, Maven, HBase, Sqoop, AWS, Kubernetes, Dynamo DB, Kinesis, Hive 2.3, HDFS, NoSQL, Pyspark, Impala, YARN, intellij, PL/SQL, Nifi, Snowflake, XML, JSON, Avro, Spark Kafka, Tableau, PostgreSQL, MySQL, Apache Flume 2.3

Confidential

Process Associate/ETL Developer

Responsibilities:

  • Responsible for design and build data mart as per the requirements.
  • Extracted Data from various sources like Data Files, different customized tools like Meridian and Oracle.
  • Worked on Views, Stored Procedures, Triggers and SQL queries and for loading the data (staging) to enhance and maintain the existing functionality.
  • Done analysis of Source, Requirements, existing OLTP system and identification of required dimensions and facts from the Database.
  • Created Data acquisition and Interface System Design Document.
  • Designed the Dimensional Model of the Data Warehouse Confirmation of source data layouts and needs.
  • Extensively used Oracle ETL process for address data cleansing.
  • Developed and tuned all the Affiliations received from data sources using Oracle and Informatica and tested with high volume of data.
  • Responsible for developing, support and maintenance for the ETL (Extract, Transform and Load) processes using Oracle and Informatica PowerCenter.
  • Created common reusable objects for the ETL team and overlooked coding standards.
  • Reviewed high-level design specification, ETL coding and mapping standards.
  • Designed new database tables to meet business information needs. Designed Mapping document, which is a guideline to ETL Coding.
  • Used ETL to extract files for the external vendors and coordinated that effort.
  • Migrated mappings from Development to Testing and from Testing to Production.
  • Performed Unit Testing and tuned for better performance.
  • Created various Documents such as Source-to-Target Data mapping Document, and Unit Test Cases Document.

Environment: Informatica Power Center 8.1/7.1.2, Erwin 4.5, Oracle 10g/9i, Teradata V2R5, XML, PL/SQL, SQL Server 2005/2000 (Enterprise Manager, Query Analyzer), Sybase, SQL* Loader, SQL * Plus, Autosys, OLAP, Windows XP/NT/2000, Sun Solaris UNIX, MS Office 2003, Visio Project, Shell scripts.

We'd love your feedback!