Cloud Solution Architect\lead Cloud Data Engineer Resume
Malvern, PA
SUMMARY
- 14+ years of IT industry experience with 4+years of Strong experience in AWS Services, Database migration from on premise to AWS RDS, cloud data architecture, Micro Services, AWS Infrastructure Deployment using CI/CD bit bucket, bamboo pipelines and Strong working experience in data analytics, Machine Learning, Sage maker, Data robot
- 4+ years of experience with AWS VPC, EC2, S3, EBS, RDS, ELB, SNS, AMI, IAM, EMR, Route 53, Cloud Watch, Aurora, DynamoDB and Lambda
- Proven experience and expertise in designing highly available architecture patterns, conducting architecture review meetings with executive management and implemented architecture\solution in production and stabilized production support process
- Deployed 100+ Aurora postgresql databases in production through CI\CD pipeline and migrated 500+ db2 mainframe tables data to 100+ Aurora postgresql databases for microservices
- Designed and implemented end - end Data Robot product in AWS infrastructure EC2 and enabled data science team to leverage data robot tool and perform feature analysis, Run Auto ML models and predict the output based on business use case and input data set in S3
- Ingested Data from multiple data sources(RDS postgresql databases, S3, Json files, DB2 tables ) into S3 Data lake (raw data layer), then applied transformation logic as per business driven lineage documents in a pyspark python program to transform the data and save it in parquet format in Target transformed S3 bucket. The transformation is being done on transient EMR clusters via pyspark logic
- Transformed data from multiple data sources is ingested in to AWS redshift warehouse for business to consume using tableau and create business reports
- Designed and implemented ECS kubernetes Architecture to migrate monolith services to micro services, Categorized application\group of services to use PODS in ECS containers which has auto scaling rules setup to scaleup and scale down horizontally as per traffic needs
- Deep knowledge and experience with reporting tools like Tableau, quick sight
- Good knowledge and understanding of ETL, Data warehousing, Snowflake, Master Data Management
- Deep understanding of Data modeling standards and practices to be followed to maintain logical data architecture and handson experience with ERWIN tool to perform data modelling as per business and data needs
- Automated manual tasks using shell scripting and python
- Proficient in ITIL methodologies, Production Support process, SLA Guidelines
TECHNICAL SKILLS
Technology: AWS RDS, Aurora postgresql, Redshift, Glue ETL,EMR, Spark, python, DB2 LUW 9.x, 10.x, DB2 BLU, cognos, Sql server,Oracle, No SQL, TSM Cloud, AWS RDS, EC2, S3, IAM,EBS, ELB, VPC, Route53,Cloudwatch, Lambda big data, data lake, Data pipeline, AWS, Kinesis, Lambda, DynamoDB, EMR, data architecture, Hadoop,, Hive, SQL, Sqoop, ETL, data modeling, database partitioning, data profiling, Replication, cloud formation, AMI, Elastic bean stack, High Availability, On premise-cloud database migration
Database Tools: pgAdmin, Workbench, Sql Developer, OEM, IBM GuardiumIBM OPTIM, Data Studio, DB2 Connect, SQL\Q- ReplicationOracle Golden gate
Operating System: AIX, Linux, Windows, Solaris, UNIX
ITIL Tools: Service Now, Maximo, Remedy
Monitoring Tools: Tivoli Monitoring, Net cool, Cloud watch
PROFESSIONAL EXPERIENCE
Confidential, Malvern, PA
Cloud Solution Architect\Lead Cloud Data Engineer
Responsibilities:
- Designed cloud formation template for Aurora postgresql DB cluster, primary & replica instances and deployed Aurora cluster across the environments
- Designed High availability Solution from RDS database perspective for application which needs 99.99% availability, Designed and implemented RDS, Route 53, Read replicas, Multi-AZ solution to span aws resources across availability zones
- Enabled Cloud data architecture for IIG and migrated data from DB2 tables to various micro service apps, cloud databases AWS Aurora PostgreSQL
- Proficient experience in writing cloudformation templates leveraging troposphere, terraform, boto3
- Currently leading a team of 4 cloud data engineers and delegating and assigning various stories as needed to accomplish project goals and MVPS, conducting knowledge sharing sessions with team to get team upskilled in all the Technical and functional areas
- Enabled Institutional SI Teams to spinup Aurora PostgreSQL databases and deploy micro services
- Designed and implemented AWS Sagemaker batch transformation from model development to predicting the output, the architecture designed and implemented has the entire pipeline to read raw data from S3 to EMR transformation cluster and transformed data will be stored from EMR to S3 bucket. Sagemaker reads input data set and performs batch transformation as per model deployed in ECS docker container and predicts output as per the designed model and trained data set, the predicted output will be consumed by Dash app via tableau
- Participated in RFP Eval proposals, Presentations, Performed POCs and Evaluated Data analytics Augmented Machine Learning tools like Data Robot, Domino Data labs, DotData, H2O.ai
- Determined metrics, designed Test strategy and performed performance testing, fail over testing for applications designed with PCF and connecting to RDS Multi-AZ instance and determined baselined to design high resilient application
- Designed RDS Snapshot restore strategy and designed cloud formation template for restoring an RDS Instance from snapshot.
- Designed cloud formation template to upgrade RDS instance from lower version(postgres 9.6) to high version(postgres 10.3) & upgrade instance class further as required
- Designed cloud formation template to move RDS Instance from one network segment to other
- Migrated DB2 mainframe tables to RDS PostgreSQL using Attunity Replication 6.1 & 6.2, created migration tasks for table full load\CDC process from DB2 Mainframe to RDS PostgreSQL for micro service
- Designed Cloud Operational Support model and stabilized teams to adhere to process and guidelines
- Performed ETL operations through AWS Glue which includes connecting to Source data sources S3, RDS and writing ETL job for transformation and loading the transformed data to target data sources like S3 or Redshift
- Utilized Attunity Enterprise Manager to monitor and maintain tasks associated with various Table migrations full load\CDC
- Designed a real time Data strategy to retrieve data from DB2 tables, transaction logs and load in to S3 buckets, perform data transfer\parsing using SQS and transform SQS messages based on FIFO and perform transformation in EMR cluster & load the transformed data to another S3 Data lake and then implemented EMR query cluster to connect to S3 data suing presto
- Designed RDS self-provisioning solution through CI/CD bit bucket /bamboo process and enabled SI teams to leverage bit bucket project to spin up RDS Instances using a button click approach
- Utilized Datarobot Auto ML tool to connect to S3 Data source and perform feature engineering, Data cleansing and run various Machine learning models and evaluate accuracy of models using AUC values and perform model selection process to deploy the model in production to predict the outcome as per business use case
- Written python Automated script to do data reconciliation between db2 and postgres tables
- Solid understanding of ETL architectures, data movement technologies, database partitioning, database optimization, and building communication channels between structured and unstructured databases.
Confidential, Malvern, PA
Senior Cloud Data Architect\Sr. Cloud Database Administrator
Responsibilities:
- Designed cloud formation templates for RDS (MySQL, postgres, Sqlserver) databases and spin up multiple databases for multiple SI app teams for micro services and client applications
- Designed a Solution to capture RDS Events and trigger an incident in service now for respective Service now Assignment group. RDS Instance is subscribed to an SNS Topic for event subscriptions, When an RDS event happens a lambda function will be triggered to capture RDS event message and route the RDS event notification to an SNS topic, SNS topic then routes the event message to Tivoli View port and Service now Assignment group and creates an incident in case of severity, otherwise creates a warning notification with required details.
- Designed cloud formation template for RDS Event monitoring solution and Implemented RDS Event monitoring solution across all LOBS and all AWS Accounts (Dev, TEST, Prod) through bamboo deployment process.
- Worked on AWS Service catalog, Created portfolio, products and launched RDS Instances using AWS Service catalog, Designed solution for Integrating Service now - AWS Service catalog.
- Designed strategy to update existing RDS Instances through update stack process and bamboo deployment process
- Designed strategy to implement lambda functions across the environments and incorporated with in cloud formation templates
- Implemented infrastructure as a code, designed cloud formation code, promoted code to various environments and spin up RDS instances with a single button click deployment approach
- Maintained infrastructure code in BITBUCKET, GITHUB repositories and promoted code through various environments (TEST, UAT, and PROD) through BAMBOO deployment process.
- Work specific to JIRA STORY, Create a branch through JIRA on bit bucket repository, modify the python code and cloud formation template in local STS and then push the changes to remote branch and then raise a pull to merge changes to master and get approvals and do the deployment in Test and production through Bamboo
- Designed and implemented cloud watch alarms and configured metrics for various RDS instances and incorporated cloud watch metrics with in the cloud formation template
- Automated database audit reports to trigger report of RDS instances and LOB details
- Setup lambda function through SNS Topic and configured Notification process whenever a new RDS Instance created, deleted, minor version upgrade, modified
- Designed self-service documents for application teams for usage of cloud formation template so that they would be able to spin up rds instances them selves
- Worked closely with application teams and helped in designing and providing various cloud solutions
- Worked on Aurora postgres and implemented read replica and high availability solutions
- Designed a solution for multi - region, multi-AZ RDS Instances for High availability and disaster recovery needs
- Configured cloud watch metrics using cloud formation template and routed alarms through sns topics and event subscriptions to Tivoli view port \omnibus and splunk and integrated with service now to generate automated incident
Confidential, Mclean VA
Lead Cloud Data Engineer\Lead Database Administrator
Responsibilities:
- Worked on AWS services like EC2, S3, RDS, IAM, EBS, ELB, VPC
- Involved in design and database migration strategy for on premise to AWS private cloud
- Design, develop, and implement Oracle database instances on AWS for the development and production environments
- Design, develop, and implement Oracle database instances on AWS for the development and production environments
- Backup and recovery (RMAN and traditional), Export and Import using data pump
- Test, validate and implement performance and resource optimization improvements in consultation with AWS development Teams
- Implemented cost optimization mechanism by designing lambda functions to auto shutdown RDS instances during non-business hours based on cloud formation tags info
- Hands-on experience working with large complex data sets, real-time/near real-time analytics, and distributed big data platforms
- Migrated around 500 databases from old to newer versions (V9.1, v.9.5 to v.9.7, 10.1, and 10.5) and applied FIXPAKS to keep DB2 at most recent level
- Good Knowledge on NoSQL Databases like Mongodb, Cassandra, HBASE, HIVE
- Loaded data into redshift tables using copy command from S3 files & Dynamo db tables .
- Leveraged data pipeline to move data in\out from redshift tables to RDS through unload\copy configured lambda to automatically copy files from s3 to redshift tables and automatically drop files from s3 buckets.
- Designed redshift tables using KEY Distribution style, used sort keys.
- Used WLM to work load management with in the cluster
- Worked on maintenance utilities like vaccum & Analyze and improved performance of rds mysql, redshift cluster tables
- Good knowledge of Building private AMI’s on VPC’s
- Migrated databases from on-premise oracle to cloud rds mysql\postgres
- Designed strategy and successfully migrated databases from on premise to AWS cloud.
- Leveraged data pipeline to move data in\out from redshift tables to RDS through unload\copy configured lambda to automatically copy files from s3 to redshift tables and automatically drop files from s3 buckets
- Designed Cloud Formation template to spin up redshift, EMR cluster
- Implemented cost saving mechanisms to auto shutdown nonprod RDS instances, EMR cluster, Redshift cluster during non-business hours\window where there is no usage
- Hands-on experience in MongoDB cluster environment and AWS redshift data warehouse
- Working on Remediation project and migrated DB2 Databases from 9.7 version to 10.5
- Migrated DB2 corporate data ware house DPF database of 45+TB and 33 logical partitions from AIX DB2 9.7 to Linux 10.5
- Worked with HPU unload\load to move data from source to target
- Worked on EMC advanced database backup & restore mechanisms using DDBOOST
- Designed and built new DPF DB2 10.5 database environments in Linux
- Migrated Oracle database databases from 11g to 12C
- Have done oracle database administration activities like Export Import using data pump
- Worked on oracle database backup & restore using RMAN
- Working with project teams to deploy their products into the AWS environment
- Supported Db2 Databases of Single family & Multi Family applications
- Performed Performance tuning, query tuning and improved performance of application
- Performed database administration activities like backup, refresh, Runstats, Reorg, Capacity planning, Query tuning, Database design & build, Database monitoring, Troubleshooting & resolving issues
- Performed performance tuning of database & improved performance of database
- Analyzed & troubleshoot issues using db2daiglog files, db2 tools like db2pd, db2top, snapshot
Confidential
Lead Database Administrator
Responsibilities:
- Proficient in DB2 Database installation and migration activity
- Experience with IBM ISAS Data warehousing housing (DPF), Managed ISAS BI Appliance database of 95TB
- Worked on Database Partitioning Features (DPF) Environment and worked on creation of range partitions and data partitions
- Experienced in implementation of IBM PDOA System
- Worked with Column based & Row based architecture tables using DB2 BLU
- Proficient in performance monitoring in DB2 and handling DB2 TSM backups
- Created DDL statements for database objects (tables, index, UDF, vies etc.)
- Set up DB2 HADR (provide High availability "HA" and scalability)
- Work closely with Application Developers, Data Modelers, Engineering, Security Administrators, Capacity Planning & Monitoring, Service Desk Scheduling and Network Administrators as needed
- Proficient in Performance tuning, oversee backups and create scripts for task automation
- Troubleshooting database issues using db2diag, db2pd, db2top, Snapshot and event monitors
- On call weekends and remote working 24x7x365
Confidential
Lead Database Administrator
Responsibilities:
- Migrated Databases from Old version to latest Versions
- Proficient in performance monitoring in DB2 and handling DB2 TSM backups
- Good Knowledge of AIX operating system and handling DB2 databases on AIX
- Performed Redirected restore using TSM & Flash copy backups methodologies
- Implemented backup & recovery strategies
- Complete database, and other IBM Informix software, installs and upgrades
- Expertise in using DB2 Movement Utilities like Export, Import, Load and db2move
- Experience in using DB2 Maintenance Utilities like Reorgchk, Reorgs and Runstats
- Performed SQL Query Tuning using db2explain and db2exfmt tools
- Troubleshooting database issues using db2diag, db2pd, db2top, Snapshot and event monitors
Confidential
Lead Database Administrator\Developer
Responsibilities:
- Performed Database Administration activities like Backup, Restore, Runstats and Reorg
- Proficient in performance monitoring in DB2 and handling DB2 TSM backups
- Proficient in Install, Configure and Manage of IBM DB2 in high available clustered environment
- Work closely with the application development team and run the data creation and data loads
- Proficient in the installation of DB2 pureScale software for a database cluster
- Migrate an existing DB2 9.7 database system to a DB2 pureScale clustered environment
- Configure database manager and database member options for a DB2 pureScale cluster
- Describe the use of multiple log streams for database members of a pureScale database
- Managed SAP Databases on SAP ECC, BW, CRM environments
- Experienced in working on BI Appliance - IBM pure data Analytics DB2 products like Netezza, ISAS
- Troubleshooting database issues using db2diag, db2pd, db2top, Snapshots and event monitors