SUMMARY

• Senior Azure/Cloudera Administrator with over 18+ years of IT experience and worked on various technologies of Java/J2EE, Cloudera and Azure Cloud Computing.
• Built and deployed various projects with Azure resources, Nifi/Kafka, Cloudera/Hortonworks Big Data Platforms.
• Coordinated with stake holders, user groups, gathered requirements, and validated capacity requirements for clusters.
• An organized team player with excellent communication skills, possess strong technical and analytic skills, business processes knowledge and provided technical solutions to enterprise level clients.
• A professional Azure/Cloudera administrator with around 7 Years of progressive experience.
• Have done installation, configuration, administration & management of complete HADOOP Services (HDFS, YARN, MRv2, HIVE, Knox, Ranger, Sqoop etc.) via Ambari Server, Agents and its Clients.
• Worked on creating/managing and provisioning access to Azure resources SQL Server instances, Storage Accounts, Containers, Synapse instances, Data Factory, Databricks, Key Vaults and AML Compute instances.
• Created action groups for Azure resources using Azure Monitoring and Alerting.
• Performed automation tasks in Azure PowerShell, Azure CLi and JSON form ARM Templates.
• Worked on Load balancer setup, Disaster Recovery setup of clusters in Ambari/Hadoop.
• Worked on Creating/Renewing SSL Certificates for the servers in a cluster and worked on Kerberos accounts in working for all domains in organization.
• Worked on Git Hub repositories and involved in setting up CI/CD using Ansible.
• Worked on Service NOW for Bug tracking & ticket processing (SR, INC, PRB, INC), JIRA, and legacy remedy systems.
• Proficient in Dashboard creations on Service Now.
• Proficient in creating alert policies on New Relic to monitor all production clusters and integrated Microsoft teams to receive critical alerts on it.
• Involved in Cloud vs Hybrid vs On - prem cluster architecture.
• Worked on various Databases including SQL Server, Oracle, Hive & also used various Data Sources for creating dashboards & dynamic reports for analysis.
• Extensive Knowledge on Data Warehousing Concepts right from Data Acquisition to Information Delivery including Extraction, Transformation & Loading.
• Exposure in IT domain across various facets of Project Management such as Analysis, Customer Support and Systems Integration.
• Exceptional communication, collaboration & team building skills with proficiency at grasping new technical concepts quickly and utilize the same in a productive manner.
• Co-ordinating the testing activities during UAT and Post Implementation.
• Experience of providing accurate Insights based on the reports created and ensuring proper coordination with offshore/onshore teams and managing project deliverables.
• Secured the HDP cluster using Kerberos and used Ranger for authorization and audit of user access.
• Experience working in Agile Scrum Software Development methodology integrating with CI/CD best practices for efficient software delivery solutions.
• Used GitHub + Jenkins setup for UAT and production systems deployments.
• Proficient in multi platforms like Unix/Linux/Solaris/Azure environments, with hands-on expertise in scripting and systems integration.
• Experience in various methodologies like Waterfall and Agile.
• Extensive work analyzing Hadoop stack and various Big Data analytic tools, including Pig and Hive, HBase database and Sqoop
• Creation of Hive databases as per service requests. Granting appropriate permissions using the Ranger policy, with additional responsibility for troubleshooting performance issues in HDFS/Hive/Hive2/beeline.
• Played a pivotal role in analyzing system failures, identifying root causes, and proposing courses of remedial action.
• Designed and implemented disaster recovery solution in HDP environment.
• Engage in collaboration with the Data Engineering team to plan and deploy effective new Hadoop environments and successfully expand existing Hadoop clusters (HDP 2.6.3)
• Ongoing cluster maintenance, including on-demand commissioning and decommissioning data nodes, cluster monitoring, troubleshooting, managing and reviewing jobs and Hadoop/Spark log files.
• Involved in creating Hive managed and external tables, and then applied HiveQL on those tables for data validation.
• Wrote shell scripts for cluster space management, data sanity check.
• Responsible for code deployment changes to UAT/PROD using Jenkins.
• Supported and configured the Azure PaaS services like Azure Key Vault, Azure Databricks, Azure BLOB and Azure Data Factory.
• Excellent communication and interpersonal skills and involved in client interactions for scoping, effort estimates and status reporting.
• Experience working in fast-paced, deadline-oriented environments.

TECHNICAL SKILLS

Databases: Oracle 9x/10g, SQL Server 2000, MySQL, DB2

Configuration Management: Visual Source Safe, ClearCase, CVS, SVN, Maven

OS: Red Hat Linux 7.2, Fedora Core 2.0/4.0, Ubuntu 5.10 Linux, Win-/95/98, Sun Solaris 5.8

Design Tools: Enterprise Architect, Rational Rose 6.1

IDEs: Eclipse 3.5, IBM Eclipse 3.0.1, IBM Rational Application Developer 6.0, IBM Rational Rose

Application Servers: IBM WebSphere Application Server 6.0, Web Logic 10.3 Languages/Frameworks C, C++, Java/Java 8, SQL, PL/SQL, Hibernate, Spring, Junit, Struts, JSF

Internet & Big Data Technologies: XML, JSP, Servlets, SOAP, Hadoop Cloudera Manager Services, HDFS, Hive, Sqoop, Hue, Zookeeper, Pig, Spark, Kafka, Nifi, HBase, Flume, Yarn, Tez

Big Data Tools: Podium Data Ingestion Tool, Talend Tool for Extraction.

CloudSkills: Microsoft Azure Data Factory, Microsoft Azure Key Vault, Azure BLOB, Azure Databricks.

Hadoop Distribution: Cloudera distributions CDH 5x, Hortonworks Distribution 2.5.3/2.6.0

Shell Programming: Bash, C, SH, KSH, Power Shell

Tools: /Utilities: Azure DevOps, Azure CLI, AzCopy,Storage Explorer, JIRA,GIT Hub, Ansible, Jenkins, Confluence, Azure Monitoring, New Relic for cluster monitoring and Alerting.

PROFESSIONAL EXPERIENCE

Confidential

Senior Data Engineer

Responsibilities:

• Successfully performed patching and upgrades of the Hadoop clusters using Ambari as well as Azure best practices.
• Also performed rolling restart of all the Kafka clusters to have minimal impact on users.
• Commissioning, Decommissioning, Balancing and managing nodes and tuning servers to gain optimal performance of the cluster.
• Regular block rebalancing of HDFS in all the clusters
• Creating Shell scripts for automating the alerts in the Hadoop Cluster.
• Analyzing Hive queries to increase the query performance.
• Managing Hadoop cluster using Horton Works HDP 2.6.5.294
• Working on ServiceNow Incidents and Change requests to perform root cause Analysis on the issues faced by the Business users.
• Tuning of Resource Pool Queues for providing Cluster Resources to various critical applications.
• Hadoop Platform Consultation and administration using Cloudera Manager/Ambari.
• Adding and removing new nodes to the cluster for scalability.
• Working with the Users to optimize the Hive queries for better utilization of the Cluster resources.
• Working on configuring the YARN queues for the allocation of resources to high priority applications.
• Using Yaml scripting to implement automation in the Hadoop Cluster.
• Creating, deleting and configuring Kafka Topics as per the requirements of the stakeholders
• Took ownership of larger support requests, including effort estimation, resource conflict resolution and time management; assuming responsibility for following existing processes to communicate updates to Manulife customers in a timely manner.
• Managed more than 650 nodes on on-Premises cluster
• Rotational On-Call: Handled IMs/hardware failures, engaging vender support as needed
• Responsible for assisting clients in strategic decision-making regarding disaster recovery and data center transformation.
• Successfully developed and implemented a disaster recovery methodology for new complex applications
• Performed semi-annual DR exercises with Database OS, SAN, Middleware and application teams, resulting in the successful recovery of over 300 servers
• Worked on data and jobs migration from on-prem Cloudera clusters to Microsoft Azure cloud
• Experience in DevOps practices and Jenkins/Azure Pipeline CI/CD pipelines, operations support
• Scheduled jobs using Oozie, CA WLA schedulers.
• Experience in leading, managing group of data engineers, QAs Working on a combination of on-prem and cloud setup.
• I am leading a group of 4 resources from onshore and 3 resources from offshore.
• Install Kerberos for added security and use Ranger for authorization to Hive tables.
• Debug the HDP (HDFS, YARN, Hive, Spark, etc.), HDF (NiFi/Kafka) and HDB cluster service performance and preventing outages for business critical applications.
• Configure and monitor cluster usage in terms of HDFS use, YARN memory consumption, Hive Server2 Interactive (LLAP) configuration tuning, host disk usage, log rotation polies etc.
• Query timings were reduced by almost 85% after tuning the servers and LLAP configurations.
• Used Jenkins and GitHub to facilitate Continuous Integration and version control.
• Created Service Now Dashboards for better tracking of Incidents/Requests/Change Requests and Problem Incidents.
• Generated multiple New Relic alert polices to monitor Enterprise Data Lake clusters.

Confidential

Full Stack Java/Hadoop Lead

Environment: Hadoop Framework, HDFS, Hive, Sqoop, Hue, Zookeeper, Big Data Cloudera CDH Apache Hadoop, Scala, Java 8, Spark, GIT, Bit Bucket, JIRA, SQL plus, Oracle Enterprise Manager (OEM), Spark, Pig, Kafka, Shell Scripting, Python Scripting, Podium Data Ingestion, Sun Solaris Linux environment, Talend TAC, AutoSys Scheduling, Rest Webservices, Python, Docker, Kubernetes Orchestration Framework.

Responsibilities:

• Involved in the Analysis, System architecture design and JAD sessions.
• Involved in ETL, Data Integration and Migration. Imported data using Sqoop to load data from Oracle to HDFS.
• Developed various automated scripts for DI (Data Ingestion) and DL (Data Loading) using python.
• And created python scripts to split the variable length mainframe files based on the record length to make it a fixed length split file and perform data ingestion as fixed length file on Podium Tool.
• Implemented Spark using Scala and Spark SQL for faster testing and processing of data.
• Implemented Spark using Scala and with Data frames and Spark SQL API for faster processing of data.
• Used Spark-SQL to load JSON data and create Schema RDD and loaded it into Hive Tables and handled Structured data using SparkSQL.
• Involved in converting Hive/SQL queries into Spark transformation using Spark RDDs, Scala and Python.
• Designed and implemented by configuring Topics in new Kafka cluster in all environments.
• Successfully secured the Kafka cluster with Kerberos
• Communicate with line of Business, gathering the Business requirements including the IDC (Incoming Data Collection) for Data Ingestion.
• Developed Spark programs using Scala APIs to compare the performance of Spark with Hive.
• Created Dual Ingestion workflow to ingest the data from enterprise data sources using both Java and using Podium Tool.
• Developed Talend Jobs to extract data to downstream systems following development standards and Git Lab Usage standards.
• Used Avro, Parquet and ORC data formats to store into HDFS.
• Developed Autosys jobs to schedule and integrate the Sqoop in jobs present on Cloudera Cluster.
• Generate, deploy and Schedule jobs using Talend Administration center using Meta Servlet API.
• Development to expose Source detail services to other components using RESTful Webservices.
• Experience designing and creating robust and secure RESTful web API and microservices.

Confidential

Senior Java/Hadoop Consultant Lead

Environment: Java8, Hadoop Framework, Cloudera Manager Services, HDFS, Hive, Sqoop, Hue, Zookeeper, Big Data Cloudera CDH Apache Hadoop, Spark, GIT, JIRA, SQL plus, Oracle Enterprise Manager (OEM), Spark, Pig, Shell Scripting, Talend Studio, Sun Solaris Linux environment, SSH, TLS, DNS (Networking).

Responsibilities:

• Responsible for Cluster maintenance, commissioning and decommissioning Data nodes, Cluster Monitoring, Troubleshooting, Manage and review data backups, Manage & review Hadoop log files.
• Monitoring systems and services, architecture design and implementation of Hadoop deployment, configuration management, backup, and disaster recovery systems and procedures.
• Installation of various Hadoop Ecosystems and Hadoop Daemons.
• Installation and configuration of Sqoop, Flume & HBase
• Managed and reviewed Hadoop Log files as a part of administration for troubleshooting purposes. Communicate and escalate issues appropriately.
• As an admin followed standard Back up policies to make sure the high availability of cluster.
• Involved in Analyzing system failures, identifying root causes, and recommended course of actions.
• Documented the systems processes and procedures for future references.
• Worked with systems engineering team to plan and deploy new Hadoop environments and expand existing Hadoop clusters.
• Configuring Load Balancer and F5 switches on Hadoop Cluster.
• Setting up SSH, HTTP, SMTP Protocols on Hadoop clusters using Linux.
• Monitored multiple Hadoop clusters environments using Zabbix. Monitored workload, job performance and capacity planning using Cloudera Manager.
• Involved in Installing and configuring Kerberos for the authentication of users and Hadoop daemons.
• Achieved customer prescribed SLAs successfully using Service Now and Jira ticket incident management.