Splunk Admin/ Production Support Analyst Resume
Owings Mills, MD
SUMMARY
- Overall, 8 years of IT experience and as ALM Tooling Engineer, Splunk Admin/Developer, Linux Admin and PL/SQL on varied projects which involves Design and Development of client/server. In platform consisting of Red Hat Linux, Windows, and Sun Solaris operating systems.
- Extensive experience in Installation, Configuration, and Migration, Troubleshooting and Maintenance of Splunk, Apache Web Server on different UNIX flavors like Linux.
- Experience with Splunk Searching and Reporting modules, Knowledge Objects, Administration, Add - On's, Dashboards, Clustering and Forwarder Management.
- Experience on Splunk Enterprise Deployments and enabled continuous integration on as part of configuration management using Ansible.
- Knowledge in using Amazon S3 as object storage built to store and receives any amount of data at a massive scale integrating as part of Splunk Frozen Bucket Archival & Restoration Process.
- Created and Managed Splunk DB connect Identities, Database Connections, Database Inputs, Outputs, lookups, access controls.
- Experience in setting up required load balancing and control between several Splunk Indexers in a Multisite Clustered Environment.
- Knowledge in setting up the Load balanced Splunk monitoring using Splunk TCP & HEC inputs with the help of the token authentication.
- Created Reports, Alerts and Dashboards by Splunk query language. Experienced in creating and running Cron Jobs for scheduled tasks.
- Managed Splunk Indexer Clusters including security, hot and cold bucket management and retention policies.
- Experience in using setting up monitoring for Splunk- Docker Logging Driver to forward the container logs to Splunk.
- Knowledge in delivering the Splunk monitoring solution for onboarding the SAP Cloud platform logs to Splunk using HEC.
- Experienced in Writing Ansible playbooks to automate our build/deployment process and do an overall process improvement to any manual processes of forwarder installation.
- Experience on Splunk Enterprise Deployments and enabled continuous integration on as part of configuration (props.conf, transforms.conf, output.conf etc...) management.
- Field Extraction, Using IFX, Rex Command and RegEx in configuration files.
- Knowledge in working with Splunk Authentication and permissions and having significant experience in supporting large scale Splunk deployments.
- Knowledge in using TFS for Continuous integration & deliver, Agile, Continuous Testing with the help for using Git Repositories.
- Good Knowledge in using JFrog Artifactory as an open-source project to speed up the development cycles using binary repositories and as an advanced repository manager creating a single place for teams to manage all the binary artifacts efficiently.
- Time chart attributes such as Span, Bins, Tag, Event types, Creating Dashboards, Reports using XML.
- Create Dashboard from search, Scheduled searches of Inline search vs scheduled search in a Dashboard.
- Experience on working with CIM in which it facilitates normalization of data from different sources, and enables applications developed by Splunk to search and display Deep Discovery logs.
- Scripting and development skills using Perl and Python with strong knowledge of regular expressions.
- Created interactive dashboards using Tableau to analyze data and can be shared, collaborated on data driven decisions.
- Proficient in PL/SQL programming - Stored Procedures, Functions, Packages, SQL tuning, and creation of Oracle Objects - Tables, Views, Materialized Views, Triggers, Sequences, Synonyms, Database Links, and User Defined Data Types.
- Worked with other IT teams, customers (users), and other managers in helping build and implement Systems and standards.
TECHNICAL SKILLS
Splunk: Splunk 5.x/6.x/7.x/8.x, Splunk Cloud, Splunk Enterprise, Splunk on Splunk, Splunk DB Connect, Splunk SIEM & ES, CIM
Web/App Servers: Apache Tomcat 7.x/6.x, WebLogic 11g/12c, WebSphere 7.0
Operating Systems: Unix/Linux (CentOS), Windows 2008/2003
Programming Language: C, C++, Java, Linux Shell Scripts, Oracle SQL and PL/SQL
Java/J2EE Technologies: J2EE, JSP, JDBC, JMS, Web Services
RDBMS: Oracle 11g/10g/9i/8i, MS-SQL Server 2005/2008/2008 R2, DB2, MS Access.
Web Technologies: HTML, DHTML, JavaScript, XML, XSL, XSLT
Web/App Servers: Apache Tomcat 6.0, web logic 8.1/9.2, web sphere 6.0
Tools: Artifactory v5.4.4, Tableau v10.2.1, Ansible Tower v2.4.1, TFS 2017, SonarQube v5.6.6
PROFESSIONAL EXPERIENCE
Splunk Admin/ Production Support Analyst
Confidential, Owings Mills, MD
Responsibilities:
- Installation and configuration of Splunk product at high level Multisite Clustered Environments.
- Configured Splunk Searching and Reporting modules, Knowledge Objects, Administration, Add-On's, Dashboards, Clustering and Forwarder Management.
- Responsible in handling the configurations and Load Balancing between different Indexers on a Multisite Clustered Environment.
- Worked on Securitysolutions (SIEM) that enable organizations to detect, respond and prevent these threats by providing valuable context and visual insights to help you make faster and smarter security decisions.
- Configured various Splunk Heavy forwarders in the Environment.
- Configured and Designed Architecture in Data indexing from various sources.
- Responsible in all On boarding process from various sources such as forwarders, HEC, API’s and with Network ports (TCP/UDP)
- Designed Access Controls for different data to each individual Application user groups.
- Responsible in configuration and deployments of different Splunk Addons.
- Configured Microsoft, Azure Addons to collect the data from different sources of Azure platform.
- Configured Splunk DB Connect to collect data and index into Splunk from SQL, MS SQL Databases.
- Configured ServiceNow in Splunk to have Incidents/Change request automatically create from the query results.
- Responsible in troubleshooting all Production issues of Splunk in environment.
- Developed Internal Application Addons using API’s and Python Scripts.
- Responsible in maintenance of Splunk Licensing for whole environment.
- Developed Ansible script to achieve automation of Splunk forwarder installation, On boarding and Deployments which saved more time in the environment.
- Integrate Amazon Cloud Watch with ECS logs for monitoring the log files and track metrics.
- Using Atlassian products like JIRA, Confluence for issue tracking.
- Used GitLab for all documentations and code integrations.
- Involved/Responsible in Splunk Buckets, KV Store and Indexer Clustering Maintenance.
- Responsible in Maintenance/Developments/Integrations of whole Splunk environment in T Rowe.
- Deployed Internal Certs to all our Splunk servers in environment to avoid Vulnerabilities.
- Involved in all Disaster Recovery Activities in environment.
- Involved in every 6 weeks On-call Rotations for the team.
Environment: Splunk Enterprise 8.x/9.x, Splunk Clustered/Distributed environment, Splunk Heavy forwarders/Universal forwarders, Splunk Addon for Servicenow, Splunk DB Connect, Splunk Addon for Microsoft windows 8.x, Microsoft Azure addon, Ansible Playbooks, python 3, Linux/Windows servers
Senior Splunk Engineer/ Splunk Developer
Confidential, Reston, VA
Responsibilities:
- Installation and configuration of Splunk forwarder on Application servers.
- Involved in Design and development of Application Dashboard Architecture.
- Responsible in On boarding data from various sources such as forwarder inputs, DB connect, Rest API, HEC, API calls.
- Involved on inputs.conf for onboarding and props.conf, transform.conf for event line breaking, timestamp extractions and time zone changes.
- Worked on syslog onboarding for few Applications.
- Configured various lookups such as lookup table files, automated lookups for extracting fields and integrated to Dashboards SPL.
- Worked on field extractions, calculated fields using Regex to extract the different fields that are to be used in Dashboard’s logics as per Business requirement.
- Developed different SPL query logics for various scenarios in Dashboard panels.
- Developed a well-designed Architecture for each Application Dashboard Development.
- Worked on many drilldowns, token values, Alerts, Reports, lookups, field extractions in each Dashboard.
- Expertise in writing complex Regex and Multikv command to extracts the fields from the log files.
- Hands on development experience in customizing Splunk dashboards, visualizations, configurations, reports and search capabilities using customized Splunk queries.
- Worked with different Application teams while onboarding the data such as involved in Java API’s, Json, DB Queries and network related data.
- OptimizedSplunkfor peak performance by splittingSplunkindexing and search activities across different machines.
- Worked and customized completed SQL queries while onboarding into splunk using Splunk DB connect.
- Worked with Splunk Addon for Rest API to get Rest API data into Splunk.
- Involved in troubleshooting various Dashboards or data related performance issues.
- Interact with Business users after each Dashboard development to explain the use cases in the Dashboards
- Created many Documentations to each Application teams after development such as technical Documentation, user Guide Documentations.
Environment: Splunk 7.x/8.x, Splunk DB Connect, Splunk Addon for Rest API, XML, Json, python 3
Senior Splunk Engineer/ Production Engineer
Confidential, Owings Mills, MD
Responsibilities:
- Installation and configuration of Splunk product at high level Multisite Clustered Environments.
- Configured Splunk Searching and Reporting modules, Knowledge Objects, Administration, Add-On's, Dashboards, Clustering and Forwarder Management.
- Responsible in handling the configurations and Load Balancing between different Indexers on a Multisite Clustered Environment.
- Worked on Securitysolutions (SIEM) that enable organizations to detect, respond and prevent these threats by providing valuable context and visual insights to help you make faster and smarter security decisions.
- Configured various Splunk Heavy forwarders in the Environment.
- Configured and Designed Architecture in Data indexing from various sources.
- Responsible in all On boarding process from various sources such as forwarders, HEC, API’s and with Network ports (TCP/UDP)
- Designed Access Controls for different data to each individual Application user groups.
- Responsible in configuration and deployments of different Splunk Addons.
- Configured Microsoft, Azure Addons to collect the data from different sources of Azure platform.
- Configured Splunk DB Connect to collect data and index into Splunk from SQL, MS SQL Databases.
- Configured ServiceNow in Splunk to have Incidents/Change request automatically create from the query results.
- Responsible in troubleshooting all Production issues of Splunk in environment.
- Developed Internal Application Addons using API’s and Python Scripts.
- Responsible in maintenance of Splunk Licensing for whole environment.
- Developed Ansible script to achieve automation of Splunk forwarder installation, On boarding and Deployments which saved more time in the environment.
- Integrate Amazon Cloud Watch with ECS logs for monitoring the log files and track metrics.
- Using Atlassian products like JIRA, Confluence for issue tracking.
- Used GitLab for all documentations and code integrations.
- Involved/Responsible in Splunk Buckets, KV Store and Indexer Clustering Maintenance.
- Responsible in Maintenance/Developments/Integrations of whole Splunk environment in T Rowe.
- Deployed Internal Certs to all our Splunk servers in environment to avoid Vulnerabilities.
- Involved in all Disaster Recovery Activities in environment.
- Involved in every 6 weeks On-call Rotations for the team.
Environment: Splunk 7.x/8.x, Splunk DB Connect, Splunk Addon for Microsoft windows 8.x, Microsoft Azure addon for splunk 3.x, Ansible 2.3x, python 3
Senior Splunk Engineer
Confidential, Norfolk, VA
Responsibilities:
- Design, support and maintain large Splunk environment in a highly available, redundant, dispersed environment.
- Requirement gathering and analysis. Interacted with team members and users during the design and development of the applications andSplunkObjects.
- Involved as a Splunk Admin in capturing, analyzing and monitoring front end and middle ware applications.
- Installation ofSplunkEnterprise,Splunkforwarders,SplunkIndexer, Apps in multiple servers with automation.
- Designing and maintaining production-quality Splunk ILPD dashboards for monitoring the feeds coming from several McAfee ePO servers.
- Participated in the creation of a DevOps enabled automated CI/CD solution that delivers tools and processes for rapid application development efforts and on-going production operations
- Knowledge in automating the Clustered Splunk Enterprise Upgrade using playbooks running on the Ansible Tower.
- Developed Splunk Dashboards, searches and reporting to support various internal clients in Security, IT Operations and Application Development.
- Responsible in handling the Failover/Failback of Splunk Instances between two Confidential during the Disaster Recovery plan of any major shutdowns on a particular site.
- Designed and implemented scalable, secure Splunk cloud architecture based on Amazon Web Services (AWS).
- Responsible in making an automated setup for moving the Splunk data into the AWS storage point making more reliable architectural design.
- Responsible in shifting the Splunk Environment into Cloud platform making cost efficient and high level architecture available in place.
- Created clustered & non-clustered indexes for increasing the performance, also monitored the indexes by troubleshooting any corrupt indexes by removing fragmentation from indexes.
- Worked on Securitysolutions (SIEM) that enable organizations to detect, respond and prevent these threats by providing valuable context and visual insights to help you make faster and smarter security decisions.
- Created several playbooks for the automating Splunk forwarder installations using Ansible Tower.
- Analyzed security based events, risks and reporting instances. Created Ansible playbooks to deploy new software and plugins as well as manage deployments to Splunk Instance.
- Pre-configureddata models to the data at search time using CIM (Common Information Model).
- Drive complex deployments of Splunk dashboards and reports while working side by side with technical teams to solve their integration issues.
- Managed Indexer Clusters including security, hot and cold bucket management and retention policies.
- Troubleshooting and resolve the Splunk - performance, search poling, log monitoring issues; role mapping, dashboard creation etc.
- Responsible for documenting the current architectural configurations and detailed data flow and Troubleshooting Guides for application support.
- Splunk configuration that involves different web application and batch, create Saved search and summary search, summary indexes.
- Designed various types of charts, alerts settings, app creations, user and role access permissions.
- Managing indexes and cluster indexes, Splunk web framework, data model and pivot tables.
- Good experience in Splunk, WLST, Shell scripting to automate and monitor the environment routine tasks.
- Managing the Task board in TFS which is useful in facilitate meeting and visualize the progress of the daily activities.
- Managing the kanban board in TFS for monitoring both Product Backlog items and Bugs as well.
- Responsible in supporting Devops Tools like Artifactory, Tableau, SonarQube etc. as part of ALM support process.
Environment: Splunk 6.x, Splunk DB Connect and other modules, TFS 2107, SonarQube V5.6.6, Tableau 10.2.1, Artifactory v5.4.4, Ansible Tower 2.4.1
Senior Splunk Admin/Developer
Confidential, St.louis, MO
Responsibilities:
- Installation and configuration of various components like indexer, forwarder, search head, deployment server, Universal and Heavy forwarder.
- Provide Regular support guidance to Splunk project teams on complex solution and issue resolution.
- Helping application teams in on-boarding Splunk and creating dashboards/alerts/reports etc.
- Most of the time worked to install Universal Forwarders but we have heavy forwarders set up to see data from syslog server side.
- Worked on installing Universal Forwarders and Heavy Forwarders to bring any kind of data fields in to Splunk.
- Involved in admin activities and worked on inputs.conf, index.conf, props.conf and transform.conf to set up time zone and time stamp extractions, complex event transformations and whether any event breaking.
- Involved in standardizing Splunk forwarder deployment, configuration and maintenance across UNIX and Windows and Created Linux containers in CoreOS and Automated system using chef platforms.
- Monitored Database Connection Health by using Splunk DB connects health dashboards, JBoss, Apache Tomcat.
- Integrate Service Now with Splunk to consume the alerts from Splunk and create service now tickets.
- Created Chef driven configuration of user account in Splunk and installed packages on Chef to manage the attributes.
- Involved in writing complex IFX, Rex and Multikv command to extracts the fields from the log files.
- Created Dashboards for various types of business users in organization and worked on creating different Splunk Knowledge objects like Macros, IFX, Calculated fields, Tags, Event Types and Look ups.
- Field Extraction, Using IFX, Rex Command and Reg Ex in configuration files.
- Use techniques to optimize searches for better performance, Search time field extractions. And understanding of configuration files, precedence and working.
- Troubleshooting of searches for performance issues by adding lookups, correct joins and using summary indexes.
- Created & deployed a tool to automate branch &project creation in SVN using Perl, Chef & Ansible scripts
- Performed troubleshooting and/or configuration changes to resolve Splunk integration issues.
- Scripting and development using Perl and Python.
- Creating and managing apps, Create user, roles, Permissions to knowledge objects.
Environment: Splunk 6.x, Splunk DB Connect and other modules, Oracle 9i/10g, Solaris 10, Sun One Web Server 6.0, Apache 2.x, python.
Splunk Admin/Developer
Confidential, Birmingham, AL
Responsibilities:
- Involved in accessing and normalizing data from multiple sources toSplunkindexer.
- Gathering various sources of syslog and XML data from devices, applications, and data bases.
- Perform daily health checks and maintain integrity of production environment by proactively resolving services impacting incidents.
- SetupSplunkForwarders for new application tiers introduced into environment and existing application.
- Work closely with Application Teams to create newSplunkdashboards for Operation teams.
- Identify pattern and trends that are indicators of routine problems.
- Troubleshooting and resolve theSplunk- performance, log monitoring issues, role mapping, dashboard creation etc.
- OptimizedSplunkfor peak performance by splittingSplunkindexing and search activities across different machines.
- Using Search Processing Language (SPL) created Visualizations to get the value out of data.
- Managed Indexer Clusters including security, hot and cold bucket management and retention policies.
- Created EVAL Functions where necessary to create new field during search run time.
- Used Ifx, Rex and Regex commands for field extraction.
- ConfiguredSplunkSearching & Reporting modules, Knowledge Objects, Administration, Regex, Dashboards, Clustering and Forwarder Management.
- Involved in writing complex IFX, Rex and Multikv command to extracts the fields from the log files.
- Create Dashboard, Reports and Alerts for events and configure alert mail.Hands on development experience in customizing Splunk dashboards, visualizations, configurations, reports and search capabilities using customized Splunk queries.
- Worked on Splunk architecture and various components (indexer, forwarder, search head, deployment server), Heavy and Universal forwarder, License model.
Environment: Splunk6.0.1, Linux, UNIX, Oracle 11g, MS SQL Server 2012, SQL, XML, Java Script, MS Excel, MS Power Point.
Splunk Admin
Confidential, Menomonee Falls, WI
Responsibilities:
- Design, support and maintain large Splunk environment in a highly available, redundant, dispersed environment.
- Requirement gathering and analysis. Interacted with team members and users during the design and development of the applications andSplunkObjects.
- Involved as a Splunk Admin in capturing, analyzing and monitoring front end and middle ware applications.
- Installation ofSplunkEnterprise,Splunkforwarders,SplunkIndexer, Apps in multiple servers with automation.
- Provided architecture validation testing, troubleshooting on issues such as Out of Memory, 100% CPU Usage hung Thread sessions, session replication, JVM Crashes.
- CreatedSplunkapp for Enterprise Security to identify and address emerging security threats through the use of continuous monitoring, alerting and analytics.
- In-depth knowledge of log analysis generated by various systems including security products.
- Assisted internal users of Splunk in designing and maintaining production-quality dashboard.
- Created Dashboards to web application admins to show user flow for End-to-End Architecture. Through which they can detect the Workflow for particular user session.
- Use techniques to optimize searches for better performance, Search time vs Index time field extraction. And understanding of configuration files, precedence and working.
- Created advanced dashboards, alerts, reports, advanced Splunk searches and visualization in Splunk enterprise.
- Installing and configuring Splunk Forwarder on both Linux and Windows servers.
- Created Splunk simple XML Dashboards with various visualizations like Pie / Bar / Line / Area charts.
- Efficiently handled Work Load Management for load balancing and failover, improving performance, reliability and scalability.
- Responsible for administering, maintaining and configuring a 24 x 7 highly available, Splunk apps for production portal environment.
- Worked as a key point of contact and advise application support team during performance and outage problems.
Environment: Splunk 6.1.x, 6.2.x, XML, SPL, Shell Scripting, Unix/Linux, Windows.
