Splunk On-Call (VictorOps)

Splunk On-Call (formerly VictorOps) makes expensive service outages a thing of the past.

It is as a central hub for alerts from multiple different sources, including Tideways.

Configuration

In your Splunk On-Call account, go into the Settings > Routing Keys section.

Either create a new routing key or write down an existing one to use with the Tideways integration.
In the Splunk On-Call "Integrations" > "REST" section, copy the integration URL, including the $routing_key variable:
In Tideways, open up the organization’s "Integration Settings" and select "Add New Integration". Select "Splunk OnCall" from the dropdown and fill the URL to notify and Routing Key fields with the information gathered in steps 1 and 2.

Select the Message Type to use in Splunk as either "CRITICAL" or "WARNING". Create two integrations for Splunk if both message types should be used for different alert notifications.

After saving, the "Preview" button allows testing the integration.

The Splunk integration can now be used in alert notifications for the organization.

Tideways attaches the following custom details to a Splunk On-Call alert:

Key

Message

Description

application

All

Name of the Tideways project in "org/project" format

type

All

Type of the alert (one of error_rate, response_time, …)

service

All

Service where the alert was triggered

environment

All

Environment where the alert was triggered

resposnse_time

response_time

Current response time

error_rate

Current failure rate

threshold

response_time, error_rate

Threshold the alert went over to get triggered

state_message

response_time, error_rate

Threshold the alert went over to get triggered

exception

new_exception

Class name of the exception being thrown

source

new_exception

File and line where the exception occurred

occurrences

new_exception

Number of times the exception occurred

Still need help? Email [email protected]