Your Canary Console is capable of posting incidents from your Canaries, to an instance of Splunk that you control. The Canary Console will post to Splunk's HTTP Event Collector (HEC) which is a data input that comes standard with all Splunk installations. There are separate sections below because setups differ for Splunk Enterprise and for Splunk Cloud users (Note, Splunk Cloud Free trial accounts are not currently supported).

Setup steps for Splunk Enterprise users

Firstly ensure that the HEC is set up on your Splunk Cloud instance by following the instructions here.  

NOTE: when creating the new HEC token that you will use, use the picture below as the guide for "Input Settings":

Enable Splunk Enterprise HEC SSL (you can skip this section if your Splunk HEC already has SSL enabled)

We require that your HEC has SSL enabled so that communications between your Canary Console and your Splunk instance are kept secure and confidential. 

In order to ensure your Splunk Enterprise instance is making use of  SSL certificates, you can follow this link to get a certificate that is signed by a trusted third-party Certificate Authority (CA).

It is preferable to use a certificate from a trusted third-party CA, but if you would like to use a self-signed certificate, then your Canary Console does support this. However, you would need to contact support@thinkst.com in order for us to get your Canary Console to explicitly trust the CA that you used when creating the self-signed certificate. The instructions to get self-signed certificates on your Splunk Enterprise instance are here.

Once you have your certificates, you can follow this link to ensure that they are formatted correctly to be used by Splunk.

When your certificates are in the correct format, you will need to follow this either of these links to configure your Splunk Indexer or Splunk Forwarder to use the new certificates you obtained above. Once this is done, then your Splunk HEC will make use of the same certificate as your Splunk Management system.

Alternatively, you could have the certificate be specific to only your Splunk HEC, in order to do that you will need to be able to edit the following file on your Splunk instance:

$SPLUNK_HOME/etc/apps/splunk_httpinput/local/inputs.conf

In that file you will need to set the following configuration values:

[http]  
 enableSSL = 1
 serverCert = <full path to your certificate chain pem file>
 sslPassword = <password for server key used in chain>

You will need to restart the Splunk process in order for this new configuration to take effect. You can do this via the CLI using the following command on your Splunk instance:

$SPLUNK_HOME/bin/splunk restart splunkd

Details to fill in on your Canary Console

Once you have HEC enabled on your instance and it is configured to use a trusted SSL certificate, then you can use the UI on your Canary Console to input the details required so that incidents get posted to your Splunk HEC.

When logged into your Canary Console, you can navigate to the Global Settings menu like seen below

Navigate to the Webhooks section:

And fill in the details for the Splunk section:

The fields are explained below:

• Port - The network port that your Splunk HEC is listening on (default is 8088 on Splunk Cloud free trials & 443 by default on Splunk Cloud instances)
• 8088 but could be changed in configuration files)
• Hostname/IP - The DNS hostname or the IP network address of the Splunk instance where the HEC is running. NOTE, the hostname/IP must match the Common Name that is expected for the SSL certificate that your HEC is configured to use.
• HEC token - The HEC Event Collector token that you intend to use, is required in order for the Canary Console to have the correct authorisation and authority to post to your HEC. More info on HEC collector tokens can be found here.

Clicking the "Add" button will then cause your Canary Console to first attempt to send a test event in order to ensure that it can correctly post to the Splunk HEC. Only once a test event gets sent successfully, do the Splunk HEC details get saved on your Canary Console. 

Setup steps for Splunk Cloud users

Firstly ensure that the HEC is set up on your Splunk Cloud instance by following the instructions here.

NOTE: when creating the new HEC token that you will use, use the picture below as the guide for "Input Settings":

Details to fill in on your Canary Console

Once you have HEC enabled on your instance, then you can use the UI on your Canary Console to input the details required so that incidents get posted to your Splunk HEC.

When logged into your Canary Console, you can navigate to the Global Settings menu like seen below

Navigate to the Webhooks section:

And fill in the details for the Splunk section:

The fields are explained below:

• Port - The network port that your Splunk HEC is listening on (default is 8088 on Splunk Cloud free trials & 443 by default on Splunk Cloud instances)
• Hostname/IP - The DNS hostname or the IP network address of the Splunk instance where the HEC is running. NOTE, the hostname/IP must match the Common Name that is expected for the SSL certificate that your HEC is configured to use. NOTE on Splunk Cloud, this always starts with "http-inputs-" as a prefix, eg: 

  http-inputs-<yourstackname>.splunkcloud.com

• HEC token - The HEC Event Collector token that you intend to use, is required in order for the Canary Console to have the correct authorisation and authority to post to your HEC. More info on HEC collector tokens can be found here.

Clicking the "Add" button will then cause your Canary Console to first attempt to send a test event in order to ensure that it can correctly post to the Splunk HEC. Only once a test event gets sent successfully, do the Splunk HEC details get saved on your Canary Console. 

Completed Setup

An example of a successfully configured Splunk Webhook is shown below:

You can test the connection again by using the "Test" button, this will send another test event through to the configured Splunk HEC. 

Clicking the × symbol will remove the currently configured Splunk Webhook.

NOTE, only one Splunk webhook is currently supported by your Canary Console.

Additional Splunk Testing:

The following curl command uses an example HTTP Event Collector token (B5A79AAD-D822-46CC-80D1-819F80D7BFB0) and uses https://hec.example.com as the hostname. Replace these values with your own before running this command.

When you make a JSON request to send data to HEC, you must specify the "event" key in the command.

curl -k  https://hec.example.com:8088/services/collector/event -H "Authorization: Splunk B5A79AAD-D822-46CC-80D1-819F80D7BFB0" -d '{"event": "hello world"}'
{"text": "Success", "code": 0}