Splunk

Log Management

Configuration

To configure Splunk log management, select “Settings → Log Management → Splunk”.

Splunk Form

By default, Splunk log management is not enabled. To enable, set the toggle to show Splunk link on Hosts, Container and Pods.

Enter the following parameters:

Parameter Description
Splunk Instance The URL or the IP address (including the port number) of the deployed instance where the logs are stored.
Index (optional) Optionally, the name of the index you have configured in the Splunk platform.

Accessing Splunk

Splunk Toggle

To access Splunk, click Go to Splunk which is located at the top right of each of these dashboards:

  • Kubernetes:

    • Host
    • Pod
    • Docker container
  • Host
  • Docker container

Accessing Instana from Splunk

There are 2 option to enable accessing Instana related entities from your logs:

  • Adjust your current dashboards: add the _raw field to the panel’s query and the drilldown to the panel section

    <drilldown>
        <link target="_blank">https://<ENVIRONMENT_URL_HERE>/#/integration/landing;config=$row._raw$</link>
    </drilldown>

    or

  • Create a new dashboard: Go to the Dashboards section in Splunk. Click Create New Dashboard, enter a name, and Save. Click Edit Dashboard, select Source, and paste the following content:

Download this file

<form theme="light">
    <label>Instana</label>
    <fieldset submitButton="false" autoRun="true">
        <input type="time" token="myTime" searchWhenChanged="true">
            <label></label>
            <default>
                <earliest>-1@h</earliest>
                <latest>now</latest>
            </default>
        </input>
    </fieldset>
    <row>
        <panel>
            <title>Events</title>
            <table>
                <search>
                    <query>sourcetype = * | table host docker.container_id kubernetes.pod_name _raw
                    </query>
                    <earliest>$myTime.earliest$</earliest>
                    <latest>$myTime.latest$</latest>
                </search>
                <option name="count">15</option>
                <option name="drilldown">row</option>
                <option name="refresh.display">progressbar</option>
                <option name="rowNumbers">false</option>
                <option name="totalsRow">true</option>
                <option name="wrap">false</option>
                <fields>["host","docker.container_id","kubernetes.pod_name","_raw"]</fields>
                <drilldown>
                    <link target="_blank">https://<ENVIRONMENT_URL_HERE>/#/integration/landing;config=$row._raw$</link>
                </drilldown>
            </table>
        </panel>
    </row>
</form>

Alert Channel

Add-On & App

Leverage the Instana & Splunk integration to access Instana metrics and events directly within Splunk.

splunk events

Configuration

Once you have the Add-On and App from Splunkbase installed, you’re ready to setup the Instana integration.

To configure, head over to “Settings → Team Settings → Events & Alerts → Alert Channels → Add Alert Channel”:

splunk alert channel

The following Splunk events are received as an HTTP POST to the configured URLs (HTTP or HTTPS).

On Open Issues/Incidents

{
  "issue": {
    "id": "53650436-8e35-49a3-a610-56b442ae7620",
    "type": "issue",
    "state": "OPEN",
    "start": 1460537793322,
    "severity": 5,
    "text": "Garbage Collection Activity High (11%)",
    "suggestion": "Tune your Garbage Collector, reduce allocation rate through code changes",
    "link": "https://XXXXXXX/#/?snapshotId=rjhkZXdNzegliVVEswMScGNn0YY",
    "zone": "prod",
    "fqdn": "host1.demo.com",
    "entity": "jvm",
    "entityLabel": "Test jvm",
    "tags": "production, documents, elasticsearch",
    "container": "test-container"
  }
}

On Close Issues/Incidents

{
  "issue": {
    "id": "6596e1c9-d6e4-4a8e-85fd-432432eddac3",
    "state": "CLOSED",
    "end": 1460537777478
  }
}

On Offline/Online/Change events

{
  "issue": {
    "id": "53650436-8e35-49a3-a610-56b442ae7620",
    "type": "presence",
    "start": 1460537793322,
    "text": "online",
    "description": "Java virtual machine on Host host1.demo.com",
    "link": "https://XXXXXXX/#/?snapshotId=rjhkZXdNzegliVVEswMScGNn0YY",
    "zone": "prod",
    "fqdn": "host1.demo.com",
    "entity": "jvm",
    "entityLabel": "Test jvm",
    "tags": "production, documents, elasticsearch",
    "container": "test-container"
  }
}