Prometheus

Introduction

Collecting metrics data with Prometheus is becoming more popular. With Instana, it’s easy to capture Prometheus metrics and correlate them using our extensive knowledge graph. A typical example is custom business metrics.

The Instana Prometheus sensor doesn’t require a Prometheus server. The sensor captures metrics directly from the endpoints that are exposed by the monitored systems. For each Instana agent, specify which Prometheus endpoints you want to poll and which metrics should be collected from them using regular expressions.

Configuration

Static configuration

For the Prometheus sensor to be able to collect data, metric sources must be defined in the agent configration file; <agent_install_dir>/etc/instana/configuration.yaml. The sensor picks up any type of metric. With a regular expression, you can define which metrics should captured for a specific metric endpoint.

# Static Prometheus Configuration
com.instana.plugin.prometheus:
  customMetricSources:
  - url: 'http://localhost:8080/metrics' # endpoint with exposed metrics
    metricNameIncludeRegex: '^sample_app_request'
  - url: 'http://localhost:9100/prometheus' # endpoint with exposed metrics
    metricNameIncludeRegex: '^sample_app_request'

Regex in the example above, configures a sensor to gather all metrics starting with sample_app_request e.g. sample_app_request_duration. In the case that metricNameIncludeRegex is not defined, the sensor collects all the exposed metrics until the defined limit.

Kubernetes configuration

Our Prometheus sensor detects if Kubernetes is being used and automatically gathers all the available pods on the node the sensor is running on. The URL field in the configuration is still required, but instead of specifying the whole URL, the path part is sufficient, e.g. /metrics.

# Kubernetes Discovery Prometheus Configuration
com.instana.plugin.prometheus:
  customMetricSources:
  - url: '/metrics' # url path part which expose metrics
    metricNameIncludeRegex: '^sample_app_request' 
  - url: '/prometheus-metrics' # url path part which expose metrics
    metricNameIncludeRegex: '^sample_app_request' 

Metrics collection

Prometheus custom metrics

Prometheus metrics appear as “Prometheus Apps” associated with the host from where they are collected. You can query Prometheus custom metrics using the “Dynamic Focus”, “Event & Alerts”, and the “Grafana Plugin” with entity.type:prometheus.

Metrics

  • Counters (max. 700)
  • Gauges (max. 700)
  • Histograms (max. 700)
  • Summaries (max. 700)