Agent Configuration

Backend

The agent needs to be configured to report to one or more backends with the file <instana-agent>/etc/instana/com.instana.agent.main.sender.Backend.cfg. By default, the agent will utilize the environment variables: INSTANA_AGENT_ENDPOINT, INSTANA_AGENT_ENDPOINT_PORT and INSTANA_AGENT_KEY.

The required values depend on your deployment.

SaaS Environment

For our SaaS Customers, INSTANA_AGENT_ENDPOINT_PORT and INSTANA_AGENT_ENDPOINT depends on the region your environment is in:

  • Europe:

    • DNS Name: saas-eu-west-1.instana.io
    • Destination Port: tcp/443
  • United States:

    • DNS Name: saas-us-west-2.instana.io
    • Destination Port: tcp/443
  • Rest of the World:

    • DNS Name: saas-us-west-2.instana.io
    • Destination Port: tcp/443

Installation for On Prem

Our on-prem Customers can configure the settings during the Installation.

When in doubt, download an agent from the portal and copy the options from the file <instana-agent>/etc/instana/com.instana.agent.main.sender.Backend.cfg.

Configuration

The Instana agent configuration can be adopted using one central configuration file at the location:

<agent_install_dir>/etc/instana/configuration.yaml

Important: It is possible to create files called configuration-abc.yaml which are merged with this file in file system order. So configuration-cde.yaml comes after configuration-abc.yaml. Only nested structures like tags are merged, values are overwritten by subsequent configurations.

The format of this file is YAML, which is very sensitive to whitespace. All indention levels only allow two blank spaces.

Additional Filesystems

By default the agent will monitor local filesystems only. To add additional filesystems, add the name of the filesystem (that is the first column in mtab or df).

For example to monitor the following filesystem:

server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr

please add the following line in the uncommented filesystems list of the host section:

com.instana.plugin.host:
  filesystems:
    - 'server:/usr/local/pub'

Tags

Specific tags for an agent can be specified in the com.instana.plugin.host section in the following format:

com.instana.plugin.host:
  tags:
    - 'production'
    - 'app1'

Alternatively, the environment variable INSTANA_TAGS=production,app1 can also be used (tags from environment variable and config file are additive).

This will add two tags to the specific agent, so now searching and filtering for tags are possible in the UI:

infrastructure tags

Installed Packages List (disabled by default)

Installed packages on operating system can be extracted once a day, if collectInstalledSoftware property is set in configuration.yaml. This is currently supported only on Debian-based linux distributions (dpkg).

com.instana.plugin.host:
  collectInstalledSoftware: false #  Valid values: true, false

Custom Zones

Instana will use Amazon Web Services, Google Compute Engine, or Openstack Nova availability zone information to group hosts by default. To customize this host grouping, the group of the specific host can be defined in the sectioncom.instana.plugin.generic.hardware using the following format:

com.instana.plugin.generic.hardware:
  enabled: true
  availability-zone: 'Demozone'

Alternatively, the environment variable INSTANA_ZONE=Demozone can be used instead (the zone from the environment variable overrides the zone from the config file).

The result is hosts grouped into zones on the map:

infrastructure zones

Custom Processes

Instana will automatically monitor process metrics of higher level sensors like Java or MySQL by default. Should you want to monitor an OS process not automatically covered by Instana, you can configure it using both the process name and/or its arguments:

com.instana.plugin.process:
  processes:
    - 'sshd'
    - 'slapd'
  arguments:
    - '/opt/script.sh'

Secrets

To filter sensitive data from collection by the agent, all sensors respect the following secrets configuration. If a key collected by a sensor matches an entry from the list, the value is redacted:

com.instana.secrets:
  matcher: 'contains-ignore-case' # 'contains-ignore-case', 'contains', 'regex'
  list:
    - 'key'
    - 'password'
    - 'secret'

Note: Generally, secrets are filtered from static data. Filtering is optimized and changes will not be applied while a process is running. To make a secret configuration effective, either restart the agent or the running monitored component, which should respect the new config.

Capturing Custom HTTP Headers

To capture custom HTTP headers you need to specify in the configuration which should be captured.

These headers are visualized in the trace view, can be searched via the UI or API and are available for Service configuration.

The configuration for that is quite simple:

com.instana.tracing:
  extra-http-headers:
    - 'x-loadtest-id'

Note: currently supported for Java, .NET, NodeJs, PHP, Python and Ruby.

Ignore processes

To ignore any monitoring of certain processes, uncomment the com.instana.ignore section and list all process names of processes that should not be monitored. Sometimes scripts or other jobs are started through the same command, but only the execution of a single command should be ignored. In this case use arguments to specify any argument that should cause a process to be ignored.

com.instana.ignore:
  processes:
    - 'java'
    - 'httpd'
  arguments:
    - '-batch-file=/tmp/batch.def'

Deactivate Self Monitoring and Code View

In order to enable self monitoring of the Agent, tailing the Agent logs, and enabling the Trace Code View, users can trigger those capabilities through the Instana UI. In case you want to deactivate these capabilities, you can set the config accordingly by adding to: <agent_install_dir>/etc/instana/com.instana.agent.main.config.Agent.cfg

backchannel.enabled = false

Note: This needs to be set before the agent is started.

SDK Configuration

Please see the Java Trace SDK documentation page for sdk configuration.

Agent Logging Configuration

By default, the Instana Agent will log to <instana_install_dir>/data/log/agent.log, which will also be rotated should the file grow too large. If you are running the agent in a container, it logs to the console instead, which is managed by Docker and accessible via docker logs.

The log level can be increased to debug by changing the level in the configuration file <instana_install_dir>/etc/org.ops4j.pax.logging.cfg from:

log4j2.logger.instana.level=INFO

to:

log4j2.logger.instana.level=DEBUG

Please ensure that there is no trailing whitespace on the line.

Older agent installs need this line to be changed from:

log4j.logger.com.instana=INFO, out, osgi:*

to:

log4j.logger.com.instana=DEBUG, out, osgi:*

Log Rotation

By default, the agent uses a log rotation of 10 times 5MB agent log files.

log4j2.appender.rolling.policy.type = SizeBasedTriggeringPolicy
log4j2.appender.rolling.policy.size = 5MB
log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy
log4j2.appender.rolling.strategy.max = 10

Configure Syslog

The Agent now uses Log4j2, which is a modern and flexible logging facility. For example, syslog can be configured like this:

log4j2.rootLogger.appenderRef.Syslog.ref = Syslog
log4j2.rootLogger.appenderRef.Syslog.level = ERROR
log4j2.appender.syslog.type=Syslog
log4j2.appender.syslog.name=Syslog
log4j2.appender.syslog.layout.type=PatternLayout
log4j2.appender.syslog.layout.pattern = ${log4j2.pattern}
log4j2.appender.syslog.facility=SYSLOG
log4j2.appender.syslog.host=localhost
log4j2.appender.syslog.port=514
log4j2.appender.syslog.protocol=UDP

Logging to STDOUT

In custom container images, you may want to log to STDOUT. You can do so by providing the following configuration in <instana-agent-install-dir>etc/org.ops4j.pax.logging.cfg:

log4j2.rootLogger.appenderRef.Console.ref = Console

log4j2.appender.console.type = Console
log4j2.appender.console.name = Console
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = ${log4j2.pattern}

Log4j2

Because the logging is using the standard log4j2 logging, it can be configured in many more ways as described in the log4j2 documentation.

Agent Proxy Setup

Background

To communicate with the backend effectively, Instana uses the HTTP/2 protocol to transfer data. In many cases, direct communication from the agent to the backend can be granted, simplifying the agents’ deployment. In some cases, one dedicated entry into, and exit out of, the network is required. For this reason, Instana can be used in combination with a variety of proxies. In general, http(s), socks4s, and socks5 proxies are supported.

Configuring the Instana Agent

For the Instana agent configuration, you need to modify two files. In the file <install-dir>/etc/mvn-settings.xml the following information must be present (and not commented out) within the <settings></settings> section:

<proxies>
  <proxy>
    <id>agent-proxy</id>
    <active>true</active>
    <protocol>http</protocol>
    <username>user-if-needed</username>
    <password>password-if-needed</password>
    <host>your-proxy-address-goes-here</host>
    <port>your-proxy-port-goes-here</port>
  </proxy>
</proxies>

This enables the Instana agent to download agent updates and additional features automatically. Furthermore, the file <install-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg needs to be reconfigured to use the proxy for the communication between the agent and the Instana backend. Please make sure the following lines are present and not commented out:

proxy.type=http
proxy.host=your-proxy-address-goes-here
proxy.port=your-proxy-port-goes-here
proxy.user=user-if-needed
proxy.password=password-if-needed
proxy.dns=true

Sample Squid Proxy Setup

This document describes the configuration of a squid proxy (www.squid-cache.org) in combination with Instana, when there is no other proxies available.

There are several ways to install Squid on your system. Most Linux distributions include Squid in their repositories and the software can be installed using the preferred package manager. In case no package is available, or you want to run Squid on Microsoft Windows, you can get Squid binaries from this location: http://wiki.squid-cache.org/SquidFaq/BinaryPackages. Once Squid is installed, an example configuration called squid.conf is created. If you want to use the proxy for the Instana communication exclusively, you can backup the default configuration and just use this minimal squid.conf:

# The tcp port squid is listening on
http_port 3128

# Please specify subnet with instana agents
acl instana_agent_net src 10.0.0.0/8

# This is the ip of the instana backend
acl instana_backend dstdomain saas-eu-west-1.instana.io
#acl instana_backend dstdomain ec2-54-144-114-141.compute-1.amazonaws.com
#acl instana_backend dstdomain saas-us-east-1.instana.io
#acl instana_backend dstdomain saas-us-east-1.instana.io

# This is the port used by Instana
acl instana_backend_port port 443

# This is the repo to download updates and additional sensors
acl instana_repo dstdomain artifact-public.instana.io
acl instana_repo_port port 80
acl instana_repo_port_secure port 443

# Protocol used for instana backend
acl instana_backend_proto proto HTTP

# Protocol used for instana backend
acl instana_repo_proto proto HTTP
acl instana_repo_proto_secure proto HTTPS

http_access allow instana_agent_net instana_backend instana_backend_port
http_access allow instana_agent_net instana_repo instana_repo_port
http_access allow instana_agent_net instana_repo instana_repo_port_secure

# DO NOT REMOVE THIS RULE!
http_access deny all

Agent Versioning and Update Management

Background

The Instana agent is optimized for little administration effort. One important capability for reducing management overhead is the automatic update capability, which automatically manages the versions of the agent and every sensor. In some scenarios this update mechanism needs more detailed configuration than just automatically updating to the latest version.

Configuring the Update Interval

By default, the agent will check for new versions every day, around 4:30 am. This automatic check can be turned off, or set to prefer a different time of day and/or only run on specific days.

The configuration is done in the file etc/instana/com.instana.agent.main.config.UpdateManager.cfg and looks, by default, like this:

# Instana Update Manager configuration.
# AUTO for automatic updates with given schedule. OFF for no automatic updates.
mode = AUTO
# DAY for daily, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY
every = DAY
# Time is hh:mm in 24 hours format.
at = 04:30

Selecting Static Versions

Every change in a sensor or agent version will pre-released to the automatic update repository as a complete set of a new “current set” of versions. All previously released sets are available at https://github.com/instana/agent-updates - in git (the version control system that keeps track of all changes to agent updates) each of those sets gets a static version number, called “sha.”

It is possible to select a specific set by setting

version = <sha>

in the etc/instana/com.instana.agent.bootstrap.AgentBootstrap.cfg file before startup. The agent then will stay at this version set and never update to a newer one.

Freezing Current Versions

Instana Repositories

The Instana agent has a set of preconfigured repositories to look for updates (and additional ones can be configured if required). In the default agent setup, the agent has a local repository located in the folder <agent-dir>/data/repo, in addition to access to a central repo over the network. This folder contains - among folders that are not important for this setup - specific folders for three Instana features:

  • /com/instana/agent-feature/
  • /com/instana/discovery-feature/
  • /com/instana/sensor-feature/

Freezing the current agent versions

Each of these features has files describing detailed information about available versions. For example, in the folder <agent-dir>/data/repo/com/instana/sensor-feature/1.0.0-SNAPSHOT there are files containing information about versions of the sensor feature.

Now you can copy all feature repos from the folder <agent-dir>/data/repo/com/instana/ to the target folder <agent-dir>/system/com/instana, from which the agent loads its features with the highest priority (e.g. cp -r <agent-dir>/data/repo/com/instana/*-feature/ <agent-dir>/system/com/instana/).

This will lead to the agent not updating itself and staying at this frozen version.

Note: The automatic updates will still continue to run in the configured interval. Should the mentioned config files be updated, the automatic update will then pull the new versions on its next run.

Limiting Agent’s CPU and Memory in different environments

In certain scenarios it is crucial to control resource consumption of processes very closely. This can become especially handy in environments that leverage resource sharing and for systems with extremely low resources. While the Instana agent is designed to consume as few resources as possible it can be limited following the instructions below as another safeguard to respect clear resource limits. All following examples limit Agent’s CPU to maximum of 50% of a single CPU and to a maximum of 512 MB memory.

systemd

Create file called /etc/systemd/system/instana-agent.service.d/20-resource_limits.conf and add the following content to it:

[Service]
CPUAccounting=true
CPUQuota=50%
MemoryAccounting=true
MemoryMax=512M

Run systemctl daemon-reload and restart instana-agent service.

Docker

Run instana-agent container with the following additional parameters:

Version 1.13 and newer:

--cpus=0.5 --memory=512m

Version 1.12 and older:

--cpu-period=100000 --cpu-quota=50000 --memory=512m

Kubernetes

Add the following configuration snippet to the agent’s container configuration:

livenessProbe:
  exec:
    command:
      - echo
      - noop
  initialDelaySeconds: 60
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 5
readinessProbe:
  exec:
    command:
      - echo
      - noop
  initialDelaySeconds: 60
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 5
resources:
  requests:
    memory: "256Mi"
    cpu: "0.5"
  limits:
    memory: "512Mi"
    cpu: "1.0"

The configuration above also requests less memory and CPU for the instana-agent container. It also tells Kubernetes to avoid hard liveness and readiness checks for the agent, so it’s not considered dead when some of its parts are not responsive.

Configuring Multiple Backends

Background

In some cases it might be desirable to have an agent report to multiple backends. For example for shared services that are used by otherwise strictly separated environments. It is possible to manually configure agents to do so. Please note, that the agent will be counted in all backends as consuming a license, and this will effectively multiply the bandwidth consumption of the agent.

Configuring the Instana Agent

The file <install-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg is used to configure a single backend the agent sends to. To use multiple backends, please rename this file to <install-dir>/etc/instana/com.instana.agent.main.sender.Backend-1.cfg and create copies named <install-dir>/etc/instana/com.instana.agent.main.sender.Backend-2.cfg, <install-dir>/etc/instana/com.instana.agent.main.sender.Backend-3.cfg and so on. Each file then can be adjusted to describe a different backend and agent key, and can even contain different proxy settings. Instead of 1, 2 and 3 any unique alphanumerical id can be used. To have the agent respect multiple backends it is important that the original file <install-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg is absent.

Agent Management

All agents reporting to Instana have their own dashboard. Each dashboard displays real-time information on the agent, lets you configure the mode, reset the agent and sensors, and initiate the agent’s self-monitoring function.

Accessing the Agent Dashboard

You can navigate to the agent dashboard either by activating the agent self-monitoring through the host dashboard:

agent dash ,

or by selecting “Agents” in the Infrastructure comparison table:

agent table

Instana Agent Dashboard

agent dash

Agent Mode

The agent mode is relevant when host-based licensing is used to decide whether the agent counts as an Infrastructure or an APM agent. This can be toggled using “Change Agent Mode”:

agent mode

This dropdown also supports the mode “off.” This turns all agent functionality off, except for the “heartbeat” to the backend. It is possible to turn the agent back on, and it will pick up again right where it left.

With Infrastructure only, all tracing related functionality is turned off.

This mode can also be set via the config file instana-agent/etc/instana/com.instana.agent.main.config.Agent.cfg:

mode = APM
# APM or INFRASTRUCTURE or OFF

Agent and Sensor Reset

The dashboard can also reset just the sensors or the complete agent as required.

Resetting a sensor when it is missing some metrics is the most lightweight corrective action available.

Resetting the whole agent is very similar to restarting the agent process, but the process remains active. This means OS level watchdogs / service scripts will not see the process id changing.

Agent Self Monitoring

Additionally, the Instana agent is capable of self-monitoring. These metrics are quite lightweight, but they are not needed most of the time, and therefore turned off by default.

agent metrics

Agent Heap Dump

Should support ask for it, you can create a heap dump of the Instana agent by executing this command

TS=`date +%s` && /opt/instana/agent/jvm/bin/jmap -dump:file=/tmp/agent-dump-$TS.hprof `cat /opt/instana/agent/agent.pid` && gzip /tmp/agent-dump-$TS.hprof