Agent Configuration

Backend

The agent needs to be configured to report to one or more backends with the file <instana-agent>/etc/instana/com.instana.agent.main.sender.Backend.cfg. By default, the agent will utilize the environment variables: INSTANA_AGENT_ENDPOINT, INSTANA_AGENT_ENDPOINT_PORT and INSTANA_AGENT_KEY.

The required values depend on your deployment.

SaaS Environment

For our SaaS Customers, INSTANA_AGENT_ENDPOINT_PORT and INSTANA_AGENT_ENDPOINT depends on the region your environment is in:

  • Europe:

    • DNS Name: saas-eu-west-1.instana.io
    • Destination Port: tcp/443
  • United States:

    • DNS Name: saas-us-west-2.instana.io
    • Destination Port: tcp/443
  • Rest of the World:

    • DNS Name: saas-us-west-2.instana.io
    • Destination Port: tcp/443

Installation for On-Premises

Our On-Premises Customers can configure the settings during the Installation.

When in doubt, download an agent from the portal and copy the options from the file <instana-agent>/etc/instana/com.instana.agent.main.sender.Backend.cfg.

Network Requirements

Some of the agent’s sensors are running outside of the agent’s process. Those sensors will connect to the agent using the local network. While this is often not a concern when agent and sensors are running on the same host, ensuring correct communication between agents and sensors may require configurations in containerized setups.

The table below lists the ports that have to be opened to reach the Agent process. Notice that, besides the default 42699 port, other ports may be necessary based on the languages that need to be traced.

Sensor Port range Configurable
Agent API 42699
Java tracing All ephemeral ports
Crystal Sensor 42699 Environment Variable
Envoy, NGINX and other Proxies tracing 42699 Environment Variable
Go Sensor 42699 Environment Variable
.NET Sensor 42657
Node.js Sensor 42699 Environment Variable
PHP Sensor 16816 PHP configuration
Node.js Sensor 42699 Environment Variable
Python Sensor 42699 Environment Variable
Ruby Sensor 42699 Environment Variable

In Kubernetes environments the network policy has to allow connection between agent pods and service pods so that external sensors can reach the agent and vice versa.

See also: Language Sensors

Configuration

The Instana agent configuration can be adopted using one central configuration file at the location:

<agent_install_dir>/etc/instana/configuration.yaml

Important: It is possible to create files called configuration-abc.yaml which are merged with this file in file system order. So configuration-cde.yaml comes after configuration-abc.yaml. Only nested structures like tags are merged, values are overwritten by subsequent configurations.

The format of this file is YAML, which is very sensitive to whitespace. All indention levels only allow two blank spaces.

Additional Filesystems

By default the agent will monitor local filesystems only. To add additional filesystems, add the name of the filesystem (that is the first column in mtab or df).

For example to monitor the following filesystem:

server:/usr/local/pub /pub nfs rsize=8192,wsize=8192,timeo=14,intr

please add the following line in the uncommented filesystems list of the host section:

com.instana.plugin.host:
  filesystems:
    - 'server:/usr/local/pub'

Tags

Specific tags for an agent can be specified in the com.instana.plugin.host section in the following format:

com.instana.plugin.host:
  tags:
    - 'production'
    - 'app1'

Alternatively, the environment variable INSTANA_TAGS=production,app1 can also be used (tags from environment variable and config file are additive).

This will add two tags to the specific agent, so now searching and filtering for tags are possible in the UI:

infrastructure tags

Installed Packages List (disabled by default)

Installed packages on operating system can be extracted once a day, if collectInstalledSoftware property is set in configuration.yaml. This is currently supported only on Debian-based linux distributions (dpkg).

com.instana.plugin.host:
  collectInstalledSoftware: false #  Valid values: true, false

Custom Zones

Instana will use Amazon Web Services, Google Compute Engine, or OpenStack Nova availability zone information to group hosts by default. To customize this host grouping, the group of the specific host can be defined in the section com.instana.plugin.generic.hardware using the following format:

com.instana.plugin.generic.hardware:
  enabled: true
  availability-zone: 'Demozone'

Alternatively, the environment variable INSTANA_ZONE=Demozone can be used instead (the zone from the environment variable overrides the zone from the config file).

The result is hosts grouped into zones on the map:

infrastructure zones

Custom Processes

Instana will automatically monitor process metrics of higher level sensors like Java or MySQL by default. Should you want to monitor an OS process not automatically covered by Instana, you can configure it using both the process name and/or its arguments:

com.instana.plugin.process:
  processes:
    - 'sshd'
    - 'slapd'
  arguments:
    - '/opt/script.sh'

Secrets

Tracing data may contain sensitive data. Therefore, the Instana agent supports the specification of patterns for “secrets”, that is, data to be redacted agent-side from the tracing data. Data treated as secrets will not reach the Instana SaaS for processing and, thus, will not be available for analysis in the UI or retrieval via API.

Instana supports the specification of secrets to match:

  • Environment variables
  • JDBC connection strings
  • Docker container info
  • HTTP query parameters (virtually for all supported runtimes, see support matrix below), e.g., https://my.domain/accounts/status?account=<secret_1>&user=<secret_2>
  • HTTP matrix path parameters, e.g., https://my.domain/accounts/account=<secret_1>;user=<secret_1>/status (support available for some runtimes, see support matrix below)
Language Secrets in HTTP Query parameters Secrets in HTTP Matrix parameters
Java
Go
.NET Framework
.NET Core
.NET Framework
Node.js
PHP
Python
Ruby
Crystal

Note: Go instrumentation is based on OpenTracing APIs, so processing secrets is on the instrumentor’s side.

Note: Instana does not support treating as secrets arbitrary segments in a URL paths, e.g., in case the URL is structured as https://my.domain/accounts/<secret>/status. The overhead of running, for example, regular expressions on URL paths to discard segments would simply be prohibitive.

(Additionally, although it is not really a matter related to secrets, the Instana agent does not capture SQL parameters from stored procedures. Similarly to the case of arbitrary path segments, we do not strip literals from SQL queries, and strongly advise the use of parametrized queries.)

Secrets are specified in the following way for all supported languages:

com.instana.secrets:
  # One of: 'equals-ignore-case', 'equals', 'contains-ignore-case', 'contains', 'regex'
  matcher: 'contains-ignore-case'
  list:
    - 'key'
    - 'password'
    - 'secret'

If a key collected by a sensor matches an entry from the list, the value is redacted and not sent to the Instana backend. If you do not specify any custom secrets configuration, the agent will use the aforementioned secrets configuration by default.

Note: Generally, secrets are filtered from static data. Filtering is optimized and changes will not be applied while a process is running. To make a secret configuration effective, either restart the agent or the running monitored component, which should respect the new config.

Note: For additional secrets redaction in selected Kubernetes resources, see Kubernetes secrets for more details.

Capturing Custom HTTP Headers

By default, Instana will not collect HTTP headers when tracing HTTP calls, but this feature can be enabled on demand. To do so, you need to specify which headers you want to have captured:

com.instana.tracing:
  extra-http-headers:
    - 'x-request-id'
    - 'x-loadtest-id'
    - ...

The values are case-insensitive. The headers thus collected will be shown in the call details (in the “callee details” section). You can also use them to search for calls and traces (in the UI as well as via the API). Finally, they can be used for service configuration.

Restrictions:

  • Instana only captures request headers on HTTP entries (HTTP calls that the instrumented process receives).
  • Instana does not capture response headers.
  • Instana does not capture headers on HTTP exits (HTTP calls where the instrumented process is the client and the HTTP server is not instrumented).
  • This feature is currently supported in Java, .NET, NodeJs, PHP, Python and Ruby.

Ignore Processes

To ignore any monitoring of certain processes, uncomment the com.instana.ignore section and list all process names of processes that should not be monitored. Sometimes scripts or other jobs are started through the same command, but only the execution of a single command should be ignored. In this case use arguments to specify any argument that should cause a process to be ignored.

com.instana.ignore:
  processes:
    - 'java'
    - 'httpd'
  arguments:
    - '-batch-file=/tmp/batch.def'

Deactivate UI Triggered Agent Capabilities

In order to tail the Instana Agent logs, perform diagnostic actions on the agent and to enable the Trace Code View, users can trigger those capabilities through the Instana UI. In case you want to deactivate all of these capabilities, you can set the config accordingly by adding to: <agent_install_dir>/etc/instana/com.instana.agent.main.config.Agent.cfg

backchannel.enabled = false

Note: This needs to be set before the agent is started.

SDK Configuration

Please see the Java Trace SDK documentation page for SDK configuration.

Agent Logging Configuration

By default, the Instana Agent will log to <instana_install_dir>/data/log/agent.log, which will also be rotated should the file grow too large. If you are running the agent in a container, it logs to the console instead, which is managed by Docker and accessible via docker logs.

The log level can be increased to debug by changing the level in the configuration file <instana_install_dir>/etc/org.ops4j.pax.logging.cfg from:

log4j2.logger.instana.level=INFO

to:

log4j2.logger.instana.level=DEBUG

Please ensure that there is no trailing whitespace on the line.

Older agent installs need this line to be changed from:

log4j.logger.com.instana=INFO, out, osgi:*

to:

log4j.logger.com.instana=DEBUG, out, osgi:*

Log Rotation

By default, the agent uses a log rotation of 10 times 5MB agent log files.

log4j2.appender.rolling.policy.type = SizeBasedTriggeringPolicy
log4j2.appender.rolling.policy.size = 5MB
log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy
log4j2.appender.rolling.strategy.max = 10

Configure Syslog

The Agent now uses Log4j2, which is a modern and flexible logging facility. For example, syslog can be configured like this:

log4j2.rootLogger.appenderRef.Syslog.ref = Syslog
log4j2.rootLogger.appenderRef.Syslog.level = ERROR
log4j2.appender.syslog.type=Syslog
log4j2.appender.syslog.name=Syslog
log4j2.appender.syslog.layout.type=PatternLayout
log4j2.appender.syslog.layout.pattern = ${log4j2.pattern}
log4j2.appender.syslog.facility=SYSLOG
log4j2.appender.syslog.host=localhost
log4j2.appender.syslog.port=514
log4j2.appender.syslog.protocol=UDP

Logging to STDOUT

In custom container images, you may want to log to STDOUT. You can do so by providing the following configuration in <instana-agent-install-dir>etc/org.ops4j.pax.logging.cfg:

log4j2.rootLogger.appenderRef.Console.ref = Console

log4j2.appender.console.type = Console
log4j2.appender.console.name = Console
log4j2.appender.console.layout.type = PatternLayout
log4j2.appender.console.layout.pattern = ${log4j2.pattern}

Log4j2

Because the logging is using the standard log4j2 logging, it can be configured in many more ways as described in the log4j2 documentation.

Agent Proxy Setup

Background

To communicate with the backend effectively, Instana uses the HTTP/2 protocol to transfer data. In many cases, direct communication from the agent to the backend can be granted, simplifying the agents’ deployment. In some cases, one dedicated entry into, and exit out of, the network is required. For this reason, Instana can be used in combination with a variety of proxies. In general, http(s), socks4s, and socks5 proxies are supported.

Configuring the Instana Agent

For the Instana agent configuration, you need to modify two files. In the file <install-dir>/etc/mvn-settings.xml the following information must be present (and not commented out) within the <settings></settings> section:

<proxies>
  <proxy>
    <id>agent-proxy</id>
    <active>true</active>
    <protocol>http</protocol>
    <username>user-if-needed</username>
    <password>password-if-needed</password>
    <host>your-proxy-address-goes-here</host>
    <port>your-proxy-port-goes-here</port>
  </proxy>
</proxies>

This enables the Instana agent to download agent updates and additional features automatically. Furthermore, the file <install-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg needs to be reconfigured to use the proxy for the communication between the agent and the Instana backend. Please make sure the following lines are present and not commented out:

proxy.type=http
proxy.host=your-proxy-address-goes-here
proxy.port=your-proxy-port-goes-here
proxy.user=user-if-needed
proxy.password=password-if-needed
proxy.dns=true

Sample Squid Proxy Setup

This document describes the configuration of a squid proxy (www.squid-cache.org) in combination with Instana, when there is no other proxies available.

There are several ways to install Squid on your system. Most Linux distributions include Squid in their repositories and the software can be installed using the preferred package manager. In case no package is available, or you want to run Squid on Microsoft Windows, you can get Squid binaries from this location: http://wiki.squid-cache.org/SquidFaq/BinaryPackages. Once Squid is installed, an example configuration called squid.conf is created. If you want to use the proxy for the Instana communication exclusively, you can backup the default configuration and just use this minimal squid.conf:

# The tcp port squid is listening on
http_port 3128

# Please specify subnet with instana agents
acl instana_agent_net src 10.0.0.0/8

# This is the ip of the instana backend
acl instana_backend dstdomain saas-eu-west-1.instana.io
#acl instana_backend dstdomain ec2-54-144-114-141.compute-1.amazonaws.com
#acl instana_backend dstdomain saas-us-east-1.instana.io
#acl instana_backend dstdomain saas-us-east-1.instana.io

# This is the port used by Instana
acl instana_backend_port port 443

# This is the repo to download updates and additional sensors
acl instana_repo dstdomain artifact-public.instana.io
acl instana_repo_port port 80
acl instana_repo_port_secure port 443

# Protocol used for instana backend
acl instana_backend_proto proto HTTP

# Protocol used for instana backend
acl instana_repo_proto proto HTTP
acl instana_repo_proto_secure proto HTTPS

http_access allow instana_agent_net instana_backend instana_backend_port
http_access allow instana_agent_net instana_repo instana_repo_port
http_access allow instana_agent_net instana_repo instana_repo_port_secure

# DO NOT REMOVE THIS RULE!
http_access deny all

Agent Versioning and Update Management

Background

The Instana agent is optimized for little administration effort. One important capability for reducing management overhead is the automatic update capability, which automatically manages the versions of the agent and every sensor. In some scenarios this update mechanism needs more detailed configuration than just automatically updating to the latest version.

Configuring the Update Interval

By default, the agent will check for new versions every day, around 4:30 am. This automatic check can be turned off, or set to prefer a different time of day and/or only run on specific days.

The configuration is done in the file etc/instana/com.instana.agent.main.config.UpdateManager.cfg and looks, by default, like this:

# Instana Update Manager configuration.
# AUTO for automatic updates with given schedule. OFF for no automatic updates.
mode = AUTO
# DAY for daily, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, SUNDAY
every = DAY
# Time is hh:mm in 24 hours format.
at = 04:30

Selecting Static Versions

Every change in a sensor or agent version will pre-released to the automatic update repository as a complete set of a new “current set” of versions. All previously released sets are available at https://github.com/instana/agent-updates - in git (the version control system that keeps track of all changes to agent updates) each of those sets gets a static version number, called “sha.”

It is possible to select a specific set by setting

version = <sha>

in the etc/instana/com.instana.agent.bootstrap.AgentBootstrap.cfg file before startup. The agent then will stay at this version set and never update to a newer one.

Freezing Current Versions

Instana Repositories

The Instana agent has a set of preconfigured repositories to look for updates (and additional ones can be configured if required). In the default agent setup, the agent has a local repository located in the folder <agent-dir>/data/repo, in addition to access to a central repo over the network. This folder contains - among folders that are not important for this setup - specific folders for three Instana features:

  • /com/instana/agent-feature/
  • /com/instana/discovery-feature/
  • /com/instana/sensor-feature/

Freezing the current agent versions

Each of these features has files describing detailed information about available versions. For example, in the folder <agent-dir>/data/repo/com/instana/sensor-feature/1.0.0-SNAPSHOT there are files containing information about versions of the sensor feature.

Now you can copy all feature repos from the folder <agent-dir>/data/repo/com/instana/ to the target folder <agent-dir>/system/com/instana, from which the agent loads its features with the highest priority (e.g. cp -r <agent-dir>/data/repo/com/instana/*-feature/ <agent-dir>/system/com/instana/).

This will lead to the agent not updating itself and staying at this frozen version.

Note: The automatic updates will still continue to run in the configured interval. Should the mentioned config files be updated, the automatic update will then pull the new versions on its next run.

Getting Current Versions

In order to get versions of the agent and sensors installed, one can use following agent API endpoint:

curl localhost:42699/version

Limiting Agent’s CPU and Memory in different environments

In certain scenarios it is crucial to control resource consumption of processes very closely. This can become especially handy in environments that leverage resource sharing and for systems with extremely low resources. While the Instana agent is designed to consume as few resources as possible it can be limited following the instructions below as another safeguard to respect clear resource limits. All following examples limit Agent’s CPU to computing resources equivalent to half a CPU (however, not all resources would necessarily come from the same CPU) and to a maximum of 512 MB memory.

systemd

Create file called /etc/systemd/system/instana-agent.service.d/20-resource_limits.conf and add the following content to it:

[Service]
CPUAccounting=true
CPUQuota=50%
MemoryAccounting=true
MemoryMax=512M

Run systemctl daemon-reload and restart instana-agent service.

Docker

Run instana-agent container with the following additional parameters:

Version 1.13 and newer:

--cpus=0.5 --memory=512m

Version 1.12 and older:

--cpu-period=100000 --cpu-quota=50000 --memory=512m

Kubernetes

Add the following configuration snippet to the agent’s container configuration:

livenessProbe:
  httpGet: # Agent liveness is published on localhost:42699/status
    path: /status
    port: 42699
  initialDelaySeconds: 75
  periodSeconds: 5
resources:
  requests:
    memory: "256Mi"
    cpu: "0.5"
  limits:
    memory: "512Mi"
    cpu: "1.0"

The configuration above also requests less memory and CPU for the instana-agent container.

Configuring Multiple Backends

Background

In some cases it might be desirable to have an agent report to multiple backends. For example for shared services that are used by otherwise strictly separated environments. It is possible to manually configure agents to do so. Please note, that the agent will be counted in all backends as consuming a license, and this will effectively multiply the bandwidth consumption of the agent.

Configuring the Instana Agent

The file <install-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg is used to configure a single backend the agent sends to. To use multiple backends, please rename this file to <install-dir>/etc/instana/com.instana.agent.main.sender.Backend-1.cfg and create copies named <install-dir>/etc/instana/com.instana.agent.main.sender.Backend-2.cfg, <install-dir>/etc/instana/com.instana.agent.main.sender.Backend-3.cfg and so on. Each file then can be adjusted to describe a different backend and agent key, and can even contain different proxy settings. Instead of 1, 2 and 3 any unique alphanumerical id can be used. To have the agent respect multiple backends it is important that the original file <install-dir>/etc/instana/com.instana.agent.main.sender.Backend.cfg is absent.

Logging Metrics or Traces to File

The agent can temporarily log metrics or traces being sent through that agent to a file on disk. This is often used to debug various issues related to metrics, traces, tracing or spans.

To enable this, locate the file <agent_install_dir>/etc/instana/com.instana.agent.main.sender.File.cfg and update it’s contents with:

# Configuration of local logging. Changes will be hot-reloaded.
# Activate logging of outgoing payloads to local disk by setting a non-empty
# prefix. The log file will be written to data/log, and the file will have the
# defined prefix followed by a timestamp.
# Note: There is no automatic rotation of those files.
prefix=locallog

# The file can be filtered to either "metrics" or "traces".
# If empty or absent, there will be no filtering.
type=traces

As noted, the changes will be hot-reloaded and be activated immediately. This should only be done temporarily as leaving this enabled has the potential to fill up all available disk space given enough time and traffic.

Allow the logging to continue for a minute or two while generating traffic to the component in question (in the case of tracing issues). Then revert the changes or comment all lines in that file. Again, changes written to disk will be hot-reloaded by the Instana agent.

If you are working with a support ticket, please attach the resulting log file to the support ticket. The log file will be located in <agent_install_dir>/data/log/locallog_*.log.

Agent Management

All agents reporting to Instana have their own dashboard. Each dashboard displays real-time information on the agent, lets you configure the mode, log level, update the agent and reset the agent and sensors.

Accessing the Agent Dashboard

You can navigate to the agent dashboard either by using the Open Agent Management button from the host dashboard:

agent dash

or by selecting “Agents” from the top level navigation’s “More” menu. Here you can search for a specific agent. By clicking on the entity link in the FQDN column you get to the agent management dashboard.

agent table

Instana Agent Dashboard

agent dash

Agent Mode

The agent mode is relevant when host-based licensing is used to decide whether the agent counts as an Infrastructure or an APM agent. This can be toggled using “Change Agent Mode”:

agent mode

This dropdown also supports the mode “off.” This turns off all agent functionality, except for the “heartbeat” to the backend. It is possible to turn the agent back on, and it will pick up again right where it left.

With Infrastructure only, all tracing related functionality is turned off.

This mode can also be set via the config file instana-agent/etc/instana/com.instana.agent.main.config.Agent.cfg:

mode = APM
# APM or INFRASTRUCTURE or OFF

Notice: The One-Liner agent installation method and the Instana agent Docker image accept an additional mode, called AWS mode. The AWS mode is simply INFRASTRUCTURE mode plus some automatic configuration of AWS data collection, as described in the Install the Instana agent to monitor AWS API data documentation.

Agent and Sensor Reset

The dashboard can also reset just the sensors or the complete agent as required.

Resetting a sensor when it is missing some metrics is the most lightweight corrective action available.

Resetting the whole agent is very similar to restarting the agent process, but the process remains active. This means OS level watchdogs / service scripts will not see the process id changing.

Sensors Info

This pop-up displays list of sensors and agent components present on the host, with name, version and state for each component. This list is searchable.

sensors_list

Agent Self Monitoring

Additionally, the Instana agent does constant lightweight self monitoring. The metrics can be used to observe agent performance and resource consumption.

agent metrics

Agent Heap Dump

Should support ask for it, you can create a heap dump of the Instana agent by executing this command

TS=`date +%s` && /opt/instana/agent/jvm/bin/jmap -dump:file=/tmp/agent-dump-$TS.hprof `cat /opt/instana/agent/agent.pid` && gzip /tmp/agent-dump-$TS.hprof