Application & Service Management

An older version of this document can be found here.

Concepts

Traditional Application Performance Management (APM) solutions are about managing the performance and availability of applications.

An application for APM tools is a static set of code runtimes (e.g. JVM or CLR) that are monitored using an agent. Normally the application is defined as a configuration parameter on each agent.

This concept, which was a good model for classical 3-tier applications, does not work anymore in modern (micro)service applications. A service does not always belong to exactly one application. Think of a credit card payment service that is used in the online store of a company and also in their Point of Sales solution. A solution to this problem could be to define every service as an application, but that would introduce some new issues:

  • Too many applications to monitor. Treating every service as an application would result in hundreds or thousands of applications. Monitoring them using dashboards will not work - just too much data for humans.
  • Loss of context. As every service is treated separately, it would not be possible to understand dependency and understand the role of the service in the context of a problem.

Instana introduces the next generation of APM with its application hierarchy of services, endpoints, and application perspectives across them. Our main goal is to simplify the monitoring of your business’ service quality. Based on the data we collect from traces and component sensors, we discover your application landscape directly from the services actually being implemented. Based upon the data collected, we can know the health of every individual service, and, subsequently, the health of the entire application landscape.

Application Perspectives

The term “application” is ambiguous, and different teams may use it to describe distinctly different things. Instead of applications, Instana provides application perspectives, in order to allow teams to capture the type of semantics that are meaningful to them.

A team that runs a multi-tenant offering likely wants to capture their tenants (tenant:one, tenant:two) as applications. Another team that runs an eCommerce site may want to capture their US (zone:us) and German (zone:eu) stores as two separate applications. While another team will want to capture their different environments (k8s.env:prod, k8s.env:dev) as applications. And finally one customer may want to group a set of services that work together functionally into an application. Each of these applications come with their own semantics and varying use cases.

Different Application Perspectives

An application perspective may overlay entire services, or just a subset of calls to those services. Application perspectives may overlap each other completely, partially, or not at all. Services and endpoints may be included within the definition of more than one application, or none at all.

Working with Application Perspectives

To create a new application perspective simply go to the application perspective inventory and select ”+ Create Application Perspective”. In the dialog select one or more tags which make up your application perspective. Everything that matches these conditions is associated to this application, as well as any downstream database services. Tags can be conjoined with operators. Users can choose between AND or OR for each tag conjoin with the following tag. When using a mix of AND and OR conjunctions, AND conjunctions have a higher bounding then OR conjunctions. For example the tag definitions tag.A AND tag.B OR tag.c AND tag.d, Instana will interpret and process it like (tag.A AND tag.B) OR (tag.c AND tag.d).

Application Creation Dialog

Within the newly created application perspective you can update or delete an application perspective by selecting the “Configuration tab”. Please note that once an application is deleted, it will still appear in the inventory as long as it existed during the selected time window.

Services

A service can be seen as a logical component that provides a public API to the rest of the system, in which the API is made up of its endpoints. A service is being monitored and makes and receives calls. A request to a service results in a single call to a particular endpoint.

Services can be considered in isolation or through the lens of an application perspective. Services will often map to one ‘unit of deployment’, such as a package or container. If multiple instances of this e.g. container operate at the same time, they will all map to the same logical service.

Service types are assigned automatically through inheritance from endpoints. For example, if a service has both HTTP and BATCH endpoints, then it is assigned both HTTP and BATCH types. KPIs (Calls, Errors, Latency) for services display an aggregate of all calls, regardless of type.

Working with Services

Instana will automatically map services based on default rules. These rules examine call and infrastructure data and create a service as soon as it’s recognized. Additionally, teams can add a custom rule. See Service Mapping below for details.

Endpoints

Endpoints define the API of a service, and allow a fine-grained view into what operations the services exposes. Every call to a service happens on a single endpoint. And every endpoint has a single, automatically discovered type: BATCH, DATABASE, HTTP, MESSAGING, RPC. Like services, endpoints can also be viewed through the lens of an application perspective.

The endpoint can be statically declared or based on call tags (in contrast to the service, which is typically determined using infrastructure tags, e.g. springboot name). For example a combination of {call.http.method} {call.http.path} would be a typical endpoint of a http service.

Another benefit for defining separate endpoints for a service is that, mainly with ‘monoliths’, services can include many different frameworks and technologies; HTTP/REST APIs, database access, message bus integration, etc. By creating at least endpoints per type of API, and possibly further breaking apart such as protocol details, metrics and statistics can be captured per type of API.

Working with Endpoints

Instana will automatically map endpoints based on default rules. See Endpoint Mapping below for details.

Endpoint Grouping - Path Templates

Instana is smart. For many frameworks, Instana will automatically group endpoints based on their REST fingerprint. As an example, endpoints such as the following (which are being served by the same application code and hence have a shared performance profile):

/hospital/1948/patient/291148
/hospital/728/patient/924892
/hospital/47/patient/25978
/hospital/108429/patient/1847

…will be grouped and reported together as:

/hospital/{hid}/patient/{pid}

This is done automatically and no user steps are required for known frameworks.

If you are writing your own tracing code (e.g. with OpenTracing), see this section on how you can achieve the same results.

Configuration

Service Mapping

Services are an integral part of APM and show a logical view on your system. Services are derived from infrastructure components like hosts, containers, and processes. The act of assigning specific infrastructure components like containers to one or more services is referred to as service mapping.

In Instana we automatically map all services based on an extensive set of predefined rules. This way an Instana user in most cases doesn’t need to do anything. Services are automatically discovered and dashboards, metrics, and rules are made available.

Customizing Service Mapping

There are three ways of customizing the default service mapping: via a custom service rule, using the service.name tag or specifying the HTTP header X-Instana-Service.

Custom Service Rule

A good example for using a custom service rule is leveraging existing meta information on infrastructure components. For example, a user is labeling it’s Docker containers with domain specific information like com.acme.service-name:myservice. To map services from this label, click on “Configure Custom Services” in the service’s inventory and add the key: docker.label plus com.acme.service-name. All calls that now pass a container with some label com.acme.service-name are associated with a service which is named by that value, e.g. myservice. There are many other tags available to create custom service mapping.

You can also add multiple keys for service mapping. Note that all keys need to match. E.g., if you want to separate your staging services based on the host zone you could add two keys: host.zone + docker.label:com.acme.service-name. That way you would separate out the two services e.g. prod-myservice and dev-myservice.

Service Configuration Dialog

Using the call tag service.name

A user can also associate very specific calls by annotating these with service which enables very tailored mappings within the users code. The service annotation will be turned into the service.name tag for searching / analyzing calls.

There are several options for adding call annotations:

For more information regarding usage of the Java Trace SDK and naming of services, also see the respective Conversion and Naming section.

Specifying the X-Instana-Service HTTP header

By setting the X-Instana-Service HTTP header on a call, the destination (service receiving the HTTP request) will be tagged with the value provided in the header.

Predefined Rules

The following rules are considered in the order top to bottom. When rule matches, the respective service is created.

Rule Tags
User defined:
Custom service rule Tags defined by the user (see custom services )
Use call tag service.name {call.service.name}
Use HTTP header X-Instana-Service {call.http.header.X-Instana-Service}
Jaeger service name {call.tag.jaeger.service}
Zipkin service name {call.tag.zipkin.service}
Infrastructure based:
AWS ECS container name {docker.label.com.amazonaws.ecs.container-name}
AWS ECS task family {docker.label.com.amazonaws.ecs.task-definition-family}
Kubernetes container name {kubernetes.container.name}
Docker Swarm service name {docker.label.com.docker.swarm.service.name}
Marathon application id {marathon.app.id-1}
Nomad task name {nomad.task.name}
Rancher 1 project service name {docker.label.io.rancher.project_service.name}
Docker image name {docker.image.name-1}
JBoss / Tomcat deployment name (parsed) {call.deployment.name-1}
Redis port Redis@ {redis.port} on {host.name}
Neo4j port Neo4j@ {neo4j.port} on {host.name}
Memcached port Memcached@ {memcached.port} on {host.name}
Varnish port Varnish@ {varnish.port} on {host.name}
Clickhouse port Clickhouse@ {clickhouse.httpPort} on {host.name}
Cassandra cluster name {cassandra.cluster.name}
ElasticSearch cluster name {elasticsearch.cluster.name}
Couchbase cluster name {couchbase.cluster.name}
MongoDB database name MongoDB@ {mongo.port} on {host.name}
Zookeeper port Zookeeper@ {zookeeper.clientPort} on {host.name}
Solr version Solr@ {solr.version} on {host.name}
Solr Solr on {host.name}
PostgreSQL port PostgreSQL@ {postgresql.port} on {host.name}
MySQL port MySQL@ {mysql.port} on {host.name}
MSSQL database sid MSSQL@ {mssql.instance}
MariaDB port MariaDB@ {mariadb.port} on {host.name}
Kafka cluster name {kafka.cluster.name}
Kafka version Kafka@ {kafka.version} on {host.name}
ActiveMQ broker name {activemq.broker.name}
RabbitMQ version RabbitMQ@ {rabbitmq.version} on {host.name}
Dropwizard name {dropwizard.name}
Spring Boot name {springboot.name}
JBoss name {jboss.server.name} {jboss.node.name?}
WebLogic name {weblogic.server.name}
WebSphere name {websphere.server.name}
JVM name (parsed) {jvm.app.name-1}
JVM name {jvm.app.name}
Node.js application with host environment {nodejs.app.name}
Python snapshot name {python.app.name}
Ruby name {ruby.app.name}
PHP using host header (parsed) when available {call.http.host-1}
PHP / PHP-FPM worker pool PHP
CLR name {clr.app.name}
Call based:
Span HTTP service with host {call.http.host-1}
Span HTTP service parsing URL {call.http.url-1}
Span Database FTP service {call.database.connection}
MongoDB database name, using span data {call.database.schema-1}
ElasticSearch database name, always use connection from span data {call.database.connection-1}
Span Database schema {call.database.schema}
Span Database connection, extract schema from URI when schema empty {call.database.connection-1}
Span Database connection, extract schema from URI specifically for SQL Server {call.database.connection-1}
Span Database connection, extract schema from URI specifically for Oracle {call.database.connection-1}
Span Database service using host {call.database.connection-1}
Span Database type (when connection and schema empty just show the type) {call.database.type}
Span Messaging service using address {call.messaging.address-1}
Span Messaging service using address {call.messaging.address}
Span Messaging type (when address is empty just show the type) {call.messaging.type}
Span SDK name SDK

Endpoint Mapping

Endpoints are automatically mapped based on the endpoint type.

For example, HTTP endpoints map automatically based on path template if accessible. When the path template is not available, the endpoints are mapped to the first path segment or can be configured as desired

Customizing Endpoint Mapping

For each service, Instana allows to have one configuration which controls how the services endpoints are getting extracted. To access the configuration, navigate to a services dashboard, go to the endpoints tab and hit “Configure Endpoints”.

Application Dependency Map Overview

Note that custom endpoint configurations are available for http based services only.

Custom Endpoint Rules

Instana comes with three default HTTP rules which are always part of the extraction chain:

  • Detected Framework: Extracts endpoints as specified in detected framework (if accessible)
  • /*: Extracts endpoints based on the first path parameter
  • Unspecified: Calls that do not match a preceding rule are assigned to this endpoint

To specify additional configuration, you can define multiple custom rules. To do so, access the custom endpoint configuration dialog from any HTTP service and hit “Add Custom HTTP Rule”. The upcoming dialog offers you the ability to define a specific path (the actual rule) and multiple test cases to check if the defined path is working as expected. For the path you can use static segments like /api or /myShopEndpoint, path parameters like /{productId}, or match any segment with /*.

In the rule tester part of the dialog, you can define multiple test cases to validate rules. For example given a query /api/*/{version}, the following test case /api/anyName/123 will match, while /otherApi/anyName/123 will not.

Application Dependency Map Overview

Sequential Evaluation

Rules are applied from top to bottom, and calls are assigned to the first matching rule. The change sequence, simply reorder rules but just drag & drop. Instana default rules can be disabled, but not reordered.

Application Dependency Map Overview

The “Unspecified” Endpoint

When there is insufficient information to automatically map a call to an endpoint (eg. manually instrumented endpoints without providing sufficient data), those calls are mapped to an “Unspecified” endpoint.

Application Dependency Map

The dependency map is available for each application and provides:

  • an overview of the service dependencies within your application.
  • a visual representation of calls between services to understand communication paths and throughput.
  • different layouts to quickly gain an understanding of the application’s architecture.
  • easy access to service views (dashboards, flows, calls and issues).

Application Dependency Map Overview

Error Messages

Error messages are all messages collected from errors happening during code execution of a service. For example, if an exception is thrown during processing and it is not caught and handled by the application code, this call together with the error message will be listed on the “Error Messages” tab. An example would be an unhandled exception in a Servlet’s doGet method that causes the request to be responded to with HTTP 500.

Log Messages

Log Messages are collected from instrumented logging libraries/frameworks (see for example the section “Logging” in the list of supported libraries). When a service writes a log message with severity WARN or higher via the logging library, the message passed to this call will be displayed on the “Log Messages” tab. In addition, writing a log message with severity ERROR or higher will mark the current call as erroneous. Log messages with a severity lower than WARN are not tracked.

Infrastructure

From the Application Perspective view or Services dashboard it is possible to navigate to the corresponding infrastructure component shown on the Infrastructure Monitoring view.

The “Unknown” Infrastructure Component

The list of infrastructure components for an application or service might sometimes show or include the “Unknown” host / container / process.

Unknown Infrastructure Component

The “Unknown” component indicates that for some or all calls to this service, we were unable to link it to a specific infrastructure component. As Services are “logical” entities, we are often able to link it to infrastructure components via the monitored process. This does not hold for example for third-party web services, which we don’t monitor but where we still create Services and Endpoints based on host-name + path. Since no host or process is known, these services would be resulting in the “Unknown” infrastructure component being shown.