Custom Tracing

Instana aims to provide fully automated out-of-the-box tracing instrumentation but in some cases this may not always possible or preferred. In such cases, users may prefer to use OpenTracing, OpenCensus, the Web Trace SDK or one of the many language based tracing SDKs to write their own custom instrumentation.

Instana provides its best value when it can rely on consistent, well-formed and complete spans and tags. While out instrumentation does it out-of-the-box, achieving the same level with manual instrumentation requires some knowledge, which this page endeavors to provide.

Please refer to Tracing Concepts page for definitions of terms used on this page.

Supported Tracing Technologies & SDKs

There are many ways you can send custom traces to your Instana dashboard. The sections below outline the currently supported technologies and SDKs.

Instana Language Tracing SDKs

Instana provides a Tracing SDK alongside many of the supported languages and an external SDK which can be utilized by any language:

OpenCensus

OpenCensus is tracing and metrics API and implementation available in several languages. Vendor-specific “exporters” allow to use OpenCensus with formats different than the default one. Currently Instana provides “Exporters” (the logic that translates spans defined via the OpenCensus model to Instana’s) for tracing data for the following languages:

OpenTracing

OpenTracing is a tracing API that relies on custom vendor-specific “tracers” to translate API invocations to spans to be sent to the backend for analysis. Currently Instana providers OpenTracing tracers for the following languages:

Tips & Best Practices

Use Well Annotated Spans for Better Processing

Instana performs advanced processing and analysis on all incoming data to best monitor, learn from and alert on your applications and infrastructure. This includes all incoming spans from all sources.

To get the most benefit out Instana’s automatic analysis and processing, adding the appropriate tags to your spans allows Instana to best analyze and act upon incoming spans.

For example, the following Python OpenTracing code provides very little information.

Note: this example uses OpenTracing to illustrate but the intention applies to all custom tracing code

import opentracing

with opentracing.tracer.start_active_span('vanilla') as pscope:
  # ...
  # do something that takes 50ms
  # ...
  pscope.span.log_kv({"foo": "bar"})  

From this code, we know only that it is a span named vanilla that took 50ms of time. We don’t know the type of span it was such as an HTTP, RPC or Messaging span. We don’t know what happened during that time (such as communicating with any other component in your infrastructure).

Instead, if you provide the appropriate contextual OpenTracing tags, Instana can better analyze and extract information from that span to act on.

import opentracing
import opentracing.ext.tags as ext

with opentracing.tracer.start_active_span('webserver') as pscope:
    pscope.span.set_tag(ext.SPAN_KIND, "entry")
    pscope.span.set_tag(ext.PEER_HOSTNAME, "localhost")
    pscope.span.set_tag(ext.HTTP_URL, "/python/simple/two")
    pscope.span.set_tag(ext.HTTP_METHOD, "POST")
    pscope.span.log_kv({"foo": "bar"})

    # ...
    # work that took 50ms
    # ...

    pscope.span.set_tag(ext.HTTP_STATUS_CODE, 204)

This well annotated span tells Instana much more about what happened in the context of this span. From the tags provided we know that this is an incoming webserver request to /python/simple/two and the resulting HTTP status code was 204.

A well annotated span such as above allows Instana to extract services, monitor connections (and their health) and overall provide a richer experience in your dashboard.

For specifics on this example, see the OpenTracing specification which defines the authoritative list of all supported OpenTracing tags. A comprehensive list of tags supported by Instana (which is a superset of OpenTracing tags) is available at the end of this page.

Always start new traces with Entry spans

For distributed tracing, there are 3 major types of spans:

  • Entry: Annotates the reception of a request, the start of a new task or consuming a message
  • Intermediate: Annotates internal work that doesn’t make or receive calls (e.g. view rendering)
  • Exit: Annotates work that makes an call to a remote service or publishes a message to a queue

The tag span.kind marks the type of span being reported. See span.kind in the table at the end of this page or alternatively in the OpenTracing specification for more details.

When starting new traces, make sure to start with an Entry span. Unless you specify the span.kind tag, the span will be considered of type Entry. Setting this tag to the appropriate value will allow for better processing, visualization and backend call extraction/mapping in Instana.

See the following example for Go:

// Start a new entry span and set 'kind' to Consumer (entry)
entrySpan := ot.StartSpan("RPCJobRunner")
entrySpan.SetTag(string(ext.SpanKind), string(ext.SpanKindConsumerEnum))

// Now the RPC exit span
spanName := fmt.Sprintf("%s %s", request.Service, request.Method)
clientSpan := ot.StartSpan(spanName, ot.ChildOf(entrySpan.Context()))
clientSpan.SetTag(string(ext.SpanKind), string(ext.SpanKindRPCClientEnum))
clientSpan.SetTag("rpc.call", request.Method)

// Make the RPC call

clientSpan.Finish()
entrySpan.Finish()

Marking a Span with an Error

Spans that represent work that contained an error (such as an exception) should include the error and message tags. For definitions of these tags, see the definition in the table below.

Passing Context Between Microservices

Distributed tracing includes a method to pass context across boundaries such as hosts, queues and services. This is usually done with carriers such as HTTP headers or message queue headers.

See our documentation on HTTP Headers for more details.

Optional: Set a Service Name

Service names are names applied to monitored application processes. In many languages, the best service name can be automatically detected based off of the Framework in use or the process command line.

If you want to override this, you can set a service name per Tracer. See the following example for the Go language.

serviceName := "default"
if *isServer {
    serviceName = "rpc-client"
} else {
    serviceName = "rpc-server"
}

opts := instana.Options{Service: serviceName})
opentracing.InitGlobalTracer(instana.NewTracerWithOptions(&opts)

All Instana OpenTracing Tracers also support this configuration via environment variable. If you set the environment variable INSTANA_SERVICE_NAME for the process, the value will be used as the service name for the process. This overrides any code level service name configuration.

See also the documentation on Instana’s overall service extraction techniques and rules.

Path Templates: Visual Grouping of HTTP Endpoints

Instana supports automatic grouping of endpoints with path templates. This is supported out of the box with Instana tracing for many frameworks. With OpenTracing, an additional step is required as described below.

Various frameworks usually have a REST path pattern similar to the following:

/api/query/1956-01-31/Guido-van-Rossum
/api/query/1976-04-18/Andrew-Ng
/api/query/1912-06-23/Alan-Turing

These similar endpoints can be grouped by reporting a http.path_tpl key in your HTTP spans with the value /api/query/{birthdate}/{name}, Instana will use this template to automatically group endpoints that match the supplied pattern. In your Instana dashboard, the HTTP endpoints will then be grouped as as single endpoint:

/api/query/{birthdate}/{name}

See the following for an example in Go:

span := ot.GlobalTracer().StartSpan("myTemplatedSpan", ext.RPCServerOption(incomingContext))
span.SetTag("http.path_tpl", "/api/query/{birthdate}/{name}")
span.SetTag(string(ext.SpanKind), string(ext.SpanKindRPCServerEnum))
span.SetTag(string(ext.HTTPUrl), req.URL.Path)
span.SetTag(string(ext.HTTPMethod), req.Method)
span.SetTag(string(ext.HTTPStatusCode), 200)

Note that this feature is Instana specific and not OpenTracing compliant.

List of Processed Tags

This section lists the specific tags that Instana looks for when identifying and processing spans. When a subset of the tags below are found for a span, Instana can then better process, analyze, visualize and alert on the incoming spans.

*Note that the tags listed below are a superset of the standard OpenTracing tags. Each tag is marked on whether it is OpenTracing compliant with either a or x if it is not.*

Some tag descriptions below are taken directly from the OpenTracing semantic conventions document.

Kind

In the distributed tracing world, there are three major categories of spans:

  1. Entry spans: spans that receives requests (such as HTTP or RPC servers), consumes messages from a queue or runs a job
  2. Exit spans: spans that make client requests (e.g. HTTP, RPC or database), push messages onto a queue or makes client database calls
  3. Intermediate spans: spans that represent work that is done in the applications such as view rendering or action/controller processing.

The span.kind tag for a span identifies the type of span that is being reported. Reporting this tag allows Instana to extract, process and map out the communications between spans. At Instana, we refer to this communication as a “call”.

Having such information allows Instana to not only monitor spans, but also monitor the calls between spans giving greater insight into the connectivity between your systems. With this in place, Instana can then also monitor and alert on the health of these calls as spans are being reported.

Tag Type OpenTracing Compliant? Description
span.kind String Either client or server for the appropriate roles in an RPC or HTTP request, and producer or consumer for the appropriate roles in a messaging scenario. Other values accepted are: entry, exit or intermediate which are not OpenTracing compliant.

HTTP

HTTP spans are those that represent an HTTP client call or HTTP server request processing.

Tag Type OpenTracing Compliant? Description
http.url String The fully qualified HTTP URL used in this HTTP client request or server processing.
http.method String The HTTP method used in this HTTP client request or server processing. Examples are “GET”, “POST”, “PUT” etc..
http.status_code Integer The HTTP status code of this HTTP request.
http.status Integer x Alternative to http.status_code. Both are supported although only one should be sent.
http.path String x The HTTP path of the request.
http.host String x The remote host if a client request or the host handling an incoming HTTP request.
http.params String x The query parameters of the HTTP request
http.error String x In the case of an error, an error message associated with this span. e.g. “Internal Server Error”
http.header String x Used to report custom headers in relation so this span such as “X-My-Custom-Header=afd812cab”
http.path_tpl String 𐄂 Allows for visual grouping of endpoints. See the Path Templates documentation for details

Notes:

  1. An appropriate span.kind should always be sent with these tags.
  2. http.host, http.path and http.params should only be sent in place of http.url. Sending a mix of these tags with http.url is not supported and the results are undefined.

RPC

RPC spans are those that represent work done in an RPC call or RPC server call processing.

Tag Type OpenTracing Compliant? Description
rpc.call String The RPC call being called or serviced (depends on value of span.kind)
rpc.host String The RPC remote host for client calls or the RPC host handling the request if an RPC server.
rpc.params String x Parameters for the RPC call
rpc.port String x Port for the RPC call
rpc.flavor String x Flavor of the RPC library in use such as XMLRPC, GRPCIO etc…
rpc.error String x Error message associated with the RPC call

Note: An appropriate span.kind should be sent with these tags.

Database

Tag Type OpenTracing Compliant? Description
db.instance String The database instance name. As an example in Java, if the jdbc.url="jdbc:mysql://127.0.0.1:3306 customers", the instance name is “customers”
db.type String For any SQL database, “sql”. For others, the lower-case database category, e.g. “cassandra”, “hbase”, or “redis”.
db.statement String A database statement for the given database type. For example, when db.type is “SQL” - SELECT * FROM user_table; when db.type is “redis” - SET mykey 'WuValue'.
db.user String Username for accessing database. E.g. “readonlyuser” or “reportinguser”

Messaging

Tag Type OpenTracing Compliant? Description
message_bus.destination String An address at which messages can be exchanged. Can be a topic or queue etc..

Peer

Tag Type OpenTracing Compliant? Description
peer.hostname String The remote host of an exit span making an outgoing call.
peer.address String The remote address of an exit span making an outgoing call.
peer.service String The service name of the remote side of an exit span making an outgoing call. Note that this is optional and may be used when the remote side is not already instrumented.

Errors

Tag Type OpenTracing Compliant? Description
error Boolean Indicates if an error was encountered during the time this span represents.
message String 𐄂 A message associated with the error

See Also