Java Trace SDK

Using the Instana Java Trace SDK (github), it is possible to manually instrument Entries and Exits that Instana does not yet know, as well as markup interesting sections of custom code. It also supports creating custom correlation across binary protocols and adding custom defined key value pairs to spans (for extracting relevant information).

Exits and Entries for a Custom Binary Protocol

The published example app (https://github.com/instana/instana-java-sdk/tree/master/instana-java-sdk-sample) contains a CustomTCP server and client, illustrating how the SDK can be used for marking entries, exits, and performing correlations.

Using the instana-java-sdk

The Java Trace SDK requires you to add a small jar file to your application. This jar file contains the annotations and helper classes used to markup the sections of code that Instana should handle. When using Maven, the dependency can be easily added with:

<dependency>
  <groupId>com.instana</groupId>
  <artifactId>instana-java-sdk</artifactId>
  <version>1.0.1</version>
</dependency>

The instana-java-sdk is available from the default Maven central repository.

When no agent is monitoring the JVM, which includes the above library and has sections marked with annotations, then they act as No-Op. It is safe to use the annotations, they do not have any impact at all as long as no agent is monitoring the JVM. To have an Instana agent actually make use of the annotations, you have to tell it which Java package names have those annotations. This is because scanning for annotations is a heavyweight process, and scanning the complete classpath of big applications can take a long time.

The required configuration.yaml section looks like this:

# Java Tracing
com.instana.plugin.javatrace:
 instrumentation:
 # By default no packages are scanned for SDK annotations.
 sdk:
 packages:
 - 'com.mycompany.backend'
 - 'com.mycompany.frontend'

The packages are scanned recursively. That is, if the com.mycompany.backend is configured to be scanned for annotations, com.mycompany.backend.impl and other sub-packages will be scanned as well.

Marking an Entry

To mark a method in an Entry span, just add the following annotation (as shown in the example at https://github.com/instana/instana-java-sdk/blob/master/instana-java-sdk-sample/src/main/java/com/acme/tcp/server/CustomTCPServer.java#L55): 

@Span(type = Type.ENTRY, value = "custom-tcp-server")

As soon as Instana sees code entering this method, a trace is started.

Marking an Exit

To mark a method in an Exit span, just add the following annotation (as shown in the example at https://github.com/instana/instana-java-sdk/blob/master/instana-java-sdk-sample/src/main/java/com/acme/tcp/server/CustomTCPClient.java#L14):

@Span(type = Type.EXIT, value = "custom-tcp-client", capturedStackFrames = 5)

Whenever this method is encountered, Instana will record a Span labeling this as an Exit call.

Correlating Exit and Entry

In many cases where an Exit and Entry are not yet known to Instana, applications use them to perform some kind of remote communication. When an Exit is encountered and no correlation is performed, the trace “breaks,” meaning two traces would be observed, one ending at the Exit, and a new one starting at the Entry.

It is possible to help Instana perform a correlation by manually transporting correlation information across the remote communication. On the exit side, the exit annotation will make sure that the required correlation identifiers exist. One can simply invoke (https://github.com/instana/instana-java-sdk/blob/master/instana-java-sdk/src/main/java/com/instana/sdk/support/SpanSupport.java#L182):

SpanSupport.addTraceHeadersIfTracing(Type.EXIT, params);

to fill the given params map with the required IDs. Should the protocol support other means of transporting other data, it is possible to get the IDs with: 

currentTraceId(type)

currentSpanId(type)

On the entry side, it is important to read the parameters and set them for the Entry Span, before the span is created by annotations:

(https://github.com/instana/instana-java-sdk/blob/master/instana-java-sdk-sample/src/main/java/com/acme/tcp/server/CustomTCPServer.java#L50):

SpanSupport.inheritNext(SpanSupport.stringAsId(trace), SpanSupport.stringAsId(span));

By doing so, the Entry Span will automatically join the remote trace and add itself as child span of the calling Exit.

Conversion and Naming

In certain situations it might be beneficial to transform SDK spans into a different type such as HTTP or RPC, to better represent the origin of these spans. This can be done by making use of the fields as defined in the Open Tracing Semantic Conventions. Conversions to the following types are possible:

  • HTTP: depends on http.url, optional are http.method and http.status_code
  • Database: depends on db.instance and db.statement, optional are db.type and db.user
  • RPC: depends on rpc.call (not defined by Open Tracing) and one of the Open Tracing peer.address, peer.hostname or peer.service fields

These fields can be set using the SpanSupport.annotate(type, name, key, value) method.

Instana will treat these fields the way automatic agent instrumentation works as soon as these fields are nested under the tags. key. So, for example, to set the http.url field, the method call would look like this;

SpanSupport.annotate(<type>, <name>, "tags.http.url", "service://my-awesome-service/some/path")

When semantic OpenTracing fields are correctly tagged, the respective rules will be used for Service and Endpoint naming as defined in Application & Service Management. When no specific fields are provided, the name of the Service for all SDK spans will be SDK. The Endpoint name will be the given name in either the @Span annotation or the SpanSupport.annotate() method.

Troubleshooting

Should the SDK traces not show up in the UI, several items are worth checking:

  • Is the instana-java-sdk jar shipped with the application? If not, the agent will silently do nothing, as it is in no-op mode.
  • Does the configuration.yaml file reference all the packages with annotations?
  • Is tracing active?
  • Is the yaml syntax correct?
  • Are other APM agents active? Most other APM agents are known to interfere with Instana, thus Instana’s agent will not perform tracing if it detects other agents. If this is the case, the following will appear in the logs:

    JVM <PID> is running the <3rdParty> agent, which is known to be causing problems for the Instana Agent. Tracing will not be enabled for this JVM.
  • Turn on DEBUG logging and look for a message like:

    2016-09-05T08:26:14.094+0200 | DEBUG | na-http-thread-1 | LoggerEndpoint                   | 48 - com.instana.agent - 1.1.194 | JVM (11403). Successfully instrumented class com.acme.resources.HelloWorldResource [sun.misc.Launcher$AppClassLoader@3d4eac69]`
    2016-09-05T08:26:14.145+0200 | DEBUG | na-http-thread-1 | LoggerEndpoint                   | 48 - com.instana.agent - 1.1.194 | JVM (11403) - Spent 188ms and 124kb transforming SDK classes. 
  • Are the annotated methods actually hit by a user/load driver? The Instana agent will interact with each JVM shortly after it has been started. As a result, transactions that happen before the agent attached are not captured. Make sure to annotate methods that are normally hit during the lifetime of the application.