Using the Instana Java Trace SDK (github), it is possible to manually instrument Entries and Exits that Instana does not yet know, as well as markup interesting sections of custom code. It also supports creating custom correlation across binary protocols and adding custom defined key value pairs to spans (for extracting relevant information).
The published example app (https://github.com/instana/instana-java-sdk/tree/master/instana-java-sdk-sample) contains a CustomTCP server and client, illustrating how the SDK can be used for marking entries, exits, and performing correlations.
The Java Trace SDK requires you to add a small jar file to your application. This jar file contains the annotations and helper classes used to markup the sections of code that Instana should handle. When using Maven, the dependency can be easily added with:
<dependency> <groupId>com.instana</groupId> <artifactId>instana-java-sdk</artifactId> <version>1.0.1</version> </dependency>
The instana-java-sdk is available from the default Maven central repository.
When no agent is monitoring the JVM, which includes the above library and has sections marked with annotations, then they act as No-Op. It is safe to use the annotations, they do not have any impact at all as long as no agent is monitoring the JVM. To have an Instana agent actually make use of the annotations, you have to tell it which Java package names have those annotations. This is because scanning for annotations is a heavyweight process, and scanning the complete classpath of big applications can take a long time.
configuration.yaml section looks like this:
# Java Tracing com.instana.plugin.javatrace: instrumentation: # By default no packages are scanned for SDK annotations. sdk: packages: - 'com.mycompany.backend' - 'com.mycompany.frontend'
The packages are scanned recursively. That is, if the
com.mycompany.backend is configured to be scanned for annotations,
com.mycompany.backend.impl and other sub-packages will be scanned as well.
To mark a method in an Entry span, just add the following annotation (as shown in the example at https://github.com/instana/instana-java-sdk/blob/master/instana-java-sdk-sample/src/main/java/com/acme/tcp/server/CustomTCPServer.java#L55):
@Span(type = Type.ENTRY, value = "custom-tcp-server")
As soon as Instana sees code entering this method, a trace is started.
To mark a method in an Exit span, just add the following annotation (as shown in the example at https://github.com/instana/instana-java-sdk/blob/master/instana-java-sdk-sample/src/main/java/com/acme/tcp/server/CustomTCPClient.java#L14):
@Span(type = Type.EXIT, value = "custom-tcp-client", capturedStackFrames = 5)
Whenever this method is encountered, Instana will record a Span labeling this as an Exit call.
In many cases where an Exit and Entry are not yet known to Instana, applications use them to perform some kind of remote communication. When an Exit is encountered and no correlation is performed, the trace “breaks,” meaning two traces would be observed, one ending at the Exit, and a new one starting at the Entry.
It is possible to help Instana perform a correlation by manually transporting correlation information across the remote communication. On the exit side, the exit annotation will make sure that the required correlation identifiers exist. One can simply invoke (https://github.com/instana/instana-java-sdk/blob/master/instana-java-sdk/src/main/java/com/instana/sdk/support/SpanSupport.java#L182):
to fill the given params map with the required IDs. Should the protocol support other means of transporting other data, it is possible to get the IDs with:
On the entry side, it is important to read the parameters and set them for the Entry Span, before the span is created by annotations:
By doing so, the Entry Span will automatically join the remote trace and add itself as child span of the calling Exit.
In certain situations it might be beneficial to transform SDK spans into a different type such as HTTP or RPC, to better represent the origin of these spans. This can be done by making use of the fields as defined in the Open Tracing Semantic Conventions. Conversions to the following types are possible:
- HTTP: depends on
http.url, optional are
- Database: depends on
db.statement, optional are
- RPC: depends on
rpc.call(not defined by Open Tracing) and one of the Open Tracing
These fields can be set using the
SpanSupport.annotate(type, name, key, value) method.
Instana will treat these fields the way automatic agent instrumentation works as soon as these fields are nested under the
tags. key. So, for example, to set the
http.url field, the method call would look like this;
SpanSupport.annotate(<type>, <name>, "tags.http.url", "service://my-awesome-service/some/path")
When semantic OpenTracing fields are correctly tagged, the respective rules will be used for Service and Endpoint naming as defined in Application & Service Management.
When no specific fields are provided, the name of the Service for all SDK spans will be
SDK. The Endpoint name will be the given
name in either the
@Span annotation or the
Should the SDK traces not show up in the UI, several items are worth checking:
- Is the
instana-java-sdkjar shipped with the application? If not, the agent will silently do nothing, as it is in
- Does the
configuration.yamlfile reference all the packages with annotations?
- Is tracing active?
- Is the yaml syntax correct?
Are other APM agents active? Most other APM agents are known to interfere with Instana, thus Instana’s agent will not perform tracing if it detects other agents. If this is the case, the following will appear in the logs:
JVM <PID> is running the <3rdParty> agent, which is known to be causing problems for the Instana Agent. Tracing will not be enabled for this JVM.
Turn on DEBUG logging and look for a message like:
2016-09-05T08:26:14.094+0200 | DEBUG | na-http-thread-1 | LoggerEndpoint | 48 - com.instana.agent - 1.1.194 | JVM (11403). Successfully instrumented class com.acme.resources.HelloWorldResource [sun.misc.Launcher$AppClassLoader@3d4eac69]` 2016-09-05T08:26:14.145+0200 | DEBUG | na-http-thread-1 | LoggerEndpoint | 48 - com.instana.agent - 1.1.194 | JVM (11403) - Spent 188ms and 124kb transforming SDK classes.
- Are the annotated methods actually hit by a user/load driver? The Instana agent will interact with each JVM shortly after it has been started. As a result, transactions that happen before the agent attached are not captured. Make sure to annotate methods that are normally hit during the lifetime of the application.
The instrumentation will call
toString() on each captured parameter and return value. This means that the values
visible within Instana directly depend on the specific
toString() method implementations of the parameter and
return value classes.
toString() is not overridden, then
java.lang.Object#toString() is eventually used. This method returns
getClass().getName() + '@' + Integer.toHexString(hashCode()) and results in the format you are seeing.
To improve visibility into captured parameters and return values, we recommend to properly override
Return values may not be available even though
captureReturn=true is set, when…
- an exception is thrown by the method,
- the method has a
voidreturn type or when
- the method returns