AWS Lambda Native Tracing

As an addition to infrastructure monitoring for AWS Lambdas (and as an alternative to X-Ray integration), Instana offers native tracing for AWS Lambda functions. This feature is currently available for AWS Lambda functions based on the Node.js runtime (Node.js 8 or later).

Native tracing of AWS Lambda functions provides three big advantages over the previous X-Ray based tracing:

  • The tracing of Lambda functions now seamlessly correlates with tracing of other components monitored by the Instana agent.
  • Instana can now diffentiate between traces coming from different versions of the same AWS Lambda function.
  • Native tracing reduces the need to pull X-Ray data from AWS, which can get expensive.

Note: At this time, native tracing for AWS Lambda functions is exclusively available for the Instana SaaS back end. For on-prem installations, this functionality will become available later.

Pre-Requesites

  1. Ensure you have set up AWS agents that monitor the AWS regions in which you deploy Lambda functions.
  2. Activate the monitoring of individual Lambda versions in the AWS Agent; see below for details.
  3. The handlers of your Lambda functions need to be configured for native tracing with Instana.

Enabling Monitoring of Individual Lambda Versions in the Agent

See our general AWS docs and the AWS Agent Installation docs for details on how to monitor AWS services with the Instana agent.

By default, the Instana AWS Lambda sensor will try to monitor each version of a Lambda function individually. This is a requirement for linking Lambda calls correctly to Lambda infrastructure. However, this requires the permission lambda:ListVersionsByFunction so please make sure that the IAM role you use in the AWS agent has that permission.

Configuring an AWS Lambda For Native Tracing

AutoTrace AWS Lambdas

The preferred way to enable Instana tracing for your AWS Lambda functions is to use the Instana Lambda layer and the provided auto-wrap handler. This approach requires no modification of the Lambda function code and is purely configurational. It is also suited to be automated or included in a Lambda deployment pipeline.

Note: AutoTrace Lambda works for version 10.x or later of the Node.js runtime. For Node.js 8.10, please use the approach outlined in the section Instana Lambda Layer & Manual Wrapping.

  1. Add the Instana Lambda layer to your function:

    1. In the configuration page for your Lambda function, click on the "Layers" box below the function name.
    2. In the "Layers" section, click on "Add a layer".

      lambda config 1 1 layer

    3. In the popup that opens, select "Provide a layer version ARN".
    4. Copy and paste the ARN for your AWS region into the ARN text input field. You can find the ARN of the Instana Lambda layer for your region in the table below.

      lambda config 1 2 layer selection

  2. Take note of what the function’s “Handler” is currently set to.

    1. Select the function name again instead of the "Layers" box to display the function code editor.
    2. Inspect the content of the field "Handler". The default value when creating a new Node.js Lambda function from scratch is `index.handler` but it might be a different value for your Lambda function.

      lambda config 2 handler old

  3. Change the value of the “Handler” field to instana-aws-lambda-auto-wrap.handler. The Lambda configuration page might display a warning like “Lambda can’t find the file instana-aws-lambda-auto-wrap.js.” This might also be displayed later when returning to the configuration page. Do not let this warning distract you. The handler is contained in the Instana Lambda layer but apparently the AWS Lambda configuration page is not smart enough to figure that out.

    lambda config 3 handler new

  4. Provide your actual handler (that is, the value you noted down in step 2) as the environment variable LAMBDA_HANDLER. If you have used the default handler index.handler previously, you can omit this environment variable as this is the default the Instana Lambda layer will fall back to. If your handler was different from index.handler, you need to provide it.
  5. Add the environment variables INSTANA_ENDPOINT_URL and INSTANA_AGENT_KEY with the required values (see below).

    lambda config 4 env vars

  6. Save the Lambda function definition by clicking on “Save”.

    lambda config 6 save

All of this can be done either via the AWS web console or any of the usual AWS management tools, like

Here is an example aws CLI command that might serve as a starting point if you want to automate the Instana integration of your AWS Lambdas:

# Do not copy and paste this verbatim!
# It will overwrite any previously defined collection of layers and
# environment variables.
aws --region $YOUR_REGION lambda update-function-configuration \
  --function-name $YOUR_LAMBDA_FUNCTION_NAME \
  --layers $INSTANA_LAYER_ARN \
  --handler instana-aws-lambda-auto-wrap.handler
  --environment ""Variables={LAMBDA_HANLDER=yourActual.handler,INSTANA_ENDPOINT_URL=...,INSTANA_AGENT_KEY=...}""

The Instana Lambda Layer

As mentioned above, Instana provides an AWS Lambda layer that takes care of automatically adding Instana tracing and metrics collection to your Lambda function, when configured appropriately.

Here are the ARNs of the latest version of the layer at the time of writing, per region:

Region ARN
eu-central-1 arn:aws:lambda:eu-central-1:410797082306:layer:instana:15
eu-north-1 arn:aws:lambda:eu-north-1:410797082306:layer:instana:15
eu-west-1 arn:aws:lambda:eu-west-1:410797082306:layer:instana:15
eu-west-2 arn:aws:lambda:eu-west-2:410797082306:layer:instana:15
eu-west-3 arn:aws:lambda:eu-west-3:410797082306:layer:instana:15
sa-east-1 arn:aws:lambda:sa-east-1:410797082306:layer:instana:15
us-east-1 arn:aws:lambda:us-east-1:410797082306:layer:instana:15
us-east-2 arn:aws:lambda:us-east-2:410797082306:layer:instana:15
us-west-1 arn:aws:lambda:us-west-1:410797082306:layer:instana:15
us-west-2 arn:aws:lambda:us-west-2:410797082306:layer:instana:15

That is, the pattern is arn:aws:lambda:${region}:410797082306:layer:instana:${layer-version}.

Please make sure to check what the latest version of the layer currently is and always use the latest versions to benefit from new features and fixes that we provide when pubishing a new version of the layer.

Configuring the Instana Back End Connection

Add the following two environment variables to your Lambda function definition:

Key Value
INSTANA_ENDPOINT_URL the reporting URL for your tenant unit (see below)
INSTANA_AGENT_KEY the Instana agent key for your tenant unit

The value for INSTANA_ENDPOINT_URL depends on where your Instana tenant unit is deployed:

Your Instana Unit’s Region INSTANA_ENDPOINT_URL
US https://serverless-us-west-2.instana.io/
EU https://serverless-eu-west-1.instana.io/

The agent key for INSTANA_AGENT_KEY is the same that you use in all installed Instana agents. If you do not know your key, please contact our sales team to help you out.

Instana Lambda Layer & Manual Wrapping

You can also add the Instana Lambda layer to your function as described above, but refrain from using instana-aws-lambda-auto-wrap.handler. In particular, this is the recommended approach for AWS Lambdas based on the “Node.js 8.10” runtime. Here are the required steps for this setup:

  1. Add the Instana Lambda layer to your function. You can find the ARN of the Instana Lambda layer in the table above.
  2. Add the environment variables INSTANA_ENDPOINT_URL and INSTANA_AGENT_KEY with the required values (see above).
  3. Modify the function’s code according to the section Manually Wrapping the Handler.

Installing @instana/aws-lambda Manually

Instead of using the Instana Lambda layer, you can also choose to install the npm package @instana/aws-lambda manually:

  1. Add the dependency @instana/aws-lambda to your project by executing npm install -S @instana/aws-lambda (or yarn add @instana/aws-lambda) in your project directory. This will add the package to your node_modules folder and save the dependency in your package.json file.
  2. Add the environment variables INSTANA_ENDPOINT_URL and INSTANA_AGENT_KEY with the required values (see above).
  3. Modify the function’s code according to the section Manually Wrapping the Handler.

Manually Wrapping the Handler

If you do not want to use the auto-wrap handler, you will need to modify the code of your Node.js AWS Lambda function slightly to enable Instana tracing for it.

Note: The code modifications described in this section are not required if you use the Instana Lambda layer and the auto-wrap handler as described in the section AutoTrace AWS Lambdas.

Note: If you bundle your Lambda handler with webpack, this approach is not recommended, since Instana’s core packge does not support being pre-processed with webpack. The AutoTrace AWS Lambdas approach is suitable for Lambdas using webpack. As an alternative, you can use manual wrapping and exclude, at a minium, the package @instana/aws-lambda (or all dependencies) from being preprocessed by webpack. See this section in the Node.js documentation; for the serverless framework please also refer to its own documentation regarding externals.

  1. Add the line const instana = require('@instana/aws-lambda'); to the very top of your handler JavaScript file.
  2. Wrap your handler function in an instana.wrap() call.
  3. Don’t forget to configure the Instana back end to connect to.

Here are some before/after examples for the different handler function styles that can be used for Node.js based AWS Lambdas:

Async Function Style Handler

If you use an async function as your handler, it should look like this:

exports.handler = async (event, context) => {
  // your code
};

The resulting code should look like this:

const instana = require('@instana/aws-lambda');

exports.handler = instana.wrap(async (event, context) => {
  // your code
}); // <- don't forget the closing ) for the instana.wrap(

Promise Style Lambda Handler

If you use a promise style handler, it should look like this:

exports.handler = (event, context) => {
  // your code, which returns a promise
};

The resulting code should look like this:

const instana = require('@instana/aws-lambda');

exports.handler = instana.wrap((event, context) => {
  // your code, which returns a promise
}); // <- don't forget the closing ) for the instana.wrap(

Callback Style Lambda Handler

If you use a callback style handler, it should look like this:

exports.handler = (event, context, callback) => {
  // your code
};

The resulting code should look like this:

const instana = require('@instana/aws-lambda');

exports.handler = instana.wrap((event, context, callback) => {
  // your code
}); // <- don't forget the closing ) for the instana.wrap(

Configuration Object

You can also pass in an optional configuration object as the first argument when wrapping your handler:

exports.handler = instana.wrap({
    // ... your configuration, for example:
    tracing: {
      stackTraceLength: 10
    }
  },
  async (event, context) => {
  // your code
});

Configuration values that are not supported in native Lambda tracing (like agentHost, agentPort, agentName, reportUncaughtException and reportUnhandledPromiseRejections) will be silently ignored.

Note that you can also use the environment variables listed on the configuration page.

Caveats

HTTP Call Attributes, API Gateway & Lambda Proxy Integration

Instana offers detailed capturing of HTTP attributes for Lambda executions that are triggerd by a trigger of type “API Gateway” or “Application Load Balancer”. This includes extracting the URL, path templates, the status code, query parameters etc. The standard enpoint extraction uses this attributes, too.

However, for API Gateway calls, HTTP attributes can only be captured if the option “Use Lambda Proxy integration” is used when defining the API Gateway’s methods. After the creation of a an API Gateway’s methods, this can be checked by inspecting the “Integration Request” box on the API Gateway configuration page. If it says “Type: LAMBDA PROXY”, it uses the Lambda Proxy integration.

These constraints do not apply to “Application Load Balancer” triggers.

Fire & Forget Calls

If you use fire-and-forget style calls in your Node.js Lambda function, Instana cannot guarantee to capture those reliable. A fire-and-forget call is any asynchronous call for which you do not wait for the result. Here is an example:

const fetch = require('node-fetch');

exports.handler = async event => {
  fetch('https://example.com/fire-and-forget'); // No await here!

  const response = await fetch('https://example.com/wait-for-me');

  return {
    message: response.text()
  };
}

This Lambda handler uses the node-fetch package to make two HTTP calls. The second call is called with the async keyword, thus execution will wait for it to finish. This call will be traced reliably.

In contrast to this, the code does not wait for the result of the first call (fetch('https://example.com/fire-and-forget')). That call might have finished by the time the Lambda execution finishes or not. If it has not finished when the Lamba execution as a whole terminates, the call will not be reported to the Instana backend. If you inspect a corresponding trace in the Instana UI you might see a warning that says “Parent span is missing for some of the entry spans in this trace during the processing.”

This example used the async/await style, but similar examples also exist for promise-style and callback-style code.

Older Node.js Lambda Runtimes

At the time of writing, Amazon offers Node.js 12, 10 and 8 as Lambda runtimes. Instana does not support older Node.js Lambda runtimes that have already been decomissioned by AWS (e.g. Node 4 and 6). If you still have Lambdas deployed that use those runtimes, please update them to a more recent Node.js Lambda runtime before configuring them for native Instana tracing.

Using the Instana API in Lambda Functions

You can access and use the entire Instana API in your Lambda code, same as in a plain vanilla Node.js app using @instana/collector. If you use the Lambda layer and the auto-wrap handler you need to add the line

const instana = require('@instana/aws-lambda');

to your code. If you use manual wrapping you will have that line already.