Skip to content

Intermittent InvalidSignatureException: Signature expired error in AWS Lambda #7605

@minh-le-q

Description

@minh-le-q

Checkboxes for prior research

Describe the bug

When using AWS SDK for JavaScript v3 from an AWS Lambda, we see intermittent InvalidSignatureException errors with messages like:

Signature expired: ... is now earlier than ... - 5 min.

From CloudWatch logs, we see these occur on the first DynamoDB request after the Lambda has been idle (frozen) for several minutes, and subsequent requests in the same warm environment succeed.
This happens despite AWS SDK v3’s documented automatic clock skew detection and correction under the correctClockSkew section here.

Regression Issue

  • Select this option if this issue appears to be a regression.

SDK version number

"@aws-sdk/client-dynamodb": "^3.614.0", "@aws-sdk/lib-dynamodb": "^3.614.0"

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Node.js 20.x on AWS Lambda constructed by the NodejsFunction construct via "aws-cdk-lib": "^2.158.0". The Lambda function is not set up with any preserved concurrency.

Reproduction Steps

Here is an example Lambda code with the handler function as the Lambda entrypoint to reproduce the issue

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, GetCommand } from "@aws-sdk/lib-dynamodb";

// Intentionally skew the clock to simulate freeze/thaw or bad offset
const ddb = new DynamoDBClient({
  region: "us-east-1",
  systemClockOffset: -3 * 60 * 60 * 1000, // -3 hours
});

const doc = DynamoDBDocumentClient.from(ddb);

export const handler = async () => {
  await doc.send(
    new GetCommand({
      TableName: "ExampleTableName",
      Key: { pk: "example" },
    })
  );
};

How to run:

  1. Deploy this Lambda with a valid IAM role that allows dynamodb:GetItem.

  2. Invoke the Lambda once.

Observed Behavior

DynamoDB returns HTTP 400 InvalidSignatureException

  • Message indicates clock skew (Signature expired)

The thrown error object:

  • includes $metadata attribute, but
  • does not include the $response attribute (which is needed to detect clock skew)

SDK retry logic does not retry (attempts: 1)
clockSkewCorrected is not set, despite the server returning a valid Date header

Timeline

  • t = t0: Lambda is invoked; DynamoDB call succeeds.
  • t ≈ t0 + 6 minutes: Lambda has been idle (environment frozen). First invocation after thaw fails with InvalidSignatureException.
  • t ≈ t0 + 6 minutes + ~30–60 seconds: Subsequent invocation succeeds in the same execution environment.

An example error from CloudWatch:

2025-12-24T18:59:14.149Z    98c58572-8298-4c7c-8bd6-7963b9f59f9b    ERROR    {
  clientName: 'DynamoDBClient',
  commandName: 'GetItemCommand',
  input: { TableName: 'ExampleTableName', Key: { pk: 'example' } },
  error: InvalidSignatureException: Signature expired: 20251224T165913Z is now earlier than 20251224T185413Z (20251224T185913Z - 5 min.)
      at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:422:20)
      at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:431:5
      at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-dynamodb/dist-cjs/index.js:2359:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:36:20
      at async /var/runtime/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/index.js:173:30
      at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:193:18
      at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:312:38
      at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
      at async getUser (/var/task/index.js:6105:27) {
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 400,
      requestId: '6IKIMD99KSTU4E4R09DN5N5NRFVV4KQNSO5AEMVJF66Q9ASUAAJG',
      extendedRequestId: undefined,
      cfId: undefined,
      attempts: 1,
      totalRetryDelay: 0
    },
    __type: 'com.amazon.coral.service#InvalidSignatureException'
  },
  metadata: {
    httpStatusCode: 400,
    requestId: '6IKIMD99KSTU4E4R09DN5N5NRFVV4KQNSO5AEMVJF66Q9ASUAAJG',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

Expected Behavior

AWS SDK v3 includes clock skew correction logic for SigV4 signing. When a request fails due to a signing timestamp being outside the allowed tolerance window (e.g., InvalidSignatureException: Signature expired) and the server’s time is available in the HTTP response (e.g., via the Date header), the SDK should be able to:

  • extract server time,
  • update its internal systemClockOffset (marking the error as clockSkewCorrected), and
  • treat the failure as transient and retry the request automatically.

This matters in Lambda because execution environments can be frozen and later thawed, and the first request after thaw can occasionally be signed with an out-of-window timestamp; automatic correction + retry would prevent these transient skew errors from failing an invocation.

Possible Solution

This does not appear to be a service-side issue but rather how the SDK’s error propagation and clock skew correction interacts with modeled service errors and the Lambda freeze/thaw lifecycle.

From our debugging, it seems that the SDK’s clock skew correction logic depends on error.ServerTime or error.$response.headers.date being available on the thrown error object, as shown in the signing middleware code.

However, for DynamoDB modeled errors (e.g., InvalidSignatureException), the raw response is not reliably attached to the exception. Only $metadata survives. Because of this:

  • the signer’s errorHandler() function cannot extract server time,
  • no clockSkewCorrected flag is set,
  • retry logic does not treat this as a transient/skew error,
  • transparent retry is not triggered.

This behavior appears tied to how the DynamoDB client (and possibly other generated clients) normalizes errors, dropping the raw response prior to the signer’s skew handling.

Possible solutions could be:

  1. attach $response (with headers.date) to the thrown error object, or
  2. attach ServerTime (or equivalent) derived from the HTTP Date header onto the error.
  3. retry on skew without $response (eg, based on Signature expired in error.message)

Additional Information/Context

No response

Metadata

Metadata

Assignees

Labels

bugThis issue is a bug.p3This is a minor priority issue

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions