-
Notifications
You must be signed in to change notification settings - Fork 652
Description
Checkboxes for prior research
- I've gone through Developer Guide and API reference
- I've checked AWS Forums and StackOverflow.
- I've searched for previous similar issues and didn't find any solution.
Describe the bug
When using AWS SDK for JavaScript v3 from an AWS Lambda, we see intermittent InvalidSignatureException errors with messages like:
Signature expired: ... is now earlier than ... - 5 min.
From CloudWatch logs, we see these occur on the first DynamoDB request after the Lambda has been idle (frozen) for several minutes, and subsequent requests in the same warm environment succeed.
This happens despite AWS SDK v3’s documented automatic clock skew detection and correction under the correctClockSkew section here.
Regression Issue
- Select this option if this issue appears to be a regression.
SDK version number
"@aws-sdk/client-dynamodb": "^3.614.0", "@aws-sdk/lib-dynamodb": "^3.614.0"
Which JavaScript Runtime is this issue in?
Node.js
Details of the browser/Node.js/ReactNative version
Node.js 20.x on AWS Lambda constructed by the NodejsFunction construct via "aws-cdk-lib": "^2.158.0". The Lambda function is not set up with any preserved concurrency.
Reproduction Steps
Here is an example Lambda code with the handler function as the Lambda entrypoint to reproduce the issue
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, GetCommand } from "@aws-sdk/lib-dynamodb";
// Intentionally skew the clock to simulate freeze/thaw or bad offset
const ddb = new DynamoDBClient({
region: "us-east-1",
systemClockOffset: -3 * 60 * 60 * 1000, // -3 hours
});
const doc = DynamoDBDocumentClient.from(ddb);
export const handler = async () => {
await doc.send(
new GetCommand({
TableName: "ExampleTableName",
Key: { pk: "example" },
})
);
};How to run:
-
Deploy this Lambda with a valid IAM role that allows dynamodb:GetItem.
-
Invoke the Lambda once.
Observed Behavior
DynamoDB returns HTTP 400 InvalidSignatureException
- Message indicates clock skew (
Signature expired)
The thrown error object:
- includes
$metadataattribute, but - does not include the
$responseattribute (which is needed to detect clock skew)
SDK retry logic does not retry (attempts: 1)
clockSkewCorrected is not set, despite the server returning a valid Date header
Timeline
- t = t0: Lambda is invoked; DynamoDB call succeeds.
- t ≈ t0 + 6 minutes: Lambda has been idle (environment frozen). First invocation after thaw fails with
InvalidSignatureException. - t ≈ t0 + 6 minutes + ~30–60 seconds: Subsequent invocation succeeds in the same execution environment.
An example error from CloudWatch:
2025-12-24T18:59:14.149Z 98c58572-8298-4c7c-8bd6-7963b9f59f9b ERROR {
clientName: 'DynamoDBClient',
commandName: 'GetItemCommand',
input: { TableName: 'ExampleTableName', Key: { pk: 'example' } },
error: InvalidSignatureException: Signature expired: 20251224T165913Z is now earlier than 20251224T185413Z (20251224T185913Z - 5 min.)
at throwDefaultError (/var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:422:20)
at /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/smithy-client/dist-cjs/index.js:431:5
at de_CommandError (/var/runtime/node_modules/@aws-sdk/client-dynamodb/dist-cjs/index.js:2359:14)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-serde/dist-cjs/index.js:36:20
at async /var/runtime/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/index.js:173:30
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/core/dist-cjs/index.js:193:18
at async /var/runtime/node_modules/@aws-sdk/node_modules/@smithy/middleware-retry/dist-cjs/index.js:312:38
at async /var/runtime/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22
at async getUser (/var/task/index.js:6105:27) {
'$fault': 'client',
'$metadata': {
httpStatusCode: 400,
requestId: '6IKIMD99KSTU4E4R09DN5N5NRFVV4KQNSO5AEMVJF66Q9ASUAAJG',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
__type: 'com.amazon.coral.service#InvalidSignatureException'
},
metadata: {
httpStatusCode: 400,
requestId: '6IKIMD99KSTU4E4R09DN5N5NRFVV4KQNSO5AEMVJF66Q9ASUAAJG',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
}
}
Expected Behavior
AWS SDK v3 includes clock skew correction logic for SigV4 signing. When a request fails due to a signing timestamp being outside the allowed tolerance window (e.g., InvalidSignatureException: Signature expired) and the server’s time is available in the HTTP response (e.g., via the Date header), the SDK should be able to:
- extract server time,
- update its internal systemClockOffset (marking the error as clockSkewCorrected), and
- treat the failure as transient and retry the request automatically.
This matters in Lambda because execution environments can be frozen and later thawed, and the first request after thaw can occasionally be signed with an out-of-window timestamp; automatic correction + retry would prevent these transient skew errors from failing an invocation.
Possible Solution
This does not appear to be a service-side issue but rather how the SDK’s error propagation and clock skew correction interacts with modeled service errors and the Lambda freeze/thaw lifecycle.
From our debugging, it seems that the SDK’s clock skew correction logic depends on error.ServerTime or error.$response.headers.date being available on the thrown error object, as shown in the signing middleware code.
However, for DynamoDB modeled errors (e.g., InvalidSignatureException), the raw response is not reliably attached to the exception. Only $metadata survives. Because of this:
- the signer’s
errorHandler()function cannot extract server time, - no
clockSkewCorrectedflag is set, - retry logic does not treat this as a transient/skew error,
- transparent retry is not triggered.
This behavior appears tied to how the DynamoDB client (and possibly other generated clients) normalizes errors, dropping the raw response prior to the signer’s skew handling.
Possible solutions could be:
- attach
$response(withheaders.date) to the thrown error object, or - attach
ServerTime(or equivalent) derived from the HTTP Date header onto the error. - retry on skew without
$response(eg, based onSignature expiredinerror.message)
Additional Information/Context
No response