s3 GetObject OOM issues with streaming large objects

[jasperlin451]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

We upgraded from 3.835 to 3.958 and now using GetObjectCommand to stream large (multi-GB) objects is causing OOM issues.

Explicitly turning responseChecksumValidation to WHEN_REQUIRED fixed the OOM issue.

I know the behavior checksum behavior changed w/ 3.729.0 but that doesn't seem to be the cause here.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-s3@3.958.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v22.19.0

Reproduction Steps

async function main() {
  const client = new S3Client({
      // responseChecksumValidation: 'WHEN_REQUIRED', -- set this to fix
    });

   const { Body: readable } = await this.client.send(
      new GetObjectCommand({
        Bucket: //bucket,
        Key: // path to very large file,
      })
    );

  if (!readable) {
    throw new Error('Failed to get readable stream');
  }

  let seenBytes = 0;
  const logEvery = 50 * 1024 * 1024; // 50 MB
  let nextLog = logEvery;

  const meter = new Transform({
    transform(chunk, _enc, cb) {
      seenBytes += chunk.length;
      if (seenBytes >= nextLog) {
        const m = process.memoryUsage();
        console.log('stream progress', {
          seenMB: Math.round(seenBytes / 1024 / 1024),
          rssMB: Math.round(m.rss / 1024 / 1024),
          heapMB: Math.round(m.heapUsed / 1024 / 1024),
        });
        nextLog += logEvery;
      }
      cb(null, chunk);
    },
  });

  const devNull = new Transform({
    transform(_chunk, _enc, cb) {
      cb();
    },
  });

  await pipeline(readable, createZstdDecompress(), meter, devNull);
}

main();

Observed Behavior

OOM

Expected Behavior

No OOM when streaming

Possible Solution

No response

Additional Information/Context

No response

[aBurmeseDev]

HI @jasperlin451 - thanks for checking in.

I tried to reproduce using code snippet you provided with a large 1GB file and logging the stream process, on "@aws-sdk/client-s3": "^3.970.0" and was unable to reproduce the reported issue. See the code and progress log below.

import { S3 } from "@aws-sdk/client-s3";
import { pipeline, Readable, Transform } from "node:stream";

async function main() {
	const client = new S3({
		// responseChecksumValidation: 'WHEN_REQUIRED', -- set this to fix
	});

	await client.putObject({
		Bucket: "s3-repro",
		Key: "large-file",
		Body: Readable.from({
			async *[Symbol.asyncIterator]() {
				for (let i = 0; i < 1024; ++i) {
					process.stdout.write(".");
					yield new Uint8Array(1024 * 1024);
				}
			},
		}),
		ContentLength: 1024 * 1024 * 1024,
	});
	console.log("");

	const { Body: readable } = await client.getObject({
		Bucket: "s3-repro",
		Key: "large-file",
	});

	if (!readable) {
		throw new Error("Failed to get readable stream");
	}

	let seenBytes = 0;
	const logEvery = 50 * 1024 * 1024; // 50 MB
	let nextLog = logEvery;

	const meter = new Transform({
		transform(chunk, _enc, cb) {
			seenBytes += chunk.length;
			if (seenBytes >= nextLog) {
				const m = process.memoryUsage();
				console.log("stream progress", {
					seenMB: Math.round(seenBytes / 1024 / 1024),
					rssMB: Math.round(m.rss / 1024 / 1024),
					heapMB: Math.round(m.heapUsed / 1024 / 1024),
				});
				nextLog += logEvery;
			}
			cb(null, chunk);
		},
	});

	const devNull = new Transform({
		transform(_chunk, _enc, cb) {
			cb();
		},
	});

	await new Promise((resolve, reject) => {
		pipeline(readable, meter, devNull, (err) => {
			if (err) reject(err);
			else resolve();
		});
	});
}

await main();

---

Here streaming progress shows that:

• Heap usage stays constant at ~9MB - The stream is processing data without accumulating it in memory
• RSS (total memory) is stable at ~1113MB - No memory leaks, the base memory footprint isn't growing
• Processing 1GB of data with minimal memory overhead

stream progress { seenMB: 50, rssMB: 1112, heapMB: 9 }
stream progress { seenMB: 100, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 150, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 200, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 250, rssMB: 1113, heapMB: 8 }
stream progress { seenMB: 300, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 350, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 400, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 450, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 500, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 550, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 600, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 650, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 700, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 750, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 800, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 850, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 900, rssMB: 1113, heapMB: 9 }
stream progress { seenMB: 950, rssMB: 1114, heapMB: 9 }
stream progress { seenMB: 1000, rssMB: 1124, heapMB: 10 }

What's the sizes of files you're working with?

[jasperlin451]

This is super weird., but my repro script no longer repros. But in our production script it still seems to make a big difference when responseChecksumValidation is set vs not so I'm a bit confused as to what is happening.

The files are 10+ Gb.

[aBurmeseDev]

I just tried with a 10Gb file, it took a bit longer but did not run into any memory issue.

it still seems to make a big difference when responseChecksumValidation

Could be because of default data integrity protection with S3 that was introduced a year ago. See here. Let me know if issue persists.

[jasperlin451]

Yeah, I guess I don't understand why flipping the data integrity check in production improves the memory but not in this test script. I will test more.