[receiver/datadog] Unbounded memory growth in high cardinality due to stale series accumulation

[daianmartinho]

Component(s)

receiver/datadog

What happened?

Description

The Datadog receiver experiences unbounded memory growth leading to OOM crashes when handling delta metrics with high cardinality. This is particularly noticeable in environments with ephemeral pods or during load tests where metric tags rotate frequently (high churn).

Deep profiling analysis revealed two underlying root causes in the MetricsTranslator:

1. Map Accumulation & Fragmentation: The lastTs map (used for delta calculation) grows indefinitely as it stores a timestamp for every unique series key encountered. The current implementation never removes stale series. Furthermore, due to Go's internal map implementation, simply adding a delete mechanism would be insufficient, as the memory used by map buckets is not released to the OS even after keys are deleted (resulting in high memory fragmentation).
2. String Pool Accumulation: The internal StringPool implementation caches metric names and tags indefinitely without a cleanup mechanism. This creates a "staircase" memory usage pattern where the baseline memory usage increases permanently with every new batch of unique tags.

Steps to Reproduce

1. Configure the OpenTelemetry Collector with the datadog receiver enabled.
2. Generate metric traffic using a tool like K6 or a script that simulates high cardinality.
3. Ensure the generated metrics utilize rotating tags (e.g., host: dynamic-id-{uuid} or job_id: {seq}) to continuously create new series identities.
4. Observe the Collector's memory usage (RSS/Heap) over time.

Expected Result

The memory usage should stabilize around a reasonable baseline. When old metric series stop reporting (become stale), the memory resources associated with them (timestamps and cached strings) should eventually be reclaimed by the Garbage Collector and returned to the system.

Actual Result

The memory usage grows linearly (or in a step-wise fashion) indefinitely until the container hits the memory limit (OOM). Even if the traffic completely stops or shifts to a new set of tags, the memory consumed by the old series is never released, resulting in a permanent "high water mark" that prevents long-running stability.

Collector version

v0.143.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

Log output

We captured 3 pprof heap profiles at different intervals while running a high cardinality load test (using rotating tags). The analysis confirms a linear and unbounded memory growth directly associated with the updateLastTsForStream function in the MetricsTranslator.

dump 1:

File: otelcol-contrib
Build ID: 6ad0a4dd7996c18453fa8979033088726a606480
Type: inuse_space
Time: 2026-01-19 12:42:53 -03
Showing nodes accounting for 149.59MB, 92.28% of 162.11MB total
Dropped 95 nodes (cum <= 0.81MB)
flat flat% sum% cum cum%
0 0% 0% 132.90MB 81.98% go.opentelemetry.io/collector/config/confighttp.(*ServerConfig).ToServer.maxRequestBodySizeInterceptor.func2
0 0% 0% 132.90MB 81.98% go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
0 0% 0% 132.90MB 81.98% go.opentelemetry.io/collector/config/confighttp.(*decompressor).ServeHTTP
0 0% 0% 132.90MB 81.98% go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP
0 0% 0% 132.90MB 81.98% go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1
0 0% 0% 132.90MB 81.98% net/http.(*ServeMux).ServeHTTP
0 0% 0% 132.90MB 81.98% net/http.(*conn).serve
0 0% 0% 132.90MB 81.98% net/http.HandlerFunc.ServeHTTP
0 0% 0% 132.90MB 81.98% net/http.serverHandler.ServeHTTP
0 0% 0% 130.61MB 80.57% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver.(*datadogReceiver).handleV2Series
0 0% 0% 77.45MB 47.77% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*MetricsTranslator).TranslateSeriesV2
68.19MB 42.06% 42.06% 68.19MB 42.06% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*MetricsTranslator).updateLastTsForStream
0 0% 42.06% 53.16MB 32.79% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*MetricsTranslator).HandleSeriesV2Payload
0 0% 42.06% 23.45MB 14.47% bytes.(*Buffer).ReadFrom
0 0% 42.06% 23.45MB 14.47% io.Copy (inline)
0 0% 42.06% 23.45MB 14.47% io.copyBuffer
0 0% 42.06% 20MB 12.34% bytes.(*Buffer).grow
20MB 12.34% 54.40% 20MB 12.34% bytes.growSlice
0 0% 54.40% 18.51MB 11.42% runtime.main
0 0% 54.40% 16.76MB 10.34% runtime.doInit (inline)
0 0% 54.40% 16.76MB 10.34% runtime.doInit1
1.50MB 0.93% 55.33% 16MB 9.87% github.com/DataDog/agent-payload/v5/gogen.(*MetricPayload).Unmarshal
0 0% 55.33% 16MB 9.87% encoding/json.(*decodeState).array
16MB 9.87% 65.20% 16MB 9.87% encoding/json.(*decodeState).literalStore
0 0% 65.20% 16MB 9.87% encoding/json.(*decodeState).object
0 0% 65.20% 16MB 9.87% encoding/json.(*decodeState).unmarshal
0 0% 65.20% 16MB 9.87% encoding/json.(*decodeState).value
0 0% 65.20% 16MB 9.87% encoding/json.Unmarshal
13MB 8.02% 73.22% 14.50MB 8.95% github.com/DataDog/agent-payload/v5/gogen.(*MetricPayload_MetricSeries).Unmarshal
0 0% 73.22% 8.26MB 5.09% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.parseSeriesProperties
0 0% 73.22% 8.26MB 5.09% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.tagsToAttributes
6.76MB 4.17% 77.39% 6.76MB 4.17% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*StringPool).Intern
0 0% 77.39% 6.05MB 3.73% google.golang.org/grpc/internal/transport.newFramer
0 0% 77.39% 5.26MB 3.25% google.golang.org/grpc.(*Server).Serve.func3
0 0% 77.39% 5.26MB 3.25% google.golang.org/grpc.(*Server).handleRawConn
0 0% 77.39% 5.26MB 3.25% google.golang.org/grpc.(*Server).newHTTP2Transport
0 0% 77.39% 5.26MB 3.25% google.golang.org/grpc/internal/transport.NewServerTransport
4.75MB 2.93% 80.31% 4.75MB 2.93% bufio.NewReaderSize (inline)
0 0% 80.31% 4.09MB 2.52% regexp.Compile (inline)
0 0% 80.31% 4.09MB 2.52% regexp.MustCompile
0 0% 80.31% 4.09MB 2.52% regexp.compile
3.01MB 1.85% 82.17% 3.51MB 2.16% runtime.allocm
0 0% 82.17% 3.51MB 2.16% runtime.newm
0 0% 82.17% 3.51MB 2.16% runtime.schedule
0 0% 82.17% 3.51MB 2.16% runtime.startm
0 0% 82.17% 3.45MB 2.13% github.com/klauspost/compress/zstd.(*Decoder).Read
0 0% 82.17% 3.45MB 2.13% github.com/klauspost/compress/zstd.(*Decoder).nextBlock
0 0% 82.17% 3.45MB 2.13% github.com/klauspost/compress/zstd.(*Decoder).nextBlockSync
0 0% 82.17% 3.45MB 2.13% go.opentelemetry.io/collector/config/confighttp.(*pooledZstdReadCloser).Read
0 0% 82.17% 3.45MB 2.13% net/http.(*maxBytesReader).Read
0 0% 82.17% 3.01MB 1.85% runtime.resetspinning
0 0% 82.17% 3.01MB 1.85% runtime.wakep
0 0% 82.17% 2.54MB 1.57% github.com/DataDog/datadog-agent/pkg/config/setup.InitConfig
0 0% 82.17% 2.54MB 1.57% github.com/DataDog/datadog-agent/pkg/config/viperconfig.(*safeConfig).BindEnvAndSetDefault
2.50MB 1.54% 83.71% 2.50MB 1.54% github.com/aws/aws-sdk-go/aws/endpoints.init
2.31MB 1.43% 85.14% 2.31MB 1.43% github.com/klauspost/compress/zstd.(*history).ensureBlock (inline)
0 0% 85.14% 2.30MB 1.42% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver.(*datadogReceiver).handleCheckRun
0 0% 85.14% 2.09MB 1.29% regexp/syntax.(*compiler).compile
2.09MB 1.29% 86.42% 2.09MB 1.29% regexp/syntax.(*compiler).inst (inline)
0 0% 86.42% 2.09MB 1.29% regexp/syntax.Compile
0 0% 86.42% 2MB 1.24% runtime.mstart
0 0% 86.42% 2MB 1.24% runtime.mstart0
0 0% 86.42% 2MB 1.24% runtime.mstart1
0 0% 86.42% 1.75MB 1.08% github.com/spf13/cobra.(*Command).Execute
0 0% 86.42% 1.75MB 1.08% github.com/spf13/cobra.(*Command).ExecuteC
0 0% 86.42% 1.75MB 1.08% github.com/spf13/cobra.(*Command).execute
0 0% 86.42% 1.75MB 1.08% go.opentelemetry.io/collector/otelcol.(*Collector).Run
0 0% 86.42% 1.75MB 1.08% go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
0 0% 86.42% 1.75MB 1.08% go.opentelemetry.io/collector/otelcol.NewCommand.func1
0 0% 86.42% 1.75MB 1.08% go.opentelemetry.io/collector/service.New
0 0% 86.42% 1.75MB 1.08% main.main
0 0% 86.42% 1.75MB 1.08% main.run (inline)
0 0% 86.42% 1.75MB 1.08% main.runInteractive
0 0% 86.42% 1.54MB 0.95% github.com/DataDog/datadog-agent/pkg/config/setup.bindEnvAndSetLogsConfigKeys
0 0% 86.42% 1.54MB 0.95% github.com/DataDog/datadog-agent/pkg/config/setup.init.0
0 0% 86.42% 1.54MB 0.95% github.com/DataDog/datadog-agent/pkg/config/setup.initConfig
1MB 0.62% 87.04% 1.52MB 0.94% github.com/DataDog/datadog-agent/pkg/config/viperconfig.(*safeConfig).BindEnv
0 0% 87.04% 1.50MB 0.93% regexp.compileOnePass
1.50MB 0.93% 87.97% 1.50MB 0.93% regexp.onePassCopy
1.50MB 0.93% 88.89% 1.50MB 0.93% go.opentelemetry.io/collector/pdata/pcommon.Map.PutStr
1.50MB 0.93% 89.82% 1.50MB 0.93% github.com/DataDog/agent-payload/v5/gogen.(*MetricPayload_Resource).Unmarshal
1.31MB 0.81% 90.62% 1.31MB 0.81% google.golang.org/grpc/internal/transport.newBufWriter (inline)
1.14MB 0.7% 91.33% 1.14MB 0.7% github.com/klauspost/compress/zstd.(*blockDec).reset
0 0% 91.33% 1.14MB 0.7% github.com/klauspost/compress/zstd.(*frameDec).next
0 0% 91.33% 1.06MB 0.66% regexp/syntax.(*compiler).quest
1.04MB 0.64% 91.97% 1.04MB 0.64% github.com/vmware/govmomi/vim25/types.Add (inline)
0 0% 91.97% 1.02MB 0.63% github.com/DataDog/datadog-agent/pkg/config/viperconfig.(*safeConfig).SetDefault
0.50MB 0.31% 92.28% 1MB 0.62% google.golang.org/genproto/googleapis/api/expr/v1alpha1.file_google_api_expr_v1alpha1_checked_proto_init
0 0% 92.28% 1MB 0.62% google.golang.org/genproto/googleapis/api/expr/v1alpha1.init.0
0 0% 92.28% 1MB 0.62% runtime.mcall
0 0% 92.28% 1MB 0.62% runtime.park_m
0 0% 92.28% 1MB 0.62% github.com/DataDog/datadog-agent/comp/otelcol/otlp/components/exporter/serializerexporter.(*factory).createMetricExporter
0 0% 92.28% 1MB 0.62% github.com/DataDog/datadog-agent/comp/otelcol/otlp/components/exporter/serializerexporter.InitSerializer
0 0% 92.28% 1MB 0.62% github.com/DataDog/datadog-agent/comp/otelcol/otlp/components/exporter/serializerexporter.InitSerializer.func2
0 0% 92.28% 1MB 0.62% github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*factory).createMetricsExporter
0 0% 92.28% 1MB 0.62% go.opentelemetry.io/collector/exporter.(*factory).CreateMetrics
0 0% 92.28% 1MB 0.62% go.opentelemetry.io/collector/service.(*Service).initGraph
0 0% 92.28% 1MB 0.62% go.opentelemetry.io/collector/service/internal/builders.(*ExporterBuilder).CreateMetrics
0 0% 92.28% 1MB 0.62% go.opentelemetry.io/collector/service/internal/graph.(*Graph).buildComponents
0 0% 92.28% 1MB 0.62% go.opentelemetry.io/collector/service/internal/graph.(*exporterNode).buildComponent
0 0% 92.28% 1MB 0.62% go.opentelemetry.io/collector/service/internal/graph.Build
0 0% 92.28% 1MB 0.62% go.uber.org/dig.(*Container).Invoke
0 0% 92.28% 1MB 0.62% go.uber.org/dig.(*Scope).Invoke
0 0% 92.28% 1MB 0.62% go.uber.org/dig.(*constructorNode).Call
0 0% 92.28% 1MB 0.62% go.uber.org/dig.defaultInvoker
0 0% 92.28% 1MB 0.62% go.uber.org/dig.paramList.BuildList
0 0% 92.28% 1MB 0.62% go.uber.org/dig.paramObject.Build
0 0% 92.28% 1MB 0.62% go.uber.org/dig.paramObjectField.Build (inline)
0 0% 92.28% 1MB 0.62% go.uber.org/dig.paramSingle.Build
0 0% 92.28% 1MB 0.62% go.uber.org/fx.(*module).invoke
0 0% 92.28% 1MB 0.62% go.uber.org/fx.(*module).invokeAll
0 0% 92.28% 1MB 0.62% go.uber.org/fx.New
0 0% 92.28% 1MB 0.62% go.uber.org/fx.runInvoke
0 0% 92.28% 1MB 0.62% reflect.Value.Call
0 0% 92.28% 1MB 0.62% reflect.Value.call
0 0% 92.28% 1MB 0.62% sync.(*Once).Do (inline)
0 0% 92.28% 1MB 0.62% sync.(*Once).doSlow
0 0% 92.28% 1MB 0.62% github.com/DataDog/datadog-agent/pkg/config/setup.initCommonWithServerless (inline)
0 0% 92.28% 1MB 0.62% github.com/DataDog/datadog-agent/pkg/config/setup.setupAPM

dump 2:

File: otelcol-contrib
Build ID: 6ad0a4dd7996c18453fa8979033088726a606480
Type: inuse_space
Time: 2026-01-19 12:58:36 -03
Showing nodes accounting for 287.60MB, 93.81% of 306.57MB total
Dropped 154 nodes (cum <= 1.53MB)
flat flat% sum% cum cum%
0 0% 0% 276.14MB 90.07% net/http.(*conn).serve
0 0% 0% 275.64MB 89.91% go.opentelemetry.io/collector/config/confighttp.(*ServerConfig).ToServer.maxRequestBodySizeInterceptor.func2
0 0% 0% 275.64MB 89.91% go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
0 0% 0% 275.64MB 89.91% go.opentelemetry.io/collector/config/confighttp.(*decompressor).ServeHTTP
0 0% 0% 275.64MB 89.91% go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP
0 0% 0% 275.64MB 89.91% go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1
0 0% 0% 275.64MB 89.91% net/http.(*ServeMux).ServeHTTP
0 0% 0% 275.64MB 89.91% net/http.HandlerFunc.ServeHTTP
0 0% 0% 275.64MB 89.91% net/http.serverHandler.ServeHTTP
0 0% 0% 272.17MB 88.78% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver.(*datadogReceiver).handleV2Series
0 0% 0% 172.44MB 56.25% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*MetricsTranslator).TranslateSeriesV2
137.09MB 44.72% 44.72% 137.09MB 44.72% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*MetricsTranslator).updateLastTsForStream
0 0% 44.72% 99.23MB 32.37% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*MetricsTranslator).HandleSeriesV2Payload
0 0% 44.72% 59.12MB 19.29% io.Copy (inline)
0 0% 44.72% 59.12MB 19.29% io.copyBuffer
0 0% 44.72% 57.20MB 18.66% bytes.(*Buffer).ReadFrom
0 0% 44.72% 52.51MB 17.13% bytes.(*Buffer).grow
52.51MB 17.13% 61.85% 52.51MB 17.13% bytes.growSlice
0 0% 61.85% 43.50MB 14.19% encoding/json.(*decodeState).array
43.50MB 14.19% 76.04% 43.50MB 14.19% encoding/json.(*decodeState).literalStore
0 0% 76.04% 43.50MB 14.19% encoding/json.(*decodeState).object
0 0% 76.04% 43.50MB 14.19% encoding/json.(*decodeState).unmarshal
0 0% 76.04% 43.50MB 14.19% encoding/json.(*decodeState).value
0 0% 76.04% 43.50MB 14.19% encoding/json.Unmarshal
35.35MB 11.53% 87.56% 35.35MB 11.53% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*StringPool).Intern
0 0% 87.56% 35.35MB 11.53% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.parseSeriesProperties
0 0% 87.56% 35.35MB 11.53% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.tagsToAttributes
0 0% 87.56% 18.51MB 6.04% runtime.main
0 0% 87.56% 16.76MB 5.47% runtime.doInit (inline)
0 0% 87.56% 16.76MB 5.47% runtime.doInit1
0 0% 87.56% 5.20MB 1.69% github.com/klauspost/compress/zstd.(*Decoder).Read
0 0% 87.56% 5.20MB 1.69% github.com/klauspost/compress/zstd.(*Decoder).nextBlock
0 0% 87.56% 5.20MB 1.69% github.com/klauspost/compress/zstd.(*Decoder).nextBlockSync
0 0% 87.56% 5.20MB 1.69% go.opentelemetry.io/collector/config/confighttp.(*pooledZstdReadCloser).Read
0 0% 87.56% 5.20MB 1.69% net/http.(*maxBytesReader).Read
4.63MB 1.51% 89.07% 4.63MB 1.51% github.com/klauspost/compress/zstd.(*history).ensureBlock (inline)
0 0% 89.07% 4.47MB 1.46% google.golang.org/grpc/internal/transport.newFramer
0 0% 89.07% 4.09MB 1.34% regexp.Compile (inline)
0 0% 89.07% 4.09MB 1.34% regexp.MustCompile
0 0% 89.07% 4.09MB 1.34% regexp.compile
0 0% 89.07% 3.68MB 1.20% google.golang.org/grpc.(*Server).Serve.func3
0 0% 89.07% 3.68MB 1.20% google.golang.org/grpc.(*Server).handleRawConn
0 0% 89.07% 3.68MB 1.20% google.golang.org/grpc.(*Server).newHTTP2Transport
0 0% 89.07% 3.68MB 1.20% google.golang.org/grpc/internal/transport.NewServerTransport
3.01MB 0.98% 90.05% 3.51MB 1.14% runtime.allocm
0 0% 90.05% 3.51MB 1.14% runtime.newm
0 0% 90.05% 3.51MB 1.14% runtime.schedule
0 0% 90.05% 3.51MB 1.14% runtime.startm
0 0% 90.05% 3.47MB 1.13% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver.(*datadogReceiver).handleCheckRun
3.16MB 1.03% 91.09% 3.16MB 1.03% bufio.NewReaderSize (inline)
0 0% 91.09% 3.01MB 0.98% runtime.resetspinning
0 0% 91.09% 3.01MB 0.98% runtime.wakep
0 0% 91.09% 2.81MB 0.92% compress/gzip.(*Writer).Write
0 0% 91.09% 2.54MB 0.83% github.com/DataDog/datadog-agent/pkg/config/setup.InitConfig
0 0% 91.09% 2.54MB 0.83% github.com/DataDog/datadog-agent/pkg/config/viperconfig.(*safeConfig).BindEnvAndSetDefault
2.50MB 0.82% 91.90% 2.50MB 0.82% github.com/aws/aws-sdk-go/aws/endpoints.init
1.76MB 0.58% 92.48% 2.30MB 0.75% compress/flate.NewWriter (inline)
0 0% 92.48% 2.09MB 0.68% regexp/syntax.(*compiler).compile
2.09MB 0.68% 93.16% 2.09MB 0.68% regexp/syntax.(*compiler).inst (inline)
0 0% 93.16% 2.09MB 0.68% regexp/syntax.Compile
0 0% 93.16% 2MB 0.65% runtime.mstart
0 0% 93.16% 2MB 0.65% runtime.mstart0
0 0% 93.16% 2MB 0.65% runtime.mstart1
0 0% 93.16% 2MB 0.65% github.com/DataDog/agent-payload/v5/gogen.(*MetricPayload).Unmarshal
2MB 0.65% 93.81% 2MB 0.65% github.com/DataDog/agent-payload/v5/gogen.(*MetricPayload_MetricSeries).Unmarshal
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/config/confighttp.(*compressRoundTripper).RoundTrip
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/config/confighttp.(*compressor).compress
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/consumer.ConsumeMetricsFunc.ConsumeMetrics
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/exporter/exporterhelper.NewMetrics.RequestConsumeFromMetrics.func2
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*obsReportSender[go.shape.interface { BytesSize() int; ItemsCount() int; MergeSplit(context.Context, int, go.opentelemetry.io/collector/exporter/exporterhelper/internal/request.SizerType, go.opentelemetry.io/collector/exporter/exporterhelper/internal/request.Request) ([]go.opentelemetry.io/collector/exporter/exporterhelper/internal/request.Request, error) }]).Send
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*retrySender).Send
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*partitionBatcher).flush.func1
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*workerPool).execute.func1
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/exporter/exporterhelper/internal/sender.(*sender[go.shape.interface { BytesSize() int; ItemsCount() int; MergeSplit(context.Context, int, go.opentelemetry.io/collector/exporter/exporterhelper/internal/request.SizerType, go.opentelemetry.io/collector/exporter/exporterhelper/internal/request.Request) ([]go.opentelemetry.io/collector/exporter/exporterhelper/internal/request.Request, error) }]).Send
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/exporter/otlphttpexporter.(*baseExporter).export
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/collector/exporter/otlphttpexporter.(*baseExporter).pushMetrics
0 0% 93.81% 1.93MB 0.63% go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Transport).RoundTrip
0 0% 93.81% 1.93MB 0.63% net/http.(*Client).Do (inline)
0 0% 93.81% 1.93MB 0.63% net/http.(*Client).do
0 0% 93.81% 1.93MB 0.63% net/http.(*Client).send
0 0% 93.81% 1.93MB 0.63% net/http.send
0 0% 93.81% 1.75MB 0.57% github.com/spf13/cobra.(*Command).Execute
0 0% 93.81% 1.75MB 0.57% github.com/spf13/cobra.(*Command).ExecuteC
0 0% 93.81% 1.75MB 0.57% github.com/spf13/cobra.(*Command).execute
0 0% 93.81% 1.75MB 0.57% go.opentelemetry.io/collector/otelcol.(*Collector).Run
0 0% 93.81% 1.75MB 0.57% go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents
0 0% 93.81% 1.75MB 0.57% go.opentelemetry.io/collector/otelcol.NewCommand.func1
0 0% 93.81% 1.75MB 0.57% go.opentelemetry.io/collector/service.New
0 0% 93.81% 1.75MB 0.57% main.main
0 0% 93.81% 1.75MB 0.57% main.run (inline)
0 0% 93.81% 1.75MB 0.57% main.runInteractive
0 0% 93.81% 1.54MB 0.5% github.com/DataDog/datadog-agent/pkg/config/setup.bindEnvAndSetLogsConfigKeys
0 0% 93.81% 1.54MB 0.5% github.com/DataDog/datadog-agent/pkg/config/setup.init.0
0 0% 93.81% 1.54MB 0.5% github.com/DataDog/datadog-agent/pkg/config/setup.initConfig

dump 3:

File: otelcol-contrib
Build ID: 6ad0a4dd7996c18453fa8979033088726a606480
Type: inuse_space
Time: 2026-01-19 13:11:28 -03
Showing nodes accounting for 386.49MB, 94.22% of 410.21MB total
Dropped 172 nodes (cum <= 2.05MB)
flat flat% sum% cum cum%
0 0% 0% 380.67MB 92.80% net/http.(*conn).serve
0 0% 0% 380.17MB 92.68% go.opentelemetry.io/collector/config/confighttp.(*ServerConfig).ToServer.maxRequestBodySizeInterceptor.func2
0 0% 0% 380.17MB 92.68% go.opentelemetry.io/collector/config/confighttp.(*clientInfoHandler).ServeHTTP
0 0% 0% 380.17MB 92.68% go.opentelemetry.io/collector/config/confighttp.(*decompressor).ServeHTTP
0 0% 0% 380.17MB 92.68% go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*middleware).serveHTTP
0 0% 0% 380.17MB 92.68% go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.NewMiddleware.func1.1
0 0% 0% 380.17MB 92.68% net/http.(*ServeMux).ServeHTTP
0 0% 0% 380.17MB 92.68% net/http.HandlerFunc.ServeHTTP
0 0% 0% 380.17MB 92.68% net/http.serverHandler.ServeHTTP
0 0% 0% 376.20MB 91.71% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver.(*datadogReceiver).handleV2Series
0 0% 0% 222.79MB 54.31% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*MetricsTranslator).TranslateSeriesV2
182.41MB 44.47% 44.47% 182.41MB 44.47% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*MetricsTranslator).updateLastTsForStream
0 0% 44.47% 153.41MB 37.40% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*MetricsTranslator).HandleSeriesV2Payload
0 0% 44.47% 82.85MB 20.20% bytes.(*Buffer).ReadFrom
0 0% 44.47% 82.85MB 20.20% io.Copy (inline)
0 0% 44.47% 82.85MB 20.20% io.copyBuffer
0 0% 44.47% 76.52MB 18.65% bytes.(*Buffer).grow
76.52MB 18.65% 63.12% 76.52MB 18.65% bytes.growSlice
0 0% 63.12% 66.50MB 16.21% encoding/json.(*decodeState).array
66.50MB 16.21% 79.33% 66.50MB 16.21% encoding/json.(*decodeState).literalStore
0 0% 79.33% 66.50MB 16.21% encoding/json.(*decodeState).object
0 0% 79.33% 66.50MB 16.21% encoding/json.(*decodeState).unmarshal
0 0% 79.33% 66.50MB 16.21% encoding/json.(*decodeState).value
0 0% 79.33% 66.50MB 16.21% encoding/json.Unmarshal
0 0% 79.33% 36.37MB 8.87% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.parseSeriesProperties
0 0% 79.33% 36.37MB 8.87% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.tagsToAttributes
35.87MB 8.74% 88.07% 35.87MB 8.74% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator.(*StringPool).Intern
0 0% 88.07% 18.51MB 4.51% runtime.main
0 0% 88.07% 16.76MB 4.09% runtime.doInit (inline)
0 0% 88.07% 16.76MB 4.09% runtime.doInit1
2.02MB 0.49% 88.57% 8.02MB 1.96% github.com/DataDog/agent-payload/v5/gogen.(*MetricPayload).Unmarshal
0 0% 88.57% 6.33MB 1.54% github.com/klauspost/compress/zstd.(*Decoder).Read
0 0% 88.57% 6.33MB 1.54% github.com/klauspost/compress/zstd.(*Decoder).nextBlock
0 0% 88.57% 6.33MB 1.54% github.com/klauspost/compress/zstd.(*Decoder).nextBlockSync
0 0% 88.57% 6.33MB 1.54% go.opentelemetry.io/collector/config/confighttp.(*pooledZstdReadCloser).Read
0 0% 88.57% 6.33MB 1.54% net/http.(*maxBytesReader).Read
6MB 1.46% 90.03% 6MB 1.46% github.com/DataDog/agent-payload/v5/gogen.(*MetricPayload_MetricSeries).Unmarshal
4.63MB 1.13% 91.16% 4.63MB 1.13% github.com/klauspost/compress/zstd.(*history).ensureBlock (inline)
0 0% 91.16% 4.09MB 1% regexp.Compile (inline)
0 0% 91.16% 4.09MB 1% regexp.MustCompile
0 0% 91.16% 4.09MB 1% regexp.compile
0 0% 91.16% 3.97MB 0.97% github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver.(*datadogReceiver).handleCheckRun
3.95MB 0.96% 92.12% 3.95MB 0.96% bufio.NewReaderSize (inline)
0 0% 92.12% 3.95MB 0.96% google.golang.org/grpc.(*Server).Serve.func3
0 0% 92.12% 3.95MB 0.96% google.golang.org/grpc.(*Server).handleRawConn
0 0% 92.12% 3.95MB 0.96% google.golang.org/grpc.(*Server).newHTTP2Transport
0 0% 92.12% 3.95MB 0.96% google.golang.org/grpc/internal/transport.NewServerTransport
0 0% 92.12% 3.95MB 0.96% google.golang.org/grpc/internal/transport.newFramer
3.01MB 0.73% 92.86% 3.51MB 0.85% runtime.allocm
0 0% 92.86% 3.51MB 0.85% runtime.newm
0 0% 92.86% 3.51MB 0.85% runtime.schedule
0 0% 92.86% 3.51MB 0.85% runtime.startm
0 0% 92.86% 3.01MB 0.73% runtime.resetspinning
0 0% 92.86% 3.01MB 0.73% runtime.wakep
1MB 0.24% 93.10% 3MB 0.73% go.opentelemetry.io/collector/pdata/internal.CopyKeyValueSlice
0 0% 93.10% 3MB 0.73% go.opentelemetry.io/collector/pdata/pcommon.Map.CopyTo
0 0% 93.10% 2.54MB 0.62% github.com/DataDog/datadog-agent/pkg/config/setup.InitConfig
0 0% 93.10% 2.54MB 0.62% github.com/DataDog/datadog-agent/pkg/config/viperconfig.(*safeConfig).BindEnvAndSetDefault
2.50MB 0.61% 93.71% 2.50MB 0.61% github.com/aws/aws-sdk-go/aws/endpoints.init
0 0% 93.71% 2.09MB 0.51% regexp/syntax.(*compiler).compile
2.09MB 0.51% 94.22% 2.09MB 0.51% regexp/syntax.(*compiler).inst (inline)
0 0% 94.22% 2.09MB 0.51% regexp/syntax.Compile

Additional context

01/16 we increased pod memory limit from 2gb to 3gb, trying to give some space for GC.. but we found later that it doesn't matter..
GOMEMLIMIT: 80% of pod memory limit

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• receiver/datadog: @boostchicken @gouthamve @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[daianmartinho]

First, i tried iterating over the map, clearing all the old items... but I soon discovered that the map remained active and the fragmented items were not being released from memory.

So the solution was recreating the lastTs map and reset the stringPool, so the GC could completely clean up the old map.
idk if this is a good approach.. but if you guys think its ok i can help with the PR

metrics_translator.go

// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package translator // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/datadogreceiver/internal/translator"

import (
	"sync"
	"time"

	"go.opentelemetry.io/collector/component"
	"go.opentelemetry.io/collector/pdata/pcommon"

	"github.com/open-telemetry/opentelemetry-collector-contrib/internal/exp/metrics/identity"
)

type MetricsTranslator struct {
	sync.RWMutex
	buildInfo   component.BuildInfo
	lastTs      map[identity.Stream]pcommon.Timestamp
	stringPool  *StringPool
	maxIdleTime time.Duration
}

func NewMetricsTranslator(buildInfo component.BuildInfo, idleTimeout time.Duration) *MetricsTranslator {
	return &MetricsTranslator{
		buildInfo:   buildInfo,
		lastTs:      make(map[identity.Stream]pcommon.Timestamp),
		stringPool:  newStringPool(),
		maxIdleTime: idleTimeout,
	}
}

func (mt *MetricsTranslator) streamHasTimestamp(stream identity.Stream) (pcommon.Timestamp, bool) {
	mt.RLock()
	defer mt.RUnlock()
	ts, ok := mt.lastTs[stream]
	return ts, ok
}

func (mt *MetricsTranslator) updateLastTsForStream(stream identity.Stream, ts pcommon.Timestamp) {
	mt.Lock()
	defer mt.Unlock()
	mt.lastTs[stream] = ts
}

// Prune removes stale series by recreating the internal map.
// It returns the number of series removed (for observability).
func (mt *MetricsTranslator) Prune() int {
	// Full lock required: we are swapping the map reference.
	// This blocks all reads (streamHasTimestamp) and writes (updateLastTsForStream) during the swap.
	mt.Lock()
	defer mt.Unlock()

	now := time.Now()
	originalSize := len(mt.lastTs)

	// Optimization: If the map is empty, skip processing.
	if originalSize == 0 {
		return 0
	}

	// Create a new, clean map.
	// We let it grow organically to avoid allocating memory for the stale series we are about to drop.
	newMap := make(map[identity.Stream]pcommon.Timestamp)

	for stream, ts := range mt.lastTs {
		tsTime := time.Unix(0, int64(ts))

		// Keep only active series (younger than maxIdleTime).
		if now.Sub(tsTime) < mt.maxIdleTime {
			newMap[stream] = ts
		}
		// Stale series are implicitly dropped (not copied to the new map).
	}

	// Map Swap: Replace the old map reference with the new one.
	// The Go GC will detect that the old map is unreachable and reclaim the memory
	// used by its buckets, effectively resolving the fragmentation issue.
	mt.lastTs = newMap

	// Pool Reset: Discard the old string pool.
	// This allows the GC to free memory used by strings from stale series (e.g., old tags).
	mt.stringPool = newStringPool()

	return originalSize - len(newMap)
}

func (ddr *datadogReceiver) Start(ctx context.Context, host component.Host) error {
	...
	go ddr.runSweeper(ctx)
	return nil
}

func (ddr *datadogReceiver) runSweeper(ctx context.Context) {
	// Run the cleaning cycle more frequently than the timeout to ensure accuracy.
        // If the timeout is 30 minutes, checking every 5 minutes is healthy.
	sweepInterval := 5 * time.Minute

	ticker := time.NewTicker(sweepInterval)
	defer ticker.Stop()
        
	ddr.params.Logger.Info("Starting memory sweeper loop",
		zap.Duration("idle_timeout", ddr.config.IdleTimeout),
		zap.Duration("sweep_interval", sweepInterval))

	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			removedCount := ddr.metricsTranslator.Prune()
			if removedCount > 0 {
                                // this can be an debug log
				ddr.params.Logger.Info("Pruned stale series from cache",
					zap.Int("removed_series", removedCount))
			}
		}
	}
}

i didn't notice any problem with the mt.Lock() for the Prune(), but we need more tests for sure..