Buffers are not serialised correctly as per the protoJSON specification

[alistairjevans]

What happened?

If I specify a "Buffer" as an attribute to an OTel object (trace/log), and I'm using the OTLP HTTP Exporter, the Buffer is serialised into the attribute as:

"bytesValue": {
    "type": "Buffer",
    "data": [
        105,
        50,
        214,
        222,
        243,
        159,
        243,
        10,
        202,
        225,
        43,
        40
    ]
}

Where it should be a base64 string, as per the spec: https://protobuf.dev/programming-guides/json/#field-representation

This problem prevents code based on the protobuf spec from being able to decode this data; e.g. in the ruby protobuf library:

Error occurred during parsing: Error parsing JSON @1:661: Expected string

Steps to Reproduce

I've created a reproduction repo here:

https://github.com/alistairjevans/node-otel-serialisation

Expected Result

All binary representations are serialised to base64 to match the protobuf specification.

Actual Result

The object is dumped using the default prototype of the object, which does not match the protobuf spec.

Additional Details

• The bug is in the toAnyValue function I believe.
• The bug impacts UIntArray as well.

Relevant log output

OTLPExportDelegate items to be sent [
  LogRecord {
    attributes: {
      'my.bytes.buffer': <Buffer 48 65 6c 6c 6f>,
      'my.string.attribute': 'normal string for comparison'
    },
    totalAttributesCount: 2,
    _isReadonly: false,
    hrTime: [ 1764942699, 829000000 ],
    hrTimeObserved: [ 1764942699, 829000000 ],
    _severityNumber: 9,
    _severityText: 'INFO',
    _body: 'Test log message with bytes attribute',
    resource: Resource {
      _attributes: [Object],
      asyncAttributesPending: false,
      _syncAttributes: [Object],
      _asyncAttributesPromise: undefined
    },
    instrumentationScope: {
      name: 'bytes-test-logger',
      version: undefined,
      schemaUrl: undefined
    },
    _logRecordLimits: { attributeCountLimit: 128, attributeValueLengthLimit: Infinity }
  }
]

Full outputted payload:

{
    "resourceLogs": [
        {
            "resource": {
                "attributes": [
                    {
                        "key": "service.name",
                        "value": {
                            "stringValue": "bytes-serialisation-test"
                        }
                    },
                    {
                        "key": "telemetry.sdk.language",
                        "value": {
                            "stringValue": "nodejs"
                        }
                    },
                    {
                        "key": "telemetry.sdk.name",
                        "value": {
                            "stringValue": "opentelemetry"
                        }
                    },
                    {
                        "key": "telemetry.sdk.version",
                        "value": {
                            "stringValue": "1.30.1"
                        }
                    }
                ],
                "droppedAttributesCount": 0
            },
            "scopeLogs": [
                {
                    "scope": {
                        "name": "bytes-test-logger"
                    },
                    "logRecords": [
                        {
                            "timeUnixNano": "1764942699829000000",
                            "observedTimeUnixNano": "1764942699829000000",
                            "severityNumber": 9,
                            "severityText": "INFO",
                            "body": {
                                "stringValue": "Test log message with bytes attribute"
                            },
                            "attributes": [
                                {
                                    "key": "my.bytes.buffer",
                                    "value": {
                                        "bytesValue": {
                                            "type": "Buffer",
                                            "data": [
                                                72,
                                                101,
                                                108,
                                                108,
                                                111
                                            ]
                                        }
                                    }
                                },
                                {
                                    "key": "my.string.attribute",
                                    "value": {
                                        "stringValue": "normal string for comparison"
                                    }
                                }
                            ],
                            "droppedAttributesCount": 0
                        }
                    ]
                }
            ]
        }
    ]
}

Operating System and Version

macOS 15.6.1

Runtime and Version

At least nodejs 22.18.0 and 22.21.1

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[JamieDanielson]

It looks like this example is still using SDK v1, are you able to get the same issue if you use SDK v2?

If it is still an issue in v2, there is also a possibility that it may be resolved by #6179 and #6192 which have not yet been released in a new version. We can try this out to see if it is still an issue but also curious if you happen to try with V2 and it works as expected.

[trentm]

I think it likely is still an issue with the latest package versions. Using the server.js from the repro repo and sending to a mock collector that I use for dev/debugging, I see this serialization using the http/protobuf exporter:

ExportLogsServiceRequest {
  resourceLogs: [
    ResourceLogs {
      scopeLogs: [
        ScopeLogs {
          logRecords: [
            LogRecord {
              attributes: [
                KeyValue { key: 'my.bytes.buffer', value: AnyValue { bytesValue: Buffer(5) [Uint8Array] [ 72, 101, 108, 108, 111 ] } },
                KeyValue { key: 'my.string.attribute', value: AnyValue { stringValue: 'normal string for comparison' } }
              ],
              timeUnixNano: Long { low: -1169131712, high: 411036596, unsigned: true },
              severityNumber: 9,
              severityText: 'INFO',
              body: AnyValue { stringValue: 'Test log message with bytes attribute' },
              droppedAttributesCount: 0,
              flags: 1,

I guess that is wrong? I don't recall what the OTLP rules are for a Buffer/binary AnyValue field

[pichlermarc]

@alistairjevans thank you for providing the repro, that was very helpful in checking if that has already been fixed on main or not.

I created a patch for our e2e tests to see if this is still a problem, and it indeed is. Unfortunately, #6179 did not fix this yet, since it only touched protobuf code, not JSON.

Here's the patch I used to verify that this bug is still present:

repro.patch

diff --git a/e2e-tests/package.json b/e2e-tests/package.json
index 9ccdc206b..6782bf0c2 100644
--- a/e2e-tests/package.json
+++ b/e2e-tests/package.json
@@ -16,9 +16,9 @@
   "dependencies": {
     "@opentelemetry/api": "^1.3.0",
     "@opentelemetry/api-logs": "0.208.0",
-    "@opentelemetry/exporter-logs-otlp-proto": "0.208.0",
-    "@opentelemetry/exporter-metrics-otlp-proto": "0.208.0",
-    "@opentelemetry/exporter-trace-otlp-proto": "0.208.0",
+    "@opentelemetry/exporter-logs-otlp-http": "0.208.0",
+    "@opentelemetry/exporter-metrics-otlp-http": "0.208.0",
+    "@opentelemetry/exporter-trace-otlp-http": "0.208.0",
     "@opentelemetry/sdk-logs": "0.208.0",
     "@opentelemetry/sdk-metrics": "2.2.0",
     "@opentelemetry/sdk-node": "0.208.0",
diff --git a/e2e-tests/test.mjs b/e2e-tests/test.mjs
index 20476c580..11e18c5ec 100644
--- a/e2e-tests/test.mjs
+++ b/e2e-tests/test.mjs
@@ -14,10 +14,10 @@
  * limitations under the License.
  */
 import { NodeSDK } from '@opentelemetry/sdk-node';
-import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-proto';
-import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto';
+import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
+import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
 import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
-import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-proto';
+import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http';
 import { SimpleLogRecordProcessor } from '@opentelemetry/sdk-logs';
 import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
 import {
@@ -79,10 +79,12 @@ async function main() {
 
   // Create a log
   const logger = logs.getLogger('example-logger');
+  // Create a Buffer (Node.js bytes representation)
+  const bufferData = Buffer.from([0x48, 0x65, 0x6c, 0x6c, 0x6f]); // "Hello" in bytes
   logger.emit({
     severityText: 'INFO',
     body: 'test-log-body',
-    attributes: { foo: 'bar' },
+    attributes: { foo: 'bar', 'my.bytes.buffer': bufferData },
     eventName: 'test-log-event',
   });
 
diff --git a/package-lock.json b/package-lock.json
index 94830fe69..54e093027 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -103,9 +103,9 @@
       "dependencies": {
         "@opentelemetry/api": "^1.3.0",
         "@opentelemetry/api-logs": "0.208.0",
-        "@opentelemetry/exporter-logs-otlp-proto": "0.208.0",
-        "@opentelemetry/exporter-metrics-otlp-proto": "0.208.0",
-        "@opentelemetry/exporter-trace-otlp-proto": "0.208.0",
+        "@opentelemetry/exporter-logs-otlp-http": "0.208.0",
+        "@opentelemetry/exporter-metrics-otlp-http": "0.208.0",
+        "@opentelemetry/exporter-trace-otlp-http": "0.208.0",
         "@opentelemetry/sdk-logs": "0.208.0",
         "@opentelemetry/sdk-metrics": "2.2.0",
         "@opentelemetry/sdk-node": "0.208.0",

I after applying, the e2e test output is still reproduces the bug:

/Users/marc.pichler/.nvm/versions/node/v20.18.0/bin/npm run test:e2e

> @opentelemetry/e2e-test@0.7.0 test:e2e
> npm run stop-collector; npm run run-collector && npm run export-telemetry && npm run verify || npm run stop-collector


> @opentelemetry/e2e-test@0.7.0 stop-collector
> docker stop otelcol-e2e

Error response from daemon: No such container: otelcol-e2e
npm error Lifecycle script `stop-collector` failed with error:
npm error code 1
npm error path /Users/marc.pichler/opentelemetry-js/e2e-tests
npm error workspace @opentelemetry/e2e-test@0.7.0
npm error location /Users/marc.pichler/opentelemetry-js/e2e-tests
npm error command failed
npm error command sh -c docker stop otelcol-e2e

> @opentelemetry/e2e-test@0.7.0 prerun-collector
> node -e "require('fs').writeFileSync('collector-output.json', '')"


> @opentelemetry/e2e-test@0.7.0 run-collector
> docker run --pull=always -d --rm --name otelcol-e2e -v $(pwd)/collector-config.yaml:/etc/otelcol/config.yaml -v $(pwd)/collector-output.json:/tmp/collector-output.json -p 4317:4317 -p 4318:4318 -w /tmp otel/opentelemetry-collector-contrib:latest --config /etc/otelcol/config.yaml && sleep 5

latest: Pulling from otel/opentelemetry-collector-contrib
Digest: sha256:b14234c4bc1b7364629af272e564913bb57bdc9736d45b8b6db5ab3417dc75f9
Status: Image is up to date for otel/opentelemetry-collector-contrib:latest
b8c5e9beceb2a89cca7f9b3cded3d27312048177c2c8f7cf1dfd9342a381f93c

> @opentelemetry/e2e-test@0.7.0 export-telemetry
> node test.mjs; sleep 5

The 'metricReader' option is deprecated. Please use 'metricReaders' instead.
{"stack":"OTLPExporterError: Bad Request\n    at IncomingMessage.<anonymous> (/Users/marc.pichler/opentelemetry-js/experimental/packages/otlp-exporter-base/build/src/transport/http-transport-utils.js:52:31)\n    at IncomingMessage.emit (node:events:531:35)\n    at endReadableNT (node:internal/streams/readable:1696:12)\n    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)","message":"Bad Request","code":"400","name":"OTLPExporterError","data":"{\"code\":3,\"message\":\"ReadStringAsSlice: expects \\\" or n, but found {, error found in #10 byte of ...|esValue\\\":{\\\"type\\\":\\\"Bu|..., bigger context ...|},{\\\"key\\\":\\\"my.bytes.buffer\\\",\\\"value\\\":{\\\"bytesValue\\\":{\\\"type\\\":\\\"Buffer\\\",\\\"data\\\":[72,101,108,108,111]}}}],\\\"d|...\"}"}

> @opentelemetry/e2e-test@0.7.0 verify
> node verify.mjs

found span
found metric
No logs found in the output
npm error Lifecycle script `verify` failed with error:
npm error code 1
npm error path /Users/marc.pichler/opentelemetry-js/e2e-tests
npm error workspace @opentelemetry/e2e-test@0.7.0
npm error location /Users/marc.pichler/opentelemetry-js/e2e-tests
npm error command failed
npm error command sh -c node verify.mjs

> @opentelemetry/e2e-test@0.7.0 stop-collector
> docker stop otelcol-e2e

otelcol-e2e

Process finished with exit code 0

I'll work on a PR.

@alistairjevans in the meantime, you could use the protobuf-based exporter @opentelemetry/exporter-logs-otlp-proto as a workaround if that works for your use-case.

[alistairjevans]

Thanks everyone! 🙏 The protoJSON spec is pretty clear that "bytes" in the proto spec should be formatted as base64; the only deviation in type format OTLP HTTP makes from the protoJSON spec is making trace/spanIds hex instead of base64.