Support Micrometer Observation

[be-hase]

I’d like to add Micrometer Observation support for Spring users.
I previously added Micrometer Tracing support ( #205 ), but for the reasons described below, I’ve learned that adding Micrometer Observation support is preferable.

Why - TL;DR

Without using Micrometer Observation, we can’t achieve tracing in Spring-related asynchronous libraries (Reactor).

Why - Detail

Micrometer Observation is a library that sits “above” Micrometer and Micrometer Tracing.
By writing a single piece of instrumentation code, it enables Tracing / Metrics / Logging.
Micrometer Observation implements the instrumentation code for Micrometer Timer and Micrometer Tracing.

• https://github.com/micrometer-metrics/micrometer/blob/main/micrometer-core/src/main/java/io/micrometer/core/instrument/observation/DefaultMeterObservationHandler.java
• https://github.com/micrometer-metrics/tracing/blob/main/micrometer-tracing/src/main/java/io/micrometer/tracing/handler/DefaultTracingObservationHandler.java

Since Spring Boot 3, Spring-related libraries generally use Micrometer Observation.

e.g.

• https://github.com/spring-projects/spring-framework/blob/ad19c1ac8229f48843a32f31905037f3a76612ec/spring-web/src/main/java/org/springframework/web/filter/ServerHttpObservationFilter.java#L107
• https://github.com/spring-projects/spring-framework/blob/ad19c1ac8229f48843a32f31905037f3a76612ec/spring-webflux/src/main/java/org/springframework/web/reactive/function/client/DefaultWebClient.java#L445
• https://github.com/spring-projects/spring-framework/blob/ad19c1ac8229f48843a32f31905037f3a76612ec/spring-web/src/main/java/org/springframework/web/client/RestTemplate.java#L746

Micrometer Observation has the concept of an Observation Context, and it stores the Tracing Context inside this Observation Context.
By propagating the Observation Context, the Tracing Context is propagated together, which makes tracing possible.

…This gets a bit nerdy from here, but there’s a library called Micrometer Context Propagation.
https://docs.micrometer.io/context-propagation/reference/

For reference, here’s an explanatory article I wrote:
https://dev.to/be-hase/about-micrometer-context-propagation-5gg9

In asynchronous libraries like Reactor, this Micrometer Context Propagation is used to propagate the Observation Context. (As a result, tracing works.)

In an environment that doesn’t use Micrometer Observation, the Observation Context doesn’t exist (in the current thread local), so tracing can’t work correctly.

What

This is implemented using Decaton’s TracingProvider / ProcessorTraceHandle / RecordTraceHandle.

In addition to unit tests, I also ran it in practice, and tracing works correctly.
(No issues with Reactor-based WebClient / Lettuce either.)

Downsides

By using Micrometer Observation, Micrometer Metrics like the following are also recorded. This slightly overlaps with Decaton’s existing metrics, but we’ll have to accept that.

# HELP decaton_processor_active_seconds  
# TYPE decaton_processor_active_seconds summary
decaton_processor_active_seconds_count{partition="1",processor="-",subscriptionId="subscriptionId",topic="topic"} 0
decaton_processor_active_seconds_sum{partition="1",processor="-",subscriptionId="subscriptionId",topic="topic"} 0.0
# HELP decaton_processor_active_seconds_max  
# TYPE decaton_processor_active_seconds_max gauge
decaton_processor_active_seconds_max{partition="1",processor="-",subscriptionId="subscriptionId",topic="topic"} 0.0
# HELP decaton_processor_processingCompletion_total  
# TYPE decaton_processor_processingCompletion_total counter
decaton_processor_processingCompletion_total{partition="1",processor="-",subscriptionId="subscriptionId",topic="topic"} 1.0
# HELP decaton_processor_processingReturn_total  
# TYPE decaton_processor_processingReturn_total counter
decaton_processor_processingReturn_total{partition="1",processor="-",subscriptionId="subscriptionId",topic="topic"} 1.0
# HELP decaton_processor_processingStart_total  
# TYPE decaton_processor_processingStart_total counter
decaton_processor_processingStart_total{partition="1",processor="-",subscriptionId="subscriptionId",topic="topic"} 1.0
# HELP decaton_processor_seconds  
# TYPE decaton_processor_seconds summary
decaton_processor_seconds_count{error="none",partition="1",processor="SomeDecatonProcessor",subscriptionId="subscriptionId",topic="topic"} 1
decaton_processor_seconds_sum{error="none",partition="1",processor="SomeDecatonProcessor",subscriptionId="subscriptionId",topic="topic"} 0.817615375
# HELP decaton_processor_seconds_max  
# TYPE decaton_processor_seconds_max gauge
decaton_processor_seconds_max{error="none",partition="1",processor="SomeDecatonProcessor",subscriptionId="subscriptionId",topic="topic"} 0.817615375

It seems possible for users to avoid recording duplicate metrics by being creative on their side.
micrometer-metrics/micrometer#5467

What should we do with the existing Micrometer Tracing support?

Spring users should use the Micrometer Observation support added here.
I haven’t seen many libraries that use Micrometer Tracing alone, so if the maintenance cost feels high, it might be fine to remove it.