Channels: Observer implementation and persistence

[stuartc]

User story

We need every proxied request to be logged with full metadata — this is the core value proposition of Channels. Build the observer that persists request/response data as each request flows through the proxy.

Details

Weir observer API not finalised. The current Weir observer behaviour and callback signatures are a working design but may change once we're deep in the integration. The acceptance criteria below reflect the current design — expect refinement during implementation.

Build Lightning.Channels.Observer implementing Weir.Observer behaviour to persist every proxied request. This story also includes the Weir modifications needed for observer state threading (deferred from the proxy story to keep scopes clean).

The observer hooks into the proxy pipeline at three points:

1. Request started — create a ChannelRequest record synchronously (if insert fails, abort the proxy)
2. Response started — capture response metadata as it arrives
3. Response finished — finalise the ChannelRequest state and create a ChannelEvent with full details

Implementation notes

Dependencies: #4401 (proxy pipeline), #4399 (schema)

Blocks: #4406 (snapshots), #4408 (history page)

Key technical considerations:

• handle_request_started creates the ChannelRequest synchronously — if insert fails, proxy is aborted (return error to client)
• Request headers must be captured from conn before the Weir call, not from handle_response_finished which doesn't include them
• started_at uses DateTime.utc_now(), not Weir's monotonic time
• Incremental body hashing (SHA256 via :crypto.hash_update/2) for both request and response — constant memory
• Body preview captured (first 64KB) for both request and response
• Completion persistence is async (via Task.Supervisor, not bare Task.start)
• If async persistence fails, ChannelRequest marked as error state and error logged
• Sensitive headers (Authorization, X-API-Key) redacted before storage
• TTFB (time to first byte) calculated and stored on event

Weir modifications (deferred from proxy story):

• Observer callbacks need to accept and thread user-defined state — the exact API (tuple option, behaviour callbacks, arity) will be finalised during implementation
• Existing Weir tests must still pass

State derivation for ChannelRequest:

State	Condition
pending	Created, no sink response yet
success	Sink returned 2xx
failed	Sink returned 4xx or 5xx
timeout	Sink connection timed out
error	Internal error (e.g., observation failed)

Release notes

N/A — internal observability infrastructure, not directly user-facing.

User acceptance criteria

• Lightning.Channels.Observer implements Weir.Observer with all three callbacks
• handle_request_started creates a ChannelRequest record synchronously; proxy aborted on failure
• Request headers captured from conn before Weir call
• started_at uses DateTime.utc_now()
• Incremental body hashing (SHA256) for request and response — constant memory
• Body preview captured (first 64KB) for request and response
• ChannelEvent created with: type, request method/path/headers/body preview+hash, response status/headers/body preview+hash, latency_ms
• ChannelRequest.state updated on completion (success/failed/timeout/error)
• TTFB calculated and stored on event
• Completion persistence is async via Task.Supervisor
• Async persistence failure marks ChannelRequest as error and logs
• Sensitive headers (Authorization, X-API-Key) redacted before storage
• Weir observer callbacks accept and thread user-defined state (API to be finalised during implementation)
• Existing Weir tests still pass