Observability — Micrometer + OpenTelemetry

Observability is opt-in and disabled by default. A fresh RestContext built with no MetricsRecorder or TracerHook bean registered never records a metric sample, never opens a span, and never adds a per-request cost — even when io.micrometer:micrometer-core and io.opentelemetry:opentelemetry-api are sitting on the classpath. Opt in by registering a @Bean MetricsRecorder and / or @Bean TracerHook on the resource. Same non-negotiable contract as Jakarta Bean Validation.

This is a deliberate departure from Spring Boot's auto-configuration, where WebMvcMetricsFilter is wired up at startup and a MeterRegistry bean is enough to turn on per-request metrics. Juneau's contract is the opposite: zero classpath cost, zero per-request cost, and zero startup cost until the user explicitly opts in.

Motivation

Juneau REST already tracks per-method execution statistics via MethodExecStats / RestContextStats. The data is there — only the wiring to the dominant observability stacks (Prometheus / Grafana via Micrometer; distributed tracing via OpenTelemetry) was missing. The new SPIs (MetricsRecorder, TracerHook) plus two opt-in bridge modules (juneau-rest-server-metrics-micrometer, juneau-rest-server-tracing-otel) close that gap without dragging either runtime into the core.

End-state developer experience:

// pom: add juneau-rest-server-metrics-micrometer + your preferred MeterRegistry (e.g. micrometer-registry-prometheus).
@Configuration
public class ObservabilityConfig {
    @Bean MeterRegistry registry() { return new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); }
    @Bean MetricsRecorder recorder(MeterRegistry r) { return new MicrometerMetricsRecorder(r); }
}
// → Each @RestOp call records a Timer sample on the registry; scrape via /actuator/prometheus.

// pom: add juneau-rest-server-tracing-otel + OTel SDK.
@Bean OpenTelemetry otel() { return GlobalOpenTelemetry.get(); }
@Bean TracerHook hook(OpenTelemetry otel) { return new OtelTracerHook(otel); }
// → Each @RestOp call becomes a SERVER span; incoming traceparent continues the upstream trace.

The off-by-default contract

The off-by-default contract has six tangible guarantees:

1. Cold-start default. A fresh RestContext built with no MetricsRecorder / TracerHook bean registered never resolves a non-NoOp recorder or tracer. The framework looks up the SPI through the RestContext's BeanStore and falls back to NoOpMetricsRecorder.INSTANCE / NoOpTracerHook.INSTANCE — both stateless singletons whose record(...) / startSpan(...) methods short-circuit immediately.
2. Per-resource granularity. A @Bean MetricsRecorder on one @Rest class never opts in a sibling resource, an unrelated resource, or the global JVM. Bean resolution is scoped to the resource's BeanStore.
3. No global "on" switch. There is no RestContext.Builder.observability(true), no juneau.observability.enabled=true system property, and no resource-level boolean attribute that enables observability across every operation in a class. Bean registration is the only opt-in surface.
4. No silent enablement via classpath presence. Even if micrometer-core and opentelemetry-api are on the classpath, no metric is recorded and no span is opened unless the user has registered a @Bean MetricsRecorder / @Bean TracerHook.
5. Lifecycle hooks are outside the observability boundary. @RestPreCall and @RestPostCall lifecycle methods are not wrapped — only the @RestOp handler invocation (the user-facing operation) is. This keeps the metric / span scoped to the call as the API consumer perceives it.
6. Hot-path overhead is short-circuited. The NoOpMetricsRecorder.record(...) and NoOpTracerHook.NoOpScope.close() methods compile to empty bodies, so the RestOpInvoker observability block costs at most one BeanStore.getBean(...) lookup plus two no-op virtual calls per request when nothing is wired up.

SPIs in juneau-rest-server

Two functional interfaces live in core juneau-rest-server. They're stable enough for users to implement directly — the Micrometer / OpenTelemetry bridges shipped in the sibling sub-modules are not the only valid implementations.

MetricsRecorder

@FunctionalInterface
public interface MetricsRecorder {
    void record(String opName, String httpMethod, String uriTemplate,
                int statusCode, Duration elapsed, Throwable error);
}

Called exactly once per @RestOp invocation (happy path and exception path). Arguments:

Field	Meaning
opName	Fully-qualified Java method name of the handler (e.g. com.acme.MyResource.findUser(java.lang.String)).
httpMethod	Uppercased HTTP method (e.g. GET, POST).
uriTemplate	The @RestOp path template (e.g. /users/{id}) — not the raw concrete URI. Using the template keeps metric / tag cardinality bounded; raw URIs (/users/123, /users/124, …) would explode the registry's series count. Empty string when the operation has no path pattern.
statusCode	HTTP response status as resolved by the framework after the handler runs (200 on success; 4xx / 5xx on the exception path).
elapsed	Wall-clock duration from just-before parameter binding to just-after framework exception mapping.
error	The thrown throwable on the exception path, or null on the happy path.

TracerHook

@FunctionalInterface
public interface TracerHook {
    Scope startSpan(RestRequest request);
}

public interface Scope extends AutoCloseable {
    void setStatusCode(int statusCode);
    void setError(Throwable error);
    @Override void close();
}

startSpan opens a per-request scope before parameter binding. The framework always closes the scope in a finally block, delivering the resolved HTTP status and any thrown throwable via setStatusCode(int) / setError(Throwable) first. close() runs exactly once per call (including parameter-resolution failures and 5xx paths) so consumers can safely end an OTel span or finalize an in-memory record inside close().

NoOp defaults

• NoOpMetricsRecorder.INSTANCE — drops events on the floor with zero allocations.
• NoOpTracerHook.INSTANCE — returns NoOpTracerHook.NoOpScope.INSTANCE. All three Scope methods are empty.

Both are process-wide singletons. The framework picks them up automatically when no @Bean of the corresponding type is registered.

Where the hooks fire

The hooks wrap the handler invocation in RestOpInvoker.invokeOp(...). The sequence per call:

1. Resolve MetricsRecorder / TracerHook from the RestContext's BeanStore (one lookup; falls back to the NoOp singletons).
2. tracer.startSpan(request) → Scope.
3. System.nanoTime() → startNanos.
4. Bind parameters, invoke the user method, run the framework's standard exception-mapping path.
5. In a finally block:
   • elapsed = Duration.ofNanos(System.nanoTime() - startNanos).
   • status = res.getStatus() (falling back to 200 / 500 if the response status is still 0).
   • scope.setStatusCode(status); scope.setError(observed) if a throwable was caught.
   • scope.close().
   • recorder.record(opName, httpMethod, uriTemplate, status, elapsed, observed).

Pre / post-call lifecycle methods (@RestPreCall / @RestPostCall) are intentionally outside the observability boundary — the metric / span describes the user-facing operation only.

Micrometer bridge — juneau-rest-server-metrics-micrometer

The Micrometer bridge ships a single MetricsRecorder implementation that records to any MeterRegistry:

public class MicrometerMetricsRecorder implements MetricsRecorder {
    public MicrometerMetricsRecorder(MeterRegistry registry) { ... }
    public MicrometerMetricsRecorder(MeterRegistry registry, String timerName) { ... }

    @Override
    public void record(String opName, String httpMethod, String uriTemplate,
                       int statusCode, Duration elapsed, Throwable error) {
        Timer.builder(timerName)        // default: "http.server.requests"
            .tag("method", httpMethod)
            .tag("uri",    uriTemplate) // bounded cardinality
            .tag("status", Integer.toString(statusCode))
            .tag("exception", error == null ? "None" : error.getClass().getSimpleName())
            .register(registry)
            .record(elapsed);
    }
}

Metric shape

The default tag set mirrors Spring Boot's WebMvcMetricsFilter convention (http.server.requests with {method, uri, status, exception}) so existing Prometheus / Grafana dashboards built for Spring Boot services keep working when those services are reimplemented on Juneau:

http_server_requests_seconds_count{method="GET",uri="/users/{id}",status="200",exception="None"} 1
http_server_requests_seconds_sum  {method="GET",uri="/users/{id}",status="200",exception="None"} 0.007

Why Spring's naming convention (not OTel-native)

OpenTelemetry's metric semantic conventions specify http.server.duration with http.request.method / http.response.status_code / http.route attribute names. We chose Spring Boot's older http.server.requests shape for the Micrometer bridge because the wider Prometheus / Grafana scrape-config and dashboard ecosystem already standardizes on it. The OTel-native attribute names live on the tracing side — see OpenTelemetry bridge — juneau-rest-server-tracing-otel below.

To override the timer name, use the two-arg constructor:

@Bean public MetricsRecorder recorder(MeterRegistry r) {
    return new MicrometerMetricsRecorder(r, "myservice.http.timer");
}

Dependency

<dependency>
    <groupId>org.apache.juneau</groupId>
    <artifactId>juneau-rest-server-metrics-micrometer</artifactId>
    <version>${juneau.version}</version>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-core</artifactId>
    <version>1.13.6</version>          <!-- consumer-supplied; provided scope on the bridge module -->
</dependency>
<!-- pick the registry you want -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
    <version>1.13.6</version>
</dependency>

The bridge declares micrometer-core in provided scope so the consumer picks the version. mvn -pl juneau-rest/juneau-rest-server dependency:tree | grep -i micrometer returns nothing — the containment requirement is verified at build time.

OpenTelemetry bridge — juneau-rest-server-tracing-otel

The OpenTelemetry bridge ships a single TracerHook implementation that opens a span per request and propagates W3C trace context:

public class OtelTracerHook implements TracerHook {

    // Option A — use the JVM-wide GlobalOpenTelemetry (recommended).
    public OtelTracerHook() { this(GlobalOpenTelemetry.get()); }

    // Option B — explicit OpenTelemetry instance (e.g. multi-tenant or test).
    public OtelTracerHook(OpenTelemetry otel) { ... }

    // Option C — explicit Tracer + TextMapPropagator (typically for tests).
    public OtelTracerHook(Tracer tracer, TextMapPropagator propagator) { ... }

    @Override
    public Scope startSpan(RestRequest request) {
        // Extract incoming W3C traceparent / tracestate.
        Context extracted = propagator.extract(Context.current(), request, RestRequestTextMapGetter.INSTANCE);

        Span span = tracer.spanBuilder(request.getMethod())   // "GET", "POST", ...
            .setSpanKind(SpanKind.SERVER)
            .setParent(extracted)
            .setAttribute("http.request.method", request.getMethod())
            .setAttribute("http.route",          /* @RestOp path template */)
            .startSpan();

        return new OtelScope(span, span.makeCurrent());
    }
}

Span shape

Each @RestOp invocation becomes one span with:

Attribute	Meaning
Span kind	SpanKind.SERVER
Span name	The uppercased HTTP method (GET, POST, …) per the OTel HTTP semantic conventions.
http.request.method	Uppercased HTTP method.
http.response.status_code	HTTP response status (set in Scope.close(), so error paths get the framework's resolved status).
http.route	The @RestOp path template (e.g. /users/{id}) — bounded cardinality, same reasoning as the Micrometer uri tag.

On the exception path:

• Span.recordException(throwable) adds an event carrying the stack trace.
• The span status is set to StatusCode.ERROR.
• An exception.type attribute carries the throwable's simple-name (e.g. IllegalStateException).

A 2xx status leaves the span status at StatusCode.UNSET per the OTel HTTP semconv (servers should not pre-color successful spans as OK).

W3C trace-context propagation

Incoming traceparent / tracestate headers are extracted via the configured TextMapPropagator (default: W3CTraceContextPropagator) so the server span continues a caller-supplied distributed trace:

Request: traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

Server span: traceId=0af7651916cd43dd8448eb211c80319c
             parentSpanId=b7ad6b7169203331
             spanId=<freshly-minted>

Downstream HTTP calls made from inside the handler pick up the active span's context automatically when the outbound HTTP client honors the OTel Context.current() (the standard OTel client instrumentations do).

Wiring

@Configuration
public class ObservabilityConfig {

    // Option A: rely on GlobalOpenTelemetry (typically wired by the otel-java-agent).
    @Bean TracerHook tracer() { return new OtelTracerHook(); }

    // Option B: explicit instance (e.g. for tests).
    @Bean TracerHook tracer(OpenTelemetry otel) { return new OtelTracerHook(otel); }
}

Dependency

<dependency>
    <groupId>org.apache.juneau</groupId>
    <artifactId>juneau-rest-server-tracing-otel</artifactId>
    <version>${juneau.version}</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.43.0</version>          <!-- consumer-supplied; provided scope on the bridge module -->
</dependency>
<!-- typically also the SDK + exporter of your choice -->
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk</artifactId>
    <version>1.43.0</version>
</dependency>

The bridge declares opentelemetry-api in provided scope so the consumer picks the version. mvn -pl juneau-rest/juneau-rest-server dependency:tree | grep -i opentelemetry returns nothing — the containment requirement is verified at build time.

Outgoing-response trace-context headers

The incoming-side propagation above continues an upstream trace into the server span. The outgoing side writes the W3C traceparent — and, when present, the tracestate — header back onto the HTTP response, so a client calling the server can read the resulting trace id off the response and correlate its own logs without running a tracing agent of its own:

Response: traceparent: 00-0af7651916cd43dd8448eb211c80319c-<server-span-id>-01
          tracestate:  vendor1=opaqueValue1,vendor2=opaqueValue2

The emitted traceparent reflects the server-started span's context (a freshly-minted span id parented to the inbound trace), not the inbound header verbatim — the same span the OTel bridge opened for the request. tracestate is written only when the active trace state is non-empty; an empty trace state never produces an empty tracestate header.

On-when-tracer, zero-cost otherwise

Response-header injection is on by default whenever a non-no-op TracerHook is active — registering a @Bean TracerHook (e.g. OtelTracerHook) is the only opt-in needed; it requires no extra wiring beyond the tracer itself. On the no-tracer path it stays zero-cost: the response processor's first action is a single request-attribute read that short-circuits when no tracer stashed a trace context, so the off-by-default contract above is preserved end-to-end.

How it works

The OTel bridge renders the W3C header values from the server-started span's context at span-start time — the only point where that context is reliably active — and stashes them as request attributes. A TraceContextResponseProcessor in juneau-rest-server's response-processor chain then reads those stashed strings during response rendering (after the span scope has closed) and writes them as response headers. This split keeps the processor free of any OpenTelemetry dependency: it works with any TracerHook bridge that populates the attributes, and core juneau-rest-server stays OTel-agnostic.

Already-committed responses

If the response has already been committed (for example a streaming / SSE handler that flushed its headers before returning), the headers can no longer be set. The processor logs a FINE message and skips the write — it never throws.

Opting out

Set the RestContext.responseTraceparent environment / system property to false to keep the processor out of the chain entirely, even when a tracer is active:

-DRestContext.responseTraceparent=false

The default is true (on-when-tracer). A per-resource @Rest-annotation opt-out is tracked as a follow-on.

MCP Trace-Context Propagation (2026-07-28)

MCP 2026-07-28 endpoints are ordinary @RestOps — POST / on McpRestServlet, POST /mcp on McpEndpoint — so everything above (the off-by-default contract, TracerHook, the OpenTelemetry bridge, response-header injection) already applies to them with no MCP-specific wiring. This section documents the one MCP-specific extension: a second, non-HTTP place a trace context can travel — the JSON-RPC request's own params._meta — and the neutral seam that makes that possible without teaching juneau-rest-server anything about MCP or JSON-RPC.

v2-only. Trace-context propagation is a 2026-07-28 feature; 2025-06-18 keeps its original HTTP-header-only tracing behavior unchanged — no _meta, no trace echo.

The neutral carrier seam

RestOpInvoker already resolves every @RestOp handler argument before it asks the active TracerHook for a span. Three neutral types in juneau-rest-server let a non-HTTP argument (like a parsed MCP JsonRpcRequest) participate in that decision without OpenTelemetry — or any tracer implementation — knowing what MCP or JSON-RPC are:

public interface TraceContextCarrier {
    String get(String key);
    Iterable<String> keys();
    void set(String key, String value);
}

public interface TraceContextExtractor {
    Optional<TraceContextCarrier> extract(RestRequest request, Object[] resolvedArguments);
    default TraceOperation operation(RestRequest request, Object[] resolvedArguments) {
        return TraceOperation.DEFAULT;
    }
}

• TraceContextCarrier is a read/write key-value view over W3C fields (traceparent/tracestate/baggage) from a source that need not be HTTP headers.
• TraceContextExtractor is a bean an adapter (like the 2026-07-28 MCP binding) publishes; it inspects the already-resolved handler arguments and, if it recognizes one, hands back a carrier plus a TraceOperation — an immutable, low-cardinality span name and attribute set the bridge should use instead of its own HTTP-derived defaults.
• Extraction only happens when a non-no-op TracerHook is active. A resource with no registered TracerHook never resolves a TraceContextExtractor either — the existing zero-allocation no-tracer fast path is unchanged.

TracerHook gains two source-compatible default methods so every existing implementation — including a bare lambda implementing only startSpan(RestRequest) — keeps compiling unchanged:

default Scope startSpan(RestRequest request, TraceContextCarrier carrier, TraceOperation operation) {
    return startSpan(request); // ignores carrier/operation unless overridden
}

default void inject(TraceContextCarrier carrier) {
    // no-op unless overridden
}

Scope gains one more source-compatible default, for observing a JSON-RPC-level error on an already-HTTP-200 response (see below):

default void recordRpcError(int code, String message) {
    // no-op unless overridden
}

params._meta — a second carrier, not a new protocol

The 2026-07-28 MCP adapter (juneau-rest-server-mcp-v20260728) publishes a stable TraceContextExtractor from both McpRestServlet and McpEndpoint (as a @Bean mcpTraceContextExtractor()). It recognizes the resolved JsonRpcRequest argument and reads its params._meta, which may carry the bare (unprefixed) W3C fields traceparent, tracestate, and baggage alongside the io.modelcontextprotocol/-prefixed negotiation keys. The extractor hands the bridge a composite carrier:

lookup(key) = params._meta[key] if present, otherwise the HTTP header of the same name

An explicit params._meta value wins over the equivalent HTTP header, independently for each of the three W3C fields. An absent metadata key simply falls back to the HTTP header. The extractor also derives a TraceOperation naming the span after the MCP method/target ("tools/call echo", "prompts/get greet", bare "resources/read", or the exact method otherwise) with mcp.*/gen_ai.* attributes — see juneau-rest-server-mcp for the full MCP-side walkthrough, including a worked request/response example.

How the OpenTelemetry bridge implements the composite

OtelTracerHook overrides the three-argument startSpan overload above. It runs the configured TextMapPropagator twice against the same Context: first against ordinary HTTP headers, then — only when the extractor recognized a carrier — a second pass against that carrier. Per the propagator contract, extraction that finds nothing usable leaves the Context from the previous pass unchanged, so an explicit params._meta value wins while an absent/invalid one falls back to whatever the HTTP pass already produced. This applies independently to the traceparent/tracestate pair and to baggage.

The bridge also overrides inject(TraceContextCarrier) (backing the neutral TraceContexts.inject(TracerHook, TraceContextCarrier) helper below) using the same configured propagator against Context.current().

Result _meta echo

At span-start, the OTel bridge already stashes the rendered traceparent/tracestate/(non-empty) baggage as request attributes for TraceContextResponseProcessor to write as HTTP response headers (see above). Before returning any successful 2026-07-28 result, the MCP adapter reads those same request-scoped attributes and copies them into the result's own _meta, alongside the server identity every success already carries:

{
  "traceparent": "00-0af7651916cd43dd8448eb211c80319c-<server-span-id>-01",
  "io.modelcontextprotocol/serverInfo": {"name": "my-server", "version": "1.0.0"}
}

So an MCP client reading the JSON-RPC result body sees the same trace identifiers an HTTP-level observer reads off the response headers. JSON-RPC errors have no result, so there's no error-response echo.

JSON-RPC errors on an HTTP-200 response

A dated MCP adapter always answers over HTTP 200, even for a JSON-RPC-level error (an unknown tool, a malformed request) — so Scope.setStatusCode(int) never sees anything but the adapter's uniform success status for those cases. Before returning a JSON-RPC error response, McpRevision calls the neutral Scope.recordRpcError(code, message) default introduced above on the request's active tracing scope. OtelTracerHook's override sets rpc.response.status_code, a low-cardinality error.type category (the JSON-RPC 2.0 reserved-code name, or "server_error" for the -32000..-32099 implementation-defined range), and StatusCode.ERROR with the JSON-RPC message as the status description — distinct from the existing thrown-exception path (Span.recordException(...) + exception.type), which still applies to a handler that actually throws.

Neutral injection helper

A dependency-free static helper stamps the active tracer's current trace context into an arbitrary carrier, without the caller depending on OpenTelemetry:

TraceContexts.inject(TracerHook tracer, TraceContextCarrier carrier);

This has no first-party MCP call site in 10.0.0 — it exists so a future MCP client (or any other non-HTTP caller) can inject the active trace context into whatever carrier it builds (for example a RequestMeta it's about to send as params._meta) using only this class and the installed TracerHook.

Pinned experimental semantic conventions

The gen_ai.*/mcp.* span attribute names, the tool/prompt/resource span-naming rules, and the error.type mapping above are pinned to the OpenTelemetry GenAI semantic-conventions snapshot at commit c739977ae690961f36e435504e5c1febaef1f7f3, plus core OpenTelemetry semantic conventions v1.43.0. All of these attributes are experimental/development — the dedicated GenAI semantic-conventions repository has no versioned release for this area yet, so a commit pin is the reproducible version. Re-check this pin before any OpenTelemetry or semantic-conventions dependency upgrade that touches MCP tracing; names and requirement levels may still change upstream.

2025-06-18 is unaffected

None of the above exists on 2025-06-18: its wire beans have no _meta-carried trace fields, its adapter publishes no TraceContextExtractor, and it continues to rely solely on the HTTP-header propagation and response-header injection described earlier on this page.

MCP client trace-context injection seam

The MCP client stack (juneau-rest-client-mcp + dated adapters) exposes an injection seam that mirrors the server-side trace propagation model without coupling client modules to server-only tracing types.

• The v2 adapter writes traceparent, tracestate, and baggage into request params._meta.
• The same adapter reads echoed values from result._meta on success.
• Injection is opt-in and builder-configurable; no tracing fields are emitted unless configured.

This seam is intentionally payload-opaque at the duplex boundary: server->client requests are dispatched as raw JsonRpcRequest envelopes through McpServerRequestHandler, with no typed sampling/elicitation projection in the client core.

Custom MetricsRecorder / TracerHook implementations

The SPIs are deliberately tiny — implement them directly when you need a backend the shipped bridges don't cover (Dropwizard Metrics, an internal time-series store, a structured-log appender, …):

@Bean public MetricsRecorder recorder() {
    return (opName, method, uri, status, elapsed, error) -> {
        myInternalMetricsBus.recordHttpCall(method, uri, status, elapsed.toMillis(), error);
    };
}

@Bean public TracerHook tracer() {
    return request -> {
        var startedAt = System.nanoTime();
        return new Scope() {
            @Override public void setStatusCode(int s) { /* stash */ }
            @Override public void setError(Throwable t) { /* stash */ }
            @Override public void close() {
                myInternalTraceBus.recordSpan(request.getMethod(), System.nanoTime() - startedAt);
            }
        };
    };
}

Per-resource and per-operation observability controls

Three annotation members added in 10.0 give fine-grained control over whether and how the observability block fires.

@Rest(observability=...) — resource-level opt-in / opt-out

Controls whether the observability block (MetricsRecorder / TracerHook) fires for all operations on a resource:

Value	Behavior
"true"	Strict opt-in: the resource requires a wired observability backend. If neither @Bean MetricsRecorder nor @Bean TracerHook is registered when RestContext is built, construction fails with a clear error. All operations have observability enabled.
"false"	Explicit opt-out: the observability block is short-circuited for every operation, even when a wired backend is present. Also suppresses TraceContextResponseProcessor header injection for the resource.
"" (default)	Inherits existing behavior: the observability block runs per-op; a missing backend silently falls back to the NoOp singleton.

Supports SVL variables (e.g. "$E{ENABLE_OBSERVABILITY,false}").

@RestOp(observability=...) — operation-level override

Same three values as the resource-level attribute; takes full precedence over @Rest(observability=...) for the specific operation. Use "false" to opt a single cheap operation out of recording while keeping the rest of the resource instrumented:

@Rest(observability="true")
public class ApiResource extends BasicRestServlet {

    @RestGet("/orders/{id}")
    public Order get(@Path String id) { ... }           // observability on

    @RestGet("/health")
    @RestOp(observability="false")
    public String health() { return "ok"; }             // observability off for this op
}

@RestOp(metricName=...) — override the derived metric name

By default the MetricsRecorder uses its own name-derivation logic (typically the fully-qualified Java method name or the opName passed from the framework). metricName overrides this for a specific operation:

@RestGet("/orders/{id}")
@RestOp(metricName="orders.get")
public CompletableFuture<Order> getOrder(@Path String id) { ... }

Supports SVL variables. Has no effect when observability is disabled for the operation.

@RestOp(metricTags=...) — additional tags

Comma-separated key=value pairs passed through to MetricsRecorder.record(...) alongside the standard tags. The MetricsRecorder implementation is responsible for parsing and applying them:

@RestPost("/orders")
@RestOp(metricTags="team=payments,region=us-east")
public Order createOrder(@Content CreateOrder body) { ... }

Supports SVL variables. Has no effect when observability is disabled for the operation.

Precedence summary

Level	Wins over
@RestOp(observability=...)	@Rest(observability=...) and the default
@Rest(observability=...)	The default (silent NoOp fallback)

Programmatic equivalent

For programmatic resource construction via AbstractRestBuilder, use the observability(String) method (part of the RestBuilder<SELF> interface) to set the resource-level value, and RestAnnotation.Builder.observability(String) for op-level configuration.

Custom (non-request) observations

Everything above instruments the request boundary — the framework opens and closes the observation around each @RestOp handler. To instrument an arbitrary block of work that has no associated HTTP request (a scheduled job, a downstream service call, a cache refresh), use the explicit programmatic observation API in org.apache.juneau.rest.server.observation.

Why explicit, not @Observed

Juneau has no method-interception / AOP substrate — no dynamic proxying of arbitrary beans, no bytecode weaving, no annotation-processor agent. Rather than introduce one to support a Spring-style @Observed annotation, custom observations are explicit: you wrap the block in a try-with-resources. This is the same explicit-over-magic stance as the curated dependency bundles (which register nothing automatically) — the active observation is the literal block you wrote, not something a classpath scan turned on.

Usage

import org.apache.juneau.rest.server.observation.*;

try (Observation o = Observations.observe("order.load", "team=payments")) {
    return loadOrder(id);
} catch (RuntimeException e) {
    o.setError(e);
    throw e;
}

Observations.observe(name, tags) records a metric timer named name (carrying the comma-separated key=value tags plus an exception tag) and opens a tracing span named name. On close() the elapsed time is recorded and the span ends; setError(...) marks both the timer's exception tag and the span's error status.

Reuses the request-path SPIs

Custom observations go through the same two SPIs the request path uses — via non-request default methods added to each:

• MetricsRecorder.record(String metricName, String metricTags, Duration elapsed, Throwable error)
• TracerHook.startSpan(String spanName) (opens an INTERNAL-kind span, nested under any active request span)

The shipped bridges (MicrometerMetricsRecorder, OtelTracerHook) override both. A bridge that doesn't override them simply skips custom observations. There is no second tag model — the metricName / metricTags strings are exactly the request-path model.

Wiring the backend

Observations is a static facade over an explicitly-installed default Observer. Install one once at startup, after your observability beans are known:

Observations.install(new Observer(metricsRecorder, tracerHook));

Until a backend is installed (and in any process that never installs one), Observations.observe(...) returns Observation.NOOP — no timestamp taken, no span opened, zero allocation beyond the try-with-resources scaffolding. Code that prefers explicit DI over a global can hold an Observer directly (resolved from the RestContext bean store) and call observer.start(name, tags).

One-dependency OTLP export — juneau-observability-otlp-bundle

The two bridge modules (-metrics-micrometer, -tracing-otel) leave the consumer to assemble a coherent exporter stack. The curated OTLP bundle does that in one dependency — pull it to export both metrics and traces over OTLP:

<dependency>
    <groupId>org.apache.juneau</groupId>
    <artifactId>juneau-observability-otlp-bundle</artifactId>
    <version>${juneau.version}</version>
    <type>pom</type>
</dependency>

It transitively brings both Juneau bridges plus a version-aligned OTLP exporter stack: opentelemetry-exporter-otlp (+ SDK + autoconfigure) for traces and micrometer-registry-otlp for metrics (which also satisfy the bridges' provided API dependencies, so you get a runnable stack from the one bundle). v1 covers metrics + traces; OTLP logs export is not bundled.

Like every Juneau bundle it is a dependency aggregator only — it registers no exporters and auto-discovers nothing. You still wire your MeterRegistry / OpenTelemetry beans and the @Bean MetricsRecorder / @Bean TracerHook explicitly, exactly as shown above. See Dependency Management (BOM & Bundles).

Log correlation — trace id in your logs

To correlate log lines with traces, Juneau's JUL LogEntryFormatter supports two optional placeholders:

Placeholder	Value
{traceId}	The active OpenTelemetry trace id, or empty when OTel is absent / no span is active.
{spanId}	The active OpenTelemetry span id, or empty when OTel is absent / no span is active.

[{date} {level}] trace={traceId} span={spanId} {msg}%n

The ids are sourced from the active OTel Span.current().getSpanContext() via an optional / reflective coupling — juneau-microservice does not hard-depend on OpenTelemetry. When OTel is absent from the runtime classpath the fields simply render empty (the placeholders never leak literally). This mirrors how MdcAsyncListener reflectively handles SLF4J.

Async correlation

When a @RestOp returns a CompletableFuture, MdcAsyncListener (gated by RestContext.isMdcAsyncPropagation()) additionally folds the active trace id / span id into the propagated SLF4J MDC snapshot under the conventional OTel appender keys (trace_id / span_id), so correlation survives the async-completion hop even when the completion thread has lost the OTel context. This too is reflective and optional.

Limitation — JUL only

LogEntryFormatter is java.util.logging-based. An application on SLF4J → Logback / Log4j2 is not served by these placeholders — wire your own MDC pattern instead (the OTel logback/log4j appenders populate trace_id / span_id in MDC, which the async-correlation enrichment above feeds). Driving every logging backend is explicitly out of scope.

What's out of scope (v1)

• Structured-logging bridges (SLF4J / Log4j2 structured appender) — the call-logger rework is tracked separately; the OTel bridge can publish a Logs event later if there's demand.
• Custom tag schemes per resource. v1 uses fixed Spring-style tags on the Micrometer side and fixed OTel HTTP semantic-convention attribute names on the OTel side. Use @RestOp(metricTags=...) for per-operation additions; a per-resource default tag set is tracked as a follow-on.
• Histogram percentile config from annotations. Let the user configure the MeterRegistry directly — that's already a Micrometer-native surface (@Timed, MeterFilter).
• StatsD / Datadog / NewRelic native bridges. Use Micrometer's registries — that's the whole point of MetricsRecorder going through MeterRegistry.

See also

• REST Server — Jakarta Bean Validation — same off-by-default contract precedent.
• REST Server — Rate-Limiting and Request-Id Propagation — sibling operational primitives. The requestId is the natural log-correlation key alongside the OTel span id.