Request Traces

A 'Request Trace' documents the progress of a request and response through a services of 'hops'; a 'hop' may be an intermediate gateway or service. Traces capture all (or most of) the dependencies of a service.

Wheres the Topology display aggregates multiple requests over a window of time and displays the dependencies, a trace samples a single request and its dependencies:

How are Traces sampled?

Traces are expensive to gather, in terms of CPU, network and storage. The Tetrate platform samples a small proportion of requests, with a default sampling rate of 1%. This capability may be fine-tuned or disabled by your platform administrator.

The Tetrate platform uses the Zipkin B3 trace headers. In particular, if you issue a request with the header X-B3-Sampled = 1, that request and its dependencies will be sampled.

Traces are 'traced' through the network of dependencies using a trace ID, which is carried in a X-B3-TraceId HTTP header. This header, along with other X-B3- headers, must be propagated by each hop in the flow. Check the documentation for your application framework for best-practice advice on propagating these headers.

View a Trace

Select the service you are interested in, and navigate to the TRACE pane:

Traces take time to process and propagate

Note that it can take several minutes, and up to 30 minutes during busy period, for trace data to be collected from all envoy proxies, processed and then presented as a full application trace.

View a recent trace:

What does this tell us?

• Edge Gateway 'edge-gw' took 37ms to process the request and response, from end-to-end
• The Edge Gateway called the bookinfo-appgw gateway, which called the productpage service
• productpage processed the request, and during processing, it called details and reviews. reviews in turn called ratings.

Note that when a service calls a dependency, for example, when productpage calls details, you will observe two trace 'spans':

These spans correspond to two samples - one from the sidecar on productpage which samples the outgoing request, and one on the sidecar in details which samples in the incoming request.

Locating Errors with Traces

Spotting errors with Traces can be challenging, as only a very small proportion of requests are typically traced.

Nevertheless, you may have noticed when we inspected the metrics and topology that some spans (edges) were colored red, indicating a high frequency of errors. These spans originated from the gateway.

If we look at traces from the perspective of the gateway, filter on 503 errors, and unfold the traces, we can see some of these error requests:

Look at the errored request in detail:

Observe that the response code is 503, and the edge gateway is attempting and failing to forward the request to reviews-old.bookinfo.svc.cluster.local:9080. This looks like a misconfiguration on the gateway!