How Distributed Tracing Works: Scrub It, Poke It, Break It

Distributed tracing is the practice of stamping every unit of work with the ID of the request that caused it, then carrying that ID across every process boundary. It turns “something is slow somewhere” into “this span, in this service, at this line”. The W3C Trace Context standard is what makes it work across competing vendors, and it does so with just two HTTP headers: traceparent and tracestate. A user in Bhubaneswar taps “Pay” on a checkout page. The request touches a gateway, an auth service, a checkout service, an inventory service and a payments service: five processes, on five machines, each writing its own logs. The page takes 148 ms. The user notices nothing. Now imagine it took 4.8 seconds instead, and it is your job to find out why. I work in observability, and I have watched people try to answer that question with grep. Even with structured logging already in place, you open five log streams, squint at timestamps and try to stitch together which lines belong to that one…