Observability on Stacknote Demo

Trace the Request Before You Tune the Service

Mon, 17 Aug 2026 00:00:00 +0000

A slow endpoint is rarely one indivisible wait. It is a chain: connection setup, queueing, application work, storage, and the trip back to the caller. The useful question is not simply “why is this request slow?” but “where did this request spend its time?”

Start with one trace

Pick one real slow request and write down its stages before changing a timeout or adding a cache.

gateway 18 ms
queue 142 ms
application 31 ms
database 24 ms
response 6 ms

The application function is not the bottleneck in this example. Making it twice as fast saves about 15 milliseconds while the request still waits 142 milliseconds before the function starts.

Measure the Tail, Not Just the Average

Mon, 03 Aug 2026 00:00:00 +0000

An average compresses a distribution into one comfortable number. Production latency is rarely comfortable or evenly distributed.

Suppose 99 requests finish in 40 milliseconds and one request takes four seconds. The average is about 80 milliseconds. That number describes almost nobody: most users saw half of it, while the unlucky user waited fifty times longer.

Keep the distribution

Track at least a few percentiles and request volume together:

Metric	Value
P50	40 ms
P95	58 ms
P99	4.0 s
Requests	100

Percentiles also need enough samples. A P99 calculated from a tiny window is mostly a story about one request, so retain histograms and compare equivalent traffic windows.