Choose monitoring signals that explain user impact

Start with important work

Identify the requests and background workflows whose failure would matter most. Consider whether users can sign in, submit work, retrieve results, and receive time-sensitive updates. These outcomes provide context for lower-level measurements.

Google’s SRE guidance describes latency, traffic, errors, and saturation as useful starting signals. Use that framework to ask concrete questions about your application, rather than copying a dashboard whose thresholds and workload assumptions belong to another service. Google SRE: monitoring distributed systems.

Preserve the distinctions that matter

Separate successful and failed requests where a combined average could be misleading. Examine important routes individually when one workflow can degrade while the rest remain healthy. Track waiting work as well as work currently executing.

Choose dimensions carefully. More detail can improve investigation, but unlimited identifiers in metrics can create operational costs and difficult-to-use data. Keep record-level context in a suitable logging or tracing system when that is the better fit.

Design the investigation path

For each important signal, document what a change may mean and where the operator should look next. Connect a request problem to the relevant release, dependency, or queue without requiring someone to assemble every clue manually.

Review monitoring after incidents and major changes. A signal that never informed a decision may need a different presentation or no longer deserve attention. A problem discovered first by users is an opportunity to ask which observable symptom could have appeared earlier.

The goal is a small set of understandable signals backed by deeper evidence. A dashboard should help an operator form and test a hypothesis, not merely confirm that instrumentation exists.

Instrument a user journey from start to finish.

For an inquiry workflow, there are at least three useful boundaries: the web request is accepted, the inquiry is durably stored, and its notification is processed. A healthy web server says little about the third boundary. Name the events and keep their meanings stable so operators do not confuse transport success with business completion.

Google’s SRE monitoring chapter gives a broader framework for choosing signals and distinguishing symptoms from causes. Use it to structure investigation, then choose measurements that actually represent your service rather than copying an unrelated dashboard.

Write the alert before choosing its threshold.

Symptom: inquiry notifications are not progressing
Affected work: accepted inquiries awaiting notification
Evidence: oldest pending item, queue size, recent outcomes
Owner: the team operating the inquiry service
First step: check worker activity and provider errors
Recovery check: pending work completes without lost inquiries

This example deliberately omits a universal numeric threshold. A delay that is acceptable for a daily report may be unacceptable for a time-sensitive workflow. Establish an expectation with the people receiving the work, observe the normal operating pattern, and choose a threshold that represents meaningful failure. Include a duration or other persistence rule where appropriate so one brief fluctuation does not create a stream of alerts.

Test the alert by stopping progress in a suitable environment and verifying both detection and recovery. A queue that drains should resolve the condition; a worker that repeatedly fails the same item should remain visible. Also check missing telemetry. No observations can mean that no requests arrived, that instrumentation broke, or that the application stopped. Treat those as different questions.

Keep identifiers useful for investigation, but avoid putting email addresses, message bodies, or other unnecessary personal data into metric labels. A small, stable set of dimensions makes comparisons easier and avoids turning monitoring into another uncontrolled copy of business records.