Scott Tompkins

Using Observability to Validate Architecture Assumptions

3 min read

Using Observability to Validate Architecture Assumptions

When architecture decisions are made without feedback mechanisms, they become expensive to unwind. You make an assumption, build on it, and by the time you discover the assumption was wrong, there’s significant work invested in the wrong direction.

I’ve started attaching each significant architecture decision to a concrete question and a measurement plan. This turns “I think this will work” into “we’ll know this works when X metric shows Y.”

The Pattern

  1. Identify the assumption. Every architecture decision rests on assumptions. “This component can handle X load.” “Latency will be under Y milliseconds.” “Tenant A’s usage won’t affect Tenant B.”

  2. Define what violation looks like. If the assumption is wrong, what would I observe? What metric would spike? What behavior would I see?

  3. Instrument it. Build the observability that would surface the violation. This might be a metric, a dashboard, an alert, or just a query I can run.

  4. Revisit. Actually look at the data. If the assumption held, great—now there’s evidence. If it didn’t, catch it early while there’s still time to adjust.

Example: Shared Data Pipeline for Multiple Tenants

The decision: Adopt a shared data pipeline for multiple tenants instead of separate pipelines per tenant.

The assumption: Noisy neighbor effects will be manageable. One tenant’s spike won’t meaningfully degrade another tenant’s experience.

The risk: If wrong, we have cross-tenant interference that undermines the value proposition.

The validation plan:

  • Define expected latencies per tenant (p50, p95, p99)
  • Define error budgets per tenant
  • Instrument per-tenant latency and throughput
  • Create dashboards that show tenant performance side by side
  • Alert when one tenant’s spike correlates with another tenant’s degradation

What this gives me: If the assumption holds, I have data to show it. If it doesn’t hold, I catch it early and can adjust—maybe adding rate limiting, maybe restructuring the pipeline, maybe reverting to per-tenant pipelines for specific cases.

Why This Matters

Metrics and traces make tradeoffs visible and reduce time spent debating assumptions. Instead of arguing about whether something will work, you instrument it and find out.

This is especially valuable for the kind of decisions that are hard to reverse: infrastructure choices, data model decisions, isolation boundaries. These are exactly the decisions where “I think it’ll be fine” isn’t good enough.

The Meta-Point

Observability isn’t just for incident response. It’s a tool for validating that systems behave as designed. If I’m confident enough in a decision to build on it, I should be confident enough to measure whether it’s actually working.