To deliver high-quality service, organizations must carefully track data flows within applications. This typically requires establishing well-designed code metrics and implementing robust monitoring systems around those metrics.

Metrics represent “the fastest observability data. They have the quickest output, can be consumed directly… require the smallest throughput, and have the smallest retention footprint.”

In service-oriented architectures, understanding how to leverage metrics at distributed integration points enables early problem detection—a critical capability explored throughout this piece.

The System Architecture

The article examines a microservice-based system featuring:

  • A Frontend API microservice (functioning partially as an API gateway)
  • Integration Aggregator (A.I.) microservices
  • Subsystem microservices specialized for domain aggregates

Understanding the Data Flow

The discussion focuses on synchronous data flows involving multiple sequential service calls. The presented sequence diagram shows a typical backend flow where:

  • Frontend API initiates requests
  • Multiple services process data sequentially
  • Responses aggregate and return to users
  • Two subflows represent error responses and timeout scenarios

Observability Perspective

Applying observability principles to the flow architecture reveals measurement opportunities. The system should expose metrics for two primary purposes:

  1. Performance measurement
  2. Flow correctness verification

Two metric types serve these purposes:

  • Counters (displayed as squares/rhomboids): Track initiated calls, completed calls, and caught errors
  • Timers (displayed as arrow bands): Measure call duration

What to Measure

Effective flow observability requires exposing:

  • Counters for every initiated and completed call with descriptive labels
  • Counters for caught errors, distinguishing between 5xx and 4xx responses
  • Timers for successful calls (error response timing is typically less relevant)

Importantly, status codes should be tracked with granular labeling, allowing later aggregation while preserving detailed data capture.

Where to Measure

Measurements should focus on:

  • Incrementing counters and starting timers when calls initiate
  • Incrementing success counters and stopping timers upon response receipt
  • Incrementing error counters when catching erroneous responses

A critical insight: “measuring the performance of every service by measuring its responses on the side of the consumer service” provides intuitive, logical system performance understanding. This approach accounts for networking, security measures, and integration overhead—factors internal service measurements miss.

Why Measure: Quality of Service

The overarching motivation centers on Quality of Service (QoS). Providing reliable service demands understanding system performance, which requires observing exposed data.

Performance Monitoring

API endpoint performance represents the most critical metric—the total duration of the complete flow. Plotting performance over time reveals degradation patterns.

Effective dashboards combine:

  • Total flow performance graphs
  • Individual segment performance graphs
  • Request volume counters

When performance degrades, these visualizations identify whether particular segments disproportionately impact response times.

Alerting approaches include:

  1. Performance budget-based alerts (e.g., responses exceeding 300ms)
  2. Historical comparison alerts (e.g., 20% slower than yesterday’s baseline)

“The same way service#1 that is calling service#2 needs to qualify the performance of service#2, your users are qualifying your application’s performance, and you are there to make sure that it is the best performance you can achieve.”

Error Rate Monitoring

All error response counters contribute to error rate tracking, supporting both quality assurance and technical flaw identification.

Error characterization distinguishes between status code ranges:

  • 4xx vs 5xx separation applies when subsystems exhibit distinct 400-range behaviors (authentication, authorization)
  • Combined tracking suits systems without special 4xx handling
  • 404 exclusion often makes sense for public-facing systems where page misses are common

The article illustrates that error impacts vary contextually:

  • Private applications with authentication: 4xx errors signal security concerns
  • Integration systems: 404s indicate upstream changes requiring immediate attention
  • Public search functionality: 404s are expected from crawlers
  • 5xx errors are always problematic

Error rate alerting should be stringent, typically based on SLOs triggering when endpoints breach service-level objectives for extended periods.

“Make sure not to waste time on staring at monitoring dashboards - set up alerts and go do something fun” — emphasizing automation over manual observation.

Summary

Key takeaways include:

  • Code instrumentation constitutes a core software development responsibility
  • System observability requires architectural consideration matching the attention devoted to system design itself
  • Consumer-side performance measurement provides the proper perspective, accounting for integration and networking factors
  • Error management remains paramount