Before diving into individual blueprints, we need a concrete system to reason about. Abstract discussions of architecture are useful up to a point — but the real complexity emerges when you apply principles to a specific problem.
This post introduces the IoT data ingestion platform that will serve as the case study throughout this series.
The System
The system receives data from a large number of distributed sensors and devices:
- Raspberry Pi arrays
- IoT sensor networks
- Mobile phones and tablets
This data flows through the system, gets processed and stored in two ways:
- A data lake for raw and transformed event data (analytics, ML)
- A relational database for structured, queryable data (operational)
On the output side, the system exposes APIs consumed by:
- Web frontends
- Mobile applications
It’s a fairly typical modern data-intensive system, but with enough moving parts to make the blueprints meaningful.
Why This System?
It represents a common class of problem:
- High ingest volume — thousands of sensors sending data continuously
- Mixed workloads — stream processing, batch processing, request-response APIs
- Multiple consumers — different clients with different latency and reliability requirements
- Greenfield — we’re designing it from scratch, which means we choose the constraints
Technology Decisions
Only one technology is strictly decided at this point: Apache Kafka as the streaming platform.
Kafka is the right choice here because:
- It decouples producers from consumers
- It provides durable, replayable message storage
- It handles high-throughput ingest gracefully
- It’s a proven choice for this class of problem
Everything else — databases, languages, frameworks, cloud provider — is left open deliberately. The blueprints should be applicable regardless of those choices.
Kubernetes is assumed as the infrastructure foundation throughout.
NFR Analysis Pass
Before building anything, it’s worth doing an explicit pass on non-functional requirements. For this system, key NFRs include:
Availability
- The sensor ingest path needs to be highly available — sensors can’t buffer indefinitely
- Web/mobile APIs need standard SLA (99.9%+)
Scalability
- Ingest must scale horizontally as sensor count grows
- Data storage must handle growing data volumes
Security
- Sensor data may be sensitive; transit encryption required
- Backend APIs must authenticate and authorise properly
- Infrastructure must follow least-privilege principles
Observability
- High-volume distributed systems are hard to debug without good telemetry
- Need metrics, logs, and traces across all components
Deployability
- Frequent releases without downtime
- Safe rollback capability
Upcoming Blueprints
Based on this system, here are the blueprints we’ll build out:
- Infrastructure blueprint — how to structure the Kubernetes workloads
- Security blueprint — Kubernetes and cloud security posture
- CI/CD blueprint — cloud-native software delivery pipeline
- Observability blueprint — metrics, logging, tracing
- Configuration management — dynamic config and circuit breakers
- DevOps tooling — the practices and tools
Each blueprint will reference back to this case study and show how the principles apply concretely.
Next: The infrastructure blueprint of the microservice project