In the previous post we introduced the IoT case study. Now let’s build the infrastructure blueprint.

The goal here is to map each component of the system to the right Kubernetes (or cloud-native) construct, and to explain the reasoning behind each decision.

First Question: What Lives in Kubernetes?

Not everything needs to be in Kubernetes. A useful first pass is to categorise components:

Kubernetes-hosted:

  • Stateless API services
  • Stream processing consumers
  • Scheduled batch jobs
  • Internal tooling

Cloud-hosted (managed services):

  • Kafka (MSK on AWS, Confluent Cloud, etc.) — operational complexity is high; managed is usually right
  • Relational database (RDS, CloudSQL, etc.) — same reasoning
  • Object storage for data lake (S3, GCS)
  • CDN for frontend asset delivery

Keeping managed services outside Kubernetes simplifies operations significantly. You give up some control, but for infrastructure that isn’t your core competency, that’s a good trade.

Ingress: Getting Traffic In

For external traffic, we use the standard three-layer Kubernetes ingress pattern:

Internet → Cloud LoadBalancer → Kubernetes NodePort → Ingress Controller → Services
  1. Cloud LoadBalancer — provisioned by the cloud provider, gives us an external IP, handles TLS termination at the edge
  2. NodePort — exposes the Ingress Controller on a port on each node
  3. Ingress Controller (nginx or similar) — routes HTTP/S traffic to Services based on host/path rules
  4. ClusterIP Services — the internal service discovery layer

This is the standard pattern and works well for HTTP/S traffic. For the sensor ingest path, we may want a separate TCP/UDP path if sensors use non-HTTP protocols — handled via a separate LoadBalancer Service.

Scaling the Sensor API

The sensor ingest API will receive high and variable traffic. We use HorizontalPodAutoscaler (HPA) to scale the Deployment automatically:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sensor-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sensor-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Key points:

  • minReplicas: 3 — we never want fewer than 3 for availability (across zones)
  • CPU at 60% target gives headroom before performance degrades
  • For real systems, consider custom metrics (requests per second) rather than CPU

Kafka Consumers: StatefulSet vs Deployment

The Kafka consumer services (stream processors) require careful thought. Should they be Deployment or StatefulSet?

It depends on whether the consumers maintain local state:

Stateless consumers → Deployment If the consumer processes each message independently and writes output elsewhere (DB, another Kafka topic), use a Deployment. Pods are interchangeable, scaling is simple.

Stateful consumers → StatefulSet If the consumer maintains local state (e.g., Kafka Streams with a local RocksDB store), use a StatefulSet. Pods have stable identities and stable storage. Kafka partition assignment can be pinned to specific pods.

For most consumers in this system, a Deployment with external state is the right default. Reach for StatefulSet only when you genuinely need stable identity.

Batch Analytics: CronJob

The analytics batch job (periodic aggregations, reports) maps cleanly to a Kubernetes CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: analytics-aggregator
spec:
  schedule: "0 * * * *"  # hourly
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: aggregator
            image: analytics-aggregator:latest
          restartPolicy: OnFailure

CronJobs are simple and reliable for periodic work. Just be careful with:

  • concurrencyPolicy: Forbid if jobs must not overlap
  • Monitoring for missed/failed runs (Kubernetes doesn’t alert on this by default)

Configuration: ConfigMaps and Secrets

Non-sensitive configuration goes into ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sensor-api-config
data:
  kafka_bootstrap_servers: "kafka.internal:9092"
  kafka_topic: "sensor-events"
  max_batch_size: "100"

Sensitive values (credentials, API keys) go into Secret:

apiVersion: v1
kind: Secret
metadata:
  name: sensor-api-secrets
type: Opaque
data:
  db_password: <base64-encoded>
  api_key: <base64-encoded>

Important caveat on Secrets: Kubernetes Secrets are base64-encoded, not encrypted by default. They’re protected by RBAC, not encryption. For production, you need either:

  • Encrypted etcd (enable at the cluster level)
  • A KMS solution (AWS KMS, GCP KMS, Hashicorp Vault)

We’ll cover this in the security blueprint.

External Service Abstraction

When your pods need to reach external services (the managed Kafka, the RDS database), use Kubernetes Service with a custom Endpoints object. This lets you treat external services as if they were internal Kubernetes services:

apiVersion: v1
kind: Service
metadata:
  name: kafka-external
spec:
  ports:
  - port: 9092
---
apiVersion: v1
kind: Endpoints
metadata:
  name: kafka-external
subsets:
- addresses:
  - ip: 10.0.1.50  # MSK broker IP
  ports:
  - port: 9092

This decouples your application code from knowing it’s talking to an external service. You can swap the external service for an in-cluster one (e.g., during development) without changing application config.

Frontend APIs: API Gateway

The web and mobile frontend APIs get an additional layer: an API Gateway.

The API Gateway handles:

  • Authentication / JWT validation
  • Rate limiting per client
  • Request routing to downstream services
  • Response caching where appropriate

Options: AWS API Gateway, Kong, Ambassador/Emissary, or nginx with lua. The choice depends on your cloud provider and team familiarity.

CDN for Static Assets

Frontend static assets (JS bundles, CSS, images) are served via CDN. This is a no-brainer:

  • Reduces latency for end users
  • Reduces load on origin servers
  • Free DDoS mitigation (to a degree)

CDN configuration is outside Kubernetes. The CDN origin points at your Ingress or a separate object storage bucket.

Summary

Component Kubernetes Object
Sensor ingest API Deployment + HPA
Stateless Kafka consumers Deployment
Stateful Kafka consumers StatefulSet
Analytics batch job CronJob
Internal services Deployment + ClusterIP Service
External traffic entry LoadBalancer + Ingress
Configuration ConfigMap
Credentials Secret + KMS
External service refs Service + Endpoints
Frontend APIs Deployment + API Gateway

Next: The Kubernetes and cloud software security blueprint