Security is rarely the most exciting topic in software engineering, but it’s one of the most consequential. This post builds the security blueprint for our IoT case study — covering network security, secrets management, RBAC, and secure container configuration.

The core principle throughout: security should be a first-class architectural concern, not a retrofit.

Network Segmentation: Protect Your Databases

The first and most important network security decision: your databases should never be directly reachable from the internet.

In cloud infrastructure, this means:

  • Databases deployed in private subnets (no public IP, no internet gateway route)
  • Only the application layer (in the private subnet or VPC) can reach the database
  • VPC peering used to connect application VPC to data VPC when they’re separate

For Kubernetes specifically, this means pods that need to talk to a database get that access via a Service (or ExternalName/Endpoints), but the database itself is isolated at the network level.

Kubernetes Network Policies

By default, all pods in a Kubernetes cluster can talk to all other pods. This is not what you want in production.

NetworkPolicies let you define ingress and egress rules at the pod level:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: sensor-api-netpol
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: sensor-api
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: ingress-controller
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: kafka
    ports:
    - protocol: TCP
      port: 9092
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53  # DNS

Start with a deny-all base policy per namespace, then add explicit allow rules. This way, any new service that gets deployed is isolated by default until you explicitly open what it needs.

Note: NetworkPolicies require a CNI plugin that supports them (Calico, Cilium, Weave Net). The default Kubernetes networking doesn’t enforce them.

DDoS Protection

For a public-facing system, you need DDoS protection at multiple layers:

  1. Cloud-level DDoS protection — AWS Shield, GCP Cloud Armor, Azure DDoS Protection. These handle volumetric attacks at the network edge before traffic reaches your infrastructure.

  2. CDN-based protection — Cloudflare, Fastly, or your cloud CDN. The CDN absorbs traffic spikes and can block malicious traffic patterns.

  3. Ingress rate limiting — Nginx Ingress supports rate limiting annotations:

nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/limit-connections: "10"
  1. API Gateway throttling — for your web/mobile APIs, configure per-client rate limits at the API Gateway level.

Layer these protections. No single layer is sufficient.

WAF (Web Application Firewall)

For the frontend APIs, a WAF provides protection against:

  • SQL injection
  • XSS attacks
  • OWASP Top 10 vulnerabilities

AWS WAF, Cloudflare WAF, or ModSecurity integrated into your Ingress controller are common options. Configure rules appropriate for your API payload shapes.

Secrets Management

As mentioned in the infrastructure blueprint, Kubernetes Secrets are base64-encoded, not encrypted by default. In production, you need a proper secrets management approach.

Option 1: Encrypted etcd + KMS

Enable encryption at rest for Kubernetes etcd, backed by a cloud KMS:

# encryption config
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - kms:
      name: aws-kms
      endpoint: unix:///tmp/socketfile.sock
      cachesize: 1000
  - identity: {}

This encrypts secrets at rest in etcd using your KMS key. The secrets still live in Kubernetes but are protected.

Option 2: Hashicorp Vault

Vault is the most fully-featured secrets management solution. It provides:

  • Dynamic secrets (short-lived credentials generated on demand)
  • Secret leasing and renewal
  • Audit log of all secret access
  • Fine-grained access policies

The Vault Agent Sidecar Injector can inject secrets into pods without the application needing to talk to Vault directly:

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "sensor-api"
  vault.hashicorp.com/agent-inject-secret-config: "secret/data/sensor-api/config"

Vault is more operationally complex than KMS-encrypted Secrets, but the dynamic secrets capability (especially for database credentials) is worth it for security-sensitive environments.

RBAC: Principle of Least Privilege

Kubernetes RBAC

Every service account should have only the permissions it actually needs. No service account should have cluster-admin unless it genuinely needs it.

Define Role (namespace-scoped) or ClusterRole (cluster-scoped) with minimal permissions:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sensor-api-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "watch", "list"]
  resourceNames: ["sensor-api-config"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
  resourceNames: ["sensor-api-secrets"]

Bind it to a dedicated ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sensor-api-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: sensor-api-sa
  namespace: production
roleRef:
  kind: Role
  name: sensor-api-role
  apiGroup: rbac.authorization.k8s.io

Cloud IAM

The same principle applies to cloud IAM. Each service’s cloud IAM role should have only the permissions it needs:

  • The sensor API needs to publish to Kafka and write to S3 → those permissions only
  • The analytics job needs to read from S3 → read-only permissions only

Use cloud provider-specific mechanisms to bind Kubernetes ServiceAccounts to cloud IAM roles (AWS IRSA, GCP Workload Identity).

Secure Container Configuration

Pod security context controls how containers run at the OS level:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sensor-api
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
      containers:
      - name: sensor-api
        image: sensor-api:1.0.0
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

Key settings:

  • runAsNonRoot: true — container must not run as root
  • readOnlyRootFilesystem: true — container can’t write to the filesystem (forces explicit volume mounts for any needed writes)
  • allowPrivilegeEscalation: false — prevents privilege escalation attacks
  • capabilities: drop: ALL — drops all Linux capabilities (add back only what’s actually needed)

These settings won’t stop every attack, but they significantly limit the blast radius of a container compromise.

Static Code Scanning

Shift security left by scanning code in CI:

  • SAST (Static Application Security Testing) — find vulnerabilities in your code before it ships (Snyk, Semgrep, SonarQube)
  • Container image scanning — scan images for known CVEs (Trivy, Clair, AWS ECR scanning)
  • Kubernetes manifest scanning — validate manifests against security policies (kube-score, Checkov, OPA/Gatekeeper)

Add these as required quality gates in your CI pipeline. A build that introduces a critical CVE should fail.

Peer Review for Security Changes

Security-sensitive changes (RBAC changes, network policy changes, secrets handling) should require mandatory peer review. Ideally, with at least one reviewer who has security expertise.

No automated tool catches everything. Human review catches what tools miss.

Summary

Concern Solution
Database isolation Private subnets + VPC peering
Pod-level network isolation NetworkPolicies (deny-all default)
DDoS protection Cloud Shield + CDN + Ingress rate limiting
Web attacks WAF (AWS WAF / Cloudflare)
Secrets at rest Encrypted etcd + KMS, or Vault
Kubernetes permissions RBAC with least privilege + dedicated ServiceAccounts
Cloud permissions Cloud IAM with least privilege + Workload Identity
Container hardening securityContext (nonRoot, readOnly, drop capabilities)
Code vulnerabilities SAST + image scanning in CI