rastko

The Death of the SDKs

Thu, 16 Jul 2026 00:00:00 +0000

For the most of the systems that ship an SDK , problem they are solving with this is shifting. We are talking dimensional shift - from solving scale problem to solving niche problem.

An SDK is knowledge about a system expressed as language-specific code — the API surface, the auth model, how to handle pagination, how to parse error responses, what the data model means. The Python library is just how you’ve chosen to ship it.

And if you ship it as a Python library, you also need a Java library. And a Node library. And something for Go. Every language your consumers use becomes another SDK to maintain, another set of docs to keep in sync, another version to cut when you change something.

What if you shipped the knowledge directly?

An integration skill. A centrally maintained, language-agnostic description of how to integrate with your system — the endpoints, the auth flow, the pagination contract, the error handling, the data model, the edge cases that bite people. A consuming team invokes it in whatever language they’re working in.

Someone needs to integrate in PHP. They invoke the skill. They get a PHP integration. Someone needs Erlang. Same skill, different output.

The SDK is not the destination — it is what you ship when you cannot yet ship the skill.

An SDK is a snapshot. You cut a version, consumers pin to it, drift happens. A skill is living knowledge — update it centrally and every consumer gets the current version. You maintain one thing, not N.

Having the upgrade command in the skill is also easy from this point on.

Once the skill exists, you can just go deeper into the agentic rabbit hole:

A skill tells you how to integrate.
An integration agent actually does it — writes the code, wires it into your service, handles the configuration.
Then an integration testing agent — verifies correctness against the same knowledge that generated the integration.

Skill → agent → testing agent.

None of this is a headstone for SDKs.

If you’re working with something sensitive — financial data, healthcare, anything where you are not comfortable routing integration through an externally hosted model — an SDK is still the right call. The same applies if you have a very specific technology stack and want every integration done identically, with no variance in how the “AI” interprets the knowledge. In those cases the SDK’s predictability is the point.

What is declining is the range of situations where an SDK is the right answer. That range is narrowing with every improvement in “AI” tooling, faster than most SDK maintainers are accounting for.

The people integrating with your system are already doing something close to this: feeding your docs and your OpenAPI spec to an “AI” and generating the integration. You’re just not in control of that process, and the quality reflects it.

You define the knowledge. You maintain it. You decide what the consuming experience looks like, regardless of what language is on the other end.

Mental Models as Medium

Sun, 28 Jun 2026 00:00:00 +0000

There’s a discipline I’ve practised for a long time that I don’t hear talked about enough: writing down how I think, not what I know.

I am talking about the actual structure of how I approach a problem — what I look for first, what questions I ask before anyone else has thought to ask them, what lenses I apply and in what order, what the shape of a good answer looks like before I know what the answer is.

It requires you to catch yourself thinking and then describe the thinking, sort of metacognition, or epistemology if you want academical term.

Most people, when asked to write down what they know about architecture, produce a list of facts, patterns, principles. That’s knowledge. What I was trying to write down was something else: the engine that produces those facts, patterns, and principles in context, on demand, in response to problems I haven’t seen yet.

The discipline of making your thinking explicit — of writing it down carefully enough that someone else could follow it — is the discipline of sharpening the thinking itself. You don’t know what you actually think until you’ve tried to write it. And once you’ve written it, you start to see the gaps, the inconsistencies, the places where your intuition outran your ability to explain it.

I’ve been doing this for years. Rather then accumulating a memory of events and decisions, I tend ot accumulate structues, mental models — a description of how an experienced architect actually sees: domains, trade-offs, stakeholder dynamics, risk surfaces, the gap between what a system does and what it should do, the difference between what’s technically possible and what’s strategically right.

The Mental model

Structured knowledge is important. This should be obvious, but it isn’t always acted on. We very often settle for having knowledge - “documentation exists, and stuff from the checklist is there” at the smaller scale and documentation attached to every service you own at a larger scale.

But structured knowledge, no matter how well organized, has a gap. The gap between the knowledge existing and applying it.

A structured mental model — one you’ve articulated carefully, refined against real problems, and organized into something navigatable — is increasing usefulless of your knowledge significantly (would say order of magnitude, but it is not liek there is actual way to measure :)).

You carry the model. You apply it to each new problem yourself. The model doesn’t meet the problem — you do, using the model. For a broader usage it becomes a bottleneck when you want someone else to benefit from it: a junior architect, a team you’re advising, a domain you’re not embedded in. You can share the documentation. You can’t share the mental model. Not in very efficient way at least… Write a blogpost? Write a book? Give a talk?

The gap is the intuition. The part of the mental model that operates faster than explanation, that pattern-matches before you’ve consciously engaged, that asks the right question before the meeting has landed on what question to ask.

You can describe intuition. Describing it doesn’t transfer it.

Every format we’ve had for knowledge transfer — books, talks, blog posts, courses, documentation — runs into this wall. They can describe the model. They cannot enact it. The reader receives the description and then does the work of turning it into practice, alone, without scaffolding, against a learning curve that most people don’t have time or support to climb.

That gap has been there for as long as people have been trying to transfer expertise. For most of history, it was simply the cost of learning — you apprenticed, you paired with someone more experienced, you made mistakes with them watching. The model transferred slowly, through exposure, through correction.

There was no other way.

Enter The “AI”

What “AI” changes is not that it makes you smarter, or faster, or able to do things you couldn’t do before. What it changes — the thing that matters most — is this:

For the first time, a written mental model can be made executable. - To a level at least.

This is, of course, LLMs, the “AI” (with large quotation marks), the Large Language Models are aweome at pattern matching and dealing with written prose - this is no brainer. Teach LLM to structure it’s pattern matching areasoning in certain way and you can have it demonstrate the analysis approach and understanding principles you are applying to the very usable level.Especially if you are not doing “rocket science”, which minimum of 99% of us dont.

You write down how you think. You structure it carefully — the lenses, the analysis frameworks, the sequence of questions, the principles that govern trade-offs. You make it explicit enough that it can be read and internalized. And then an “AI” agent internalizes it and applies it — to your codebase, your system, your problem, in your context — in real time.

Important - for my fellow SciFi freaks out there, this is why I mentioned specific cases (this time code, architecture etc) - no, you cant really generally teach it how to think like you, but for some niche things and one abstract layer you carefully choose…

The model doesn’t sit on the shelf. It works. It meets the problem you bring it. It asks the questions you would ask. It surfaces the considerations your experience would surface (under your control). Not because the “AI” is replacing your judgment — but because the written model is now an active participant in the thinking, not a static reference you consult.

I am describing potential way to try and cover the gap between structured knowledge and applied expertise — the gap that every knowledge transfer format has always had to leave open may be narrowed using Large Language Models and well crafted harnesses (from industries mega products like Codex, Claude Code to your local machine friendly qwens and gemmas and harness you wrote in python)

This is what I have tried to build one winter week in 2025.

I finally found use for the things I tried to wrte before — an architecture book I never finished, a solution architect’s guide that stalled, the article about analysis patterns — weren’t wasted. Turns out they were drafts of the mental model itself.

These were attempts to make the thinking explicit before the capability to make it executable existed. The blueprint concept, the two scales, the cognitive load argument, the themes that kept recurring in everything I wrote — they’re all now in quantum-toolbox.

quantum-toolbox

quantum-toolbox is a markdown-based executable mental model — no runtime dependencies, no plugins, no API calls — that ships as a git submodule into any project. “AI” “agents” read it at session start and gain the ability to try and do some architecture work.

quantum-toolbox on GitHub

The core is what I call the mental model as engine. Rather than building dozens of separate skills that each explain architecture from scratch, there’s one foundational layer that encodes the mental model itself — domains, stakeholder analysis, gap analysis, principles, the way an experienced architect actually structures a problem. Every skill inherits from this. The mental model is the engine - the archtecture thinking core; the skills are the gears.

It covers the full range:

Analysis — codebase, architecture, security (OWASP, NIST, NIS2, ISO 27001), nonfunctional requirements, fitness functions
Architecture — the complete TOGAF ADM cycle, C4 modeling
Development — software design, tech stack decisions, code conventions
Output — architecture docs, product specs, ArchiMate, presentations, compliance reports

27 skills across 4 categories. Pure markdown. Works with any “AI” agent that reads text.

The research tool, the analysis tool, the creation tool - well, mostly analysis tool, but there is potential (I do use it like this). They’re the same mental model operating at different phases of the work. You research with it, you analyse with it, you create with it. The consistency comes from the engine underneath.

What’s Next

This article is the “why” — the argument for why executable mental models matter and what they change. The interesting stuff is in the how.

Coming up in this series:

How the mental model engine actually works — the architecture-thinking layer that makes everything else possible
Practical TOGAF through “AI” — what happens when you give an agent the full ADM cycle
Security analysis as a thinking framework — not a checklist, a lens
How to customize the mental model for your organization
The determinism of the result and token economics of teaching an “AI” HOW to “think”

Write things down. Structure the thinking, not just the conclusions. Make it explicit enough that it could be followed.

That’s been the practice — quantum-toolbox is where it landed.

Optimizing the Impact of Architects

Sat, 27 Jun 2026 00:00:00 +0000

There’s an XKCD comic that gets passed around in engineering circles.

xkcd.com/2347 — CC BY-NC 2.5

In March 2016, a developer named Azer Koçulu — working out of Oakland, California — unpublished 250 npm packages after a dispute with npm Inc. over a package name. One of those packages was left-pad: eleven lines of JavaScript that padded strings on the left. Within hours, React wouldn’t build. Babel wouldn’t build. Thousands of projects that had never heard of Azer Koçulu, that had never made a conscious decision to depend on him, were broken.

Nobody designed this. Nobody governed it. The dependency crept in, package by package, until a Turkish developer in California with a legitimate grievance had more impact on the global software supply chain than any single engineering organization.

That’s the open-source internet, and it operates under its own rules.

But the same thing happens inside large enterprises very often. Just slower, less dramatic, and without the tweets.

How enterprises are organized

Medium and large enterprises divide by business domain. Production. Retail. Digital. B2B. HR. Each domain has its own budget, its own leadership, its own roadmap, its own engineers.

This is correct. I am not saying politically — structurally correct. Conway’s Law is not a suggestion: the systems you build reflect the communication structure of the organization building them. Domain-oriented organizations produce domain-coherent systems. That’s what you want.

Domain architecture operates across four layers within each domain:

Technology enables applications, applications produce and consume data, data enables business capability. This is well understood, which is a problem since the edges are not smooth.

The no man’s land

When you divide into domains, you create boundaries. And boundaries, left unmanaged, become no man’s land.

No man’s land is what happens when something important doesn’t clearly belong to any single domain — and so nobody owns it.

In practice it looks like this:

Too many business domains, and some units fall outside any of them — nobody claimed them, nobody maintains them, and they persist in a state of benign neglect until something breaks.

A unit gets assigned to the closest domain even though the fit is wrong. It gets the attention of a domain that doesn’t understand it and doesn’t have the right incentives to invest in it.

Analytical data that crosses domain boundaries has no clear owner. Multiple domains use it; none of them maintains it. Everyone assumes someone else is responsible.

Tooling that enables a transformation programme lives in one domain, but the programme spans three. When the original sponsor moves on, the tool becomes an orphan — technically alive, practically unsupported.

Systems get upgraded and modernised within their domains; the integration interfaces between them don’t. No single domain has the mandate or the budget to own the crossing. The interfaces hang in maintenance mode, a thread holding up a wall.

There’s a version of this I’ve used in presentations: imagine Mufasa showing Simba the kingdom from Pride Rock, the savanna stretching out in every direction. Simba points to a dark patch on the horizon. “Dad, what is that area?” Mufasa doesn’t look. “Son, we don’t go there. That part was created by Steve from accounting.”

Everyone laughs. Everyone in the room has a Steve from accounting.

The grey zone problem is organizational, not technical. The thing in the dark patch isn’t broken — it just has no owner, no mandate, no one who sees it as their job to understand it. Domain teams can’t fix this. Each domain is optimizing for its own area by design. Nobody in Production has the incentive, the mandate, or the full picture to govern something that touches Production, Retail, and Finance simultaneously.

Steve sometimes does, and generally people do not mind, because they dont know what steve does, they jsut knwo thsi is not presenting the problem.

At least one function needs to transcend the boundaries

This is where architecture comes in — specifically because it is a multidisciplinary function. The BDAT stack is a map of everything a domain team manages internally and everything that falls apart at the seams between them. An architecture function that operates across all four layers — business, data, application, technology — and across all domains is structurally positioned to see what no single domain team can see.

That’s the job - ensure nothing stays in the grey zone urecognized, and unmanaged. To say: this crosses boundaries, which means it belongs to architecture until it belongs somewhere else. To make the invisible visible, assign ownership, and then get out of the way.

Domain teams will optimize for their domains. That’s what they’re designed to do, and it’s right that they do it. You can’t ask a Production domain team to take ownership of the integration layer between Production and Retail — that’s not their scope, not their mandate, and not their incentive.

Someone else has to hold the picture that no single domain can hold. That function is enterprise architecture.

The challenges of governance

This is where most architecture functions fail.

Standards that are agreed but not enforced don’t hold. Decision-making authority that isn’t clear doesn’t hold. Architecture reviews that can be bypassed at will don’t hold.

The challenge: the things you’re governing are moving. Teams ship. Domains grow. New capabilities appear. The governance model has to be firm enough to provide consistency and fast enough not to become a bottleneck.

Most organizations fail in one of two directions. Governance is so light it doesn’t actually constrain anything — you have principles, not decisions, and everyone nods at them and does what they were going to do anyway. Or it’s so heavy that teams route around it — the process costs more than violating it costs, so violation becomes standard practice.

Governance needs to be affordable and firm. Can you achieve that?

The test is simple: can someone get a clear answer to a significant technical proposal from your architecture function before the window for that decision has closed? If the answer is usually no — either because the function doesn’t engage or because it engages too slowly to matter — governance is broken. The direction of the failure tells you how to fix it.

Three questions

Any significant technical initiative eventually needs three questions answered:

You can’t design how until you know whether. You can’t commit to when until you know how.

The architecture function’s job is to own the first question and structure the path to the second.

The first phase — business modeling and assessment — establishes the what and why, then produces a technical verdict: feasible or not, on what terms, with what constraints. This is where architecture earns its place at the business table. Not by showing up to validate decisions that have already been made, but by engaging early enough to shape them.

The second phase — the architecture creation — works from data architecture through application architecture to technology architecture, in that order:

The “when” falls out of planning once the architecture decisions exist — not before, and not by someone guessing.

What this means for how you set up the function

A few things that distinguish architecture functions that have real impact from:

The no man’s land inventory. Someone in the architecture function should be able to answer, at any given time: what exists in this organization that isn’t clearly owned by any domain? If nobody has that list, nobody is governing it.

Upstream, not downstream. Architecture that arrives at the end of a project to validate decisions is audit, not governance. The function needs to be in the room when the “can it be done” question is first asked — not when the answer has already been baked into a budget commitment.

Cross-boundary ownership is explicit. Every integration interface, every shared data asset, every cross-domain capability needs a named owner with a named mandate.

The three-question model is a forcing function. When a business stakeholder asks “when will this be done?” before anyone has established “can it be done?”, the architecture function needs to redirect, not answer the wrong question faster. Part of the job is training the organization to sequence correctly.

Left-pad broke the internet in an afternoon. That was a governance failure at ecosystem scale — nobody owned the dependency graph, so nobody could protect it.

Enterprise architecture exists to solve the same problem at organizational scale. Not to prevent domains from making their own decisions — that would break the thing that makes domain teams work.

Job is to ensure that what lives between the domains, at the crossings, in the integration layer, in the shared capabilities that everyone uses and nobody maintains — that those things have owners, have governance, and don’t become your organization’s left-pad moment.

The function is not glamorous. The decisions it makes are often invisible even if they prevent a problem that nobody ever knew was coming. That invisibility is the point. The architecture function is working when nothing breaks unexpectedly at the boundaries.

The Role of the Architect

Tue, 18 Oct 2022 00:00:00 +0000

The word “architect” is overloaded to the point of near-meaninglessness.

In ancient Rome, Vitruvius defined architecture through three principles: firmitas (strength), utilitas (usefulness), and venustas (beauty). A building must be structurally sound, serve its purpose, and be aesthetically pleasing. All three, not any one.

Software architects, in theory, apply the same thinking: systems must be technically sound, serve business needs, and be elegant (or at least maintainable). The role requires deep technical knowledge combined with broad contextual awareness — business, organisational, operational.

In practice, the role has been stretched into shapes that bear little resemblance to this.

Part 1: Where the Term Comes From

The “software architect” title emerged in the 1990s as software systems grew complex enough to require dedicated people thinking about structure, interfaces, and technical direction beyond what individual teams could do.

The classic architect role:

Sets technical direction and architectural standards
Makes key design decisions with long-term implications
Identifies and manages technical risk
Connects technical decisions to business outcomes
Enables other teams to build well

This requires a specific and rare combination: deep technical ability, systems thinking, communication skills, and enough organisational awareness to understand what tradeoffs actually matter.

The problem is that this rare combination is being applied to many roles that don’t actually fit it.

Part 2: Three Case Studies in Architect Role Misuse

Case Study 1: “Partnering with Customers”

At several large technology vendors (cloud providers, platform companies), the “Solutions Architect” role carries a quota.

The job is not to design solutions. The job is to convince potential customers to buy the platform. The architect does technical demos, answers RFPs, writes proof-of-concept code, and serves as a trusted technical voice in the sales process.

This is a sales engineering role. It requires different skills, different incentives, and a different performance model than an architecture role. The person in this role is optimised for closing deals, not for sound technical decision-making.

Calling it “Solutions Architect” serves the vendor because it sounds technical and trustworthy. But the customer should understand what incentives are driving the advice they receive.

This isn’t a criticism of the people in these roles — it’s a valuable function. But it’s not architecture. The mislabelling creates confusion.

Case Study 2: Product Owner / Business Analyst Creep

In many product organisations, the most technically literate Product Owners and Business Analysts get promoted or rebranded into “Solution Architect” roles.

The pattern goes like this: a PO who can read API docs, attend technical refinement sessions, and write semi-coherent Jira acceptance criteria gets recognised as “technical”. At some point, they’re re-titled.

The resulting role is essentially a technical PO or senior BA — someone who bridges business requirements and technical implementation. This is genuinely valuable, but it is not architecture.

Architects in this mould often lack:

Hands-on experience designing systems under real constraints
Deep understanding of failure modes and non-functional concerns
The authority to say “no, this design is wrong” when the product owner wants something
Any experience operating what they design

The resulting “architecture” is often product requirements dressed up in technical language — a Confluence diagram that says “the system will do X” without any serious engagement with how.

Case Study 3: Architect Per Feature Team

In a scaling engineering organisation, there’s pressure to embed architects everywhere. Every feature team gets an architect. Every domain gets a domain architect.

The problem: the work available to an architect embedded in a feature team is, by definition, feature-team-level work. The team’s architectural concerns are scoped to their service, their API, their database schema.

This work may be important, but it’s not architecture in the sense of setting direction, managing cross-cutting concerns, or making decisions with long-term systemic implications. It’s detailed technical design — which engineers on the team can and should do.

Architects embedded in feature teams tend to drift into:

Being a senior engineer who does more design and less coding
Being a bottleneck for technical decisions that the team should own
Losing visibility of the broader architectural concerns that only they can see

Meanwhile, the org has consumed its architecture capacity on feature-level concerns and has no bandwidth for real architectural work.

Part 3: Why This Happens

These patterns aren’t random. They emerge from specific pressures:

Hypergrowth scaling: When a company doubles headcount in 18 months, processes and role definitions can’t keep up. Titles get applied loosely. People get promoted faster than their scope can grow. Architecture is a function that scales poorly — you can’t just double the architect headcount and get double the architectural output.

Title inflation: “Engineer” sounds less impressive than “Architect” to clients, to candidates, to organisational charts. There’s pressure to upgrade titles even when the scope doesn’t change.

Filling gaps: When a team lacks a strong technical senior engineer, attaching an “architect” to the team feels like a fix. Often it is — but it’s treating a symptom (weak team capability) rather than the cause.

What Actually Scales

Architecture capacity doesn’t scale linearly with headcount. You probably need fewer architects than you think.

What you do need:

Architects with genuine scope — cross-cutting concerns, platform decisions, long-horizon thinking
Engineers who can do local technical design well (don’t call them architects, train them as engineers)
Clear interfaces between architectural decisions (set by architects) and implementation decisions (owned by teams)
An architecture function that’s connected to both technical reality and business strategy

The firms that do this well tend to have senior architects operating across multiple teams, setting direction, and deliberately transferring knowledge and decision-making authority to engineering teams as they mature. The architect’s job is to work themselves out of the loop on decisions that teams can handle — and to focus on the decisions that genuinely require their scope.

That’s a harder job than being embedded in a feature team. It’s less visible on a short timescale. But it’s the job.

Written from experience as an Engineering and Architecture leadership consultant. The patterns described are composites of situations I’ve observed across multiple large organisations; no single organisation is described here.

Dynamic configuration management and circuit breakers in Kubernetes

Tue, 27 Apr 2021 00:00:00 +0000

Feature flags and circuit breakers sound straightforward. In a single-server application, they are. In a distributed system running across multiple Kubernetes clusters in multiple regions, the problem of how do you get a configuration change to all pods simultaneously becomes non-trivial.

This post presents an architecture for dynamic configuration management and circuit breakers in Kubernetes that has run in production on business-critical, high-scale systems.

The Problem

You have microservices deployed across multiple Kubernetes clusters — say, eu-west, us-east, ap-southeast. Each cluster has multiple pods of each service.

You need to:

Flip a feature flag for all pods in all clusters simultaneously
Trip a circuit breaker when a downstream integration starts failing
Restore the circuit breaker when the integration recovers

You can’t just update a database record and have all pods pick it up — there’s no shared state between clusters. You can’t use Kubernetes ConfigMaps as the source of truth — they’re per-cluster, and there’s no built-in sync mechanism.

You need a proper distributed configuration architecture.

The Architecture

The solution has three layers:

Layer 1: Central Config API + Storage

A single, globally accessible Config API (REST) backed by persistent storage (PostgreSQL or similar). This is the single source of truth for all configuration.

Key operations:

GET /config/{service}/{environment} — read current config
PUT /config/{service}/{environment} — update config
GET /config/history/{service}/{environment} — audit log

The Config API is your control plane. Operators interact with it (directly or via a UI) to change flags and configuration.

Layer 2: Consul KV Store Per Cluster

Consul provides a distributed key-value store with watch/notify capabilities. Deploy a Consul cluster per Kubernetes namespace (or per cluster).

Consul is the local config cache for each cluster. Pods watch Consul for changes rather than polling the central API. This gives:

Low latency config propagation (sub-second when a key changes)
Resilience — if the central API is unavailable, pods continue with last-known config
No direct coupling between pods and the central API

Layer 3: Config Service (Reconciler)

A Config Service (a microservice or CronJob) runs in each cluster and reconciles config from the central storage into the local Consul:

Central Config API → Config Service → Consul KV Store → Pods

The Config Service:

Polls the central Config API on a regular interval (e.g., every 30 seconds)
Compares the remote config with what’s currently in Consul
Pushes changes to Consul

For time-sensitive changes (circuit breaker trips), the interval can be reduced or the Config API can push to the Config Service via webhook.

How Pods Consume Config

Pods watch the Consul KV Store for changes to their config key. When the key changes, they update their in-memory config without restarting.

Most languages have Consul client libraries that support this pattern:

import consul

c = consul.Consul(host='consul.production.svc.cluster.local')

# Watch for changes
index, data = c.kv.get('config/sensor-api/production', wait='5m', index=0)
config = json.loads(data['Value'])

When Consul notifies the pod of a change, the pod reloads config immediately.

Circuit Breaker Integration

Here’s where it gets interesting. The circuit breaker is driven by Prometheus alerts rather than in-service logic.

The flow:

Prometheus detects that a downstream integration is degraded (error rate > threshold, latency > threshold)
Prometheus fires an alert to Alertmanager
Alertmanager routes the alert to a webhook endpoint on the Config API
The Config API automatically flips the circuit breaker flag for the affected service
The Config Service propagates the change to all clusters’ Consul stores
All pods pick up the change within seconds

# Prometheus alert rule
groups:
- name: circuit-breakers
  rules:
  - alert: PaymentServiceDegraded
    expr: |
      rate(http_requests_total{service="payment-service",status=~"5.."}[5m])
      / rate(http_requests_total{service="payment-service"}[5m]) > 0.1
    for: 2m
    annotations:
      circuit_breaker_target: "payment-service-integration"
      circuit_breaker_action: "open"

The alertmanager webhook configuration:

receivers:
- name: circuit-breaker-webhook
  webhook_configs:
  - url: 'https://config-api.internal/circuit-breaker/trigger'
    send_resolved: true

When the alert resolves (the integration recovers), Alertmanager fires the send_resolved webhook, which closes the circuit breaker automatically.

This is a metrics-driven, automated circuit breaker — the system heals itself without human intervention.

Resilience Design

A few important resilience decisions in this architecture:

Full Config Objects, Not Patches

When the Config Service pushes to Consul, it always pushes the full config object for a service, not a partial update or patch.

Why: partial updates can leave a service in an inconsistent state if there’s a failure mid-update. Full object pushes are idempotent — if the same object is pushed twice, the result is the same.

Sensible Defaults

Every service must define sensible defaults for all config values. If Consul is unavailable at startup, the service starts with defaults. If a config value is missing from Consul, the service uses the default.

Never let a missing config cause a startup failure.

Defaults as Deployed Config

The Consul config for a service in a fresh cluster starts as the deployed defaults (the same values the service uses if Consul is unavailable). This ensures a new cluster is immediately operational without a config sync cycle.

No Partial Config Updates

The Config API validates the full config object before accepting a write. You can’t write half a config. The schema is versioned, and the API rejects configs that don’t match the current schema.

When to Use This Architecture

This is a relatively complex system. It’s appropriate when you have:

Multiple Kubernetes clusters (multi-region or multi-environment)
Services that need sub-second config propagation
Circuit breakers that need to trip automatically based on metrics
Strict audit requirements for config changes

For single-cluster deployments or simpler requirements, Kubernetes ConfigMaps with a restart-or-reload mechanism may be sufficient.

Summary

Operator / Prometheus Alert
          ↓
    Config API (REST) + Storage
          ↓ (via Config Service / reconciler)
    Consul KV Store (per cluster)
          ↓ (watch/notify)
    Pod in-memory config

The key properties:

Single source of truth (central Config API)
Low-latency propagation (Consul watches)
Resilience (Consul as local cache; sensible defaults)
Automated circuit breaking (Prometheus → Alertmanager → Config API → Consul)
Auditability (all changes logged in Config API)

The 7 (and a half) tools to assist you on your DevOps journey

Sat, 10 Apr 2021 00:00:00 +0000

This post is a catalog. Not a deep dive into any single tool, but a map of the tools and concepts that meaningfully increase DevOps maturity. Think of it as a checklist you can work through incrementally.

Seven tools, one half-tool. Let’s go.

1. ChatOps

What it is: Bringing CI/CD control into your team’s chat platform (Slack, Teams, etc.).

Instead of logging into dashboards, you interact with your deployment pipeline via chat commands:

/deploy sensor-api v1.4.2 to production
> Deploying sensor-api v1.4.2 to production...
> ✅ Deployment complete. sensor-api v1.4.2 running in production.

Why it matters:

Full team visibility into deployments without context-switching to dashboards
Chat serves as audit log (who deployed what, when)
Low-friction operations reduce deployment anxiety and encourage more frequent deploys
Incident response is faster when everyone can see what’s happening

How to implement: Hubot (GitHub), Lita, or custom Slack bots integrated with your CI/CD platform (Jenkins, CircleCI, GitHub Actions, etc.).

2. Feature Flags

What it is: Conditional code paths controlled by runtime configuration, not deployments.

if feature_flag("new-sensor-processing-algo", user_context):
    return new_algorithm(data)
else:
    return legacy_algorithm(data)

Why it matters: Feature flags decouple code deployment from feature release. You can:

Deploy code to production before turning the feature on
Release to a subset of users first (percentage rollout, specific regions, specific users)
Turn off a feature instantly if it causes problems — no rollback needed
Run A/B tests without redeploying

Dimensions to flag on:

Environment (dev / staging / production)
User cohort (beta users, internal users, paying users)
Region or data center
Percentage of traffic

Tools: LaunchDarkly, Unleash (open source), Flagsmith, or a simple database-backed config service.

Pitfall: Feature flags accumulate technical debt. Have a process for cleaning up flags after features are fully released.

3. Circuit Breaker

What it is: An automatic switch that disconnects a failing integration to prevent cascading failures.

Inspired by electrical circuit breakers. Three states:

Closed — normal operation, requests flow through
Open — integration is failing, requests are blocked immediately (fail fast, no waiting for timeout)
Half-open — probe state, allow some requests through to check if the integration recovered

Why it matters: In distributed systems, slow or failing dependencies can exhaust your thread pools and connection pools, causing your service to fail even for unrelated requests. Circuit breakers contain the blast radius.

Implementation: The circuit breaker watches metrics (error rate, latency) and trips when thresholds are exceeded. In our Kubernetes system, we can drive this via Prometheus alerts → webhook → config service (covered in the next post).

Libraries: Resilience4j (Java), Polly (.NET), Hystrix (Netflix, now maintenance mode).

4. Canary and Shadow-Prod Releases

Canary Releases

What it is: Route a small percentage of production traffic to a new version before rolling it out fully.

With Kubernetes and nginx ingress:

annotations:
  nginx.ingress.kubernetes.io/canary: "true"
  nginx.ingress.kubernetes.io/canary-weight: "10"

This routes 10% of traffic to the canary version, 90% to stable. Monitor the canary for errors, latency, and business metrics. If it looks good, increase the weight gradually. If it looks bad, remove the canary with zero user impact.

Shadow Production Testing

What it is: Mirror production traffic to a new version without affecting the response the user receives.

User request → stable version → user response
             ↓ (mirrored)
           new version → response discarded

The new version processes the same real traffic but its responses are thrown away. You can compare responses to check for regressions.

This is powerful for validating algorithm changes or significant refactors before they see production traffic.

5. Rollback

What it is: The ability to revert to the previous version of a service instantly.

In Kubernetes, a rollback is as simple as:

kubectl rollout undo deployment/sensor-api

Or more precisely, deploy the previous known-good image:

kubectl set image deployment/sensor-api sensor-api=sensor-api:git-sha-of-previous-version

Why it matters: Rollback is your last line of defence. When everything else fails — canary didn’t catch it, feature flag can’t fix it — you need to be able to revert to the previous version in under 2 minutes.

Make sure rollback is:

Practiced regularly (add it to your runbooks and fire drills)
Automated where possible (GitOps systems like ArgoCD make this trivial)
Not blocked by database migrations (design migrations to be backward-compatible)

6. Seeds (Project Bootstrap Templates)

What it is: Opinionated project templates that come pre-wired with all the standard tooling.

Starting a new microservice from scratch takes days of setup: CI/CD configuration, Dockerfile, Kubernetes manifests, observability libraries, code structure, testing setup. Seeds compress this to minutes.

Three types of seeds:

Microservice seed: A template repository for each language/framework you use. Contains:

Dockerfile and .dockerignore
Kubernetes manifests (Deployment, Service, HPA, NetworkPolicy)
CI/CD pipeline configuration
Health check endpoint
Observability hooks (metrics, structured logging, tracing)
Testing scaffolding

CICD seed: A template for the CI/CD pipeline itself. Standardises pipeline stages across all services.

Cloud bootstrap seed: Terraform modules for spinning up standard infrastructure components (VPC, EKS cluster, RDS, S3, etc.) with sensible defaults and security baseline.

New projects instantiate from these seeds. The result: every service starts life with good practices already in place, rather than accumulating them (or not) over time.

6.5. Shared Libraries (the half tool)

What it is: Internal libraries that encapsulate cross-cutting concerns — HTTP clients, logging, instrumentation, feature flag clients, etc.

This is the half-tool because it’s not a standalone thing — it’s the companion to seeds. Seeds include these libraries by default.

What to put in shared libraries:

HTTP client with retry logic, circuit breaker, tracing headers
Structured logging (JSON, with standard fields)
Metrics instrumentation (counter, histogram, gauge)
Feature flag client
Health check endpoint implementation

Why it matters: Without shared libraries, each team implements these things differently (or not at all). With shared libraries, the standard becomes the path of least resistance.

Pitfall: Shared libraries can become a bottleneck. Keep them small and well-maintained. Versioning is critical — don’t break all services by pushing a breaking change to a library they all depend on.

7. DevOps Checklist

What it is: A crystallised list of the essential DevOps capabilities — a tool for assessing maturity and driving improvement.

The adidas engineering organisation open-sourced their DevOps maturity framework. It covers:

Source control — branching strategy, commit hygiene
CI/CD — pipeline automation, quality gates, deployment frequency
Testing — coverage, test pyramid, integration testing
Observability — metrics, logging, alerting, SLOs
Security — SAST, image scanning, RBAC, secrets management
Reliability — SLOs, incident response, runbooks, chaos engineering
Documentation — architecture docs, runbooks, onboarding

Use a checklist like this in two ways:

Assessment — where is your team on each dimension today?
Roadmap — which gaps are most important to close first?

The checklist doesn’t tell you how to improve — that’s what the blueprints are for. But it tells you where to look.

These seven-and-a-half tools don’t need to be implemented all at once. Pick the ones with the highest leverage for your current situation, implement them well, then move to the next.

The compound effect over 12-18 months is significant.

Cloud Native CICD

Mon, 05 Apr 2021 00:00:00 +0000

If there’s one capability that separates high-performing engineering organisations from struggling ones, it’s software delivery. Not talent, not tooling, not architecture — delivery.

Continuous integration and continuous delivery (CI/CD) is the engineering practice that makes frequent, safe delivery possible. This post builds the CI/CD blueprint for cloud-native systems.

Why Frequent Small Releases Win

The research in Accelerate (Forsgren, Humble, Kim) is unambiguous: high-performing teams deploy more frequently and have lower change failure rates than low-performing teams. These aren’t trade-offs — more frequent deployment is correlated with better stability.

The intuition: a deployment of 10 small changes is less risky than a deployment of 100 changes. Each small change is easier to understand, easier to review, easier to roll back if something goes wrong.

Big bang releases are a risk accumulation strategy. Avoid them.

Cloud-Native in Eight Elements

Before getting into the pipeline, it’s worth stating what “cloud-native” means from a delivery perspective. Four central ideas:

Containerisation — every workload is a container
Orchestration — Kubernetes manages the containers
DevOps — shared ownership of the full SDLC
Continuous delivery — software is always in a deployable state

And eight elements that make it real:

Immutable infrastructure
Declarative configuration
Microservices
Dynamic scheduling
Automated delivery pipelines
Observability
Self-healing
Policy-as-code

SCM as SDLC Tempo Driver

Source control management (Git) is the heartbeat of the SDLC. Every significant step — feature start, code review, quality gate, deployment — should be triggered by a Git event.

We use GitHub Flow as the branching strategy:

main is always deployable
Feature work happens on short-lived branches (hours to days, not weeks)
PRs are the unit of review and the trigger for quality gates
Merge to main triggers deployment to dev; tagged releases trigger staging/production

GitHub Flow is simple and it forces small, frequent merges. Long-lived feature branches are a smell — they mean large, risky merges.

The Containerisation Pipeline

Every service gets built into a container image in CI:

git push → CI trigger → build container image → tag with git SHA → push to registry

Key principles:

Build once — the same image that passes tests gets deployed to every environment
Tag with git SHA — gives you exact traceability from production back to the commit
Never use latest — in production, always pin to a specific image SHA or version tag

Dev Environment Per Branch

One of the most powerful patterns in cloud-native CI/CD: a full development environment per branch.

With Kubernetes, it’s feasible to spin up a complete stack for every feature branch:

feature-branch → CI → build image → deploy to dev/feature-xyz namespace → accessible at feature-xyz.dev.yourdomain.com

This gives developers and reviewers a live, isolated environment to test against before merging. It eliminates the “works on my machine” problem and makes PR review much more effective.

The namespace is torn down when the branch is deleted.

Quality Gates on PRs

Before any code can merge to main, it must pass a set of automated quality gates:

1. Unit Tests

Fast, isolated tests of individual components. Must pass with high coverage. Target: under 5 minutes.

2. Integration Tests

Tests that exercise the service against real (or realistic) dependencies. Slower, but catch integration bugs that unit tests can’t.

3. Static Security Scan (SAST)

Automated scanning for security vulnerabilities in code and dependencies. Fail the build on critical findings.

4. Code Quality Scan

Linting, static analysis, code smell detection. Sonarqube, CodeClimate, or similar.

5. Peer Review

At minimum one human review of every change. Security-sensitive changes require a security-aware reviewer.

All five gates are required. A merge that skips any of them is a risk that will eventually manifest as a production incident.

Build Once, Promote Everywhere

The golden rule of CI/CD: the same artifact that was tested is the artifact that runs in production.

Do not rebuild images between environments. Rebuilding introduces the possibility of differences — different dependency versions, different build environment state, different code if something was wrong with the first build.

The promotion pipeline looks like:

commit → build image (tag: git SHA) → push to registry
                                              ↓
                                    deploy to dev (automatic on merge)
                                              ↓
                                    automated test suite in dev
                                              ↓
                                    promote same image to staging (manual or automatic)
                                              ↓
                                    acceptance tests in staging
                                              ↓
                                    promote same image to production (manual approval)

The image SHA is what gets promoted. The environments are different; the artifact is not.

ChatOps

ChatOps brings your deployment pipeline into your team’s communication tool (Slack, Teams, etc.).

Instead of logging into a CI dashboard to trigger or check deployments, you interact via chat commands:

/deploy sensor-api v1.4.2 to staging
/rollback sensor-api in production
/status sensor-api

The bot executes the commands, reports back, and logs everything in the channel. This gives the whole team visibility into what’s being deployed and when, without anyone needing to context-switch to a separate tool.

ChatOps also serves as an audit log: every deployment, every rollback, is recorded in the chat channel with a timestamp and the person who triggered it.

Infrastructure as Code

The Kubernetes manifests, Helm charts, Terraform configurations — all of this is code, and it lives in Git.

Changes to infrastructure go through the same review and quality gate process as application code. Separate repositories for app code and infrastructure config (GitOps pattern) is common and works well.

Use ArgoCD or Flux to apply infrastructure changes from Git automatically (GitOps). The cluster state is always reconciled with what’s in Git.

Summary

A complete cloud-native CI/CD pipeline:

Developer pushes to feature branch
CI builds image, runs unit tests
Dev environment deployed from branch
PR raised → quality gates run (unit, integration, security scan, code quality)
PR reviewed and approved
Merge to main → automatic deploy to dev
Automated tests run in dev
Manual promotion to staging → acceptance tests
Manual approval → promote to production
ChatOps notification to team

The goal is a pipeline where deploying to production is boring. Boring is good.

The Kubernetes and cloud software security blueprint

Thu, 04 Feb 2021 00:00:00 +0000

Security is rarely the most exciting topic in software engineering, but it’s one of the most consequential. This post builds the security blueprint for our IoT case study — covering network security, secrets management, RBAC, and secure container configuration.

The core principle throughout: security should be a first-class architectural concern, not a retrofit.

Network Segmentation: Protect Your Databases

The first and most important network security decision: your databases should never be directly reachable from the internet.

In cloud infrastructure, this means:

Databases deployed in private subnets (no public IP, no internet gateway route)
Only the application layer (in the private subnet or VPC) can reach the database
VPC peering used to connect application VPC to data VPC when they’re separate

For Kubernetes specifically, this means pods that need to talk to a database get that access via a Service (or ExternalName/Endpoints), but the database itself is isolated at the network level.

Kubernetes Network Policies

By default, all pods in a Kubernetes cluster can talk to all other pods. This is not what you want in production.

NetworkPolicies let you define ingress and egress rules at the pod level:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: sensor-api-netpol
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: sensor-api
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: ingress-controller
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: kafka
    ports:
    - protocol: TCP
      port: 9092
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53  # DNS

Start with a deny-all base policy per namespace, then add explicit allow rules. This way, any new service that gets deployed is isolated by default until you explicitly open what it needs.

Note: NetworkPolicies require a CNI plugin that supports them (Calico, Cilium, Weave Net). The default Kubernetes networking doesn’t enforce them.

DDoS Protection

For a public-facing system, you need DDoS protection at multiple layers:

Cloud-level DDoS protection — AWS Shield, GCP Cloud Armor, Azure DDoS Protection. These handle volumetric attacks at the network edge before traffic reaches your infrastructure.
CDN-based protection — Cloudflare, Fastly, or your cloud CDN. The CDN absorbs traffic spikes and can block malicious traffic patterns.
Ingress rate limiting — Nginx Ingress supports rate limiting annotations:

nginx.ingress.kubernetes.io/limit-rps: "100"
nginx.ingress.kubernetes.io/limit-connections: "10"

API Gateway throttling — for your web/mobile APIs, configure per-client rate limits at the API Gateway level.

Layer these protections. No single layer is sufficient.

WAF (Web Application Firewall)

For the frontend APIs, a WAF provides protection against:

SQL injection
XSS attacks
OWASP Top 10 vulnerabilities

AWS WAF, Cloudflare WAF, or ModSecurity integrated into your Ingress controller are common options. Configure rules appropriate for your API payload shapes.

Secrets Management

As mentioned in the infrastructure blueprint, Kubernetes Secrets are base64-encoded, not encrypted by default. In production, you need a proper secrets management approach.

Option 1: Encrypted etcd + KMS

Enable encryption at rest for Kubernetes etcd, backed by a cloud KMS:

# encryption config
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - kms:
      name: aws-kms
      endpoint: unix:///tmp/socketfile.sock
      cachesize: 1000
  - identity: {}

This encrypts secrets at rest in etcd using your KMS key. The secrets still live in Kubernetes but are protected.

Option 2: Hashicorp Vault

Vault is the most fully-featured secrets management solution. It provides:

Dynamic secrets (short-lived credentials generated on demand)
Secret leasing and renewal
Audit log of all secret access
Fine-grained access policies

The Vault Agent Sidecar Injector can inject secrets into pods without the application needing to talk to Vault directly:

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "sensor-api"
  vault.hashicorp.com/agent-inject-secret-config: "secret/data/sensor-api/config"

Vault is more operationally complex than KMS-encrypted Secrets, but the dynamic secrets capability (especially for database credentials) is worth it for security-sensitive environments.

RBAC: Principle of Least Privilege

Kubernetes RBAC

Every service account should have only the permissions it actually needs. No service account should have cluster-admin unless it genuinely needs it.

Define Role (namespace-scoped) or ClusterRole (cluster-scoped) with minimal permissions:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sensor-api-role
  namespace: production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "watch", "list"]
  resourceNames: ["sensor-api-config"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
  resourceNames: ["sensor-api-secrets"]

Bind it to a dedicated ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sensor-api-binding
  namespace: production
subjects:
- kind: ServiceAccount
  name: sensor-api-sa
  namespace: production
roleRef:
  kind: Role
  name: sensor-api-role
  apiGroup: rbac.authorization.k8s.io

Cloud IAM

The same principle applies to cloud IAM. Each service’s cloud IAM role should have only the permissions it needs:

The sensor API needs to publish to Kafka and write to S3 → those permissions only
The analytics job needs to read from S3 → read-only permissions only

Use cloud provider-specific mechanisms to bind Kubernetes ServiceAccounts to cloud IAM roles (AWS IRSA, GCP Workload Identity).

Secure Container Configuration

Pod security context controls how containers run at the OS level:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sensor-api
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
      containers:
      - name: sensor-api
        image: sensor-api:1.0.0
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

Key settings:

runAsNonRoot: true — container must not run as root
readOnlyRootFilesystem: true — container can’t write to the filesystem (forces explicit volume mounts for any needed writes)
allowPrivilegeEscalation: false — prevents privilege escalation attacks
capabilities: drop: ALL — drops all Linux capabilities (add back only what’s actually needed)

These settings won’t stop every attack, but they significantly limit the blast radius of a container compromise.

Static Code Scanning

Shift security left by scanning code in CI:

SAST (Static Application Security Testing) — find vulnerabilities in your code before it ships (Snyk, Semgrep, SonarQube)
Container image scanning — scan images for known CVEs (Trivy, Clair, AWS ECR scanning)
Kubernetes manifest scanning — validate manifests against security policies (kube-score, Checkov, OPA/Gatekeeper)

Add these as required quality gates in your CI pipeline. A build that introduces a critical CVE should fail.

Peer Review for Security Changes

Security-sensitive changes (RBAC changes, network policy changes, secrets handling) should require mandatory peer review. Ideally, with at least one reviewer who has security expertise.

No automated tool catches everything. Human review catches what tools miss.

Summary

Concern	Solution
Database isolation	Private subnets + VPC peering
Pod-level network isolation	NetworkPolicies (deny-all default)
DDoS protection	Cloud Shield + CDN + Ingress rate limiting
Web attacks	WAF (AWS WAF / Cloudflare)
Secrets at rest	Encrypted etcd + KMS, or Vault
Kubernetes permissions	RBAC with least privilege + dedicated ServiceAccounts
Cloud permissions	Cloud IAM with least privilege + Workload Identity
Container hardening	securityContext (nonRoot, readOnly, drop capabilities)
Code vulnerabilities	SAST + image scanning in CI

The infrastructure blueprint of the microservice project - Kubernetes centric

Thu, 19 Nov 2020 00:00:00 +0000

In the previous post we introduced the IoT case study. Now let’s build the infrastructure blueprint.

The goal here is to map each component of the system to the right Kubernetes (or cloud-native) construct, and to explain the reasoning behind each decision.

First Question: What Lives in Kubernetes?

Not everything needs to be in Kubernetes. A useful first pass is to categorise components:

Kubernetes-hosted:

Stateless API services
Stream processing consumers
Scheduled batch jobs
Internal tooling

Cloud-hosted (managed services):

Kafka (MSK on AWS, Confluent Cloud, etc.) — operational complexity is high; managed is usually right
Relational database (RDS, CloudSQL, etc.) — same reasoning
Object storage for data lake (S3, GCS)
CDN for frontend asset delivery

Keeping managed services outside Kubernetes simplifies operations significantly. You give up some control, but for infrastructure that isn’t your core competency, that’s a good trade.

Ingress: Getting Traffic In

For external traffic, we use the standard three-layer Kubernetes ingress pattern:

Internet → Cloud LoadBalancer → Kubernetes NodePort → Ingress Controller → Services

Cloud LoadBalancer — provisioned by the cloud provider, gives us an external IP, handles TLS termination at the edge
NodePort — exposes the Ingress Controller on a port on each node
Ingress Controller (nginx or similar) — routes HTTP/S traffic to Services based on host/path rules
ClusterIP Services — the internal service discovery layer

This is the standard pattern and works well for HTTP/S traffic. For the sensor ingest path, we may want a separate TCP/UDP path if sensors use non-HTTP protocols — handled via a separate LoadBalancer Service.

Scaling the Sensor API

The sensor ingest API will receive high and variable traffic. We use HorizontalPodAutoscaler (HPA) to scale the Deployment automatically:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: sensor-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: sensor-api
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Key points:

minReplicas: 3 — we never want fewer than 3 for availability (across zones)
CPU at 60% target gives headroom before performance degrades
For real systems, consider custom metrics (requests per second) rather than CPU

Kafka Consumers: StatefulSet vs Deployment

The Kafka consumer services (stream processors) require careful thought. Should they be Deployment or StatefulSet?

It depends on whether the consumers maintain local state:

Stateless consumers → Deployment If the consumer processes each message independently and writes output elsewhere (DB, another Kafka topic), use a Deployment. Pods are interchangeable, scaling is simple.

Stateful consumers → StatefulSet If the consumer maintains local state (e.g., Kafka Streams with a local RocksDB store), use a StatefulSet. Pods have stable identities and stable storage. Kafka partition assignment can be pinned to specific pods.

For most consumers in this system, a Deployment with external state is the right default. Reach for StatefulSet only when you genuinely need stable identity.

Batch Analytics: CronJob

The analytics batch job (periodic aggregations, reports) maps cleanly to a Kubernetes CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: analytics-aggregator
spec:
  schedule: "0 * * * *"  # hourly
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: aggregator
            image: analytics-aggregator:latest
          restartPolicy: OnFailure

CronJobs are simple and reliable for periodic work. Just be careful with:

concurrencyPolicy: Forbid if jobs must not overlap
Monitoring for missed/failed runs (Kubernetes doesn’t alert on this by default)

Configuration: ConfigMaps and Secrets

Non-sensitive configuration goes into ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sensor-api-config
data:
  kafka_bootstrap_servers: "kafka.internal:9092"
  kafka_topic: "sensor-events"
  max_batch_size: "100"

Sensitive values (credentials, API keys) go into Secret:

apiVersion: v1
kind: Secret
metadata:
  name: sensor-api-secrets
type: Opaque
data:
  db_password: <base64-encoded>
  api_key: <base64-encoded>

Important caveat on Secrets: Kubernetes Secrets are base64-encoded, not encrypted by default. They’re protected by RBAC, not encryption. For production, you need either:

Encrypted etcd (enable at the cluster level)
A KMS solution (AWS KMS, GCP KMS, Hashicorp Vault)

We’ll cover this in the security blueprint.

External Service Abstraction

When your pods need to reach external services (the managed Kafka, the RDS database), use Kubernetes Service with a custom Endpoints object. This lets you treat external services as if they were internal Kubernetes services:

apiVersion: v1
kind: Service
metadata:
  name: kafka-external
spec:
  ports:
  - port: 9092
---
apiVersion: v1
kind: Endpoints
metadata:
  name: kafka-external
subsets:
- addresses:
  - ip: 10.0.1.50  # MSK broker IP
  ports:
  - port: 9092

This decouples your application code from knowing it’s talking to an external service. You can swap the external service for an in-cluster one (e.g., during development) without changing application config.

Frontend APIs: API Gateway

The web and mobile frontend APIs get an additional layer: an API Gateway.

The API Gateway handles:

Authentication / JWT validation
Rate limiting per client
Request routing to downstream services
Response caching where appropriate

Options: AWS API Gateway, Kong, Ambassador/Emissary, or nginx with lua. The choice depends on your cloud provider and team familiarity.

CDN for Static Assets

Frontend static assets (JS bundles, CSS, images) are served via CDN. This is a no-brainer:

Reduces latency for end users
Reduces load on origin servers
Free DDoS mitigation (to a degree)

CDN configuration is outside Kubernetes. The CDN origin points at your Ingress or a separate object storage bucket.

Summary

Component	Kubernetes Object
Sensor ingest API	Deployment + HPA
Stateless Kafka consumers	Deployment
Stateful Kafka consumers	StatefulSet
Analytics batch job	CronJob
Internal services	Deployment + ClusterIP Service
External traffic entry	LoadBalancer + Ingress
Configuration	ConfigMap
Credentials	Secret + KMS
External service refs	Service + Endpoints
Frontend APIs	Deployment + API Gateway

Next: The Kubernetes and cloud software security blueprint

The blueprints - case study, specification and architecture review (intro)

Thu, 12 Nov 2020 00:00:00 +0000

Before diving into individual blueprints, we need a concrete system to reason about. Abstract discussions of architecture are useful up to a point — but the real complexity emerges when you apply principles to a specific problem.

This post introduces the IoT data ingestion platform that will serve as the case study throughout this series.

The System

The system receives data from a large number of distributed sensors and devices:

Raspberry Pi arrays
IoT sensor networks
Mobile phones and tablets

This data flows through the system, gets processed and stored in two ways:

A data lake for raw and transformed event data (analytics, ML)
A relational database for structured, queryable data (operational)

On the output side, the system exposes APIs consumed by:

Web frontends
Mobile applications

It’s a fairly typical modern data-intensive system, but with enough moving parts to make the blueprints meaningful.

Why This System?

It represents a common class of problem:

High ingest volume — thousands of sensors sending data continuously
Mixed workloads — stream processing, batch processing, request-response APIs
Multiple consumers — different clients with different latency and reliability requirements
Greenfield — we’re designing it from scratch, which means we choose the constraints

Technology Decisions

Only one technology is strictly decided at this point: Apache Kafka as the streaming platform.

Kafka is the right choice here because:

It decouples producers from consumers
It provides durable, replayable message storage
It handles high-throughput ingest gracefully
It’s a proven choice for this class of problem

Everything else — databases, languages, frameworks, cloud provider — is left open deliberately. The blueprints should be applicable regardless of those choices.

Kubernetes is assumed as the infrastructure foundation throughout.

NFR Analysis Pass

Before building anything, it’s worth doing an explicit pass on non-functional requirements. For this system, key NFRs include:

Availability

The sensor ingest path needs to be highly available — sensors can’t buffer indefinitely
Web/mobile APIs need standard SLA (99.9%+)

Scalability

Ingest must scale horizontally as sensor count grows
Data storage must handle growing data volumes

Security

Sensor data may be sensitive; transit encryption required
Backend APIs must authenticate and authorise properly
Infrastructure must follow least-privilege principles

Observability

High-volume distributed systems are hard to debug without good telemetry
Need metrics, logs, and traces across all components

Deployability

Frequent releases without downtime
Safe rollback capability

Upcoming Blueprints

Based on this system, here are the blueprints we’ll build out:

Infrastructure blueprint — how to structure the Kubernetes workloads
Security blueprint — Kubernetes and cloud security posture
CI/CD blueprint — cloud-native software delivery pipeline
Observability blueprint — metrics, logging, tracing
Configuration management — dynamic config and circuit breakers
DevOps tooling — the practices and tools

Each blueprint will reference back to this case study and show how the principles apply concretely.

Next: The infrastructure blueprint of the microservice project

The two scales of DevOps

Mon, 02 Nov 2020 00:00:00 +0000

DevOps is one of the most misused and misunderstood terms in software engineering. Ask ten engineers what DevOps means and you’ll get ten different answers. Some will say CI/CD. Some will say shared responsibility. Some will say it’s just a job title for ops people who can code.

I want to introduce a mental model that I find useful: the two scales of DevOps.

Scale 1: The Scale of Development

The first scale is about developer flow — how fast and how safely can your engineering organisation ship software?

Key metrics (from the DORA research):

Lead Time for Change — from commit to production
Deployment Frequency — how often you deploy
Change Failure Rate — how often deployments cause incidents

The Scale of Development encompasses everything that affects these metrics:

CI/CD

The pipeline that takes code from commit to production. Build, test, scan, deploy. The goal is to make this fast, reliable, and automated. A slow CI/CD pipeline kills flow.

Developer Experience (DX)

Everything that affects a developer’s ability to work effectively: local development environments, tooling, documentation, onboarding, cognitive load of the codebase.

A team with poor DX will move slowly regardless of how good the pipeline is.

Flow Efficiency

Derived from Lean manufacturing. Of the total time from “idea” to “shipped”, what fraction is actual work vs waiting? In most organisations, the ratio is terrible — 10-20% active work, 80-90% wait time (in queues, review, etc.).

Improving flow efficiency often has more impact than improving individual speed.

Emergent Design

In fast-moving systems, big-upfront design doesn’t work. You need the capacity for continuous, incremental design improvement — refactoring, abstraction, simplification — as a first-class engineering activity.

Scale 2: The Scale of Life

The second scale is about production reliability — when things are running in production, how do you keep them running?

This is the domain of Site Reliability Engineering (SRE).

Key metrics:

MTTD — Mean Time to Detect (how fast do you know something is wrong?)
MTTR — Mean Time to Resolve (how fast can you fix it?)
SLO/SLI/SLA — Service Level Objectives, Indicators, Agreements

The Four Golden Signals

Google’s SRE book defines four signals that, if monitored, give you a complete picture of production health:

Latency — how long requests take
Traffic — how much demand is hitting the system
Errors — rate of failed requests
Saturation — how “full” the system is (CPU, memory, connections)

SLOs and Error Budgets

SLOs (Service Level Objectives) define the target reliability of a service — e.g., “99.9% of requests succeed in under 200ms”. The gap between 100% and the SLO is the error budget.

Error budgets align incentives: development teams want to ship features fast; SRE teams want stability. The error budget is the negotiated zone where both can happen. If the budget is being consumed too fast, you slow down releases. If there’s plenty of budget, you can move faster.

DevOps vs SRE

DevOps and SRE are often treated as competing philosophies. They’re not — they’re complementary.

DevOps is a cultural movement: shared ownership, collaboration between dev and ops, automation of toil. SRE is Google’s implementation of that culture, with specific practices and metrics.

You can think of it this way:

DevOps defines the what (goals, culture, principles)
SRE provides the how (practices, metrics, tooling)

Cognitive Load

One concept that cuts across both scales is cognitive load — the mental burden placed on engineers.

Team Topologies (Skelton & Pais) makes this the central constraint of team design. Teams that own too much, or work with systems too complex to reason about, move slowly and make more mistakes.

Architectural decisions — how services are split, how teams are structured, what shared platforms exist — should be evaluated in terms of their cognitive load impact as much as their technical properties.

Books worth reading

Accelerate — Forsgren, Humble, Kim — the research behind DORA metrics
The SRE Book — Google — the canonical SRE reference
Team Topologies — Skelton & Pais — team structure and cognitive load

Next: The blueprints - case study, specification and architecture review

Native to the cloud

Thu, 22 Oct 2020 00:00:00 +0000

This is the first post in the Cloud-Native Blueprints series — a set of deep dives into the non-functional aspects of cloud-native projects. The series is Kubernetes-centric, but the concepts apply broadly.

How we got here

Cloud computing didn’t just give us cheaper servers. It gave us a fundamentally different way to think about software architecture. Before cloud, infrastructure was slow to provision, expensive to scale, and painful to manage. Architectural decisions were made with those constraints baked in.

Cloud changed the constraints. Suddenly you could provision resources in minutes, scale horizontally on demand, and pay only for what you used. Architectural thinking had to catch up.

Out of this shift emerged cloud-native — a design philosophy and set of practices built to fully exploit what cloud infrastructure makes possible. Not “lift and shift” of old workloads, but systems designed from the ground up for elastic, distributed, automated environments.

Kubernetes as the foundation

Of all the technologies that emerged from the cloud-native movement, Kubernetes has become the de facto standard for cloud-native infrastructure. It provides:

Container orchestration at scale
Declarative configuration of the desired state
Self-healing and automated rollouts
A common abstraction layer across cloud providers

Kubernetes doesn’t solve everything — it introduces its own complexity — but it epitomises cloud-native infrastructure. This series will use Kubernetes as the primary lens.

What this series is about

Running software on Kubernetes is one thing. Running it well — at scale, reliably, securely, and efficiently — is another.

Most engineering attention goes to functional requirements: the features, the business logic, the data. Non-functional requirements (NFRs) — reliability, security, performance, observability, deployability — often get treated as afterthoughts.

They shouldn’t be. In distributed systems, the NFRs are frequently harder than the functional requirements, and failures in them tend to be catastrophic rather than graceful.

This series will systematically cover the non-functional blueprints for cloud-native systems:

Infrastructure — how to structure Kubernetes workloads
Security — Kubernetes and cloud security posture
CI/CD — cloud-native software delivery
Observability — monitoring, logging, tracing
Configuration management — dynamic config and circuit breakers
DevOps tooling — the practices and tools that make it work

Each blueprint will be grounded in a concrete case study: an IoT data ingestion platform. This gives us a realistic, complex system to reason about.

A word of caution

Cloud-native comes with real complexity costs. Not every system needs Kubernetes. Not every team has the capability to own the full SDLC end-to-end.

The right question isn’t “should we use cloud-native?” but “which parts of cloud-native make sense for our context, our team, and our problem?”

Throughout this series, I’ll try to be honest about that. The goal isn’t to sell you on a stack. It’s to give you the blueprints to make good decisions.

Don’t break stuff whatever others say.

Next: The two scales of DevOps

Monitoring Data Flows in a Microservice System

Sat, 12 Sep 2020 00:00:00 +0000

To deliver high-quality service, organizations must carefully track data flows within applications. This typically requires establishing well-designed code metrics and implementing robust monitoring systems around those metrics.

Metrics represent “the fastest observability data. They have the quickest output, can be consumed directly… require the smallest throughput, and have the smallest retention footprint.”

In service-oriented architectures, understanding how to leverage metrics at distributed integration points enables early problem detection—a critical capability explored throughout this piece.

The System Architecture

The article examines a microservice-based system featuring:

A Frontend API microservice (functioning partially as an API gateway)
Integration Aggregator (A.I.) microservices
Subsystem microservices specialized for domain aggregates

Understanding the Data Flow

The discussion focuses on synchronous data flows involving multiple sequential service calls. The presented sequence diagram shows a typical backend flow where:

Frontend API initiates requests
Multiple services process data sequentially
Responses aggregate and return to users
Two subflows represent error responses and timeout scenarios

Observability Perspective

Applying observability principles to the flow architecture reveals measurement opportunities. The system should expose metrics for two primary purposes:

Performance measurement
Flow correctness verification

Two metric types serve these purposes:

Counters (displayed as squares/rhomboids): Track initiated calls, completed calls, and caught errors
Timers (displayed as arrow bands): Measure call duration

What to Measure

Effective flow observability requires exposing:

Counters for every initiated and completed call with descriptive labels
Counters for caught errors, distinguishing between 5xx and 4xx responses
Timers for successful calls (error response timing is typically less relevant)

Importantly, status codes should be tracked with granular labeling, allowing later aggregation while preserving detailed data capture.

Where to Measure

Measurements should focus on:

Incrementing counters and starting timers when calls initiate
Incrementing success counters and stopping timers upon response receipt
Incrementing error counters when catching erroneous responses

A critical insight: “measuring the performance of every service by measuring its responses on the side of the consumer service” provides intuitive, logical system performance understanding. This approach accounts for networking, security measures, and integration overhead—factors internal service measurements miss.

Why Measure: Quality of Service

The overarching motivation centers on Quality of Service (QoS). Providing reliable service demands understanding system performance, which requires observing exposed data.

Performance Monitoring

API endpoint performance represents the most critical metric—the total duration of the complete flow. Plotting performance over time reveals degradation patterns.

Effective dashboards combine:

Total flow performance graphs
Individual segment performance graphs
Request volume counters

When performance degrades, these visualizations identify whether particular segments disproportionately impact response times.

Alerting approaches include:

Performance budget-based alerts (e.g., responses exceeding 300ms)
Historical comparison alerts (e.g., 20% slower than yesterday’s baseline)

“The same way service#1 that is calling service#2 needs to qualify the performance of service#2, your users are qualifying your application’s performance, and you are there to make sure that it is the best performance you can achieve.”

Error Rate Monitoring

All error response counters contribute to error rate tracking, supporting both quality assurance and technical flaw identification.

Error characterization distinguishes between status code ranges:

4xx vs 5xx separation applies when subsystems exhibit distinct 400-range behaviors (authentication, authorization)
Combined tracking suits systems without special 4xx handling
404 exclusion often makes sense for public-facing systems where page misses are common

The article illustrates that error impacts vary contextually:

Private applications with authentication: 4xx errors signal security concerns
Integration systems: 404s indicate upstream changes requiring immediate attention
Public search functionality: 404s are expected from crawlers
5xx errors are always problematic

Error rate alerting should be stringent, typically based on SLOs triggering when endpoints breach service-level objectives for extended periods.

“Make sure not to waste time on staring at monitoring dashboards - set up alerts and go do something fun” — emphasizing automation over manual observation.

Summary

Key takeaways include:

Code instrumentation constitutes a core software development responsibility
System observability requires architectural consideration matching the attention devoted to system design itself
Consumer-side performance measurement provides the proper perspective, accounting for integration and networking factors
Error management remains paramount

Planning Observability of Your System

Sat, 29 Aug 2020 00:00:00 +0000

Introduction

Observability is the ability of your system to expose data about its performance and behavior so we can further monitor, analyze, and alert on certain trends or changes. Making your system observable requires certain effort in your project.

Why Should You Care

Observability of your system is the top non-functional requirement. In the author’s opinion, not even application performance tops observability. It is absolutely necessary that you know what is happening in your system at any given time and in level of detail necessary to be able to keep everything running smoothly by adequately reacting to change in the system.

Of course, there is a possibility that you do not share this opinion, in which case you would probably rather develop your product, do a bit of testing, UAT, and push it out. Probably because it requires a certain amount of work and is considered non-essential as you start. If Google Analytics is all you need, refer back here when you change your mind. If, however, you want to dive into this topic and start planning, the author wants to share some insights that should help you pin down stuff that is most important for you and prioritize.

There are quite a few aspects of observability in service-oriented architecture, and this article focuses on discussing every aspect with a little bit of depth in order to help you better understand, focus and plan developing your observability. To help you plan and have a broader conversation with all the people involved in planning your project, the author will provide some insight into the most popular stacks as well and give insights into the most important defining factors.

Your system being observable can mean multiple things on as many levels, but generally, it means that system exposes data that we can consume to:

Monitor system in real-time.
Alert or perform some automated task based on defined data state, or rather a change of state.
Analyze and predict behaviors, requirements etc…
Do a detailed analysis of the incident or any distinct event in the system.
Any combination of the above.

Since it is all about using proper tools, let’s spend some time on quick discussion about what we need and what tool/type of observability we choose.

Characterizing Capabilities

When we talk observability of the system, the way we are planning it really depends on what we want to achieve. Ultimately, we would love to achieve everything, but some ability to focus and plan would be nice. Let us take a look at some factors to take into account when prioritizing observability.

Main Types of Data

Logs — Old, but gold. The first that ever existed has all the data about what is going on, but slow, hard to process, and needs huge system throughput to bring usage close to real-time. You can still go to the file that was generated last and grep it for a particular phrase or just look for the time when the error happened, which will forever be a favorite thing learned (apart from making pizza from scratch).

Metrics — Quick, flexible, made for quick availability and low throughput requirement, easy and long retention. Not too much data per unit of information, real-time trend analysis and reaction.

Traces — Everything you would like to know about the call you made, seems like magic especially in a well-instrumented distributed system. Sometimes slower than logs, but allows for deeper analytics.

Performance Qualities

Usability — Do metrics allow a good way to monitor and analyze them, with flexibility in data architecture to give enough versatility.

Speed/throughput requirements — how much metrics system can take before receiving some performance penalty. Also, how fast are the metrics available for consumption/use (both manual and automated).

Way We Plan to Consume Data

Monitoring — real-time plotting and observation of system state.

Alerting and automatic triggers — reacting to events by broadcasting notification or triggering events.

Analytics — analyzing collected data to gather useful learnings about the system and/or perform root cause analysis (RCA) of the problem in the past (usual case) or ongoing problem (rarer since Analytics is mostly about applying complex algorithms to big data sets to draw conclusions, and that takes time).

Quick Comparison Table

Type	Performance / Throughput needs	Recommended for
Logs	Medium / Medium - still lots of text	RCA/Monitoring/Alerting
Metrics	Fast / Low - smallest unit of data	Monitoring/Alerting/Analytics
Traces	Slow / High - contains lots of data	RCA, Alerting in some setups

Creating Observability

As you are building a system, no matter if it is a huge monolithic application, or a multiservice-something, planning observability you want to achieve will get you places. For that, you’ll have to know what kind of tools you need and what additional infrastructure will it take… of course, but there is also a choice of where to start. It is less obvious, but deciding to do everything might be a costly decision depending on how much time you got to deliver the first version of your system.

For most of the stuff that is written, the assumption is your goal is full DevOps, and that your team is building everything and running it in all the environments.

If you are getting something pre-built, you have an army of engineers or reasonably long timelines there might be a way to get it all in time.

Logging

Generally, you do not need to do anything in order for your system to start gathering logs, logs will be available to you anywhere from the default log directory for the OS of choice to Kubernetes logs. What you will need is everything else - literally.

Going from just a stack of call logs inside a gazillion line file on the disc to a searchable log database that can also be used for a certain level of alerting and many different applications is the matter of planning. The author usually recommends two well-known stacks, that differ in the log collection segment:

ELK — Elastic, Logstash, Kibana which is more standard one
EFK — Elastic, Fluentd, Kibana which is a more flexible one in terms of data flow management and further design scalability. It is also recommended for Kubernetes.

Everyone loves 3-letter abbreviations so nobody bothered to add F at the start of both since it became a standard. F for Filebeat.

Basically, you need to setup EXK and Filebeat, Filebeat will pick up the logs as they arrive to any of the mentioned standard log outputs and send them to a log collector, that gets stored into the Elastic database so logs will be indexed and made searchable and visualizable through Kibana.

Obviously, the more activity you got in your system, the more logs there is, more data throughput becomes crucial for your logs to get to you — Filebeat needs to send logs to Elastic that then needs to index them (on the way there is a throughput of whatever you use). There have been cases of hundreds of terabytes of logs a day with need of month to 3 months retention… It becomes a whole separate system to maintain.

The good thing is that, when overwhelmed, these setups are quite good in the degrading performance but keeping data, so if you do not run out of physical hardware and everything just halts, your logs may be late to Kibana because elastic can’t index, or they might be stuck in some queue, but they will not be lost. That is, at least, something.

In the Wild

Stack itself is relatively easy to deploy with lots of resources available online, from tutorials to docker-composed sets, Terraform submodules and the Helm charts to do it for you. The first hard thing to do is set everything right the first time from the delivery of logs perspective. The second hard thing to do is not to kill your logging stack with too much or too big logs.

Good luck.

Metrics

On the metrics side of things are a lot different than on the logs side of things — although there is a general majority choice of a tool, and thus an approach to it, there are actually two different approaches based on a way metrics are exposed/collected:

Pull — used by wider accepted and CNCF adopted Prometheus
Push — used by StatsD

Based on two stand out representatives of two groups, a quick comparison is provided.

Pull — Prometheus

Prometheus is a current metrics collection stack of choice, developed by SoundCloud and fully promoted to open source in 2015, although it was available for longer.

Pull means that the metrics collector is scraping services that are being monitored by it. In turn, the system requires that any service that needs to be monitored by Prometheus exposes metrics. Data exposition format allows “condensation” of the metrics that are exposed, so they basically can accumulate until scraped for both variable scraping intervals, but also the longer periods in case of incidents with collector.

Metrics are kept in the service’s memory until the collector pulls (scrapes) the metrics.

The architecture of the metrics collection and way Prometheus is built gives us two important architectural advantages:

Resilience — If Prometheus collector dies for some time, metrics are not lost because they are kept in the service’s memory until scraped
Federation — the same way the service exposes metrics to be collected by Prometheus, Prometheus itself can expose aggregated metrics to be collected by other Prometheus system, allowing for metrics collection scaling by creating a tree hierarchy collection structure known as Federation in the Prometheus ecosystem

Prometheus has a full set of needed components within its ecosystem — it collects metrics in form of time-series and stores them, it also has a alert manager to handle alerts and push gateway to support sort lived services (jobs, container orchestration, serverless…).

There is a certain complexity to using pull metrics that you have to take into consideration — since it scrapes services for metrics you would need service discovery to always know what to scrape, especially in container orchestration space.

In the Wild

Prometheus is a multi-component system that is relatively easy to deploy thanks to the community, but it has lots of quirks for you to handle. It will be super useful, and is recommended wholeheartedly, but be ready to do some maintenance on it as it is a fast data collection and parsing system having many resilience points which often cause stuff like memory leaks and hard drive space overload.

Every time you want to plot any real-life scenario data, you are making mid to high complexity query to Prometheus, which can overload it, especially in high-resolution intensive plots we can make Prometheus less efficient for its main purpose. Luckily, you can record (sort of index/cache) certain queries (rules) and make the whole thing more performant.

Prometheus can be used as a source directly in Grafana, which is the best open source monitoring+analytics frontend dashboard tool out there.

Push — StatsD

The first monitoring stack that could seriously be called that was StatsD, developed by Etsy.

Push means that your system is pushing the signals to StatsD all the time.

This simplifies lots of architectural requirements since service discovery is not needed, the service itself needs to know where the StatsD server (also called daemon) is, and that is that it can receive signals from many sources.

The author personally thinks that price in a bit of complexity on the Prometheus side is worth it.

Looking at the stack itself, where Prometheus has everything in one place, StatsD has its daemon and clients (for every language), but everything else needs a custom thing to plug in:

To read and transform data you need something like Graphite, that you can later also use as a source for Grafana.
For alerting you need another module.

In the Wild

Fact that the daemon gets hit for every event in your service and that your service needs to make a request every time the event you are measuring is done leads to new requirements on the software engineering side if the number of request to your app grows. You need to sample every metric if you do not want most of your resource consumption to go into metrics. There is this really good anecdote from Etsy that they actually had alerts on metrics sampling, so Ops guys would come for you if you did not sample or you did not sample enough.

The best way to visualize in this case is also Grafana, but in this case you can not use StatsD as a data source directly, you need Graphite.

Sampling

The technique of reducing number of metrics in a way that you are only sending samples of data (that is why it is also known as downsampling). What you do is send one in 100 metrics generated (one in every 100 requests gets measured).

Sampling helps in both push and pull case, but it is more useful in push case since it reduces resource consumption in the collection and storage space needed for retention. In Pull case, it just saves you disk space and some resources for more complex queries.

Tracing

As the data boom hit the industry it did not miss the observability of stuff we are building. Now we can expose every small detail about the long term behavior of our app in the form of a huge amount of data that tracks any request end to end, even in distributed systems.

The main difference between the two large-volume observabilities is that logs in their essence represent a collection of individual events and traces are designed to represent set of connected events and their holistic picture for further analysis. The base of tracing observability is code instrumentation and there are many ways it is done (including connecting log events using request ids), but the currently accepted global standard is OpenTracing API (now part of OpenTelemetry).

Two tools to mention are Zipkin and Jaeger, developed by Twitter and Uber respectively, both using OpenTracing API and both supporting every programming language in almost every form (framework) out there.

Zipkin is simpler to deploy a more conventional client-server collection system that is oriented towards less complex, non-container-orchestrated systems. Jaeger is more complex, distributed tracing oriented, and preferably deployed to Kubernetes.

In the Wild

Tracing tools logically need more complex setups in their backends with fast databases, map/reduce systems, and special scaling needs in order to provide us with the best trace grouping and analysis capability. These are always complex systems of many small and big components to deploy and maintain across the space.

In return, we get the ability to see requests travel through our system, and the capability to identify the exact place where any of the erroneous ones failed and which logs the failure produced etc…

Data inflation!

Summary

Logs have all the data, but you need a delivery system, database, indexing, and search/dashboard tool in order to be able to use them beyond merely skimming the text in a file on a disk somewhere
Metrics do not have too much data but they provide an essential way of measuring your system’s performance and reacting to it in real-time
Tracing is giving us deep insight into the system’s behavior, and holistic analysis of every request. Price is a complex system to gather, store, analyze, and store to be searched and visualized.
Prometheus is the best
Use Grafana.

P.S.

This was written as a separate article since the author realized that intro to another observability article that had been written can actually be planning and tools insight guide for a broader audience — aiming for all specialties like engineers, architects, tech POs, QA engineers… Hopefully it worked :D

Software in Time of Apocalypse

Mon, 20 Apr 2020 00:00:00 +0000

In the current moment, amid the COVID-19 crisis, isolation and heavy reliance on software for work, communication, entertainment, wellbeing, and information have made it increasingly difficult to comprehend the evolution of software perception over the last 10-15 years.

Only nine years before this essay, Marc Andreessen published his influential piece on why software was consuming the world. At that time, Netflix, Uber, and Airbnb were already transforming commerce, though the full scope of digital disruption was still unfolding. Software rapidly altered the commercial landscape, bringing businesses online and enabling new service-based models.

Nowhere to Run

Now facing global pandemic and physical isolation, society is confronting an unfamiliar need: to persist, sustain life in its best form, and prioritize wellness above all else. The author describes this moment as “the great pause”—a time when people cannot escape into constant activity.

The pause has shifted purchasing patterns away from experiences like concerts and rideshare services toward goods like furniture, flowers, food, and record players. More significantly, it has encouraged genuine human connection, both with ourselves and others.

New Digital Wave

A profound wave of digital transformation is occurring. Small businesses are rapidly establishing web shops. Museums and galleries are launching virtual tours. Painting lessons, film festivals, and theater productions are being streamed online. Musicians perform free shows to global audiences from their homes.

Education has transformed through cloud-based systems, videos, games, and video calls. Children adapted immediately to Zoom classrooms and Google Meet gatherings, maintaining social interaction despite physical distance.

A Proper Wellness

Remote yoga sessions, Instagram live streams of meaningful content, guided meditations, and online therapy services have proliferated. Cooking tutorials, fitness challenges, and wellness content that previously remained niche now serve genuine needs rather than marketing-driven desires.

The author observes that people are learning what they need to be well through digital means—ancient meditation practices alongside modern fitness programs, dancing therapies, and virtual fitness challenges.

After the Great Pause

This pandemic represents a learning experience about our greatest modern invention: software and digital transformation. Society is realizing that the same tools used to accelerate consumption and sensory overload can facilitate healing and connection.

The crucial insight: “Screens we are looking at are not the problem — we are the problem. Screens were not alienating us, it was us who were alienating us.” Now, stuck to screens more than ever, people use them for connection rather than isolation. Society has finally recognized that software serves human flourishing.

Un-busy Your Team

Sun, 27 Oct 2019 00:00:00 +0000

The rituals

How many companies you know are doing “the agile”?

How many are going through some kind of agile transformation at the moment?

What are the rituals of your agile team? Do you have any?

Did those rituals get added on top of your regular agenda or did they streamline it and “merge” it (in a sense)?

I know most of you are cringing by now.

The sync

Being in the tech industry for a while now, I got a pretty good understanding of how easy all the good process management and organizational ideas descend into the spiral of never ending syncs, reports, daily meetings… It all seems logical - to gain visibility of every small process, you have to have many meetings to align with everyone or read same amount of reports. Small price to pay to gain transparency and agility.

And when we achieve nothing, which is logical result of not doing anything, we conclude that we need the retrospective. Or even better — a few retrospectives.

But, is not doing anything actually the worst thing about it? What does it mean “to do” in this case?

The team

In our industry (any engineering and science, really, but let’s stick to what we know) your team is your “think tank”, it is carefully chosen group of talented people who have tools to do the job.

And the job is … problem solving, creating, hacking… It is fight for that new idea, for knowledge and inspiration.

IRQ

In a computer, an interrupt request is a hardware signal sent to the processor that temporarily stops a running program and allows a special program, an interrupt handler, to run instead. Hardware interrupts are used to handle events such as receiving data from a modem or network card, key presses, or mouse movements.

Wikipedia (and any microprocessor architecture 101 course)

Interruption works more or less the same with people — you send a request for new piece of work, if it is perceived as more important current work will pause and new work will begin.

However, no real life work is as simple and as straight forward to jump on as pre-processed, compiled and linked code sent straight to microprocessor that then does what it is best in. Microprocessor has queue for this stuff as well…

The busy

When person is interrupted while working, by new piece of work or just new piece of information, many things happen cognitively and emotionally before we even get to doing some of it. When new communication comes in, especially if it is relate to other similar, but not the same piece of work, context change hits and, for some period of time, takes over significant part of our cognitive capacity. If this is, as it is usually the case, followed by emotional reaction due to being interrupted, we have significant negative impact on both cognitive capacity and spirit. People tend to be stressed out if ability to focus is taken away from them, or when priorities keep changing.

This is where real problem is shaped — today, people are often busy even before they start doing something, and they sometimes stay so busy that they do not actually do anything… And in many cases it is our failure as leaders — we did not protect our teams from this.

Focus at work

Focus is a lot about strategic choice, about systematically avoiding context switches. Shifting your mind from one thing to another cost us lowered productivity time — very often low productivity and high frustration periods.

More precisely — in most of the cases it costs us days of significantly lowered productivity every week.

Some idle time

Make sure to know that having idle time is essential for creative work. Stopping your mind from racing, blurring the focus and just hanging around yields lots of beautiful thought processes. Idleness is in the root of EUREKA! moment.

Your team needs focus and idleness, you need focus and idleness, your company’s mission and vision depends on your focus and idleness.

The busy busy

What, unfortunately, happens is that we are busy, which turns out to be quite the opposite to actually doing work.

This phenomena is semi-recognized through quite proverbially paraphrasing “Busy is a new stupid” everywhere and trying to make some sense out of our schedules and how to effectively manage our time.

What is there to do, anyways?

It is time to recognize the following:

Not all interruptions are work requests but all the interruptions are making your mind busy.
Not all time consuming tasks are work, but they are making you busy
You can not hack and create while you are busy, unless you are busy hacking and creating.
You can not learn new thing while you are busy, unless you are busy learning.
When you stop being busy, your mind needs time to settle before becoming creative (not to mention sugar).

Leaders

It is job of a leader to make sure things are moving forward, motivating team and finding that spark that puts everyone where they want to be. You got to know where everyone wants to be and what everyone wants to do.

Additionally, leader needs to get team’s work where it needs to be according to general vision. Vision, marching orders — tomayto, tomahto… Your only job is to enable everyone in the team to be their best selves, to focus, move forward and thrive. Leader should protect the team from distractions in order to let them do what they do best.

All in all it is rather conundrum-a-day then I-got-this situation, but one thing is certain.

In order to be able to do everything that needs to be done, we have to help team be less busy.

Our teams of engineers and creatives need to not be busy. We need to commit our time to protect the precious minds that create our worlds.

We need to unbusy our teams.

Internal and External Connectivity in Kubernetes Space

Sat, 13 Jul 2019 00:00:00 +0000

As you navigate through your application’s development stages and consider Kubernetes deployment, understanding how components connect internally and externally becomes essential.

“Knowledge you will get from this article also covers ‘services & networking’ part of CKAD exam, which currently takes 13% of the certification exam curriculum.”

Services

Kubernetes services enable networking between cluster components and external systems. The article covers three main types:

NodePort — Exposes pods through a node’s port, involving three port layers:

targetPort (application listening port)
Port (service port)
NodePort (external access port, range 30000-32767)

Example NodePort service:

apiVersion: v1
kind: Service
metadata:
 name: k8s-nodeport-service
spec:
 type: NodePort
 ports:
 - targetPort: 6379
   port: 6379
   NodePort: 30666
selector:
 db: redis

Services use selectors to link to pod sets, functioning as load balancers distributing traffic randomly across matching pods.

ClusterIP — Provides internal cluster communication through unified IP/name for pod groups, preventing instability from pod recreation and eliminating manual load-balancing complexity.

apiVersion: v1
kind: Service
metadata:
 name: api-1
spec:
 type: ClusterIP
ports:
- targetPort: 9090
  port: 9090
selector:
 app: api-1
 type: backend

ClusterIP serves as the default service type when unspecified.

LoadBalancer — Provisions external cloud load balancers with advanced features and standard port mapping. Each exposed service requires its own load balancer, potentially increasing complexity and costs.

apiVersion: v1
kind: Service
metadata:
 name: lb1
spec:
 externalTrafficPolicy: Local
 type: LoadBalancer
 ports:
 - port: 80
   targetPort: 80
   protocol: TCP
   name: http
 - port: 443
   targetPort: 443
   protocol: TCP
   name: https
 selector:
   app: lb1

“The external traffic will not be equally load balanced across pods, but rather equally balanced at the node level.”

Ingress

Managing multiple web servers with different pod sets using traditional services becomes complex in real-world scenarios. This complexity motivated Kubernetes to introduce Ingress as a Layer 7 load balancer solution.

Ingress provides load balancing, SSL termination, and name-based virtual hosting for HTTP/HTTPS traffic.

Ingress Controller Setup

Two components are necessary:

Ingress Controller — Manages ingress rules (nginx, GCE officially supported)
Ingress Resources — Defines HTTP routing rules

Example nginx ingress controller deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 name: nginx-ingress-controller
spec:
 replicas: 1
 selector:
  matchLabels:
   name: nginx-ingress
 template:
  metadata:
   labels:
    name: nginx-ingress
  spec:
   containers:
   - name: nginx-ingress-controller
     image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
     args:
     - /nginx-ingress-controller
     - configMap=$(POD_NAMESPACE)/ingress-config
     env:
     - name: POD_NAME
       valueFrom:
         fieldRef:
          fieldPath: metadata.name
     - name: POD_NAMESPACE
       valueFrom:
        fieldRef:
         fieldPath: metadata.namespace
     ports:
     - name: http
       containerPort: 80
     - name: https
       containerPort: 443

ConfigMap deployment:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration

A NodePort service exposes the ingress controller:

apiVersion: v1
kind: Service
metadata:
 name: nginx-ingress
spec:
 type: NodePort
 ports:
 -targetPort: 80
  port: 80
  protocol: TCP
  name: http
 -targetPort: 433
 port: 433
 protocol: TCP
 name: https
selector:
 name: nginx-ingress

Service accounts require specific permissions:

apiVersion: v1
kind: ServiceAccount
matadata:
 name: nginx-ingress-serviceaccount

Ingress Resources

Simple ingress routing all traffic to one service:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: simple-api-ingress
spec:
 backend:
  serviceName: simple-api-service
  servicePort: 8080

Path-based routing example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: proper-api-ingress
spec:
 rules:
 -http:
   paths:
   -path: /api/search
    backend:
     serviceName: search-api-service
     servicePort: 8081
   -path: /api
    backend:
     serviceName: api-service
     servicePort: 8080
   -path: /
    backend:
     serviceName: frontend-service
     servicePort: 8082

Host-based routing with subdomains:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
 name: proper-api-ingress
spec:
 rules:
 -host: api.example.com
  http:
   paths:
   -path: /search
    backend:
     serviceName: search-api-service
     servicePort: 8081
   -path: /
    backend:
     serviceName: api-service
     servicePort: 8080
 -host: app.example.com
  http:
   paths:
   -path: /
    backend:
     serviceName: frontend-service
     servicePort: 8082

The article emphasizes maintaining all configurations as code in version control repositories, following infrastructure-as-code principles for application reliability.

Local Kubernetes Setup with Minikube on Mac OS X

Tue, 01 Jan 2019 00:00:00 +0000

Minikube serves as an excellent tool for establishing a local Kubernetes environment to test and experiment with deployments. This guide walks through installation, configuration, and integration with essential tools.

Minikube Installation

Minikube operates through a virtual machine and supports various hypervisor options. VirtualBox is the recommended choice for macOS.

Installation via Homebrew:

$ brew cask install virtualbox minikube

macOS security features may block VirtualBox installation on Mojave and later versions. Access System Preferences > Security & Privacy and approve Oracle software installation on the General tab before rerunning the installation command.

Running and Accessing the Cluster

Start the minikube cluster with:

$ minikube start

Stop the cluster when not in use to conserve battery life:

$ minikube stop

Access the Kubernetes dashboard:

$ minikube dashboard

For users managing multiple cluster contexts, list and switch to minikube:

$ kubectl config get-contexts
$ kubectl config use-context minikube

Ingress Controller

Enable the ingress add-on for deployments requiring ingress resources:

minikube addons enable ingress

Configure local hostnames in /etc/hosts to match ingress rules:

$ echo "$(minikube ip) local.host" | sudo tee -a /etc/hosts

Container Registry

Local Docker Registry

Point your Docker context to minikube:

$ eval $(minikube docker-env)

To revert: $ eval $(docker-machine env -u)

Start a local registry container:

$ docker run -d -p 5000:5000 --restart=always --name registry registry:2

Build and tag images for local registry:

$ docker build . -t <your_tag>
$ docker tag <your_tag> localhost:5000/<your_tag>:<version>

Use localhost:5000/<your_tag>:<version> as the image reference in deployments.

Remote Container Repository

For remote repositories (ECR, GCR, Docker Registry), use the registry-creds addon for authentication:

$ minikube addons configure registry-creds
$ minikube addons enable registry-creds

Note on ECR: If using AWS ECR without a role ARN, provide a placeholder value like “changeme” rather than leaving it blank to prevent deployment failures.

Reference the pull secret in deployments:

imagePullSecrets:
  - name: awsecr-cred

Helm

Helm functions as a package manager and configuration tool for Kubernetes. Installation and initialization:

$ brew install kubernetes-helm
$ helm init

Helm currently uses Tiller as its backend, deployed during initialization. Verify deployment with:

$ kubectl describe deploy tiller-deploy --namespace=kube-system

This setup provides a complete local Kubernetes environment for testing deployments before production deployment.

Building General Purpose Blockchain

Sun, 08 Jul 2018 00:00:00 +0000

towards blockchain solution for any micro service ecosystems

The author describes an academic experiment to develop a general-purpose blockchain system that goes beyond introductory tutorials. The goal is creating a blockchain that functions as a supplementary backend component within multi/microservice architectures, removing single points of failure while strengthening data security through decentralization and replication.

What is General Purpose Blockchain

The author references extensive documentation resources and notes the industry is still defining blockchain’s actual use cases beyond financial applications. They propose that blockchain functions as specialized storage for sensitive data and transactions, integrated with existing microservice ecosystems through command and event buses.

Base Set of Features

Tool, Not Star

The solution should be:

Easy to deploy
Highly configurable to adapt to system needs
Simple to integrate via RESTful API

Multiple Consensus Algorithms

The author recommends supporting several approaches:

PoW (Proof-of-work): Computationally challenging to create but easy to verify
DPoS (Delegated proof-of-stake): Representative nodes chosen by network clusters
DBC (Delegated block creation): Scheduled resource consumption for private networks

Multi-chain Capability

Support for multiple, differently-structured chains within a single application.

Denormalization

Convert blockchain data into conventional databases or reporting systems.

Built-in Encryption

Default encryption with configurable field-level protection.

DevOps Integration

Docker support, manifests, and versioned API contracts.

Administrative Features

Thin UI for user management, chain forking, encryption control
Analytics dashboard tracking usage, chain sizes, and performance metrics

Nice-to-Have Features

Custom code execution (“contracts”)
Socket integration
CLI client

Implementation Approach

The author notes the project exists at https://github.com/metaphorical/quantum-blockchain with documentation, Trello board, and contribution guidelines. Development continues part-time toward an MVP version 1.0, welcoming community contributors.

Blockchain as a Solution — Not the Blockchain Guide You Want, But the One You Deserve

Wed, 28 Feb 2018 00:00:00 +0000

This is part two of a short blockchain series. The first article explored foundational concepts in cryptocurrency.

Blockchain as a Niche of Safety

The author emphasizes timing’s importance in technology adoption. Artificial neural networks, though 75 years old as a computational model, only became practically viable when supporting infrastructure matured sufficiently. This same perspective applies to blockchain’s future applications—some are currently feasible while others await further technological development.

Death, Taxes and Real Estate — Break Free From Bureaucracy

Public records worldwide suffer from inconsistencies: missing ownership documentation, vague tax recordings, and tampered birth/death records. Access control remains problematic—who can view public real estate data, who grants access, and how is it managed?

Blockchain addresses these concerns through its original design. The technology emerged as a solution for notary databases and has since evolved into its current form.

Keeping Track of Changes — Untouchable Data

Real estate functions primarily as an investment, making ownership history critical. This information requires privacy protection despite regulatory oversight. Public notification should only occur when properties are in transition or dispute.

In a typical transaction, data keys exchange between parties to verify property value and ownership details. Once confirmed, a validation process checks for irregularities, disputes, or conditions affecting the transaction. Upon clearing all checks, new ownership records on the blockchain while other transaction elements (money transfer, etc.) occur outside it.

For birth, marriage, and death certificates, blockchain provides clean, immutable records without requiring extensive node networks—though crucially, different stakeholders must operate separate nodes.

Tax Collection and Insurance Claims — Triggering the Actions

Taxation and insurance present greater complexity. Beyond storing untouchable data, meaningful system improvements require substantial development. Major insurance companies and governments are researching this space through initiatives like the b3i blockchain initiative and China’s blockchain tax research.

Current research centers on Ethereum’s “smart contracts”—the blockchain’s capacity to execute preprogrammed actions when specific conditions materialize.

Global Distribution of Local Controls

Implementation requires interconnected blockchain networks operating at global scale, performing varied validations and inter-network exchanges to determine conditions for executing complex financial contracts.

Consider this scenario: A person relocates mid-year, changing employment and residence while tax obligations span two countries with different regulations. Birth records transfer to residence-processing blockchains, connecting visa dates to contract dates. Taxation blockchains exchange data applying new rules. Identity data links to newly formed files in global private record systems. The complete system recalculates, validates, stores, and executes tax returns accounting for work in multiple jurisdictions, potentially including property ownership and recent mortgages.

“It would just require you to request this change with your private encryption key (which can be retinal scan).” This thought experiment, while technologically feasible, requires unrealistic expectations regarding coordination.

The future likely involves either distributed global networks providing granular role-based access to private and governmental data ledgers, or centralized global governance. Neither scenario appears imminent.

“Bare in mind that it took several months (maybe even years?) for people to adopt P2P networks as a way to safely download entertainment content.” Implementing more complex, less essential systems will require decades (assuming humanity avoids self-destruction).

Every Day Blockchain Application — Importance of Truth

Modern commerce and other sectors require enhanced auditing, verification, and event tracking reliability. Current experiments aim to integrate blockchain into these areas, providing greater trustworthiness.

Humanitarian Aid: This sector desperately needs spending controls, transport verification, and origin-to-destination tracking. Blockchain could provide secure, distributed identity solutions benefiting UN bodies, NGOs, and international organizations. Progress appears slow, though UNICEF members have discussed blockchain with experts.

Fair Trade and Direct Trade: These initiatives’ core purpose involves maintaining records proving products benefit their producers directly. Distributed data systems can verify complete product paths from production through distribution. However, “Educating people to reward fair and direct trade products can’t be handled through blockchain.”

Organic Food: Similar tracking applies here, emphasizing locations, seeds, fertilizers, and distribution networks.

Freshness: Shelf time significantly affects fruit, vegetable, and flower pricing. This domain sees substantial data forgery due to financial incentives. Major US food retailers support blockchain initiatives for tracking and automatic expiration recall.

Vehicle History: Mileage, maintenance records, and tune-ups present persistent tracking challenges, particularly regarding odometer fraud during vehicle sales.

Virtually any sector requiring reliable, immutable, distributed databases could benefit from blockchain implementation.

Blockchain Beyond Database — Remember SETI

Code execution during block creation enables distributed computing. Ethereum and similar implementations allow blockchain node operators to opt into code execution participation.

Current applications primarily involve contracts—checking network condition satisfaction before executing inter-party transactions. However, blockchain-based code execution could revive projects like SETI@home. Many SETI@home volunteers redirected computing power toward Bitcoin and cryptocurrency mining, but distributed systems could theoretically provide supercomputer-scale processing for scientific endeavors globally.

Technical maturity obstacles remain significant.

Cryptocurrencies Are the Solution and a Problem

Cryptocurrencies enable important technological concepts addressing major gaps, but progress carries costs.

Direct Cost: Freedom and anonymity generate illegal marketplaces for drug trafficking, weapons, hired violence, human trafficking, and worse. The Silk Road marketplace exemplified this problem. The FBI seized over 200,000 Bitcoin from it—representing one direct cost of progress.

Hidden Cost: The cryptocurrency community should consolidate technologies and resources. Thousands of currencies cannot address Bitcoin mining’s energy inefficiency. Mining remains computationally intensive, with processors consuming maximum power.

“Bitcoin mining in the world is consuming almost as much power as Switzerland.” This year’s consumption likely exceeds Switzerland’s total energy use. Since mining concentrates in low-electricity-cost regions burning coal for power generation, Bitcoin carries a substantially larger carbon footprint than Switzerland.

“Our children do not need Bitcoin (crypto currencies in general) if mining does not get lot more energy optimized, but I think they can find pretty good use for a blockchain based solutions.” Blockchain research and adoption remain essential, necessitating continued field exploration.