Engineering journal
Insights.
Notes on cloud architecture, delivery, platforms and the systems underneath them, written for the people who build and run them.
- DevOpsAlerting on error budget burn rateHow multi-window, multi-burn-rate alerts in Prometheus page on real user impact, turn slow degradation into tickets and stay quiet the rest of the time.October 13, 20264 min read
- Platform EngineeringGolden paths are an interface, not a templateWhy the most important design decision in an internal developer platform is what developers declare, and what the platform decides for them.October 13, 20263 min read
- Infrastructure as CodeStructuring Terraform for many environmentsHow to lay out modules, state and environment definitions so that adding an environment is a small, reviewable change rather than a copy-paste exercise.October 13, 20263 min read
- Platform EngineeringAn internal LLM gateway as a platform capabilityWhy a single internal endpoint in front of all model providers simplifies keys, quotas, logging, cost attribution and failover, and how to introduce one without slowing teams down.September 29, 20265 min read
- AI InfrastructureConnecting AI assistants to internal systems with the Model Context ProtocolHow MCP servers expose internal tools and data to AI assistants, and how to authenticate, scope, deploy and audit them so that an assistant can only do what the user could do.September 1, 20266 min read
- SecuritySecuring LLM applicationsA practical guide to prompt injection, unsafe output handling, over-privileged tools, data leakage and logging in applications built on large language models, mapped to the OWASP Top 10 for LLM Applications.August 4, 20266 min read
- AI InfrastructureSelf-hosting open-weight models vs using hosted model APIsHow cost structure, utilization, latency, data control and operational load differ between running open-weight models on your own GPUs and calling a hosted model API, and when a mix of both makes sense.July 7, 20266 min read
- Infrastructure as CodeManaging many cloud accounts with TerraformHow to keep state, credentials and modules organized when Terraform manages dozens of AWS accounts, and how to plan and apply across all of them from CI.June 9, 20265 min read
- AutomationAutomating back-office document workflows with LLMsHow to turn invoices, contracts and forms into validated structured data with a language model, a JSON schema, deterministic checks and a human review queue that catches what the model gets wrong.May 12, 20266 min read
- Platform EngineeringKubernetes autoscaling from pods to nodesHow the Horizontal Pod Autoscaler, KEDA and Karpenter fit together, how to choose the metric each one scales on, and the interactions that cause slow scale-ups and flapping.April 14, 20265 min read
- AI InfrastructureLetting AI agents act on production safelyThe controls that make it reasonable to give an AI agent access to real infrastructure, from scoped tool permissions and dry runs to human approval, audit logs, rate limits and a kill switch.March 17, 20265 min read
- DevOpsRolling out OpenTelemetry in the right orderA practical order of work for adopting OpenTelemetry across many services, from a Collector gateway and auto-instrumentation to tail sampling, semantic conventions and backend choice.February 24, 20265 min read
- CloudDisaster recovery that works when you need itHow to set recovery point and recovery time objectives per system, choose between backups and replication, and prove recovery with scheduled restore tests and game days.January 27, 20266 min read
- AI InfrastructureTesting LLM features like the rest of your softwareHow to build an offline evaluation set, combine deterministic checks with graded ones, and run them in CI whenever a prompt or model changes, while tracking cost and latency.December 30, 20255 min read
- AutomationAuto-remediation with EventBridge and Lambda, without the surprisesHow to build event-driven operational automation on AWS that fixes known problems on its own, survives duplicate and failed events, and cannot take production down with it.December 2, 20253 min read
- SecurityRolling out default-deny NetworkPolicies in KubernetesHow to move a cluster from allow-all to default-deny networking without an outage, covering rollout order, DNS egress, namespace selectors, CNI support and how to test the result.November 4, 20256 min read
- Platform EngineeringDORA metrics without the gamingHow to compute deployment frequency, lead time, change failure rate and recovery time from data you already have, the pitfalls that make them meaningless, and how to use them to help teams rather than rank them.October 7, 20255 min read
- AI InfrastructureUsing an LLM to triage alerts, carefullyHow to give a language model the context it needs to summarize and triage alerts, from runbooks to recent deploys, while keeping it read-only, keeping a human in charge, and measuring whether it actually helps.September 9, 20255 min read
- Infrastructure as CodePolicy as code for Terraform plansChecking Terraform plan JSON with OPA and Conftest in CI, starting with a handful of rules that prevent real damage, and handling exceptions without turning policy into a blocker.August 19, 20255 min read
- Platform EngineeringKubernetes cluster upgrades without dramaFinding deprecated APIs before they break, respecting version skew, tuning node pool surge and choosing between in-place upgrades and blue/green clusters.July 22, 20255 min read
- DevOpsDatabase schema migrations without downtimeHow to change a PostgreSQL schema under live traffic with expand and contract, batched backfills, lock-aware DDL and a deploy order that never leaves old code facing a schema it cannot read.June 24, 20255 min read
- AI InfrastructureThe infrastructure behind retrieval-augmented generation in productionWhat it takes to run RAG beyond a prototype, from ingestion pipelines, chunking and embeddings to choosing between pgvector and OpenSearch, keeping the index fresh, enforcing access control and measuring retrieval quality.May 27, 20256 min read
- CloudCloud cost allocation and showback that teams trustHow to attribute cloud spend to the teams that cause it, with a small tagging standard that is enforced, account-per-team boundaries, activated cost allocation tags and a fair split of shared Kubernetes clusters.April 29, 20256 min read
- SecuritySigning container images with cosign and enforcing them at admissionHow to sign images keylessly in CI with Sigstore cosign, verify who built them, and stop unsigned or wrongly signed images from running in Kubernetes with the Sigstore policy controller or Kyverno.April 1, 20255 min read
- AutomationRunbooks as codeHow to turn the manual steps in incident runbooks into versioned, reviewed automation with approvals and audit trails, starting with the tasks on-call repeats most often.March 4, 20255 min read
- Platform EngineeringEphemeral preview environments for every pull requestHow to give each pull request its own short-lived environment on Kubernetes, with a namespace, a URL, seeded data and automatic teardown, without letting the bill grow with every open branch.February 4, 20255 min read
- AI InfrastructureSizing GPUs for LLM inferenceHow to estimate GPU memory for serving a large language model, from weight size by precision to KV cache per token, and how batch size, context length and quantization trade against each other.January 14, 20255 min read
- Platform EngineeringStructuring GitOps with Argo CD: app-of-apps or ApplicationSetsHow to organize Argo CD applications across many services and clusters, when app-of-apps still fits, how git and cluster generators replace hand-written Applications, and how sync waves and separate config repositories keep it manageable.December 17, 20245 min read
- DevOpsStructured logging that helps during incidentsHow to make application logs useful when something is broken, with JSON output, a consistent field schema, request and trace IDs, sensible levels, and a lid on cost.November 19, 20245 min read
- SecurityLeast-privilege IAM in practice: start broad, then tightenA workable process for AWS IAM least privilege using Access Analyzer policy generation, last-accessed data, permission boundaries and policy checks in CI.October 22, 20245 min read
- DevOpsMaking CI fast with caching that stays correctHow to cache dependencies and Docker layers in GitHub Actions, design cache keys that invalidate at the right time, and avoid caches that slow builds down or break them.September 24, 20245 min read
- CloudPlanning VPC IP address space before you need itHow to size VPC CIDR blocks, keep ranges unique across accounts and on-premises networks, lay out subnet tiers, and leave room for Kubernetes pod addresses.August 27, 20245 min read
- Infrastructure as CodeRefactoring Terraform state with moved and import blocksHow to rename resources, extract modules, adopt existing infrastructure and split state files using moved, import and removed blocks, with plans you can review.July 30, 20245 min read
- Platform EngineeringLiveness, readiness and startup probes done rightWhat each Kubernetes probe should check, why liveness probes on dependencies turn small incidents into outages, and how to choose timing values.July 9, 20245 min read
- AutomationWriting operational scripts that are safe to rerunHow to write shell scripts for operations work that can be run twice without damage, fail loudly when something goes wrong, and know when they have outgrown Bash.June 11, 20246 min read
- Platform EngineeringWhat to put in a Backstage software catalog, and how to keep it trueWhich entities and fields belong in catalog-info.yaml, how to model ownership, systems and APIs, and the automation that stops the catalog from drifting out of date.May 14, 20245 min read
- DevOpsContainer image hygiene for productionMulti-stage builds, minimal and distroless base images, non-root users, pinned digests, immutable tags and scanning, and how they fit together into one build pipeline.April 16, 20245 min read
- CloudDesigning for availability-zone failureHow to keep a service running when one availability zone goes away, from static stability and per-zone capacity to load balancer behavior, data replication and actually testing a zone loss.March 19, 20246 min read
- SecurityMoving secrets out of env files and CI variablesHow to put application secrets in AWS Secrets Manager or Vault, sync them into Kubernetes with External Secrets Operator, rotate them without redeploying by hand, and keep each team's access narrow.February 20, 20245 min read
- Platform EngineeringPodDisruptionBudgets and safe node drainsHow PodDisruptionBudgets interact with node drains and cluster upgrades, why minAvailable and maxUnavailable behave differently, and the configurations that block upgrades or fail to protect anything.January 23, 20245 min read
- Infrastructure as CodeVersioning Terraform modules without breaking every consumerHow to publish shared Terraform modules with semantic version tags, consume them through a registry or pinned git refs, and roll upgrades out across many stacks.January 2, 20245 min read
- DevOpsTrunk-based development: keeping main releasableHow short-lived branches, feature flags and enforced checks let a team merge to main many times a day without breaking what ships.December 5, 20235 min read