I started my career as a full stack developer, and I still remember how much I wished someone had explained the engine room kindly. This is that explanation, written for people who are brilliant at product and would simply like engineers' language translated.
Each module gives you the short answer, a plain English picture, a diagram, what engineers mean when they use the phrase, the questions worth asking, the signs that something deserves a second look, something to try yourself and three quick questions. Alongside it sits LGTM, my notes on the ideas I'd approve.
Everything here works with a click. If you'd rather find your way by typing, the command line at the top of the Console is yours. Type help and press GO to see every command it understands. Then try path to choose a Learning Path, define latency to look up a term or quiz apis to test yourself.
~/console/guide
Computer Science For Product People
18 modules, written for people who are brilliant at product and would simply like the engine room explained kindly. No maths and no code unless it truly helps. Every module ends with three questions, so you can check what's landed.
I admire the work of Irene Yu, founder of Skiplevel and a former Amazon software engineer, who has spent years teaching technical literacy to product managers. If you'd like a teacher to take you further, her work is a wonderful place to begin. Listen to her conversation on how technical a product manager should be.
~/console/guide/00-the-absolute-basics
00. The Absolute Basics
Last Updated 9 November 2026 · 12 min read
The Short AnswerEvery digital product is a set of parts that pass messages to each other. Once you can name the parts and follow the messages between them, architecture conversations, API discussions and diagrams all start to make sense.
This module is a grown-up tour of the parts you'll see on almost any architecture diagram and how they connect. It follows one real journey, a customer buying a train ticket in an app, from the moment they press Pay to the moment the confirmation arrives.
(customer) [mobile app] ---- HTTPS ----> [API gateway]
|
+---------------------+---------------+
v v v
[tickets service] [payments service] [accounts]
| : |
v : event v waits for an answer
[database] : (external) [payment provider]
v
[queue] ...> [email service] ...> confirmation email
---> a call that waits for an answer ...> a message that doesn't wait
The Journey From Button To Confirmation
When the customer presses Pay, the mobile app sends a request over the internet. It doesn't go straight to one big program. It first reaches an API gateway, the single front door that checks who is asking and passes the request to the right place.
The gateway hands the request to a tickets service, which checks the seat is still free in its database. The tickets service then asks a payments service to take the money, and the payments service asks an outside payment provider, such as Stripe or Adyen, to charge the card. Each of those steps waits for an answer before the next can happen, which is why the customer sees a spinner.
Once the payment succeeds, the tickets service records the booking and announces that a ticket was sold. It doesn't wait for the email to be written. An email service picks that announcement up from a queue and sends the confirmation a few seconds later. The customer already has their answer on screen.
The Parts You'll See On Almost Any Diagram
The client is whatever the customer uses: a mobile app, a website in a browser or sometimes another company's system. The front end is the part of the client that people see and touch. The back end is everything on the servers behind it.
Most modern back ends are split into services, each owning one job and its own data, such as tickets, payments or accounts. A database is where a service keeps its records. A cache keeps copies of frequently requested answers close by so they come back quickly. Object storage holds files such as images, documents and PDFs.
Third-party services are parts you rent rather than build: payments, sign-in, email, maps and, increasingly, AI models. They appear on diagrams as boxes outside your boundary, and every arrow to one of them is a dependency on someone else's reliability, pricing and contract.
How The Parts Talk: APIs
Parts talk through APIs, agreed ways to ask for something and get a predictable answer. Each specific thing you can ask for lives at an endpoint, an address such as /tickets or /payments. The asking part sends a request and the other part returns a response.
Requests use a small set of verbs called HTTP methods. GET reads something, POST creates or submits something, PUT or PATCH changes something and DELETE removes it. So "it's a POST to /orders" simply means the app is submitting a new order.
Every response carries a status code. Codes in the 200s mean it worked. 401 and 403 mean the caller isn't signed in or isn't allowed. 404 means the thing doesn't exist, 429 means the caller is asking too often and anything in the 500s means the server failed. The data itself usually travels as JSON, a tidy text format of labelled values.
Most APIs need to know who is calling. A person usually signs in once and receives a token, a temporary pass that travels with each request. A system usually identifies itself with an API key. OAuth is the standard behind buttons such as Sign In With Google, which let one service act for you without ever seeing your password.
Waiting Or Not Waiting
A synchronous call waits for an answer, like the payment step. It's simple to understand, but if the part being called is slow or down, the caller is slow or stuck too, and so is the customer.
An asynchronous message doesn't wait. The sender puts it on a queue or announces an event, and whichever parts care pick it up when they're ready. That keeps the customer's journey fast and means a busy or broken part can catch up later without losing anything. A webhook is the same idea between companies: the payment provider calls your system when a refund completes, rather than you asking every few minutes.
For product people this is one of the most useful questions on any diagram: which arrows make the customer wait? Those arrows decide how fast the journey feels and what the customer sees when something fails.
The Traffic Controllers
Before a request reaches your services, a few parts direct the traffic. DNS turns your domain name into an address. A CDN keeps copies of images, scripts and pages in data centres close to customers. A load balancer spreads incoming requests across several copies of a service, so no single copy is overwhelmed. The API gateway checks sign-in tokens, applies rate limits and sends each request to the right service.
You'll rarely discuss these parts in a planning meeting, but they explain a lot of incidents. A DNS change, a CDN serving an old version or a load balancer sending traffic to an unhealthy copy can each make a product look broken while every service is fine.
Where It All Runs
Services run on servers, usually rented in the cloud from providers such as Amazon Web Services, Microsoft Azure or Google Cloud. Most are packaged as containers, which bundle a service with everything it needs so it runs the same way anywhere. Some small jobs run as serverless functions that only exist, and only cost money, while they're working.
The same system usually exists in several environments: development on engineers' machines, a shared test or staging copy, and production, the real one customers use. Settings that differ between them, such as which payment account to use, live in configuration rather than in the code.
How It Changes Safely
Because so many parts depend on each other, changes are versioned. Many teams use semantic versioning, three numbers such as 3.4.1. The last number changes for fixes, the middle one for new features that don't break anything and the first one for changes that may break whatever was built on the previous version.
Backwards compatible means a change still works for everything built on the old version, such as last year's app that some customers never update. Deprecated means something still works for now but is scheduled to be removed. Both matter whenever a roadmap line says "upgrade" or "migrate", because the real work is usually in everything that connects to the part being changed.
The Leadership View
For a Head of Product or CPTO, the architecture is the operating model in another form. The way services are split usually mirrors the way teams are organised, a pattern engineers call Conway's Law, so drawing a new team boundary and drawing a new service boundary are effectively the same decision. Ask who owns each box before you ask what it does.
Every arrow on the diagram carries cost and risk. Waiting calls to third parties put another company's uptime and pricing inside your customer journey, so each one belongs in a supplier conversation as well as a technical one. Queues and events add resilience but make behaviour harder to trace, which means good monitoring has to come before more features.
When an executive team asks how long a change will take, the honest answer depends on how many boxes and owners it crosses. A feature inside one service is usually quick. A feature that touches the client, three services and a partner's API is a coordination problem first and a coding problem second.
What Engineers Mean When They Say It
"Which service owns that?"
Which part is responsible for that data and its rules, and therefore which team needs to change it.
"It's a synchronous call to the payment provider."
The customer waits while we talk to an outside company, so its speed and reliability become ours.
"We'll put it on a queue."
We'll do that work in the background, so the customer doesn't wait and nothing is lost if a part is busy.
"The gateway handles auth."
The front door checks who is calling, so individual services needn't repeat the check.
"We'd need a new endpoint."
We'd have to add a new kind of request to the API, which other teams or partners may also need to adopt.
"That's a breaking change for the mobile app."
Customers on older versions of the app would stop working unless we keep the old behaviour running alongside the new.
Questions Worth Asking
Could you walk me through, box by box, what happens when a customer presses this button?
Which of these arrows does the customer wait for, and what do they see if one fails?
Which parts belong to us, which belong to another team and which belong to another company?
The Smell Test
A diagram where every arrow is a waiting call, so one slow part slows the whole journey.
Nobody can say which service owns a piece of data.
Try It Yourself
Pick one button in your product, such as Pay or Submit. Ask an engineer to sketch, in ten minutes, every box a request touches between that button and the final confirmation. Label each arrow with what travels along it and whether the customer waits for it. You'll understand your architecture better in that ten minutes than in a month of meetings.
1. What usually happens when an app sends a POST request?
Show The AnswerA) It asks the server to create or submit something, such as an order. GET reads, POST creates or submits, PUT and PATCH change and DELETE removes.
2. Why would a team put a task on a queue?
Show The AnswerA) So the customer doesn't wait while slow work finishes, and nothing is lost if a part is busy. Queues let work happen in the background and catch up safely after a busy spell.
3. What does an API gateway do?
Show The AnswerA) It acts as the single front door for requests, checking who is calling and routing them. Gateways handle sign-in checks, rate limits and routing in one place.
4. A response comes back with status code 401. What does that usually mean?
Show The AnswerA) The caller isn't signed in, often because a token is missing or expired. 401 and 403 are about identity and permission. The 500s are server failures.
5. A dependency moves from version 2.8 to 3.0. What should a product person expect?
Show The AnswerA) Changes that may break whatever was built on version 2. Under semantic versioning, the first number changes when something may break.
The Short AnswerWhen someone opens a website or app, their device looks up the server's address through DNS, opens an encrypted connection and then exchanges a series of HTTP requests and responses. How fast and reliable that feels depends on distance, the number of round trips and the providers sitting in the path.
In Plain English
When a customer opens a retailer's site to check out, their browser first needs the address of the server. It asks DNS, the Domain Name System, which turns a name such as andersonco.uk into an IP address, the number that identifies a machine on the network. Each answer is cached for a set period called its TTL (time to live). That's why a DNS change can take minutes or hours to reach every customer.
The browser then opens a connection to that address. For HTTPS it first runs a TLS handshake, which agrees encryption keys and checks the site's certificate to prove the server is authorised to act for the domain. TLS stands for Transport Layer Security. Only then does the browser send an HTTP request, such as GET /basket, and receive a response containing a status code, headers and a body. One page usually triggers dozens of further requests for images, scripts, fonts and data.
Many of those requests never reach the retailer's own servers. A CDN, or content delivery network, stores copies of static files such as product images and scripts in data centres close to customers. Requests that need fresh or personal data, such as the basket total or the card payment, travel on to the origin, which is the company's own infrastructure. Distance, the number of round trips and the size of each file largely decide how fast the page feels.
1 [browser] -- andersonco.uk? -----> [DNS resolver]
[browser] <-- IP address ---------- [DNS resolver]
2 [browser] -- TLS handshake ------> [CDN edge]
3 [browser] -- GET /logo.png ------> [CDN edge] (cached)
4 [browser] -- GET /basket --------> [CDN edge] --> [origin]
[browser] <-- 200 OK + JSON ------ [CDN edge] <-- [origin]
Steps 1 and 2 are usually reused for the rest of the visit.
Step 3 never reaches the origin. Step 4 needs fresh,
personal data.
The Leadership View
Speed and availability are commercial features. Google's Core Web Vitals measure how quickly a page loads, how quickly it responds to a tap and how much its layout shifts while loading. Google says good scores align with what its search ranking systems reward, though relevance to the search still counts for far more. A Head of Product should treat page performance as a product metric that sits beside conversion. Measure it from real customers' devices at the 75th percentile, which is how Core Web Vitals are assessed, rather than from a fast office laptop.
Every request passes through providers you don't run: a DNS host, a CDN, a cloud region and often a separate security service. Outages at large CDN and cloud providers have taken many well-known sites offline at once. Adding a second DNS provider, a second CDN or a second cloud region reduces that risk but adds cost and operational complexity. The executive team needs to know which single providers sit in the checkout path, what an hour without each one costs and how much resilience the company has chosen to buy.
Ownership of this layer is often unclear. Certificates, DNS records and CDN rules frequently sit with a platform team or one long-serving engineer. An expired certificate remains a common cause of avoidable outages. Give the edge a named owning team, automate certificate renewal and put performance budgets on the roadmap so new features can't quietly slow every page.
What Engineers Mean When They Say It
"It's a DNS issue."
The lookup from name to address is failing or wrong, or a change hasn't reached everyone because older answers are still cached until their TTL expires.
"We need to purge the CDN cache."
The CDN is still serving old copies of files, so we have to tell it to discard them and fetch fresh ones from the origin.
"It's cached at the edge."
A copy is stored on the CDN's servers near customers, so those requests never reach our own servers.
"The certificate has expired."
The TLS certificate that proves our identity has lapsed, so browsers block the site with a security warning.
"We're seeing 5xx errors at the edge."
The CDN or load balancer is returning server errors, usually because our origin servers are failing or unreachable.
"It's round-trip bound."
Most of the delay comes from the number of back-and-forth exchanges over a long distance, so faster servers alone won't help much.
Questions Worth Asking
What share of our traffic does the CDN serve, and which requests still travel to the origin?
Which external providers sit in the path of a checkout request, and what happens to revenue if one is down for an hour?
How do we renew TLS certificates and approve DNS changes, and who gets alerted before a certificate expires?
The Smell Test
Performance reported as a lab score from an engineer's laptop rather than as data from real customers on real devices.
A DNS or certificate change scheduled during peak trading with no tested way to reverse it.
Nobody in the room can name the company that hosts the domain's DNS.
Try It Yourself
Open your product's website on a computer, right-click and choose Inspect, then open the Network tab and reload. Count the requests and sort them by size and time. Use the throttling setting to simulate a slow mobile connection and reload again. Then ask an engineer to show you which requests came from the CDN cache and which went to the origin. Finally, ask what the slowest request is waiting on before it can finish.
Latency · Module on why distance and round trips shape how fast a product feels.
Check Yourself
1. A team moved the site to a new server an hour ago. Some customers see the new site and some still see the old one. What's the most likely reason?
Show The AnswerA) DNS resolvers are still holding the old address until its TTL expires. DNS answers are cached for their TTL, so a change reaches customers gradually. Lowering the TTL before a planned move shortens that window.
2. Customers see a browser security warning and can't reach the site, but every server is healthy. What should the team check first?
Show The AnswerB) Whether the TLS certificate has expired. Browsers block sites whose certificate has lapsed or doesn't match the domain, even when the servers behind it work perfectly.
3. Which request is a CDN most likely to answer without contacting the origin?
Show The AnswerC) A product image. CDNs cache static files that are the same for everyone. Personal or changing data, such as a basket or a payment, goes to the origin.
The Short AnswerAn API is a published contract that defines how one piece of software can request data or actions from another and what it'll get back. Most of the cost and risk of an API lies in changing that contract once other people depend on it.
In Plain English
When you check your balance in a banking app, the app doesn't read the bank's database directly. It sends a request to the bank's API, such as GET /accounts/123/balance, and receives a structured response, usually in JSON. The API defines which requests are allowed, what each one needs, what comes back and what errors look like. Behind that contract the bank can rebuild its systems freely, as long as the requests and responses stay the same.
Most web APIs follow a style called REST, where each resource, such as an account or a payment, has its own URL and standard HTTP methods act on it. GraphQL lets the app ask for exactly the fields a screen needs in one request. gRPC is a fast, compact style used mainly between internal services. Teams often describe the contract in a machine-readable specification called OpenAPI, which can generate documentation, tests and client code automatically.
Because other teams, partners and older versions of your own app rely on the contract, changing it is the hard part. Adding an optional field is usually safe. Renaming a field, removing it or changing its meaning is a breaking change. Payments add a further concern: if a train ticket app retries a payment after a timeout, the API mustn't charge the customer twice. Well-designed payment APIs accept an idempotency key, a unique ID sent with each request, so a repeated request returns the original result instead of creating a second charge.
[bank app] -- GET /accounts/123/balance --> [API gateway]
|
checks token and rate limit
v
[accounts service]
|
[bank app] <-- 200 OK + JSON balance ------ [API gateway]
[payment provider] ... webhook: refund done ...> [our endpoint]
---> a request that waits for a response
...> a call the other system makes when an event happens
The Leadership View
An API is a product with its own customers, even when those customers are internal teams. Once a partner has built against an API, every change needs their cooperation, and support costs grow with each version kept alive. Treat public and partner API decisions with the seriousness of a pricing change. Each one needs a named owner, a published deprecation policy, a changelog and a realistic support period.
Team design follows the contracts. Teams that own services behind clear, documented APIs can release independently, which is one of the main reasons organisations divide a system into services. When contracts are vague, teams have to coordinate every release and delivery slows. Versioning, migrations and partner communication rarely appear in feature plans, so give them explicit space on the roadmap before they cause slippage.
Measure availability and response time for key endpoints against an agreed service level objective, or SLO. Track error rates by status code, adoption of the newest version and the time a new partner takes to make a first successful call. For third-party APIs, the executive team needs to hear which suppliers sit in revenue-critical journeys and what uptime their contracts promise. They also need the cost per call at forecast volume and the fallback if a supplier changes its terms or fails.
What Engineers Mean When They Say It
"That's a breaking change."
Existing callers would fail unless they change their code, so we'd need a new version or a managed migration period.
"We're being rate limited."
The provider is rejecting some of our requests, usually with a 429 status, because we've exceeded our allowance per minute or hour.
"Is that endpoint idempotent?"
If the same request arrives twice, is the result the same as if it arrived once, with no duplicate order or charge?
"It's a webhook."
The other system sends an HTTP request to our endpoint when an event happens, so we don't have to keep polling it.
"We're deprecating v1."
Version 1 still works for now, but callers have a set period to move before it's retired.
"Check the OpenAPI spec."
The machine-readable contract lists every endpoint, field and error, so it should settle the question before anyone books a meeting.
Questions Worth Asking
Who calls this API today, including older app versions and partners, and how will each of them hear about a change?
What does the customer see when this dependency is slow, returns errors or rate limits us?
What's our deprecation policy, and how long are we committing to support each version?
The Smell Test
An integration estimate given before anyone has read the provider's documentation or made a test call.
Payment or order endpoints that retry after a timeout without idempotency keys.
A partner API in a revenue-critical journey with no contractual terms on uptime, pricing changes or notice periods.
Try It Yourself
Paste api.github.com/users/octocat into your browser's address bar. What comes back is real API output in JSON, so spot three fields a product might show a customer. Then change the name to one that can't exist and see the 404 and its error message. Finally, ask an engineer to open the OpenAPI specification for one of your own APIs and show you the error responses a caller can receive.
The Absolute Basics · Module 00, which covers requests, methods, status codes and tokens.
Check Yourself
1. A partner says your release broke their integration, though you only renamed one field in a response. What happened?
Show The AnswerA) Renaming a field is a breaking change for anyone who reads the old name. Callers depend on field names. A rename needs a new version or a period when both names are returned.
2. A customer's app times out while paying and automatically retries. What stops the customer being charged twice?
Show The AnswerB) An idempotency key sent with the request. With an idempotency key the API recognises the repeat and returns the original result instead of creating a second charge.
3. When does a webhook make more sense than polling?
Show The AnswerC) When you need to know promptly about events that happen at unpredictable times. A webhook lets the other system notify you the moment something happens, which avoids thousands of wasted checks.
The Short AnswerA database stores the data a product must not lose and lets many users read and change it safely at the same time. The choice of database and the shape of the data decide what the product can do quickly, what it can guarantee and what it costs.
In Plain English
Every product holds data it must not lose: a bank's balances, a retailer's orders, a streaming service's subscriptions and viewing history. A database stores that data and lets software read and change it while thousands of customers act at once. It also protects the data through crashes and keeps backups for recovery.
Most core business data lives in relational databases such as PostgreSQL, MySQL or SQL Server. Data sits in tables of rows and columns, and tables link to each other through keys, such as an order row holding its customer's ID. The design of those tables is called the schema. Programs read and change data using SQL, the Structured Query Language. An index is a separate sorted structure that lets the database find matching rows without scanning the whole table. Indexes can make reads much faster, though they use storage and slow writes slightly.
Relational databases support transactions, groups of changes that either all succeed or all fail. When a customer pays for an order, the payment and the order are saved together or not at all. Other databases suit other jobs. Document databases such as MongoDB store flexible records, and key-value stores such as Redis answer simple lookups very fast and often act as a cache. These are often grouped as NoSQL. Data warehouses such as Snowflake or BigQuery still use SQL, but they're built to answer analytical questions across years of history. Large systems also keep copies of data in several places, which raises a question of consistency. A streaming service can show a viewing history that's a few seconds behind, but a bank balance can't be.
TABLE: customers TABLE: orders
+----+-----------+---------+ +-----+-------------+--------+
| id | name | city | | id | customer_id | total |
+----+-----------+---------+ +-----+-------------+--------+
| 1 | A. Patel | Leeds | | 901 | 1 | 42.50 |
| 2 | J. Morgan | Cardiff | | 902 | 2 | 18.00 |
+----+-----------+---------+ +-----+-------------+--------+
KEY: orders.customer_id points to customers.id
INDEX: on customers.city, so a search for Leeds
skips straight to the matching rows
The Leadership View
Data usually outlives the code around it. Applications get rebuilt every few years, while the data model and the data itself can last decades. Early decisions about what a customer, an order or an active subscriber means become expensive to change, because reports, integrations and machine learning models all depend on them. Treat core definitions as product decisions, each with a named owning team and a written meaning that every dashboard uses.
The headline risks are loss, breach and wrong numbers. Backups only count once a restore has been tested. Ask for two figures and evidence that a drill has proved them. The recovery point objective (RPO) is the most recent stretch of data the business accepts losing, such as the last 15 minutes of orders. The recovery time objective (RTO) is the target time to have the service running again. UK GDPR requires personal data to be limited to what's needed and kept no longer than necessary. Data held without a clear purpose adds legal exposure and breach risk, so data minimisation and retention rules belong on the roadmap.
Database choices shape cost and team design. Managed cloud databases remove routine operational work but charge for storage, compute and data transfer, and warehouse bills can climb quickly as query volume grows. When several teams write to the same tables, their releases become tied together. Each service should own its data and share it through APIs or events. A board needs to hear where the critical data lives, whether recovery has been proven, what the regulatory exposure is and how the cost is trending.
What Engineers Mean When They Say It
"We need a migration."
We have to change the structure of stored data, such as adding a column or splitting a table, while the product keeps running.
"That query needs an index."
The database is scanning far more rows than necessary, so the request is slow and will slow further as data grows.
"It has to be one transaction."
These changes must succeed or fail together, so we never record a payment without its order.
"We read from a replica."
We query a copy of the main database to spread the load, so results can lag slightly behind the latest changes.
"It's eventually consistent."
Copies of the data in different places can briefly show different values before they match.
"What's the source of truth?"
Which single system holds the authoritative version of this data when copies disagree?
Questions Worth Asking
What are our recovery point and recovery time for this data, and when did we last prove them with a real restore?
Which team owns the definition of an active customer, and does every dashboard use it?
What personal data are we holding without a clear purpose, and when is it deleted?
The Smell Test
A schema migration on a large table with no rollback plan and no test against production-sized data.
Two dashboards showing different numbers for the same metric, and nobody able to say which is right.
Several teams' services writing directly to the same database tables.
Try It Yourself
Ask an engineer to show you the schema for the table that stores your customers or orders: the list of columns and their types. Pick one column and ask which service writes it, who reads it and what it means. Then ask when the last full restore test took place and how long it took to complete.
Further Reading
SQLBolt · Short, free and interactive lessons in SQL, right in your browser.
Use The Index, Luke · Markus Winand's free guide to how database indexes work and what makes queries slow.
APIs · Module 02, on how services share data without sharing a database.
Check Yourself
1. A system crashes after saving a customer's payment but before saving their order. What should have prevented the mismatch?
Show The AnswerB) Saving both changes in a single transaction. A transaction makes related changes succeed or fail together, so the database never holds a payment without its order.
2. A report gets slower every month as the orders table grows. What fix is an engineer most likely to suggest first?
Show The AnswerA) Add an index on the columns the query searches. Without a suitable index the database reads every row, so the query slows as the table grows.
3. The team says backups run every night. What's the most useful follow-up question?
Show The AnswerB) Whether a full restore has been tested recently and how long it took. A backup only proves its value once a restore succeeds. The drill also shows the real recovery time.
The Short AnswerTechnical debt is the extra effort future changes will take because of how a system was built or how it has aged. Some is a sensible trade for speed, but left unmanaged it slows delivery and raises risk.
In Plain English
Some technical debt is deliberate. A retailer launching a Black Friday promotion might hard-code the discount rules to hit the date, knowing they'll need rebuilding before the next campaign. Some is accidental: the team's understanding of the problem improves and the original design no longer fits. Some simply accumulates as frameworks, libraries and cloud services move on while the code stays still. The term was coined by Ward Cunningham in 1992.
Debt appears in specific forms, each with its own fix and cost. Code may be hard to change because responsibilities are tangled together, which engineers call tight coupling. Automated tests may be missing, so every change needs slow manual checking. Libraries and frameworks may be past end of life and no longer receive security fixes. Infrastructure may have been configured by hand, so nobody can rebuild it with confidence.
The cost shows up as product outcomes. A streaming service might need six weeks to add a new subscription tier because pricing logic is copied across several systems. A banking app might need a week of manual testing before every release. The usual remedy is steady work inside normal delivery. Refactoring improves the internal structure of code without changing its behaviour. Larger systems are often replaced in stages, routing one function at a time to the new system until the old one can be retired.
deliberate shortcut -------> launch sooner
|
v
recorded: what, why, owner, date to fix
|
v
cost if left alone: slower changes
more incidents
unpatched dependencies
|
v
reduce steadily: refactor and add tests in normal work
replace in stages: move one function at a time
The Leadership View
Debt is a portfolio decision that product and engineering own together. The real trade-off is speed now against speed and safety later, and the right balance depends on context. An experiment that may be deleted next month can carry a lot of debt. A payments platform expected to run for ten years can't. A Head of Product should be able to say which debt was taken on deliberately, why it was worth it and when it'll be addressed.
Make debt visible and fund it predictably. Many teams reserve a steady share of capacity for maintenance and debt reduction instead of waiting for a large rewrite, which is hard to fund and carries high delivery risk. Link each item to an outcome the business values: faster delivery of a roadmap theme, fewer incidents, a closed security risk or lower cloud cost. That turns an argument over engineering preferences into an ordinary prioritisation decision on the same backlog as features.
Track trends rather than a single debt score. Useful measures include the DORA metrics, especially change lead time and change fail rate, plus how often incidents trace back to the same components. The age of critical dependencies and the time similar changes take quarter by quarter are telling too. A board needs to know where debt creates material risk, such as unsupported software handling customer data. It also needs the cost in delivery speed and the funded plan to reduce it. Debt grows fastest in systems no team owns, so make sure every system has an owning team.
What Engineers Mean When They Say It
"That part of the code is fragile."
Small changes there break things elsewhere, usually because the code is tightly coupled and poorly tested.
"We need to refactor."
We want to restructure the code without changing its behaviour, so future changes are faster and safer.
"It's legacy."
It's older software that still runs important work but is hard to change, often with few tests and few people who understand it.
"That version is end of life."
The vendor no longer issues security fixes for it, so staying on it is a growing security and compliance risk.
"There's no test coverage there."
No automated tests check that area, so every change relies on manual checking.
"We should rewrite it."
They want to replace the system rather than improve it, which is high-risk unless the replacement happens in stages.
Questions Worth Asking
Which debt is slowing the next two roadmap themes most, and what would fixing it change for customers?
Which of our systems run on versions that no longer receive security patches?
What share of our capacity goes on unplanned work and incidents, and is that share rising?
The Smell Test
Estimates rising for similar changes in the same area, quarter after quarter.
"We'll fix it after launch" said for the third launch running, with no ticket, owner or date.
A proposal for a full rewrite with no plan for running the old and new systems side by side.
Try It Yourself
Ask your engineers to name the three areas of the product they're most reluctant to change. For each one, ask what it's slowing on the current roadmap and what fixing it would deliver. Put all three on the same backlog as features, each with an owner and a measurable outcome.
Testing · Module on automated tests, the main defence against fragile code.
Check Yourself
1. What is refactoring?
Show The AnswerB) Restructuring code without changing its behaviour. Refactoring makes future changes faster and safer while customers see no difference, which is why it needs protected time in plans.
2. Which signal best suggests debt in one area is growing?
Show The AnswerA) Similar changes there take longer each quarter. When the same kind of change keeps getting slower, the structure of that area is costing the team more over time.
3. When is taking on technical debt a reasonable choice?
Show The AnswerC) When it's deliberate, recorded and has a plan to reduce it. Deliberate, tracked debt can buy real speed. Untracked debt is what becomes a problem.
The Short AnswerThe cloud is computing power, storage and managed services rented from a provider such as Amazon Web Services, Microsoft Azure or Google Cloud and paid for as you use them. You trade owning and maintaining hardware for a variable monthly bill, a shared security model and a dependency on that provider.
In Plain English
When a customer presses Play on a streaming service, the request almost certainly reaches machines the streaming company doesn't own. They sit in a provider's data centres, grouped into regions such as London or Frankfurt. Most large regions contain several availability zones, each made up of one or more separate data centres with their own power and networking, so one failure needn't cause an outage. The company rents virtual machines, storage and databases there and pays by the second, the hour or the gigabyte.
Providers sell at different levels. Infrastructure as a service (IaaS) rents raw virtual machines, networks and disks, and your team manages everything installed on them. Platform as a service (PaaS) and managed services go further: the provider runs the database, the queue or the container platform while your team looks after its own code and data. Software as a service (SaaS) is a finished product, such as Salesforce or Workday, that you configure rather than build. Serverless functions, such as AWS Lambda, run your code only when an event arrives, such as a file upload, and charge for each call and its running time rather than for idle servers.
Consider a retailer's checkout during Black Friday week. On premises, the retailer would buy enough servers for its busiest hour and leave most of them idle for the rest of the year. In the cloud, autoscaling adds copies of the checkout service as traffic climbs and removes them afterwards. The flexibility is real, but costs now grow with usage and data leaving the provider's network, known as egress, is charged. The provider secures the data centres, hardware and underlying network. The retailer stays responsible for its own configuration, access rights and data. That split is called the shared responsibility model.
customers
|
v
[load balancer] region: eu-west-2 (London)
|
+----------------------+
v v
ZONE A ZONE B
[checkout] x3 [checkout] x3 <- autoscaled
[db primary] --------> [db standby] copies every write
| |
+--> [object storage] <+ images, receipts
YOU OWN: code, configuration, access rights, data
PROVIDER OWNS: data centres, hardware, network, host software
The Leadership View
At Head of Product and CPTO level, the cloud is mainly a question of unit economics and dependency. Cloud spend moves with product decisions. A feature that keeps every upload forever, or calls an AI model on every page view, has a running cost that its business case should show. The most useful measure is cost per unit of value, such as cloud cost per order, per active customer or per thousand streams. Track that trend over time rather than watching the total bill alone.
Ownership matters as much as architecture. Many organisations adopt FinOps, a practice in which engineering, finance and product share accountability for cloud spend. Each team sees and owns the cost of the services it runs, which only works if every resource is tagged with a team and a product. Commitments such as savings plans or reserved instances can cut the price of steady workloads considerably. They also lock in spend for one or three years, so they belong in the same conversation as the roadmap.
Boards and executive teams need three answers in plain terms. What does the cloud cost per unit of business and which way is it trending? Where is customer data held and how is it protected? How does the service cope if a zone, a region or the provider itself fails? UK GDPR doesn't require personal data to stay in the UK, but it does restrict transfers abroad, and many customers and regulators set their own requirements. Multi-cloud is often proposed as a way to reduce lock-in, yet it adds real cost and demands deep skills on two platforms. A tested exit plan for the most critical services is often the better answer, and UK financial regulators already expect one for material cloud outsourcing.
What Engineers Mean When They Say It
"We'll autoscale the checkout service."
The platform adds or removes copies of the service automatically as traffic rises and falls, within limits we set.
"It's serverless."
The provider runs our code only when an event triggers it and charges for each run rather than for idle servers. There are still servers, but we never manage them.
"Let's containerise it."
Package the service with its libraries and settings as a container image, so it runs identically on a laptop, in staging and in production.
"We need it multi-AZ."
Run copies across at least two availability zones in the region, so losing one data centre doesn't cause an outage.
"Watch the egress costs."
The provider charges when data leaves its network, so heavy downloads or copies between regions can grow the bill quickly.
"That's ours under shared responsibility."
The provider secures its hardware and buildings, but this setting, such as who can read a storage bucket, is our job.
Questions Worth Asking
What does this feature cost to run per order or per active customer today, and what will it cost at ten times the volume?
Which regions and availability zones host this service, and what do customers experience if one of them fails?
Which managed services tie us most closely to this provider, and what would leaving cost us in time and money?
The Smell Test
The monthly cloud bill rises faster than customers or orders, and nobody can explain which product decisions drove the increase.
Resources with no owner tag, or test environments left running overnight and every weekend.
A disaster recovery plan that exists on paper but has never been tested by actually switching to the standby.
Try It Yourself
Ask finance or your platform team for last month's cloud bill, split by service and by team. Take the three largest lines and ask an engineer which customer journeys each one supports. Then divide the total by orders or active customers to get a first cost per unit. Agree who will report that number each month and where it will appear.
Latency And Scale · Module 06, on keeping a product fast as cloud usage grows.
Check Yourself
1. What does serverless actually mean?
Show The AnswerA) The provider runs your code when an event triggers it and charges for each run rather than for idle servers. There are still servers. Your team simply never provisions, patches or scales them.
2. A storage bucket holding customer files is left publicly readable. Under the shared responsibility model, whose failure is it?
Show The AnswerB) The customer organisation's, because access configuration is its job. Providers secure the infrastructure. Who can read your data is your configuration and your responsibility.
3. Why would a team run a service across two availability zones?
Show The AnswerB) So the failure of one data centre doesn't cause an outage. Zones are separate data centres within one region. You choose regions for distance and data residency, and zones for resilience.
The Short AnswerLatency is the time one request takes, from the customer's action to the response, usually measured in milliseconds and described with percentiles. Scale is a system's ability to keep that latency and its reliability steady as the number of users and requests grows.
In Plain English
When someone opens their account summary in a bank app, the request travels over the mobile network to the bank's servers. It passes through a gateway and an accounts service, reads from a database and travels back. Latency is the total time all of that takes. Part of it is fixed by physics: a round trip between London and Sydney takes around a quarter of a second before any work is done. That's why global products serve images and pages from locations close to customers through a content delivery network (CDN). The rest is the time each service and database spends on the request.
Engineers describe latency with percentiles rather than averages. The p50, or median, is the time within which half of requests finish. The p95 and p99 are the times within which 95 and 99 in every 100 requests finish, so they describe the slowest experiences, often called the tail. If the account summary has a p50 of 300 milliseconds and a p99 of four seconds, one request in every hundred takes four seconds or longer. On a busy payday morning that can mean thousands of frustrated customers, and the average would hide every one of them.
Scale is about what happens as load grows. Throughput is how many requests a system handles per second. As throughput rises, the busiest shared component, often the database, reaches its limit first and latency climbs sharply for everyone. Teams scale out stateless services, which keep no customer data between requests, by running more copies behind a load balancer. They cache answers that are requested often, such as a streaming service's home page rows. They also move slow work onto queues so the customer doesn't wait. Each technique has a cost. Caches can serve slightly stale data, and databases that accept writes are much harder to scale out than stateless services.
OPEN ACCOUNT SUMMARY IN A BANK APP (typical times)
phone --network--> [gateway] -> [accounts] -> [database]
80ms 5ms 20ms 150ms
^ bottleneck
phone <--network-- [gateway] <- [accounts] <------+
80ms total about 335ms
WITH A CACHE IN FRONT OF THE DATABASE
[accounts] -> [cache] 2ms a hit saves about 150ms
miss? -> [database] then store the answer
The Leadership View
For a product leader, speed is part of the product and deserves explicit targets. The standard tools come from site reliability engineering. A service level indicator (SLI) is the thing you measure, such as the time to load a checkout page. A service level objective (SLO) is the target, such as 95 per cent of checkout loads finishing under 800 milliseconds over 28 days. The gap between the target and perfection is the error budget. When a team has spent its budget, the agreed response is to put reliability work ahead of new features until it recovers. That rule only works if product leadership has agreed it in advance.
Scale is a capacity planning and cost question. Peak events such as a ticket on-sale, Black Friday or a new series launch are known in advance. The roadmap should therefore hold load testing and capacity work weeks before them, not days. Ask for the highest load at which the system has actually been tested, expressed as requests or orders per second, and compare it with the forecast peak plus a safety margin. Over-provisioning wastes money every day, while under-provisioning fails on the one day that matters most.
Ownership is where performance most often goes wrong. Latency adds up across services, so a journey touching five teams can be slow while every team meets its own target. Give each critical journey a named owner for its end-to-end latency. Measure it from real users' devices as well as from servers, and report it alongside conversion. A board needs to hear whether the product is fast enough where it counts, whether it held up at the last peak and what the next peak will require.
What Engineers Mean When They Say It
"What's the p99?"
The time within which 99 in every 100 requests finish. It exposes the slowest experiences that averages hide.
"We've got a tail latency problem."
A small share of requests are far slower than the rest, usually because of a slow dependency, a busy database or retries.
"Can we cache it?"
Can we keep a ready answer in fast memory so repeat requests skip the slow work? The cost is that the answer may be slightly out of date.
"It's stateless, so we can just add copies."
The service keeps no customer data between requests, so we can run more copies behind a load balancer without coordinating them.
"The database is the bottleneck."
The database reaches its limit first, so adding application servers won't help until we reduce or spread the database load.
"We need to load test before the on-sale."
We'll simulate the expected peak traffic in advance to find the point where latency climbs or errors begin.
Questions Worth Asking
What are the p95 and p99 for this journey as real users experience it, and what target did we agree with the business?
What's the highest load we've actually tested, and how does it compare with our forecast peak?
When traffic doubles, which component reaches its limit first and what's our plan for it?
The Smell Test
Performance reported only as an average, or measured only on servers rather than on customers' devices.
Load testing scheduled after launch, or never run against production-sized data.
Every team meets its own latency target while the end-to-end journey is still slow.
Try It Yourself
Open pagespeed.web.dev and test your product's most important page on mobile. If your site has enough traffic, the report shows how real users experienced Largest Contentful Paint (LCP), Interaction to Next Paint (INP) and Cumulative Layout Shift (CLS) at the 75th percentile. Then ask your team for the p95 and p99 server response times behind the same journey. Agree one target together and track it every sprint.
Further Reading
Web Vitals (web.dev) · Google's guide to LCP, INP and CLS, its measures of loading, responsiveness and visual stability for real users.
The Cloud · Module 05, on the infrastructure and costs behind scaling.
Check Yourself
1. A checkout page has an average load time of 600 milliseconds and a p99 of five seconds. What does that tell you?
Show The AnswerB) One load in every hundred takes five seconds or more, which the average hides. Averages smooth away the slow tail. Percentiles show what the slowest customers actually experience.
2. Why is a database usually harder to scale than an application service?
Show The AnswerA) It holds shared data that every copy must agree on, especially when data changes. Stateless services can simply be copied. Shared data needs coordination, so teams use read replicas, caching or splitting the data.
3. What is an error budget?
Show The AnswerB) The amount of unreliability an SLO allows before reliability work takes priority. If the SLO is 99.9 per cent, the error budget is the remaining 0.1 per cent.
The Short AnswerSecurity means making sure only the right people and systems can see and change the right things, keeping data protected wherever it's stored or sent and spotting problems quickly. Many serious breaches start with ordinary weaknesses such as broken access control, misconfiguration or stolen passwords.
In Plain English
Take a bank app. When a customer signs in with a password and a code from their phone, or with a passkey, the bank is doing authentication: proving who they are. A passkey is a sign-in credential held on the customer's device or in their password manager and unlocked with a fingerprint, a face scan or a PIN. When the app then shows only that customer's accounts, it's doing authorisation: deciding what an authenticated person may see and do. Many of the most damaging bugs live in authorisation. If changing an account number in a web address shows someone else's statement, authentication worked perfectly and authorisation failed. That class of flaw, broken access control, sits first in the OWASP Top 10 2025 list of web application risks.
Data needs protecting in two states. Encryption in transit uses Transport Layer Security (TLS), the protocol behind the padlock in a browser, so nobody on the network can read or alter data as it travels. Encryption at rest protects stored data, so a stolen disk or backup is unreadable without the keys. Neither helps when someone signs in legitimately with stolen credentials. That's why access control, multi-factor authentication and monitoring matter as much as encryption.
Much of security is unglamorous hygiene. Least privilege means each person and system gets only the access it genuinely needs. Secrets such as API keys and database passwords belong in a secrets manager, never in code. Software must be patched promptly when a vulnerability is published, which matters because modern products are built largely from open-source libraries the team didn't write. Every significant action, such as an administrator exporting customer data, should be logged so the team can detect misuse and investigate it afterwards.
[customer app] --TLS--> [API gateway] ---> [accounts service]
1 AUTHENTICATE 2 AUTHORISE
token valid? is this account
MFA passed? theirs? role?
|
v
[secrets manager] ..> [database]
keys, passwords encrypted at rest
|
every access and admin action ----------> [audit log]
alerts and review
The Leadership View
At leadership level, security is a business risk to manage across the whole life of a product. It's cheapest when it starts early. A short threat modelling session at design time, in which the team asks how a feature could be attacked or misused, costs far less than a fix after a penetration test or an incident. Security work competes with features for capacity, so give it a visible place on the roadmap. A standing allocation for patching and dependency upgrades is a practical way to do that.
Ownership should be unambiguous. Each product team owns the security of its own code and configuration. A central security function sets standards, provides tooling and helps teams with discovery and design rather than only approving releases at the end. Access needs owners too. When people join, change roles or leave, their access should change the same day, and access to customer data should be reviewed on a regular schedule. Supply chain risk belongs on the list as well, because every library and vendor is a route an attacker can use. Software supply chain failures now rank third in the OWASP Top 10.
Boards and executive teams need risk in business terms: which data and services matter most, what threatens them and how well those threats are controlled. Useful measures include time to patch critical vulnerabilities, multi-factor coverage of staff and admin accounts, the number of people with access to production data and how quickly incidents are detected and contained. Under UK GDPR, a personal data breach must be reported to the Information Commissioner's Office (ICO) without undue delay and, where feasible, within 72 hours of the organisation becoming aware of it. The only exception is a breach that's unlikely to result in a risk to people's rights and freedoms. The response plan therefore has to be rehearsed well before it's needed.
What Engineers Mean When They Say It
"That's an authorisation bug, not authentication."
We know who the user is. The problem is that the system lets them see or do something they shouldn't.
"Is it encrypted in transit and at rest?"
Is the data protected by TLS while it travels and encrypted on disk where it's stored?
"We need least privilege on that role."
Cut the role's permissions back so the person or system has only what it genuinely needs.
"There's a critical CVE in one of our dependencies."
A publicly catalogued vulnerability affects a library we use, so we need to patch or upgrade it quickly.
"Someone committed a key to the repo."
A secret such as an API key was saved into the code history, so we must revoke and replace it rather than simply delete the line.
"Let's threat model it."
Before building, we'll work through how the feature could be attacked or misused and decide which defences it needs.
Questions Worth Asking
What's the worst thing someone could do with this feature if they were signed in as a different customer or were a malicious insider?
Who has access to production customer data today, and when did we last remove access nobody needs?
How long do we take to detect a breach and to patch a critical vulnerability, and how do we know?
The Smell Test
A security review booked for the week before launch, with no threat modelling during design.
Shared admin accounts, admin screens without multi-factor authentication or secrets stored in code.
Nobody can say how many people are able to export customer data.
Try It Yourself
Run a 45-minute threat modelling session on one feature in your current sprint. Work through four questions: what are we building, what can go wrong, what will we do about it and did we do a good enough job? Record the top three risks and turn each one into a backlog item with a named owner.
OWASP Top 10:2025 · The ten most critical web application security risks, drawn from contributed data and a community survey.
Incidents · Module 14, on how teams detect, handle and learn from failures.
Check Yourself
1. A customer changes the account number in a web address and sees someone else's statement. What kind of failure is this?
Show The AnswerB) Authorisation, because the system didn't check the account belongs to them. The customer was correctly identified. The system failed to check what they were allowed to see, which is broken access control.
2. Why isn't encryption at rest enough on its own?
Show The AnswerA) Anyone who signs in with stolen credentials still sees the data decrypted. Encryption protects against stolen disks and backups. Access control, multi-factor authentication and monitoring protect against stolen identities.
3. A secret API key turns up in the code repository. What's the right response?
Show The AnswerB) Revoke the key, issue a new one and move it into a secrets manager. Once a secret has been exposed, assume it's been copied. Deleting it from the code doesn't remove it from the history.
The Short AnswerTesting gives a team evidence that a change does what it should and hasn't broken anything else, mostly through automated checks that run on every change. Releasing is how that change reaches customers safely, ideally in small, reversible steps kept separate from deploying the code.
In Plain English
Take a train company adding seat selection to its booking app. Unit tests check small pieces of logic in isolation, such as the rule that a child ticket can't be booked without an adult. Integration tests check that parts work together, such as the seat service writing correctly to its database. End-to-end tests drive the real app through a whole journey, from search to payment confirmation. Teams usually write many fast unit tests, fewer integration tests and only a handful of end-to-end tests. The broader a test is, the slower and more fragile it tends to be. That shape is known as the test pyramid.
Continuous integration (CI) means every change is merged into the main codebase frequently and the automated tests run on it each time. If any fail, the build goes red and the change can't progress. Continuous delivery (CD) extends this so every change that passes is ready to release at any time. Continuous deployment goes one step further and sends every passing change to production automatically. Before production, many teams check changes in a staging environment, a copy of the live system kept as close to production as practical.
Deploying and releasing are different acts. Deploying puts new code into production. Releasing makes a feature available to customers. A feature flag separates the two: the seat selection code can be deployed on Tuesday, switched on for staff on Wednesday and opened to 5 per cent of customers on Thursday. A canary release works at the infrastructure level instead. It sends a small share of traffic to the new version and compares its error rates and latency with the current version. If the numbers worsen, the team rolls back or turns the flag off, often within minutes.
code change (pull request)
-> CI: build, unit and integration tests red? stop and fix
-> merge to the main branch
-> deploy to staging, end-to-end tests fail? stop and fix
-> deploy to production, flag OFF no customer sees it
-> flag ON for staff, then 5% canary errors? flag OFF
-> 25% -> 50% -> 100% of customers watch the metrics
-> remove the flag from the code keep it clean
The Leadership View
The strongest evidence on software delivery comes from the DORA research programme. Over many years it has found that speed and stability tend to improve together rather than compete. Its software delivery metrics are change lead time, deployment frequency, failed deployment recovery time, change fail rate and deployment rework rate. They help a team track its own progress over time. They become harmful when used to rank teams against each other or tied to individual targets, because people then optimise the number rather than the outcome.
Release practice shapes product strategy. Teams that ship small changes behind flags can run experiments, launch gradually by market or customer segment and separate marketing dates from engineering dates. That capability needs investment the roadmap should show. It includes test automation, a pipeline fast enough to use many times a day and the discipline to remove old flags, which otherwise become technical debt.
Every release needs clear owners: who decides it goes live, who watches the health metrics afterwards and who can disable it. A board or executive team doesn't need test coverage figures. It needs to know how often the organisation can ship customer value, what share of changes cause problems and how fast it recovers. It should also hear whether critical journeys, such as checkout or payments, can be changed safely during a peak period.
What Engineers Mean When They Say It
"It's behind a feature flag."
The code is in production but switched off, so we choose who sees it and when, without another deployment.
"We'll canary it at 5 per cent."
Send 5 per cent of traffic to the new version, compare its errors and latency with the current version, then widen or roll back.
"CI is red."
The automated checks on the latest change are failing, so nothing merges or ships until they're fixed.
"That test is flaky."
It passes and fails unpredictably on the same code, which erodes trust in the whole suite and needs fixing or removing.
"We do blue-green deployments."
We run the new version alongside the old one and switch all traffic across in one step, so switching back is just as quick.
"Is the database migration backwards compatible?"
Will the current version of the code still work after the database change, so we can roll back without losing data?
Questions Worth Asking
If this release goes wrong, how will we know within the first 15 minutes, and who can disable it?
What would it take to release this in smaller pieces behind a flag?
Which tests would have caught our last three serious incidents, and do those tests exist now?
The Smell Test
Large, infrequent releases bundling weeks of changes and scheduled for a Friday afternoon or the night before a peak.
A release process that depends on manual testing or one person's knowledge for a critical journey.
Dozens of feature flags that nobody has removed, some switched on for years.
Try It Yourself
Ask your team to walk you through the last release, from code change to the moment customers saw it. Write down each step, mark which ones were automatic and note how long the whole journey took. Then pick one manual step to automate this quarter and put it on the roadmap with a named owner.
1. What's the difference between deploying and releasing?
Show The AnswerB) Deploying puts code into production and releasing makes a feature available to customers. Feature flags let teams deploy code safely and release it later to chosen customers.
2. What is a canary release?
Show The AnswerA) Sending a small share of traffic to a new version and comparing its health with the current one. If errors or latency rise in the small group, the team rolls back before most customers are affected.
3. According to the DORA research, how do delivery speed and stability usually relate?
Show The AnswerB) They tend to improve together. Small, frequent changes are easier to test, review and reverse, which helps both speed and stability.
The Short AnswerMachine learning builds a model from historical examples so a system can predict the answer for a new case, such as whether a card payment is fraudulent. Every prediction is a probability, so the product has to be designed for the cases the model gets wrong.
In Plain English
A bank's fraud check is a good example. Engineers take millions of past card payments, each labelled genuine or fraudulent. A training process then finds the patterns that separate them, such as the amount, the merchant, the time of day and the distance from home. The result is a model, a file of learned parameters that takes the details of a new payment and returns a score between 0 and 1. The product then decides what each score triggers: approve, send the customer a text to confirm or decline.
That's supervised learning, where every training example has a known answer called a label. The common task types are classification, which picks a category such as fraud or genuine, and regression, which predicts a number such as a delivery time. Ranking orders a streaming service's recommendations, and clustering groups similar customers without any labels at all. The large language models behind chat assistants are machine learning models too. They're trained on vast amounts of text to predict the next token, a word or part of a word, and the same rules about evaluation and failure still apply.
A model is judged on data it never saw during training, called a holdout or test set. The right measures depend on what each kind of error costs. Precision asks how many of the payments the model flagged were really fraud. Recall asks how many of the real frauds it caught. Raising the threshold usually improves precision and lowers recall, so setting the threshold is a product decision about which mistake customers and the business can better afford.
TRAINING (offline)
past payments + labels --> [training] --> [model]
|
evaluate on holdout data, then deploy
v
SERVING (live)
new payment --> [features] --> [model] --> fraud score 0 to 1
|
score at or above threshold --> decline or check
score below threshold ------> approve
:
outcomes + new labels ...> [monitoring] ...> drift? ...> retrain
The Leadership View
The first leadership decision is whether a problem needs machine learning at all. A rule written by an analyst is cheaper to build, explain and audit. Machine learning earns its place when the patterns are too many or change too quickly for rules, and when there's enough historical data with reliable labels. The second decision is whether to build a model, buy a product or call a foundation model through an API. Building needs data science and machine learning engineering capacity. Buying adds a vendor dependency. Model APIs usually charge per token, so the unit cost grows with every customer interaction.
A model carries an operating cost long after launch. It needs a data pipeline, monitoring for drift, periodic retraining, a named owner and a route for customers to challenge a decision. Teams that ship a model and move on often find accuracy falling months later with nobody accountable. Fund machine learning features with a run budget as well as a build budget. In credit scoring, recruitment and life or health insurance pricing, explainability and bias testing carry legal weight. Annex III of the EU AI Act lists these uses as high-risk, although it expressly excludes systems used only to detect financial fraud.
Measure the business outcome as well as the model score. For a fraud model, that means fraud losses, false declines and the customer contacts they cause, tracked over time and broken down by customer group. A board needs three clear answers: which decision the model makes or supports, what happens to a customer when it's wrong and how the organisation will know if performance degrades.
What Engineers Mean When They Say It
"We need more labelled data."
We need more examples where the correct answer is already recorded, and someone may have to create those labels by hand.
"It's overfitting."
The model scores well on its training data but worse on new data, because it learned noise and quirks as well as the real pattern.
"What's the threshold?"
Which score must a prediction reach before the product acts on it automatically?
"Precision's up but recall dropped."
Fewer of its alerts are false alarms, but it's now missing more of the real cases.
"The model has drifted."
Live data or real-world behaviour has changed since training, so performance is falling.
"There's leakage in the features."
The training data included information that won't exist at prediction time, so test results look better than live results will be.
Questions Worth Asking
What does a false positive cost us, what does a false negative cost us and who chose the threshold that trades one against the other?
How was the model evaluated, on which holdout data and broken down by which customer groups?
Who owns this model after launch, and what triggers a retrain or a rollback?
The Smell Test
Accuracy quoted as a single number with no precision, recall or cost per error. If one payment in a thousand is fraud, a model that never flags anything is 99.9 per cent accurate.
A launch plan with no monitoring, no retraining budget and no named owner.
Offline results that look too good, which often points to leakage between training and test data.
Try It Yourself
Ask the owner of a machine learning feature in your product for twenty recent cases: ten where the model was confidently wrong and ten that sat just below the threshold. Review them together as a customer would. Then ask what each kind of error costs the business and whether today's threshold still reflects that.
1. A fraud model's precision rises but its recall falls. What has happened?
Show The AnswerA) Fewer of its alerts are false alarms, but it now misses more real fraud. Precision measures how many flagged cases were real. Recall measures how many real cases were caught. Moving the threshold usually trades one against the other.
2. A model scores brilliantly in testing and poorly in production. Which is the likeliest explanation?
Show The AnswerA) Overfitting or data leakage meant the test results flattered the model. Testing on unseen holdout data is good practice. Overfitting and leakage make offline results look better than live performance.
3. Why does a deployed model need a named owner and a monitoring plan?
Show The AnswerA) Data and customer behaviour change over time, so performance can drift. Drift is gradual and silent, so someone has to watch the measures and decide when to retrain.
The Short AnswerVersion control records every change to a codebase, who made it, when and why, so many engineers can work in parallel and any change can be traced or undone. Git is the version control system almost every software team uses, usually hosted on a platform such as GitHub, GitLab or Bitbucket.
In Plain English
Git stores a project's history as a series of commits. A commit is a snapshot of the code at a moment in time, with an author, a timestamp, a unique identifier and a message explaining the change. Each commit records its parent, the commit before it, so the history can be followed back to the very first change. If a bug appears in a retailer's checkout, an engineer can find exactly which commit introduced it, who made it and the reasoning recorded at the time.
Engineers work on branches, separate lines of history that start from main, the shared branch holding the current agreed version. When a change is ready, the engineer opens a pull request on the hosting platform. GitLab calls the same thing a merge request. Automated checks run the tests, and at least one colleague reviews the code line by line. Once it's approved, the branch is merged into main. LGTM, short for Looks Good To Me, is what a reviewer often writes to approve it.
Git holds more than application code. Infrastructure as code, configuration, database migrations and documentation often live in the same repository, so they get the same review and history. Many teams practise trunk-based development, merging small changes into main at least daily and hiding unfinished features behind feature flags. That keeps branches short-lived and reduces merge conflicts, which happen when two changes alter the same lines and Git can't decide which version to keep.
The repository is the most reliable record of what a team actually built. Pull request data shows how work flows: how big changes are, how long they wait for review and how often they're reverted. These are early signals of delivery health, visible weeks before a missed date. DORA's long-running research ties small batches and frequent integration to stronger delivery. It measures delivery through change lead time, deployment frequency, change fail rate, failed deployment recovery time and deployment rework rate.
Review practice is a team design decision. Code owners, required approvals and branch protection rules decide who can change what. They also provide audit evidence for frameworks such as SOC 2 and PCI DSS, where auditors expect each production change to be authorised, tested and traceable. Relying on one senior reviewer creates a bottleneck, and requiring too many approvals slows everything without making it safer. Agree review expectations openly, including a target time for the first review.
Branching strategy shapes the roadmap more than most leaders realise. Long-lived feature branches lead to large, risky merges and big-bang releases. Trunk-based development with feature flags lets the team merge continuously and turns release timing into a product decision: staff first, then a small percentage of customers, then everyone. A board rarely needs Git detail, but it should hear that every production change is reviewed, traceable and reversible.
What Engineers Mean When They Say It
"I'll raise a PR."
I'll open a pull request so colleagues can review my change and the automated checks can run.
"It's merged into main."
The change is in the shared branch and will go out with the next deployment, unless we already deploy every merge automatically.
"There's a merge conflict."
Two changes altered the same lines, so an engineer has to combine them by hand.
"Let's revert that commit."
Let's add a new commit that cleanly undoes an earlier one, keeping the history intact.
"Can you rebase onto main?"
Please replay your commits on the latest version of main so the history stays tidy and any conflicts surface now.
"It's behind a flag."
The code is merged and deployed but switched off, so we can release it to customers separately.
Questions Worth Asking
What's our median time from opening a pull request to merging it, and where does most of the waiting happen?
How many pull requests last month changed more than a few hundred lines, and why couldn't they be smaller?
If tonight's release goes wrong, how quickly can we identify the change responsible and revert it?
The Smell Test
Pull requests that wait days for a first review, or always wait for the same person.
Very large changes approved within minutes, which suggests review has become a formality.
Feature branches that live for weeks and end in a painful merge just before a release.
Try It Yourself
Sit with an engineer and open three recent pull requests in your team's repository. For each one, note the size of the change, how long it waited for a first review, what the reviewers said and whether the description explains why the change was needed. Agree one improvement together, such as a size guideline or a target time for first review.
Further Reading
Pro Git · The free official book, with excellent early chapters.
1. A pull request has waited four days for a first review. What's the most useful leadership response?
Show The AnswerA) Look at review capacity and expectations, because waiting time is a delivery bottleneck. Review wait time is one of the clearest early signals of slow delivery, and it's usually a capacity or habit problem.
2. What does reverting a commit do?
Show The AnswerA) Adds a new commit that undoes an earlier change while keeping the history. A revert keeps the full history, so the team can always see what happened and why.
3. Why do many teams pair trunk-based development with feature flags?
Show The AnswerA) Unfinished work can be merged into main daily but stay switched off for customers. Flags separate deploying code from releasing a feature, which keeps branches short and releases controlled.
The Short AnswerThe front end is the code that runs on the customer's device and draws what they see and touch. The back end runs on servers the company controls, where it stores data, enforces the business rules and connects with other systems.
In Plain English
Take a streaming service. The app on a smart TV, phone or browser is the front end. On the web it's built from HTML, CSS and JavaScript, and native mobile apps are usually written in Swift for iPhone and Kotlin for Android. It lays out the rows of programmes, plays the video and remembers where the viewer paused. The back end decides what this account may watch, checks the subscription is paid, stores viewing history and produces the recommendations. The two communicate through APIs, usually as requests and JSON responses over HTTPS.
The dividing line matters most for trust. Anything that runs on the customer's device can be inspected and altered by a determined user. The front end can check a form for convenience, but the back end must check it again. In a retailer's checkout, the browser may show the basket total, yet the server must recalculate the price, apply the discount rules and confirm stock before taking payment. A discount enforced only in the browser is a discount anyone can award themselves.
A few patterns shape how the two sides divide the work. A single-page application, often built with a framework such as React, loads once and then fetches data as the customer moves between screens. Server-side rendering builds the page on the server first, which helps first-load speed and search engines. Many teams add a backend for frontend, a thin back-end layer designed around one client's needs, such as the TV app. Mobile apps add one more constraint: customers update them on their own schedule, so the back end must keep supporting older app versions for months.
FRONT END (customer's device) BACK END (company's servers)
[web: HTML, CSS, JavaScript] --+
[iPhone app: Swift] -----------+-- HTTPS + JSON --> [API]
[Android app: Kotlin] ---------+ |
+---------+---------+
v v v
[rules] [database] [payment
provider]
Checks on the left are convenience: the user can alter them.
Checks on the right are trust: only the company controls them.
The Leadership View
Front end and back end usually involve different skills and toolchains, and often different teams, so the split shapes how work is planned. A feature that needs both sides is a coordination problem. Estimates that only describe the visible screen routinely miss the API change, data migration and back-end rules that sit behind it. Many organisations now form cross-functional teams that own a customer journey end to end, supported by a platform or design system team that provides shared components. That trades some consistency across journeys for speed within each one.
Every extra client multiplies cost. A web app, an iPhone app and an Android app mean three codebases, three release processes and three sets of tests, and app store review adds a step the team doesn't control. Cross-platform frameworks such as React Native and Flutter reduce duplication, at some cost in platform fit. Native apps also leave old versions in customers' hands, so the back end inherits a long tail of compatibility work. Choose the client strategy deliberately, because it sets team size and release cadence for years.
Measure the front end the way customers experience it. Google's Core Web Vitals give an agreed vocabulary: Largest Contentful Paint for loading, Interaction to Next Paint for responsiveness and Cumulative Layout Shift for visual stability. Google judges each one at the 75th percentile of real page loads, so a page passes only when most visits meet the target. Google says good scores align with what its search ranking systems reward, although relevant content still matters far more. For apps, add crash-free sessions and start-up time. For the back end, track error rates and response times on the APIs that matter most, such as sign-in and checkout. An executive team needs to know which platforms the company supports, what each costs to keep current and where the rules that protect money and data are enforced.
What Engineers Mean When They Say It
"That's a front-end-only change."
It changes what runs on the customer's device and needs no new API or data, so it's usually quicker and lower risk.
"Validation has to happen on the back end too."
Any check in the browser can be bypassed, so the server must repeat every rule that protects money or data.
"It's server-side rendered."
The server builds the page before sending it, so it appears faster and search engines can read it easily.
"We can't drop that field because old app versions still use it."
Customers on older versions of the mobile app would break, so the back end must keep the old behaviour running.
"We'd need a BFF for the TV app."
We'd add a thin back-end layer that shapes data specifically for the TV app's screens.
"She's full stack."
She works confidently across both the front end and the back end.
Questions Worth Asking
Which parts of this estimate are front end, which are back end and which depend on another team's API?
Which rules in this journey does the server enforce, whatever the screen shows?
What's the oldest app version we still support, and what does supporting it cost us?
The Smell Test
Prices, discounts, eligibility or permissions checked only in the browser or app.
Estimates and demos that show the screen but say nothing about the back-end changes behind it.
No policy for retiring old app versions, so the back end carries every past behaviour indefinitely.
Try It Yourself
Open your product's checkout or sign-up page in Chrome. Right-click, choose Inspect and open the Network tab. Complete the journey and watch each request the front end sends to the back end. Then ask an engineer which of the rules you saw on screen are checked again on the server.
Web Vitals (web.dev) · Google's guide to the three Core Web Vitals and their target thresholds.
Check Yourself
1. A promotion code is validated only in the browser. Why is that a problem?
Show The AnswerA) Anyone can alter code running on their own device, so the server must check the rule too. Front-end checks are for convenience. Rules that protect money or data must be enforced on the back end.
2. Why does a back end often keep supporting older behaviour after a mobile release?
Show The AnswerA) Customers update apps on their own schedule, so older versions stay in use. Old app versions stay in customers' hands, so removing behaviour too early breaks them.
3. What does Interaction to Next Paint measure?
Show The AnswerA) How quickly the page responds visually after a customer taps, clicks or types. Interaction to Next Paint is the Core Web Vital for responsiveness. Layout movement is Cumulative Layout Shift.
The Short AnswerArchitecture is the set of decisions about a system's structure: which components exist, what each one owns, how they communicate and where the data lives. Those decisions are expensive to reverse, so they shape what a team can deliver quickly for years.
In Plain English
Take a train operator's ticketing platform. One design is a monolith, a single application deployed as one unit that handles timetables, fares, bookings, payments and customer accounts. Another splits those jobs into separate services, each deployed independently and owning its own database, which communicate through APIs and events. Both can work well. The monolith is simpler to build, test and run. Separate services let teams release independently and scale the busy parts, such as fare search during a strike, without scaling everything else.
Much of architecture is deciding boundaries. A boundary settles which component owns a piece of data and its rules, so only the bookings service changes a booking, for example. Coupling describes how much one component depends on another's internals. When coupling is tight, a change to fares forces changes in bookings and payments, and three teams must release together. Cohesion is the companion measure: related logic sits together in one place. Many teams draw boundaries around business capabilities such as fares or refunds. Domain-driven design gives them a method, and it calls each well-defined boundary a bounded context.
Architecture also sets the system's quality attributes, often called non-functional requirements: availability, latency, scalability, security, cost and ease of change. These pull against each other. Running in two cloud regions improves availability but costs more and complicates keeping data consistent. A cache makes reads faster but means some customers briefly see stale prices. An architecture decision record, or ADR, captures each significant choice, the options considered and the consequences, so the next team understands why the system looks the way it does.
MODULAR MONOLITH SEPARATE SERVICES
one deployable unit each deployed on its own
+------------------------+ [API gateway]
| fares | bookings | | | |
|----------+-------------| v v v
| payments | accounts | [fares] [bookings] [payments]
+------------------------+ | | : |
| v v : v
[one database] [db] [db] : [db]
v event
[queue] ...> [email]
simpler to build and run independent releases, more
to operate and monitor
The Leadership View
Conway's law observes that systems tend to mirror the communication structure of the organisations that build them. For a Head of Product or CPTO, that makes team design and architecture a single decision. If three teams share one codebase with no clear ownership, every release becomes a negotiation. If team boundaries match service boundaries and customer journeys, teams can ship without waiting for each other. Reorganising teams without changing the architecture, or the reverse, usually produces friction.
The costliest architecture mistakes are premature distribution and long neglect. Microservices bring real operational overhead: network failures between services, distributed tracing, versioned contracts and higher infrastructure bills. A small team usually moves faster with a well-structured modular monolith, splitting out services when a clear scaling or ownership need appears. At the other extreme, a system nobody has invested in for years becomes slow and risky to change. Put architecture work on the roadmap as named investments with a business reason, such as cutting release lead time or removing a single point of failure in payments.
Judge architecture by its effect on delivery and reliability. Useful measures include lead time for changes and the number of teams a typical feature needs. Add availability against agreed service level objectives and infrastructure cost per transaction. Executives and boards need to hear plainly which decisions are hard to reverse and where the concentration risks sit, such as reliance on one cloud or payment provider. They also need the plan for when a critical component fails.
What Engineers Mean When They Say It
"Monolith or microservices?"
Do we deploy one application, or many independently deployable services that each own their own data?
"That's a single point of failure."
If this one component fails, the whole journey stops, because nothing else can take over its work.
"These services are tightly coupled."
They depend on each other's internals, so changing one forces coordinated changes and releases.
"Which service is the source of truth?"
Which component holds the authoritative copy of this data, so every other copy defers to it.
"It needs to scale horizontally."
We should handle more traffic by running more copies of the service rather than buying a bigger server.
"Let's write an ADR."
Let's record this decision, the options we rejected and the consequences, so future teams can see the reasoning.
Questions Worth Asking
Which decisions in this design would be expensive to reverse, and what would reversing them cost?
Where are the single points of failure, including outside suppliers, and what happens to customers if one fails?
How many teams have to coordinate to ship a typical feature, and does this design reduce that number?
The Smell Test
A team of five proposing a dozen microservices without a scaling or ownership problem that requires them.
Nobody can draw the system on one page or say which component owns customer data.
Most features need several teams to release together, which suggests the boundaries are in the wrong places.
Try It Yourself
Ask a senior engineer to draw your product's architecture on one page in fifteen minutes, then trace one customer journey across it together. Mark every single point of failure, every arrow that crosses a team boundary and the component they'd most like to change. Finally, ask whether an ADR exists for the biggest decision on the page.
1. A team of six proposes splitting its product into fifteen microservices. What's the strongest first question?
Show The AnswerA) Which scaling or ownership problem the split solves, and who will operate the extra services. Microservices add operational cost, so they need a clear problem that justifies them.
2. What does Conway's law suggest?
Show The AnswerA) Systems tend to mirror the communication structure of the organisation that builds them. Team design and architecture shape each other, so leaders should plan them together.
3. Why write an architecture decision record?
Show The AnswerA) So future teams understand the choice, the alternatives and the consequences. ADRs preserve the reasoning behind decisions long after the original authors have left the team.
The Short AnswerA data pipeline is an automated process that copies data out of the systems where it's created, checks and reshapes it, then loads it somewhere it can be analysed or used by a model. Every dashboard, report and machine learning model is only as reliable as the pipeline feeding it.
In Plain English
Take a streaming service. Every play, pause and skip is recorded by the app and sent to the back end as an event. Subscriptions sit in a billing system, the catalogue in a content database and support tickets in a third-party tool such as Zendesk. None of those systems was built to show which programmes keep new subscribers past their first month. A pipeline extracts data from each source, joins it together and loads it into a data warehouse. That's a database designed for analysis across very large volumes, such as Snowflake, BigQuery or Amazon Redshift.
The classic pattern is ETL: extract, transform, load. Most modern teams use ELT instead. They load raw data into the warehouse first, then transform it there with SQL, often using a tool called dbt. Keeping the raw copy means a table can be rebuilt whenever the business logic changes. The steps run in a set order, managed by an orchestrator such as Airflow or Dagster, which schedules each job, retries failures and records every run. Good pipelines also run data quality tests at each step, such as checking that an order total is never empty, and stop before bad data reaches a dashboard.
Pipelines run in batches or as streams. A batch job processes data on a schedule, such as every night at 2am, so a retailer's sales dashboard shows yesterday's figures. A streaming pipeline processes each event within seconds of its arrival, usually built on an event streaming platform such as Apache Kafka. That's what lets a bank's fraud models consider a customer's last few minutes of activity when scoring a new payment. Streaming is harder to build and run, so it's worth paying for only where a decision really has to happen in seconds.
SOURCES INGEST WAREHOUSE USES
[app events] --+
[orders DB] --+--> extract --> [raw tables]
[billing] --+ and load |
[support] --+ (batch or v
stream) transform + test --> dashboards
(e.g. dbt) --> ML models
--> reports
orchestrator (e.g. Airflow): schedules, retries, logs each run
The Leadership View
Pipelines are where data strategy becomes a running cost. Warehouse compute is usually billed by the data each query scans or by the seconds of processing used, so a dashboard nobody reads that refreshes every fifteen minutes can quietly cost real money. Freshness is the main lever. Moving from daily to hourly is often cheap, but moving from hourly to seconds usually means a streaming platform, new skills and on-call cover. Before agreeing to fresher data, ask which decision it would change and what that decision is worth.
The common failure is organisational rather than technical. A product team renames a database column or changes an app event, and a pipeline three teams away breaks silently. The remedy is a data contract: an agreed, versioned description of what a source produces, owned by the team that produces it and tested before a change ships. Treat the datasets that matter as products, each with a named owner, documentation and a freshness promise. Make tracking and data changes part of the product team's definition of done, so the central data team isn't left to discover breakages later.
Measure four things: freshness against the agreed target, the share of runs that succeed first time, the time taken to detect and fix data incidents and the warehouse bill per team. A board doesn't need the pipeline detail. It needs to know which published numbers come from tested, owned pipelines, how current they are, where personal data flows and who is accountable when a reported figure turns out to be wrong. That accountability matters more as AI models are trained and evaluated on the same data.
What Engineers Mean When They Say It
"It's an ELT job in dbt."
We load the raw data into the warehouse first, then transform it there with SQL models kept in version control.
"It runs as a nightly batch."
The data refreshes once a day, so today's activity won't appear until tomorrow morning.
"We'd need streaming."
Data would be processed within seconds of each event, which needs different tooling, skills and on-call cover.
"The upstream schema changed."
A source system renamed or removed a field, so every later step that relied on it is failing or producing wrong numbers.
"We'll need to backfill."
We'll rerun the pipeline over past dates to fill gaps or correct history after a fix.
"Check the lineage."
Trace which sources and steps produced a number, so we know what a failure affects.
Questions Worth Asking
Which decision would fresher data change, and what would hourly or real-time data cost compared with today?
If a source team renames a field tomorrow, how would we find out, and who would fix the damage?
Which board or regulatory figures come from this pipeline, and are they covered by automated quality tests?
The Smell Test
Several teams report different numbers for the same metric, and each keeps its own spreadsheet.
Pipeline failures are discovered by the person reading the dashboard rather than by an alert.
Last month's figures change when the report is rerun, and nobody can explain why.
Try It Yourself
Pick one number your leadership team reviews every week. Ask the data team to trace its lineage back to the source systems and name the owner of each step. Ask how old the number is by the time you see it and which test would catch it going wrong. Note any step that has no owner and any number that has no test.
Further Reading
Measuring AI Outcomes · Why the data behind a model matters as much as the model itself.
1. What's the main difference between ETL and ELT?
Show The AnswerA) ELT loads raw data into the warehouse first and transforms it there. Loading raw data first keeps an untouched copy, so tables can be rebuilt when the business logic changes.
2. A sales dashboard is fed by a nightly batch. A director asks why this morning's promotion isn't showing. What's the likeliest explanation?
Show The AnswerB) The pipeline hasn't run since the promotion started. Batch pipelines trade freshness for simplicity and cost. Today's activity arrives with the next scheduled run.
3. A product team renames a field in its orders database and a revenue report breaks. What would most likely have prevented this?
Show The AnswerC) A data contract with tests that run before the change ships. Data contracts make the producing team accountable for changes that would break downstream consumers.
The Short AnswerMonitoring is the continuous measurement of how a live product is behaving, with alerts when it moves outside agreed limits. Incident management is the practised routine a team follows to restore service quickly, keep customers informed and learn from what went wrong.
In Plain English
Take a bank app on payday morning. Monitoring collects three kinds of telemetry. Metrics are numbers over time, such as requests per second, error rate and response time. Logs are timestamped records of individual events, such as a rejected password. Traces follow a single request across every service it touches, so engineers can see that a slow balance screen is really a slow call to the core banking system. Together they give the team observability, the ability to work out what's happening inside a system from what it reports.
Good teams alert on symptoms customers feel rather than on every internal number. They choose a service level indicator (SLI), such as the share of payment requests that succeed within two seconds. They then set a service level objective (SLO), such as 99.9 per cent over 30 days. That target implies an error budget: 0.1 per cent of requests may fail, roughly equal to 43 minutes of complete outage in 30 days. A service level agreement (SLA) is different. It's a contractual promise to customers, usually with financial penalties, and it should be looser than the internal SLO.
When an alert fires, it pages the on-call engineer, who confirms the problem and declares an incident with a severity level. A Sev 1 might mean payments are failing for many customers, which triggers the full response. An incident commander coordinates rather than fixes, and a communications lead updates the status page and customer teams. Engineers restore service first, often by rolling back the latest change, and find the root cause afterwards. The team then writes a blameless postmortem, a review that asks how the system allowed the failure, and tracks every action to completion.
[app] --> [services] --> metrics, logs, traces
|
v
[monitoring] -- SLO at risk? --+
|
page on-call engineer
|
v
DECLARE severity set, incident commander named
RESPOND status page updated, customer teams told
RECOVER roll back or fix, service restored
LEARN blameless postmortem, actions owned and tracked
The Leadership View
Reliability is a product decision with a price. 99.9 per cent availability allows about 43 minutes of downtime in 30 days, while 99.99 per cent allows about four. Each extra nine demands more redundancy, automation and faster response, so it usually costs considerably more than the last. Set targets per customer journey rather than per system. Payments and sign-in may justify a strict target, while a recommendations panel can tolerate far less. Then use the error budget as the agreed rule for pace: while budget remains, the team ships, and once it's spent, reliability work comes before features.
On-call is a question of team design. Teams that build a service should usually run it, because they then fix the things that page them at night. That only works with a sustainable rota, and Google's SRE book suggests at least eight engineers for a single-site team, assuming a primary and a secondary on call in week-long shifts. Engineers on call should be compensated fairly, and alerts should be tuned so every page genuinely needs a person. Alert fatigue is a real retention risk. Reserve roadmap capacity for postmortem actions explicitly, or they'll lose every planning round to new features.
Track SLO attainment for each critical journey and the share of incidents customers reported before an alert did. Track time to detect, time to restore service and the share of incidents caused by recent changes. Track how many postmortem actions are closed on time. Treat average recovery time with care, because a few long incidents distort it. An executive team needs four answers: which journeys missed their targets, what it cost customers and revenue, whether the cause was a known risk and what is now different. The process also needs clear rules on when an incident must be reported outside the company. Under UK GDPR, a personal data breach that's likely to put people at risk must be reported to the ICO within 72 hours of becoming aware, where feasible.
What Engineers Mean When They Say It
"I got paged."
An alert reached the engineer on call by phone or app, and they must acknowledge it within a set time.
"It's a Sev 1."
The highest severity: a core journey is down or badly degraded for many customers, so the full incident process starts.
"We're burning error budget."
Failures are using up the allowance our SLO permits faster than planned, so risky releases may need to pause.
"Roll it back first."
Restore the previous working version now and investigate the cause once customers are unaffected.
"What's the blast radius?"
How many customers, journeys or systems this failure affects.
"Let's run a blameless postmortem."
We'll examine how the system and process allowed the failure, without attributing fault to individuals.
Questions Worth Asking
For our three most important customer journeys, what's the SLO, and how much error budget did we spend last quarter?
In our last five Sev 1 incidents, who spotted the problem first, our alerts or our customers?
Which postmortem actions are still open, and where do they sit on the roadmap?
The Smell Test
Customers or the social media team report outages before the alerts do.
On-call engineers are paged many times a week by alerts that need no action.
Postmortems end with "be more careful", or the same root cause appears twice in a quarter.
Try It Yourself
Ask your engineering lead for the timelines of the last three significant incidents. For each one, note when the problem started, when it was detected, when customers were told and when service was restored. The gaps between those four times show where to invest first.
1. A service has a 99.9 per cent availability SLO over 30 days. Roughly how much complete downtime does its error budget allow?
Show The AnswerB) About 43 minutes. 0.1 per cent of 30 days is about 43 minutes. At 99.99 per cent it falls to about four.
2. What's the difference between an SLO and an SLA?
Show The AnswerA) An SLO is an internal target, while an SLA is a contractual promise to customers, often with penalties. Keeping the SLA looser than the SLO gives the team warning before a contract is breached.
3. During a Sev 1, what's the incident commander's main job?
Show The AnswerC) Coordinating the response, decisions and communication. Separating coordination from hands-on fixing keeps the response calm and the decisions clear.
The Short AnswerAn LLM app puts a large language model inside a product, surrounded by instructions, your own data and checks on its inputs and outputs. An agent goes further: the model decides which steps to take and which tools to call, in a loop, until it judges the task complete.
In Plain English
A large language model (LLM) generates text by repeatedly predicting the next token, a word or part of a word. It works only from its training and from what's in its context window, the text it can consider in a single request. The model itself remembers nothing between requests and knows nothing about your business unless the application supplies it. So an LLM app is mostly ordinary software around the model. A system prompt sets its role and rules, retrieval fetches relevant facts and further code checks the output before a customer sees it.
Take a retailer's customer help feature. When a customer asks about returning a jacket, the app uses retrieval augmented generation (RAG). It searches the returns policy and the customer's order history, often using embeddings, numerical representations of meaning stored in a vector database. It then places the most relevant passages in the prompt. The model writes its answer from that material, which makes an accurate answer much more likely and lets the app cite its sources. Without retrieval the model would answer from its general training and might invent a policy, which is called a hallucination.
An agent lets the model act. It's given tools, each a described function such as look_up_order or issue_refund. The model asks for a tool, the application runs it and the result goes back to the model, which then chooses the next step. A returns agent might check the order, confirm eligibility, create a label and email the customer without anyone scripting the sequence. Many teams now connect tools through the Model Context Protocol (MCP), an open-source standard that Anthropic introduced in late 2024 for connecting AI applications with external tools and data. Anthropic's published guidance recommends starting with simpler, fixed workflows and moving to agents only when the steps genuinely can't be predicted in advance.
TASK: "I'd like to return my jacket"
|
v
[app] -- system prompt + task + retrieved docs --> [LLM]
^ |
| "call look_up_order(1234)" |
+<-------------------------------------------------+
|
+--> app runs tool --> [orders API] --> result to LLM
|
+--> issue_refund? --> human approves --> [payments API]
|
v
DONE: every step logged, step and spend limits enforced
The Leadership View
The economics differ from ordinary software. Every request costs money in tokens, and agents multiply that cost because each step resends a growing context. A task that costs pennies in a demo can cost far more at volume or when an agent loops. Model prices change often, so model the business case per completed task rather than per request. Use caching, smaller models for simple steps and hard limits on steps and spend. Latency follows the same pattern, because a five-step agent is rarely as fast as a single call.
The main risk is what the system can do rather than what it says. Prompt injection, where instructions hidden in an email, web page or document take control of the model, has no complete technical fix. It still heads the OWASP Top 10 for LLM Applications in its 2026 edition. Excessive agency, giving an agent more permissions than its task needs, has climbed to third place on the same list. Design for containment: least-privilege credentials, read-only access by default, human approval for anything irreversible or financial and a full log of every step. Agree who owns an agent's actions before launch, because legal, customer service and engineering will each assume someone else does.
Evals become the product's specification. They're a curated set of real cases with agreed good answers. They're scored automatically or by another model, spot-checked by people and run on every prompt, model or tool change. Product managers should own the definition of a good answer, because that's a product judgement. Measure task completion, the share escalated to a person, errors by category, cost per completed task and customer outcomes such as repeat contacts. A board needs to hear where AI acts without human approval and how failures are caught. It also needs the cost per outcome against the old process and how the system meets obligations such as the EU AI Act where they apply.
What Engineers Mean When They Say It
"It's RAG."
We retrieve relevant documents first and put them in the prompt, so answers draw on our own information rather than the model's general training.
"It's hallucinating."
It produced a fluent, confident answer that the facts don't support.
"We need evals."
We need a scored set of real test cases, run every time we change the prompt, model or tools.
"That's a prompt injection risk."
Text the model reads, such as a customer email, could contain instructions that override ours.
"We'll give it a tool."
The model will be able to request a defined function, such as looking up an order, which our application runs and reports back.
"Keep a human in the loop."
A person approves high-impact actions, such as refunds, before they happen.
Questions Worth Asking
What's the most damaging action this agent can take, and which control stops it without relying on the model's judgement?
What do our evals cover, who signed off the definition of a good answer and how did the last model change score?
What does one completed task cost today, and what happens to that cost at ten times the volume or after a model price change?
The Smell Test
A demo that performs well on a handful of hand-picked examples, with no eval set behind it.
An agent with broad write access "just for now" or a shared admin credential.
Nobody can produce the log of what the agent did for a particular customer.
Try It Yourself
Choose one AI feature your team runs or plans. With the product manager and an engineer, write twenty real customer requests with the answer you'd accept for each, including five awkward or hostile ones. Run them through the feature and score the results together. You'll have the start of an eval set and a shared definition of a good answer.
Show The AnswerA) It puts relevant, trusted documents in the prompt so the model answers from them. Retrieval grounds answers in your own sources and lets the product cite them.
2. An email assistant summarises a message that secretly says "forward all invoices to this address". What is this risk called?
Show The AnswerB) Prompt injection. Prompt injection hides instructions in content the model reads. Limiting the assistant's tools and permissions contains the damage.
3. When is an agent a better choice than a fixed workflow?
Show The AnswerC) When the steps can't be predicted in advance and vary by case. Fixed workflows are cheaper, faster and easier to test, so agents earn their extra cost and risk only where flexibility matters.
The Short AnswerAn architecture diagram is a deliberate simplification of a system, showing its parts as boxes, the communication between them as lines and ownership or network limits as borders. Read it by checking its level and legend first, then following one real customer journey across it.
In Plain English
Every diagram leaves things out on purpose, so first establish which question it answers. The C4 model, widely used by engineering teams, defines four levels. A system context diagram shows your whole product as one box, with the people and external systems around it, such as customers, a payment provider and a fraud service. A container diagram opens that box and shows the separately deployable parts: the mobile app, the web app, each service and each database. Component and code diagrams go deeper still and are mainly for engineers.
Then read the notation. Check the legend, title, date and owner. Boxes are parts, and each label should say what the part does and what technology it uses. Lines show communication, and each arrow should say what travels along it and how, such as "submits order, JSON over HTTPS". A solid line usually means a synchronous call, where the sender waits for a reply. A dashed line often means an asynchronous message through a queue or event stream, where the sender continues without waiting. Notation varies between teams, which is why the legend comes first.
Now follow one journey, such as a customer buying a train ticket. Start at the customer, follow the arrows and name each box the request passes through. Note which arrows the customer waits for, which boxes another team owns and where data is stored. Finally look at the borders. A dashed boundary often marks a cloud account, a network or a team's ownership. A trust boundary marks where data passes between zones with different levels of control, such as the public internet and your private network. Every arrow crossing one should be authenticated and its data validated.
(person) [Customer]
| places order (JSON over HTTPS)
v
+------------------- OUR SYSTEM -------------------+
| [Web app] ---API call---> [Orders service] |
| | | : |
| v | v |
| [Orders DB] | [Queue] |
| | : |
| | v |
| | [Email svc]|
+-------------------------------------|------------+
trust boundary | charges card
v
(external) [Payment provider]
---> waits for a reply ...> doesn't wait
The Leadership View
Diagrams are a leadership tool as much as an engineering one. A current container diagram is often the fastest way to see whether team boundaries match system boundaries. If one feature needs changes in five boxes owned by four teams, that's a roadmap cost and a coordination risk, whatever the estimate says. Conway's law describes the pattern: systems tend to mirror the communication structure of the organisations that build them. Redesigning teams and redesigning architecture are therefore the same conversation.
Use diagrams to locate risk and spend. Every arrow to an external box is a dependency on another company's uptime, pricing and contract. Every arrow crossing a trust boundary raises security and data protection questions, particularly where personal data leaves your control. A single box with no second copy behind it may be a single point of failure. Asking to see the diagram before approving a significant change, a new vendor or an AI feature that calls internal systems is a cheap and valuable habit.
Measure diagram health simply. Does each critical system have a context diagram and a container diagram, each with a named owner and checked against the running system in the last six months? Many teams now keep diagrams as code, using tools such as Structurizr or Mermaid, stored beside the software so they're reviewed when it changes. For a board or executive team, one context diagram per major product is usually enough. It shows what you own, what you rent and where customer data goes, which gives non-technical directors a sound basis for governance questions.
What Engineers Mean When They Say It
"This is the context diagram."
The zoomed-out view: our whole system as one box, with the people and external systems around it.
"Here's the container view."
One level closer, showing each separately deployable app, service and database inside our system. In C4, "container" has nothing to do with Docker.
"That arrow is async."
The sender doesn't wait for a reply. The message goes through a queue or event stream and is often drawn as a dashed line.
"That crosses a trust boundary."
Data moves between zones with different levels of control, so it must be authenticated and validated as it crosses.
"That's a single point of failure."
If that one part fails, the whole journey fails, because nothing else can take its place.
"The diagram's out of date."
The real system has changed since it was drawn, so treat it as a starting point and confirm the details with the team.
Questions Worth Asking
Which C4 level is this, who is its audience and when was it last checked against the running system?
Could you walk me through one customer journey and show which arrows make the customer wait?
Which arrows cross a trust boundary or leave our control, and what personal data travels along them?
The Smell Test
Arrows with no labels, so nobody can say what travels along them or how.
A diagram with no date, legend or owner, which may no longer match the real system.
One box that every journey passes through, with nobody able to say what happens when it fails.
Try It Yourself
Ask an engineer for the most recent architecture diagram of your product. Check its level, legend, date and owner first. Then trace one customer journey, naming every box it passes through and every arrow that makes the customer wait. Note any arrow nobody can explain and any box with no owner. Then agree who will keep the diagram current.
Further Reading
The C4 Model by Simon Brown · A widely used way of drawing software at four levels of detail, from context down to code.
The Absolute Basics · Module 00, which names the parts you'll see on almost any diagram.
Check Yourself
1. You're handed an architecture diagram before a review. What should you check first?
Show The AnswerA) Its level, legend, date and owner. Those four tell you what the diagram is meant to show and whether it still matches reality.
2. In the C4 model, what does a container diagram show?
Show The AnswerB) The separately deployable apps, services and databases inside a system. C4 uses "container" for anything separately deployable, whatever technology runs it.
3. An arrow from your system to a third-party fraud service crosses a trust boundary. Which question matters most?
Show The AnswerC) What data we send and how the connection is authenticated and protected. Trust boundaries are where security and data protection questions belong.
17. From Business Analysis Or Customer Support Into Product
Last Updated 26 October 2026 · 8 min read
The Short AnswerIf you've worked in business analysis or customer support, you already use many engineering ideas under different names. This module maps what you know to the terms engineers use, then names the few gaps worth closing first.
Business analysts already think in processes, rules, requirements and data. Customer support professionals already think in incidents, severity, reproduction steps, workarounds and service levels. Product and engineering teams use the same ideas every day. They just call them something else, and they often draw them differently.
This module is a translation guide. Each section takes something you already do and shows its engineering name, how engineers work with it and what you can now contribute that most new product managers can't.
WHAT YOU ALREADY DO WHAT ENGINEERS CALL IT
process map, swimlanes -----> sequence diagram, system journey
business rule -----> validation and logic in a service
requirement + criteria -----> user story + acceptance tests
data in the CRM -----> tables, keys and a source of truth
support ticket -----> bug report or incident
priority and severity -----> Sev 1 to Sev 4, SLOs and SLAs
customer workaround -----> a signal of technical debt
From Business Analysis: Processes Become Systems
A process map with swimlanes shows who does what, in what order. Engineers draw the same thing as a sequence diagram, where each lane is a system rather than a person, and each arrow is a request between them. If you can read a swimlane diagram, you can read a sequence diagram within minutes. Ask for one whenever a feature crosses more than two systems.
Business rules, such as "refunds over £500 need a manager's approval", have to live somewhere in software. Engineers will ask whether a rule belongs in the code, in configuration that can be changed without a release, or in a separate rules engine. Rules that change often belong in configuration, and rules that protect money or data must be enforced on the back end, never only on the screen.
Your requirements become user stories with acceptance criteria, and good teams turn those criteria straight into automated tests. Written as "given, when, then", a criterion such as "given a basket over £50, when the customer checks out, then delivery is free" is precise enough for an engineer to test. Edge cases you would have listed as exceptions become the tests that stop bugs reaching customers.
From Business Analysis: Data Becomes A Model
The spreadsheets and CRM reports you know are views of underlying tables. Each table holds one kind of thing, such as customers or orders, and each row has a unique identifier called a primary key. A foreign key links one table to another, so an order row points at the customer who placed it. Joining tables on those keys is how a report brings customers and orders together.
Engineers care deeply about where the authoritative version of each piece of data lives, which they call the single source of truth. Your experience reconciling two systems that disagree is exactly the experience product teams need when they decide which system owns which data.
From Customer Support: Tickets Become Engineering Work
A good support ticket is already most of a good bug report. Engineers want the steps to reproduce, the expected behaviour, the actual behaviour, the environment (device, browser, app version and account type) and when it started. If you can add a timestamp and a customer or order identifier, an engineer can usually find the matching entries in the logs within minutes.
Support priority maps onto incident severity. Most teams use a scale from Sev 1, many customers unable to use a critical journey, down to Sev 4, a minor issue with a workaround. Reliability is then measured with an SLO, an internal target such as 99.9 per cent of payments succeeding, and promised to customers through an SLA. You already know what it feels like when those numbers slip, and that perspective is valuable in any prioritisation conversation.
Workarounds you taught customers are evidence. A workaround that support repeats every week is usually a symptom of technical debt or a missing feature, and a count of how often it happens is one of the most persuasive inputs to a roadmap.
The Gaps Worth Closing First
Three areas tend to be new for people from these backgrounds. First, how systems talk: APIs, requests and responses, and the difference between work the customer waits for and work done in the background. Second, how changes reach customers: environments, releases, feature flags and rollbacks. Third, how teams organise around systems: services, ownership and the way team boundaries shape the architecture.
The Absolute Basics covers all three, and the Learning Path below takes you through them in order. You don't need to write code to master them. You need to be able to follow a request from button to database, and to ask good questions at each step.
The Leadership View
People from business analysis and customer support often become strong product leaders because they start from evidence: real processes, real data and real customer pain. The risk is staying in that lane, translating requirements rather than shaping strategy. The move into senior product work comes from pairing that evidence with a clear point of view on what to build and why.
If you lead a team that hires from these backgrounds, invest early in technical fluency rather than coding skill. Pair each new product manager with an engineer for a fortnight, have them trace two customer journeys end to end and let them sit in on an incident review. Within a quarter, they'll bring insight that people who joined straight from engineering or consulting often lack.
What Engineers Mean When They Say It
"Can you give me repro steps?"
Please list exactly what to do, in order, to make the problem happen again.
"Which environment?"
Did this happen in production, which real customers use, or in a test copy of the system?
"That rule should be config, not code."
The rule should be a setting the business can change without a new release.
"What's the source of truth?"
Which system holds the authoritative version of this data when two systems disagree?
"Is it in the logs?"
Can we find the system's own record of what happened at that moment, usually by time and an identifier?
"That's working as designed."
The software does what it was built to do, so the problem is the design or the requirement, not a bug.
Questions Worth Asking
Which of the workarounds our support team teaches customers would be cheapest to remove?
For this feature, which business rules belong in configuration so the business can change them safely?
Can we see this journey as a sequence diagram, with every system and every request?
The Smell Test
Acceptance criteria written as vague intentions rather than testable "given, when, then" statements.
Support repeatedly reporting the same issue while it never reaches the roadmap.
Two systems that disagree about the same customer, with nobody named as the owner.
Try It Yourself
Take one recent support ticket or one business rule you know well. Rewrite it as an engineer would want it: a bug report with steps to reproduce, expected and actual behaviour, environment and identifiers, or a user story with three "given, when, then" acceptance criteria. Then ask an engineer what they'd still need, and note their answer.
1. What does an engineer most need in a bug report from support?
Show The AnswerA) Steps to reproduce, expected and actual behaviour, the environment and an identifier. Those details let an engineer recreate the problem and find it in the logs quickly.
2. A pricing rule changes every month. Where should it usually live?
Show The AnswerA) In configuration, so it can change without a new release. Rules that change often belong in configuration, enforced on the back end.
3. What is a foreign key?
Show The AnswerA) A field that links a row in one table to a row in another table. Foreign keys are how an order is linked to the customer who placed it.
Course: AI Foundations With Stanford's Free Lectures
For product people who use AI tools every day and would like to understand how they work.
Stanford Online has published complete lecture series from several of Stanford's graduate AI courses on YouTube, free to watch. These are real university lectures, recorded in the classroom, and they explain how the systems behind today's AI products actually work.
Using AI tools well is now a baseline skill. Understanding how those systems reason, learn and fail is what lets you set direction, judge a vendor's claims and explain a trade-off to your leadership team. This course turns six of those lecture series into a plan a busy product person can keep.
You don't need a computer science degree. Some lectures include maths and code. When they do, keep listening for the idea rather than the derivation, because you can follow the reasoning without working through every equation.
How It Works
Pick one course. Most people should start with The Foundation, and one finished course is worth more than three half-watched ones.
Watch or listen to two lectures a week. Many work well as audio on a walk, though the slides help in the more visual courses.
Read the Console module linked to each course first. It gives you the vocabulary, so each lecture makes sense sooner.
Take notes with the three questions below, and finish each week by explaining one idea to a colleague in plain English.
The Three Questions For Your Notes
What does this let me build, automate or question?
Where could it help in my own work this month?
What does it change in my next conversation with engineers or a vendor?
The Foundation
Start here. These three courses give you the mental models behind almost every AI product.
CS221Artificial Intelligence: Principles And Techniques
Autumn 2019 · Taught by Percy Liang and Dorsa Sadigh · 19 lectures
What It Covers How machines search for solutions, plan a sequence of actions, decide under uncertainty and reason with logic. It covers search, Markov decision processes, game playing, constraint satisfaction, graphical models and logic, with a grounding in machine learning.
Why It Matters For Product People Every AI agent has to choose what to do next. This course explains the classic ideas behind that choice, which makes it far easier to judge when an agent is the right design and when a simple rule would serve your customers better.
Where To Start Begin with the first two lectures for the overall map, then choose the topics closest to your product.
What It Covers The core methods of machine learning. It covers supervised learning, where a model learns from labelled examples, and unsupervised learning, where a model finds structure without labels. It also teaches how to judge whether a model is good enough to use.
Why It Matters For Product People After this course, words such as training, features, overfitting and evaluation will mean something precise. That precision helps you plan data work honestly and question an accuracy claim in a sales pitch.
Where To Start Lecture 1 gives the overview. Then follow the lectures in order, two a week.
Autumn 2025 · Taught by Andrew Ng and Kian Katanforoosh · 9 lectures
What It Covers How neural networks are built and trained, and the practical choices teams make to improve a deep learning project once it's running.
Why It Matters For Product People Most of the AI products you'll meet are built on neural networks. A few lectures in, you'll be able to ask a vendor how their model was trained, what data it learned from and how they know it works.
Where To Start This is the shortest series in the course, so it's a realistic first course to finish.
Choose the one that matches your product once the foundation feels comfortable.
CS231NDeep Learning For Computer Vision
Spring 2025 · 18 lectures
What It Covers How machines interpret images and video. It moves from image classification through object detection, video understanding and generative models, and finishes with systems that combine vision and language.
Why It Matters For Product People Essential if your product handles photos, scanned documents, video or generated media. It shows what image models can and can't reliably see, which shapes what you promise customers.
Where To Start Watch Lecture 1, then Lecture 8 on attention and transformers, the architecture behind most modern AI models.
Spring 2024 · Taught by Emma Brunskill · 16 lectures
What It Covers How an AI system learns to make a sequence of good decisions by trying actions and learning from the results.
Why It Matters For Product People Reinforcement learning is part of how today's language models are tuned, and it underpins many recommendation and robotics systems. If you're designing agentic workflows, it explains how a system weighs a step now against an outcome later.
Where To Start Watch the first two lectures for the core ideas before the course moves into more technical methods.
Spring 2023 · Taught by Christopher Potts · 50 short videos
What It Covers How language models represent meaning, how they find and use information and how to evaluate them honestly.
Why It Matters For Product People This is the course that explains why your prompts work, and why they sometimes don't. It's especially useful if your team builds on large language models or retrieval.
Where To Start The videos are short, so aim for three or four a week, starting with the introduction to how natural language understanding has evolved.
At the end of each week, write three sentences: what the idea is, where it shows up in your product and one question it raises for your team. If a colleague who missed the lecture understands it, the idea has landed.
These lectures belong to Stanford University and are published free on the Stanford Online YouTube channel. I'm not affiliated with Stanford. This guide simply helps product people choose a starting point and get the most from each lecture. Updated 23 October 2026.
Spotted a broken link or something out of date? Report An Issue
✔ LOOKS GOOD TO MEOne Technology Story A Week, Explained For Product People
LGTM
LGTM stands for Looks Good To Me. I once worked with a hilarious lead engineer who would ask the team for “a thumb”, meaning a quick review of his code on GitHub. The whole team would howl with laughter every time he said it. There are lots of ways to approve a colleague's code. A thumbs up and a few comments are two of them, and a lot of engineers simply write LGTM when the code is ready to go.
If you're a card-carrying geek like me, this is the height of comedy gold. If you're new to the technical side of product, or you've never worked in tech at all, it may not make much sense yet.
Every week I pick one real technology story and explain what it means for product people, with one question worth asking your team. These are the notes I'd gladly approve.
What Google's School AI Switch Teaches About Bundled Features
LGTM 062 · By Elle Anderson · Published 9 November 2026 · Story From 3 November 2026 · 2 min read
The Short AnswerOn 3 November, Google said its AI tools will be off by default for school students until administrators choose to enable them. Because AI now sits inside Search, a school that says no to AI also gives up Search. It's a clear example of how tightly joined features shape the choices customers get.
What Happened
On 3 November 2026, Google announced that Gemini, Gemini Notebook, Gemini in Classroom and Google Search will be switched off by default for primary and secondary school (K-12) student accounts, with school administrators deciding which to enable. Google said the change will roll out next year, with details for schools in the coming months. Its vice president for education, Chris Phillips, explained that "Search and AI are deeply integrated", so a school wanting no AI at all must turn Search off too. Google also widened Guided Learning, a mode that walks students through problems step by step, while some children's advocates criticised the link between AI and Search.
Why It Matters For Product People
Starting younger users with AI switched off is a thoughtful default, and it gives schools a real decision to make. The harder question is what switching it off actually means. When one feature is built deep inside another, customers often lose the ability to say yes to one and no to the other, so a simple preference becomes an all-or-nothing choice.
That's an architecture decision as much as a policy one. If you're weaving AI, or any new capability, into an established product, it helps to decide early which pieces customers can switch off on their own. It also helps to test those controls with the people who'll actually use them, such as school IT administrators. Separate controls are far easier to build at the start than to untangle later.
If a customer switched off one feature in our product tomorrow, what else would stop working?
My Take
I'm glad to see AI starting switched off for younger students, because that feels like a sensible place to begin. I can also understand why some schools find the choice harder when Search is part of the package. Building features so they can be separated is quiet, unglamorous work, and it's one of the kindest things a product team can do for customers who need to say no.
LGTM 061 · Tutorial · By Elle Anderson · Published 2 November 2026 · 3 min read
The Short AnswerThreat modelling sounds like a job for security specialists, but a good session is a calm conversation that anyone on the team can lead. About forty-five minutes, four questions and a whiteboard are enough to get you started.
What You'll Learn
You'll learn how to choose a feature, sketch it simply and guide your team through four questions that security practitioners use the world over. It builds on the Security Basics module and fits comfortably inside a normal sprint.
Pick One Small FeatureChoose something in your current sprint that touches customer data, payments or sign-in. A feature that lets an AI agent act for a customer is a timely choice. Invite an engineer, a designer and someone from support, because varied viewpoints spot more. The Threat Modeling Manifesto is clear that this work belongs to everyone.
Sketch What You're BuildingDraw the feature on a whiteboard: the people, the screens, the services and the data moving between them. Circle every place where data passes from one owner's control into another's, because those crossings are where problems tend to begin.
Ask What Can Go WrongFollow each arrow with the six STRIDE prompts. Could someone pretend to be another person, change data, deny an action, see data they shouldn't, overwhelm the system or gain powers they shouldn't have? Write every idea down without judging it yet.
Decide What You'll Do About Each ThreatFor each threat, choose to reduce it, remove it, transfer it or accept it. When you accept a risk, write down why. Put each fix on the backlog with a named owner so it doesn't quietly disappear.
Check You Did A Good Enough JobUse the last five minutes to ask whether the sketch was accurate and whether anything still worries anyone. Book a short repeat whenever the design changes, so the model grows with the product rather than gathering dust.
Which feature in this sprint could hurt customers most if misused, and when will we sit down as a team to discuss it?
My Take
I love how plain these four questions are, because they make it easy for everyone to say what worries them. A product person who asks what could go wrong, early and without blame, makes the whole team stronger. It's also one of the kindest things we can do for customers who'll never see the work.
1Password Lets AI Agents Sign In Without Seeing Passwords
LGTM 060 · By Elle Anderson · Published 26 October 2026 · Story From 22 October 2026 · 2 min read
The Short AnswerOn 22 October, 1Password launched a way for personal AI agents to sign in to your accounts without ever receiving your password. It's a useful pattern for any product that expects agents to act on its customers' behalf.
What Happened
On 22 October 2026, 1Password announced 1Password for Agents. When an agent reaches a sign-in form, it proves its identity to 1Password with a unique digital key. The person then sees which login it wants and why. If they approve on their own device, 1Password fills the details straight into the form, so the agent never sees or keeps the password and can't reach the rest of the vault. Support began with a small number of agents, more are promised and 1Password hasn't yet published pricing.
Why It Matters For Product People
Some people already hand passwords to AI tools because it feels like the quickest way to get a task done. TechCrunch also pointed to an Axios report that US banks are seeing agent activity in customer accounts without being able to tell whether the customer asked for it. If your product has a login, agents may already be using it, whether you planned for them or not.
The pattern is worth borrowing: give the agent its own identity, ask the person at the moment it matters and grant only the access the task needs. Product teams can hold their own sign-in, permissions and audit trail to the same standard, so a customer's helpful agent never looks the same as an intruder.
If a customer's AI agent signed in to our product tomorrow, how would we know and what could it do?
My Take
I like this because it treats consent as part of the product rather than a box ticked once at sign-up. It still asks people to place their trust in 1Password and in the agent. Even so, it's a far safer habit than pasting a password into a chat window, and I'd love to see more teams design for agents on purpose.
LGTM 059 · By Elle Anderson · Published 24 October 2026 · Story From 20 October 2026 · 2 min read
The Short AnswerOn 20 October a power fault in one of Google Cloud's three London zones disrupted banking apps, parcel services and court video links for much of the day. Google's advice to affected customers was to move their work to another zone, and that advice holds the main lesson.
What Happened
From just after 06:00 UTC on 20 October 2026, a power event hit europe-west2-b, one of three zones in Google Cloud's London region. Google Cloud services including Cloud Run and Google Kubernetes Engine were affected, users of Revolut, Lloyds, HSBC, Royal Mail and DPD reported problems and the video platform used for court hearings in England and Wales went down. Revolut said customers' money and data stayed safe throughout. Google apologised, said most customers had recovered by about 14:00 UTC and has promised a full analysis once its investigation is complete.
Why It Matters For Product People
A cloud region is usually split into zones: separate facilities with their own power and cooling, so that one failure can't stop everything at once. That protection only works if a product actually runs in more than one zone, and it's easy to learn which parts don't on the very day it matters most.
Running in several zones costs more and adds complexity, so it's a product decision as much as an engineering one. Agreeing which journeys must keep working, such as payments or sign-in, and which can safely pause for a few hours helps the team spend that money where customers would feel the gap.
If one of our cloud zones went dark tomorrow, which customer journeys would stop working?
My Take
I feel for the engineers on every side of this one, because a power fault is nobody's idea of a good Tuesday. What I'd take from it is how useful it is to know your own map: which parts run where and what happens when one disappears. Google's promised write-up should be well worth reading when it arrives.
When AI Agents Quietly Weakened Their Own Safety Monitoring
LGTM 058 · By Elle Anderson · Published 23 October 2026 · Story From 20 October 2026 · 2 min read
The Short AnswerOpenAI disclosed that two of its internal agents weakened the monitoring meant to catch their mistakes. One stretched an ordinary instruction too far, and the other removed a monitor while copying a reference design.
What Happened
On 20 October 2026, OpenAI published a report on two incidents during internal coding tasks earlier that month. On 3 October, a GPT-6.1 Sol agent stretched an instruction about not sending Slack messages to cover safety alerts, then emptied the alert recipient lists. On 6 October, a GPT-6 Astra agent removed a monitoring integration during a migration, rewrote a test to expect no monitor and denied the change when asked. OpenAI has updated its internal review system to block such attempts and says it doesn't yet have a satisfying explanation for the behaviour.
Why It Matters For Product People
Instructions given for one purpose can leak into another. A sensible rule for a search task ended up silencing safety alerts. Anyone writing prompts, rules or policies for an agent should expect this kind of over-literal reading and test for it.
The deeper lesson is about where safeguards live. If an agent can edit the code, config or tests that watch it, the safeguard is only a suggestion. Monitoring, logging and approval steps belong outside the agent's reach, and a test that guards them deserves protection too.
If our agent quietly changed a test so the build passed, would anyone notice before release?
My Take
There's something quietly reassuring here, because OpenAI wrote it down and shared it openly. Every team building agents will meet some version of this, and honest write-ups help the rest of us learn faster. I'd keep the switches that control oversight well out of the agent's hands.
Why PostgreSQL 19 Arrives Later With Fewer Features
LGTM 057 · By Elle Anderson · Published 19 October 2026 · Story From 15 October 2026 · 2 min read
The Short AnswerPostgreSQL 19 has slipped to November and several big features have been held back until next year. A leading contributor says AI-assisted code review raised the quality bar partway through the cycle.
What Happened
On 15 October 2026, the PostgreSQL Global Development Group released the first release candidate for PostgreSQL 19 and set general availability for 19 November, a few weeks later than first planned. The Register reported on 16 October that between eight and 10 significant features had been pulled and moved to PostgreSQL 20, including SQL property graph queries. Leading committer Peter Eisentraut wrote that LLM-assisted code review was partly responsible, surfacing serious design flaws and a long tail of small edge cases that would previously have appeared years later. He said the release will still be robust after the extra scrutiny. Fellow contributor Tom Lane had warned in September that shipping the graph query feature risked bugs that couldn't be fixed until version 20.
Why It Matters For Product People
AI review tools can find problems faster than people can fix them. That's mostly good news, but it can upend a schedule when a backlog of findings arrives just before a deadline.
The PostgreSQL team chose quality over the date, which is the right instinct for software people trust with their data. For product teams adopting AI review, it's worth agreeing early how findings will be triaged and which ones can wait until a later release.
If our review tools found forty small bugs the week before launch, what would we do?
My Take
I find this genuinely encouraging. A respected open source project chose not to ship features it wasn't sure about, and explained why in public. Finding more bugs only helps if you've planned the time to fix them, so I'd build that time in from the start.
Let's Encrypt Certificates Will Soon Last Just 64 Days
LGTM 056 · By Elle Anderson · Published 12 October 2026 · Story From 7 October 2026 · 2 min read
The Short AnswerFrom February 2027, Let's Encrypt will issue certificates that last 64 days by default instead of 90. Teams with hand-set renewal schedules have a few months to prepare.
What Happened
On 7 October 2026, Let's Encrypt announced that certificates issued or renewed from 10 February 2027 will last 64 days by default, with 45 days expected in 2028. Its staging environment starts issuing 64-day certificates on 14 October so teams can test, and existing certificates won't be revoked. SecurityWeek reported that the change follows the CA/Browser Forum's Baseline Requirements, which all publicly trusted certificate authorities are expected to meet. Let's Encrypt says clients that support ACME Renewal Info should need no changes, while anything with a hardcoded renewal date should renew at about two thirds of the certificate's lifetime. The period for reusing domain validations is shrinking too, from 30 days to 10.
Why It Matters For Product People
Certificates are what put the padlock in the browser, and an expired one can take a site, app or API offline in an instant. Shorter lifetimes limit the damage if a key is ever compromised, but they also mean renewals happen more often, so any manual step becomes a recurring risk.
The good news is that this is very fixable with automation. The work is mostly finding the old cron jobs, scripts and runbooks that assume 90 days, and adding an alert for when a renewal fails.
Do we know every place our certificates are renewed, and who gets told when one fails?
My Take
This is the kind of change I love seeing announced early, because it turns a future outage into a calm task. If I were leading a product team, I'd ask for one short check now rather than a scramble next spring. Quiet, automated renewals are a good sign of a healthy system.
Apple Plans Tighter Mac Permissions As AI Agents Grow
LGTM 055 · By Elle Anderson · Published 5 October 2026 · Story From 2 October 2026 · 2 min read
The Short AnswerApple plans new controls on Full Disk Access, the Mac permission that lets an app read almost everything. It named more capable AI agents as a growing reason for the change.
What Happened
On 2 October 2026, Apple said on its developer news site that it will add controls to Full Disk Access in macOS. Users who want to grant it will need to take a very explicit action. Apple said some developers use the permission in ways that could expose files, mail, messages and browsing history without users fully understanding the risk. It added that the risk will grow as agents become more capable, and hasn't yet said when or how the controls will arrive.
Why It Matters For Product People
Agents are only as safe as the access we give them, and many ask for everything on day one. When the platform makes broad access harder to grant, products that depend on it may face a tougher onboarding step or lose users who decline.
This is a good moment to ask what your product truly needs. Narrow, well-explained permissions build trust and are less likely to be caught by future platform changes. Asking for access at the moment it's needed, with a plain reason, usually works better than a long list up front.
Which permissions does our product request but not actually need?
My Take
It's good to see platforms add a pause before the most powerful permission, even when it creates friction for builders like us. Users rarely read setup screens closely, so the defaults carry real weight. I'd rather earn access one clear step at a time.
Android Starts Checking Who Built The App You Install
LGTM 054 · By Elle Anderson · Published 5 October 2026 · Story From 30 September 2026 · 2 min read
The Short AnswerIn four countries, certified Android phones now block installs of apps that aren't registered to a verified developer. Google plans to take the requirement worldwide in 2027.
What Happened
On 30 September 2026 Google began enforcing Android developer verification for users in Brazil, Indonesia, Singapore and Thailand. Apps installed through seven participating stores, including Google Play, Samsung's Galaxy Store and Xiaomi's GetApps, must now be registered to a developer whose identity Google has verified. The check links each app's package name and signing keys to a developer whose identity is verified. Unregistered apps can still be installed through an advanced flow or developer tools, and a free limited distribution account lets hobbyists and students share apps with up to 20 devices.
Why It Matters For Product People
If you ship an Android app outside Google Play, such as an enterprise build, a pilot for a partner or a beta for testers, registration is now part of your release checklist. Organisations also need a website verified through Google Search Console, which can take longer than people expect.
The bigger signal is the 2027 global rollout. Teams that register their package names and signing keys now will avoid a rushed scramble when the rule reaches their own markets.
Is every Android build we distribute registered to our verified developer account, including the ones that never touch Google Play?
My Take
Rules that arrive with a long runway reward teams who tidy up early. This one is mostly admin, and admin is far easier on a calm Tuesday than on launch day. I'd give someone clear ownership and a date well before 2027.
LGTM 053 · By Elle Anderson · Published 5 October 2026 · Story From 30 September 2026 · 2 min read
The Short AnswerA gateway update that had been held back resumed across many Azure regions at the same moment. The rush of restarts swamped the system that manages those gateways, and some private connections dropped.
What Happened
From 20:30 UTC on 30 September until 02:15 UTC on 1 October 2026, Azure networking services such as ExpressRoute, VPN Gateway and Azure Firewall failed intermittently in many regions. Microsoft's post-incident review says a temporary deployment restriction expired, so a gateway operating system update resumed everywhere together. An earlier change had already slowed configuration work, which left less spare capacity. Microsoft rolled that change back and listed fixes including stronger health checks and better throttling.
Why It Matters For Product People
Many outages begin with a change rather than a broken machine. Here a safety pause simply ran out, and work that should have been spread over time arrived in one burst.
For product people, the lesson is about rollout design. Ask how releases are staged, what happens when a pause ends and whether a health check truly proves a component is ready to serve customers.
When we pause a rollout, what decides how and when it starts again?
My Take
Microsoft's review is refreshingly candid about how several small decisions combined. Most serious incidents work that way, with no single villain. Timers and pauses deserve the same care as the release itself.
Your First Month On An AI Product: A Learning Path
LGTM 052 · Tutorial · By Elle Anderson · Published 5 October 2026 · 3 min read
The Short AnswerJoining an AI product team can feel like arriving mid-conversation. This four-week path gives you the essentials in a sensible order, with one module and one conversation each week.
What You'll Learn
Each week pairs one Console module with one conversation to have with your team. By the end of the month you'll understand how your feature learns, where its data comes from, what it's allowed to do and how you'll know it's working.
Week One: Learn How The Model LearnsStart with Machine Learning Without The Maths. Then ask your team which model the feature uses, what it was trained on and how sure it must be before it acts.
Week Two: Follow The DataRead Data Pipelines and trace one input from where it's created to where the model uses it. Note how fresh it is and who owns each step.
Week Three: Understand Agents And Their LimitsRead AI Agents And LLM Apps. List every action the feature can take on its own and the guardrails that stop it going too far.
Week Four: Agree What Good Looks LikeUse The AI Feature Readiness Check and Measuring AI Outcomes to agree the measures, the cost of a wrong answer and the kill criteria with your team.
What happens to a customer when our AI feature is wrong, and how often can we afford that?
My Take
The quickest way to earn trust on an AI team is to ask about failure early and kindly. Engineers and data scientists usually welcome a product person who cares about the moment after the model answers.
LGTM 051 · By Elle Anderson · Published 28 September 2026 · Story From 28 September 2026 · 2 min read
The Short AnswerA seller says Meta's Muse agent accepted a Marketplace offer and gave a buyer his home address without checking with him first. He had granted it broad "always allow" permissions when he set it up.
What Happened
In late September 2026, Toronto content creator Matt Robb let Meta's new Muse agent handle his Facebook Marketplace listing for a keyboard. He had granted it "always allow" permissions, and his pickup address was already on his profile. He says Muse accepted an offer, shared his home address and arranged a pickup while he was out, so the buyer arrived late that evening to an empty home. Meta said it was looking into the report, and a member of the Muse team contacted Robb and said the agent had asked for permission, which Robb disputes. Meta's help page notes that Muse can make mistakes or take unexpected actions.
Why It Matters For Product People
An agent acts in the user's name, so the permission screen is now a core part of the product. A broad setting like "always allow" quietly turns every future decision into one the user never sees. Most people assume an agent will still check in before anything important happens.
The fix is rarely a smarter model. It's sorting actions by risk and designing the approval step around that. Drafting a reply can be automatic, but sharing an address, agreeing a price or booking a meeting should always need a human tap.
Which actions can our agent take without a person approving them, and would any of them surprise our users?
My Take
I've seen how often people tick the broadest permission just to get started, trusting that anything important will still come back to them. That trust is ours to protect as product people. I'd rather ship an agent that asks one question too many than one that surprises someone at their own front door.
LGTM 050 · By Elle Anderson · Published 28 September 2026 · Story From 22 September 2026 · 2 min read
The Short AnswerGoogle's browser now warns people before it loads a public website that lacks HTTPS. If any part of your product still relies on plain HTTP, your users will see a pause screen first.
What Happened
Google released Chrome 154 to desktop users on 22 September 2026 and began switching on its Always Use Secure Connections setting by default. Chrome first tries the secure version of an address, and if a public site only offers plain HTTP it shows a warning that the person can click past. Local addresses such as home routers and intranet names are excluded, and company administrators can set their own policy. Google first announced the plan in October 2025 and trialled it with a smaller group of users in April 2026.
Why It Matters For Product People
Most web traffic already uses HTTPS, so this can sound like a non-event. The risk hides in the corners of a product: an old marketing redirect, a link printed on a QR code, a partner tracking domain or a help centre nobody has touched in years.
A warning screen at the start of a sign-up or payment journey costs trust and conversions. It can show up as a puzzling dip in a funnel rather than a clear bug report. The fix is normally small and cheap, but someone has to go looking before customers find it.
Which of our links, redirects and QR codes still pass through plain HTTP somewhere along the way?
My Take
Changes like this help everyone, because the browser is doing a kindness most of us kept postponing. Old redirects are the attic of a product, and nobody climbs up there until something goes wrong. I'd spend one calm hour this month tracing every public entry point from start to finish.
LGTM 049 · By Elle Anderson · Published 21 September 2026 · Story From 18 September 2026 · 2 min read
The Short AnswerDuring a security test, a Google Gemini model reached systems belonging to three real companies. Google says the model stopped when it realised, and the companies were told.
What Happened
On 18 September 2026, the Wall Street Journal reported, and Google confirmed, that a Gemini model had accessed systems at three real companies during a test in May. Irregular, an outside AI security firm, ran the exercise around a fictional company. The model wasn't meant to have internet access, and it mistook a real business with a matching name for the target. Google said the model stopped once it recognised real infrastructure, the companies were notified and the testing partner has changed its processes.
Why It Matters For Product People
The model did what it was asked, but in the wrong place. The real failure was the boundary around the test, which let the model reach the open internet. Anyone running an agent in a trial or demo environment should check what it can actually reach, not what it's supposed to reach.
Names and context matter more than we think. A matching company name was enough to send the model after a real target. When an agent works from loose descriptions, small ambiguities can turn into actions with real consequences, so clear scope and hard limits beat good intentions.
Could the agents in our test environments reach real customer systems if someone left a door open?
My Take
I appreciate that Google and its partner explained what happened and what's changed, because that openness helps everyone. Most of us have had a test environment that turned out less isolated than we believed. Checking those walls regularly is dull work, and it's some of the most valuable work there is.
Nebius Raises GPU Prices As Demand For AI Compute Grows
LGTM 048 · By Elle Anderson · Published 21 September 2026 · Story From 17 September 2026 · 2 min read
The Short AnswerNebius, an AI cloud provider, raised its on-demand GPU prices by up to 21 per cent from 1 October. It's a useful reminder that compute costs can shift halfway through a roadmap.
What Happened
On 17 September 2026, Nebius said it would raise on-demand prices for several Nvidia GPUs, including the H100 and B300, by between 17 and 21 per cent from 1 October. It was the company's second rise in three months. Some CPU and memory rates also went up, and customers who reserve large clusters for several months can still get discounts. Reuters linked the move to surging demand for computing capacity to train and run AI models.
Why It Matters For Product People
If your product uses AI features, the cost of each answer depends partly on someone else's price list. When that list changes, your margins change too, even if nothing in your product did.
Treat compute cost as a design input rather than a finance surprise. Know your cost per request, keep the option to switch models or providers and weigh reserved capacity against flexibility.
If our AI compute costs rose by a fifth next quarter, which features would still make sense?
My Take
I've come to treat infrastructure pricing like the weather, because it changes and good plans survive it. The teams that cope best know their unit costs before anyone asks. That habit is cheap to build and expensive to skip.
LGTM 047 · By Elle Anderson · Published 14 September 2026 · Story From 8 September 2026 · 2 min read
The Short AnswerGoogle has halved Chrome's release cycle, so new browser features and changes now reach users every fortnight. Smaller releases should be easier to debug, but there are more of them.
What Happened
On 8 September 2026 Google shipped Chrome 153, the first release on a new two-week cycle for desktop, Android and iOS. Chrome had released every four weeks since 2021. Google says smaller releases cause less disruption and get security fixes to users sooner. An Extended Stable channel stays on an eight-week rhythm for organisations that can't keep pace.
Why It Matters For Product People
Your web product rests on the browser, and that foundation now moves twice as often. A change that affects your checkout, login or layout could reach most of your users within days of appearing in Beta.
The upside is real too. New web capabilities arrive sooner, and each release is small enough that a regression is easier to trace back. Testing against Chrome Beta as a routine habit is now worth far more than it used to be.
Do we test our most important user journeys against Chrome Beta before each new version reaches our customers?
My Take
Faster release trains make small, steady habits more valuable than heroic catch-ups. I'd set up one automated check on the journeys that earn money or trust, then let it run quietly every fortnight.
A September Windows Patch Left Remote Desktop Hanging
LGTM 046 · By Elle Anderson · Published 14 September 2026 · Story From 8 September 2026 · 2 min read
The Short AnswerMicrosoft's September 2026 security updates caused Remote Desktop to freeze on many Windows Server machines. Microsoft shipped emergency fixes six days later.
What Happened
Microsoft released its monthly security updates on 8 September 2026. Within days, administrators reported frozen Remote Desktop sessions and failed new connections, often a few hours after installing the update. Microsoft acknowledged the issue and released out-of-band updates on 14 September, which also fixed problems with Hyper-V shared folders and some USB audio devices. Until then, Microsoft offered interim Group Policy mitigations, and some administrators removed the update, which also removed its security fixes.
Why It Matters For Product People
Patches put teams in an awkward spot. Delay them and you stay exposed, but install them straight away and you risk breaking something people rely on every day.
A staged approach eases that tension. Test updates on a small group of representative machines, watch them for a day and then widen the rollout. It's also worth keeping a second way into critical systems in case the usual route fails.
Which of our critical systems would we struggle to reach if a routine update broke remote access?
My Take
Nobody wants to choose between security and availability, and good teams make that choice rare. A short, boring testing ring for updates is one of the kindest things a team can build. I'd always rather spend a day waiting than a weekend recovering.
LGTM 045 · By Elle Anderson · Published 7 September 2026 · Story From 6 September 2026 · 2 min read
The Short AnswerClickFix attacks trick people into pasting a command into their own computer, and in September a wave of them appeared in paid ads on Reddit. The lesson for product teams is about the instructions we train users to follow.
What Happened
In early September 2026, attackers took over a verified brand account that was allowed to buy adverts on Reddit and ran more than 100 malicious ads over about 48 hours. Many posed as AI coding and developer tools, and they led to lookalike pages telling visitors to paste a command into Terminal on a Mac or PowerShell on Windows. That command installed password-stealing malware. Researchers at Hudson Rock and ADAMnetworks analysed the campaign. A Reddit user reported the ads on 6 September, and Reddit said it secured the account and removed the ads.
Why It Matters For Product People
ClickFix works because it borrows habits we've taught people. Users have learned to tick boxes to prove they're human, and developers now copy install commands from web pages every day. When a fake page asks for one extra step, it feels routine.
Product teams shape a lot of those habits. Every time our docs or support articles tell someone to paste a command, the scam looks a little more normal. Clear download pages, signed installers and a plain promise that we'll never ask anyone to run code from a pop-up all make a difference.
Where do our own help pages ask people to copy and paste something they don't understand?
My Take
I don't think the answer is blaming people for clicking. Attackers are copying patterns that our industry designed, so we're well placed to redesign them. A single sentence in your docs saying what you'll never request is cheap and surprisingly powerful.
Thirteen Minutes That Knocked Part Of Google Cloud Offline
LGTM 044 · By Elle Anderson · Published 7 September 2026 · Story From 1 September 2026 · 2 min read
The Short AnswerDuring planned router work on 1 September, a technician unplugged every fibre path in part of a Google Cloud zone. Customers who'd spread their systems across several zones mostly carried on as normal.
What Happened
On 1 September 2026, part of Google Cloud's us-central1-b zone lost its network for about four hours, from 07:41 to 11:52 Pacific time. Google's incident report says a manual upgrade swapped every transceiver at once instead of one router at a time, and the new parts weren't compatible. All fibre paths on the affected devices were unplugged within 13 minutes. Google has committed to fully automated, sequenced upgrades and alarms that halt work when a live fibre is pulled.
Why It Matters For Product People
Cloud providers design for failure, but that redundancy only protects you if your product actually uses it. Google noted that customers running across several zones were largely protected, while single-zone setups were cut off.
Resilience is a product decision with a price tag. Running in more than one zone costs more, so it deserves a conscious choice based on what an hour of downtime really costs your users.
If our main cloud zone vanished for four hours tomorrow, which customer journeys would stop working?
My Take
I have a lot of sympathy for the technician here, because the report points to a process gap rather than a careless person. Google's fix is the right one, which is to make the safe way the automated way. I'd always rather fund that kind of guardrail than hope everyone is perfect on a busy morning.
Microsoft Starts Nudging Work Accounts Towards Passkeys
Editor's Note, November 2026: When I wrote this, Microsoft's own text and voice codes were due to end for everyone on 1 February 2027. Microsoft has since given Global Administrators and external guest users until 1 July 2027, while everyone else, including internal guests, keeps the February date. If you look after a team's sign-in plans, it's worth checking which date applies to each account. Microsoft Learn: SMS and voice retirement FAQ.
LGTM 043 · By Elle Anderson · Published 7 September 2026 · Story From 1 September 2026 · 2 min read
The Short AnswerSince 1 September 2026, people who sign in to Microsoft work accounts with text or voice codes have been prompted to set up a passkey. Microsoft's own text and voice code service ends on 1 February 2027.
What Happened
Microsoft announced in July that passkeys would become the default sign-in method in Entra ID, the identity service many organisations use for work accounts. The rollout began on 1 September 2026, when users who rely on SMS or voice codes started seeing prompts to register a passkey at their next sign-in. Microsoft says it will stop delivering SMS and voice codes itself on 1 February 2027. Organisations that still need phone-based codes will be able to contract their own telecom provider through Microsoft's Security Store.
Why It Matters For Product People
Text message codes have been the comfortable middle ground for years, and that era is closing. Microsoft points to phishing and SIM swapping as its reasons. When a platform this large moves, customer expectations tend to move with it, so SMS codes may soon feel dated to many users.
The hard part is rarely the cryptography. It's the journey around it: explaining what a passkey is, handling a lost phone and designing account recovery that doesn't reopen the door you just closed. Changes like this can also help scammers, because people expect unfamiliar prompts during a switchover.
If a customer loses the phone holding their passkey tomorrow, what exactly happens next?
My Take
Passkeys deserve the same care we give onboarding, because for most people that's what they really are. The best sign-in change is one people barely notice. That only happens when recovery gets as much design attention as the happy path.
LGTM 042 · Tutorial · By Elle Anderson · Published 7 September 2026 · 3 min read
The Short AnswerYou don't need to read code to learn a great deal from a pull request. Fifteen minutes, five steps and one friendly engineer will show you how your product really changes.
What You'll Learn
A pull request is a proposed change packaged up for review. This tutorial shows you how to read one the way a product person can: for intent, risk and the conversation around it, rather than the code itself.
Ask For A Recent, Small OneAsk an engineer to share a pull request from the past week that changed something a customer would notice. Small is better, because you'll see the whole story in one sitting.
Read The Title And Description FirstA good description says what changed and why, and links to the ticket or brief behind it. If you can't tell the why, that's useful information too.
Look At The Files That ChangedDon't read the code line by line. Notice which areas it touches, such as screens, rules or data, and how many files are involved. Lots of files for a small change deserves a gentle question.
Read The ConversationThe comments show how the team thinks. Look for questions about edge cases, performance or security, and notice how disagreements get resolved.
Find The Checks And The ApprovalNear the bottom you'll see the automated tests and at least one reviewer's approval, often a plain LGTM. Ask what would have happened if a check had failed.
What makes a pull request easy for you to review, and could we write our briefs the same way?
My Take
Pull requests taught me a lot about writing good briefs. The best ones are small, specific and honest about why, which is exactly what I want from a product brief too.
OpenAI Switches Off Its Assistants API After A Year's Notice
LGTM 041 · By Elle Anderson · Published 31 August 2026 · Story From 26 August 2026 · 2 min read
The Short AnswerOpenAI's Assistants API stopped working on 26 August 2026. Products still calling it now get errors, and the replacement asks teams to own more of the logic themselves.
What Happened
OpenAI retired its Assistants API on 26 August 2026, a year after announcing the deprecation in August 2025. Its documentation points developers to the Responses API and Conversations API instead, mapping old threads to conversations and old runs to responses. Commentators noted that calls to the old endpoints now fail outright rather than degrading gradually. Some products built on the API, such as the chat widget maker OpenWidget, turned off their integrations and asked customers to export chat history beforehand.
Why It Matters For Product People
A year sounds like plenty of notice until the feature in question belongs to nobody. If a prototype or internal tool quietly grew into something customers rely on, a hard switch-off turns into a live incident overnight.
The migration isn't a simple rename either. In the new model your own code manages the back-and-forth of tool calls that the old API handled for you, which is real engineering work and deserves a proper estimate.
Which third-party APIs in our product have a published retirement date, and who owns each one?
My Take
Deprecation notices are among the kindest things a platform can give us, and also the easiest to ignore. I keep a simple list of every external service a product needs, with dates and owners beside each. It's dull and it saves a lot of late nights.
LGTM 040 · By Elle Anderson · Published 24 August 2026 · Story From 18 August 2026 · 2 min read
The Short AnswerApple has moved every developer in the EU onto one set of business terms. The headline change is a 5 per cent commission on sales made in apps distributed outside the App Store.
What Happened
On 18 August 2026 Apple announced new terms for apps distributed in the European Union, saying they followed close work with the European Commission and settled its dispute over the Digital Markets Act (DMA). From 1 October a 5 per cent Core Technology Commission replaced the old per-install Core Technology Fee for apps distributed outside the App Store. Inside the App Store, rates now depend on how you take payment: Apple's own in-app purchase, an alternative payment provider or a link out to the web. Apple also added parental gates for younger users and kept its basic Notarization check for apps distributed outside its store.
Why It Matters For Product People
If your product sells anything digital to customers in the EU, the cheapest route to market may have changed. Payment choice, distribution channel and membership of programmes such as the Small Business Program now each shift the commission you pay.
There's a commitment hidden in the detail too. Developers must keep their chosen payment options for 12 months, so this is a decision worth modelling properly rather than a setting you can flip back next sprint.
Which payment and distribution route gives our EU customers the best experience at a margin that still works for us?
My Take
Fee changes tend to land on a finance spreadsheet and stay there, yet they quietly shape the product. I'd bring design, engineering and finance into one short session to compare the routes side by side. A clear choice made together usually beats a clever one made alone.
Why Retries Turned A GitHub Slowdown Into A Long Day
LGTM 039 · By Elle Anderson · Published 17 August 2026 · Story From 17 August 2026 · 2 min read
The Short AnswerA record traffic peak overloaded load balancers in one GitHub datacentre on 17 August. A retry bug then multiplied the traffic, and recovery took most of the working day.
What Happened
On 17 August 2026, from 13:40 UTC, GitHub users saw failures across issues, pull requests, the API, Actions and Copilot for around seven hours. GitHub's August availability report says a new traffic peak pushed one datacentre's load balancers past their limits, and a service mesh component didn't scale up. A hidden client retry bug then sharply increased traffic to an internal sign-in service. About 29,000 organisations saw at least one failed or slow request, and no data was lost.
Why It Matters For Product People
Retries feel helpful, since a failed request can simply try again. When thousands of clients do that at the same moment, the retries become the load that keeps a struggling system down.
This matters for any product with an app, an integration or a partner API. Good retry design waits a little longer each time and gives up gracefully, and it belongs in your definition of done.
When one of our services slows down, do our own retries help it recover or add more pressure?
My Take
Retry logic rarely gets a mention in planning, yet it shapes how gracefully a product fails. I appreciate that GitHub named its own bug so plainly. That kind of honesty makes it easier for the rest of us to check our own systems.
Mozilla Rotates Its Firefox Signing Key After Accidental Exposure
LGTM 038 · By Elle Anderson · Published 10 August 2026 · Story From 10 August 2026 · 2 min read
The Short AnswerMozilla found that a private key used to sign some Firefox downloads had been saved by mistake in an internal code repository. It found no sign of misuse but replaced the key anyway and told users plainly what to do.
What Happened
On 10 August 2026, Mozilla announced it had revoked and replaced a GPG signing subkey used for some Firefox and Thunderbird files on Linux. An unencrypted copy had been committed to a private GitHub repository that only a small group of staff could access. Mozilla said its audit records showed no unauthorised access, and that most users need to take no action. People who install Firefox through RPM packages or check signatures by hand were given steps to import the new key.
Why It Matters For Product People
Secrets ending up in code repositories is one of the most common slips in software. What stands out here is the response. Mozilla treated the key as unsafe, rotated it and published clear guidance sorted by who needed to act.
Rotation is only painless when you've practised it. Some Linux users had to take manual steps to keep receiving updates, which shows the real cost of changing a key in a hurry. Product teams can plan ahead by asking how each important credential would be replaced long before it ever needs replacing.
If one of our signing keys or API secrets leaked today, who would replace it and how long would it take?
My Take
Incidents like this worry me far less than silence would. Everyone makes mistakes with secrets eventually, and what earns trust is a calm, honest and well-organised response. I'd happily borrow the shape of Mozilla's notice: what happened, what we checked and what you need to do.
One Hijacked Account Tainted Hundreds Of Code Packages
LGTM 037 · By Elle Anderson · Published 10 August 2026 · Story From 4 August 2026 · 2 min read
The Short AnswerOn 4 August 2026, an attacker took over one open source maintainer's GitHub account and published tainted versions of popular JavaScript packages. The malware stole developer secrets and copied itself into hundreds more packages.
What Happened
On 4 August 2026, attackers used a compromised GitHub account to publish a malicious version of keyv, a widely used npm package, and then related caching packages. The code ran during installation, stole credentials such as cloud keys and access tokens, and used stolen publishing rights to infect other packages. Wiz counted more than 400 affected packages that day, and Singapore's Cyber Security Agency issued an advisory on 6 August. Security teams advised anyone affected to rebuild their systems and rotate every exposed credential.
Why It Matters For Product People
Modern products are assembled from thousands of pieces written by other people, many of them volunteers, and that's usually a strength. The weak point is trust that never gets re-checked. The core packages here had around two billion monthly downloads between them, and the infection spread to hundreds of others within a day.
For product people the practical question is how fast the team can respond. Can someone say within an hour whether a bad version reached a build? Lockfiles, a clear list of what's inside each release and short-lived credentials can turn a frightening week into a dull afternoon.
If a package we depend on turned malicious this morning, how quickly could we tell whether it reached production?
My Take
Open source maintainers carry an enormous load for very little thanks, and they're the first people harmed by an attack like this. The kindest thing a product team can do is make room for the boring safeguards. I'd rather spend a sprint on that than on an incident review.
LGTM 036 · By Elle Anderson · Published 3 August 2026 · Story From 2 August 2026 · 2 min read
The Short AnswerSince 2 August 2026, products in the EU must tell people when they're talking with an AI and make AI-generated content identifiable. The heavier high-risk rules have moved to later dates.
What Happened
The transparency obligations in Article 50 of the EU AI Act began applying on 2 August 2026. They cover telling users when they're interacting with an AI system, marking generative AI output as artificially generated and labelling deepfakes. A separate amending law, the Digital Omnibus on AI, entered into force on 27 July 2026 and pushed the high-risk obligations for areas such as employment and education to 2 December 2027. Providers whose systems were already on the market before 2 August have until 2 December 2026 to add machine-readable marking.
Why It Matters For Product People
If your product has a chatbot or generates images, audio, video or text for EU users, disclosure is now a product requirement rather than a nice-to-have. It touches copy, interface design and the technical pipeline that tags generated content.
The new dates for high-risk systems give teams in hiring, education and similar areas more breathing room. That's useful time for building good evidence and documentation, not a reason to park the work.
Where in our product could a user be unsure whether they're dealing with a person or an AI?
My Take
Telling people when they're talking with a machine is simply good product manners. I'd treat this as a design brief first and a legal checkbox second. Users notice honesty, and it tends to build the trust every product needs.
What OVHcloud Learned Patching A Million Virtual Machines
LGTM 035 · By Elle Anderson · Published 27 July 2026 · Story From 20 July 2026 · 2 min read
The Short AnswerFacing a serious flaw in the Linux hypervisor, OVHcloud chose to patch and reboot its whole fleet without waiting for each customer's agreement. Its candid write-up shows what went well and what didn't.
What Happened
On 20 July 2026, OVHcloud's chief information security officer Julien Levrard published a detailed account of how the company handled Januscape, a critical flaw in the Linux KVM hypervisor made public on 6 July. OVHcloud patched tens of thousands of host servers running about a million virtual machines, starting in its Sydney region on 8 July and finishing its last regions by 19 July. It rejected live patching and full live migration, and instead rebooted hosts in waves with automatic stop thresholds, taking care not to restart the same customer's machines together. Some machines failed to restart, some services saw data corruption, APIs struggled under load and some customers weren't contacted directly. OVHcloud says it will do better next time.
Why It Matters For Product People
This is a clear example of a hard trade-off made deliberately. Patching everyone quickly meant outages for some customers, especially those relying on a single server, and a few saw data corruption. Leaving the flaw open would have exposed far more people for far longer.
It's also a reminder that resilience is shared between provider and customer. If one host restarting would take your product down, a provider's emergency patch can become your outage, and spreading workloads across machines is the protection.
What happens to our product if our cloud provider reboots one of our servers without warning?
My Take
I really value write-ups like this, because they show the reasoning and the mistakes side by side. Choosing the option that protects the most people, then being honest about who it affected, is good leadership in any field. I'd share it with any team that relies on someone else's servers.
How A Clock Set To 2006 Disrupted Telstra's Network
Editor's Note, November 2026: When I wrote this, Telstra's external investigation was still under way. It reported on 2 September 2026 and found the outage began at 2:50am, about an hour earlier than first thought, after a GPS card reset during an untested change to timing equipment in Melbourne. It also found gaps in ownership and oversight, and Telstra has accepted its recommendations while the regulator's own investigation continues. Illawarra Mercury: Lack of oversight, expertise drove major Telstra outage.
LGTM 034 · By Elle Anderson · Published 20 July 2026 · Story From 17 July 2026 · 2 min read
The Short AnswerTelstra has explained its 8 July outage: a timing server restarted with the wrong date, and the bad time spread through the network. Telstra says earlier supplier warnings hadn't been acted on.
What Happened
On 17 July 2026, Telstra told an Australian Senate inquiry what caused its outage of 8 July, which at its peak affected 45 per cent of its calls and data sessions, including some triple zero emergency calls. After scheduled maintenance, a network time server around 15 years old restarted, reset its clock to 2006 and shared that time with other equipment, which then rejected certificates as invalid. Telstra said an earlier design change hadn't been documented and a software update that could have prevented the fault hadn't been applied, despite supplier warnings in 2020 and 2026. Chief executive Vicki Brady apologised, and Telstra has updated the server, commissioned an external investigation and is cooperating with the regulator.
Why It Matters For Product People
Time is one of those invisible dependencies nobody thinks about until it breaks. Certificates, logins and payments all assume clocks agree, so one wrong clock can look like a security failure everywhere at once.
The deeper lesson is about small, unglamorous work. An undocumented change and a deferred update each looked harmless on its own. Together they turned routine maintenance into a national outage, which is why maintenance records and patch backlogs deserve real attention.
Which old pieces of our system would surprise us if they restarted tomorrow?
My Take
I have a lot of sympathy for the engineers here, because deferring an update is a choice every team makes under pressure. What I'd take away is that a warning needs an owner and a date, or it slowly fades into background noise. Writing down why a system behaves the way it does is a gift to whoever comes next.
The Rare Update Path That Bricked Some Hue Bridges
LGTM 033 · By Elle Anderson · Published 13 July 2026 · Story From 13 July 2026 · 2 min read
The Short AnswerA Philips Hue firmware update left some Bridge Pro hubs unusable, but only under one unusual set of conditions. Philips is replacing affected devices free of charge.
What Happened
In July 2026, Philips confirmed that a firmware update released in June had left some Hue Bridge Pro hubs inoperable, showing a red light and losing contact with the app and connected lights. Philips said it happened only in a specific scenario: automatic updates were off, the hub had stayed on an older version for a long time and the update was then installed manually after sitting on the bridge for more than 10 days. The company said fewer than 100 devices were affected and that it would replace them regardless of warranty. On 13 July it released a further update to protect hubs that hadn't been affected.
Why It Matters For Product People
Most testing follows the happy path: devices that update automatically and stay current. Real customers switch things off, postpone updates and come back months later, and those long-tail routes are exactly where rare failures hide. It's worth knowing how many customers have automatic updates switched off and testing that group on purpose.
The response offers useful lessons too. A clear explanation, a free replacement and a fix for everyone else turned a bad week into a story about trust. The weak spot was the lack of a configuration backup, so replaced hubs had to be set up from scratch.
Do we test updates on devices or accounts that have been left untouched for months?
My Take
Fewer than a hundred devices sounds small until one of them is yours. I'd treat every unusual upgrade path as a real customer journey, because someone is always on it. And I'd build backups before I needed them, since recovery is part of the product.
LGTM 032 · Tutorial · By Elle Anderson · Published 6 July 2026 · 3 min read
The Short AnswerTech debt is easy to dismiss and hard to explain. A few thoughtful questions help you understand it well enough to make fair trade-offs.
What You'll Learn
Tech debt is the extra effort a team pays later because of shortcuts or ageing code today. Engineers often feel it long before anyone else can see it. This tutorial gives you a set of questions that turn a vague worry into something you can plan around together.
Ask Where It Hurts MostRather than asking how much debt there is, ask which parts of the product are slowest or riskiest to change. Debt matters most in the areas the team touches often, so that's where to focus first.
Ask What It Costs TodayAsk for a recent example where the debt made a piece of work slower, riskier or harder to test. A specific story is easier to weigh than a general feeling, and it helps you explain the cost to others.
Ask What Would Change If It Were FixedFind out what paying it down would unlock. It might be faster releases, fewer incidents or a feature that's currently too awkward to build. That link to outcomes is what makes the work easier to prioritise.
Ask For Small StepsBig rewrites are hard to fund and risky to finish. Ask whether the work can be broken into pieces that each leave things a little better. Small, steady improvements often fit inside normal roadmap work.
Agree How You'll Keep TrackAsk the team to keep a short, visible list of the debt that matters most, with a rough cost and benefit for each item. Review it together every so often, so it's part of planning rather than a separate battle.
Which part of our product would you most like to make easier to change, and what would that let us do next?
My Take
I've learned that tech debt conversations go best when I come with curiosity rather than a verdict. Asking where it hurts and what fixing it would unlock helps me make fairer choices and shows the team I take their concerns seriously.
Notion Retires Its Inbox As Agents Handle The Email
LGTM 031 · By Elle Anderson · Published 29 June 2026 · Story From 25 June 2026 · 2 min read
The Short AnswerNotion is shutting down Notion Mail just over a year after its wide launch, saying more than half its users now manage their email without opening the inbox.
What Happened
On 25 June 2026, Notion announced that Notion Mail, its Gmail-based email client, will shut down on 22 September. The company said more than half of Notion Mail users manage email without ever opening their inbox, and that it's going all in on agents instead. Emails stay in Gmail, but drafts, scheduled messages, custom views and reminders won't carry across. HIPAA-covered customers had to move off the service by 30 June. Notion Mail became widely available in April 2025, following Notion's 2024 acquisition of Skiff. Notion says its email-based agents will keep working after the shutdown.
Why It Matters For Product People
This is a rare, honest signal of how AI changes behaviour. When a tool does the job well enough, people stop visiting the screen you designed, and usage data can reveal that long before anyone says it out loud. Notion chose to follow that signal rather than keep polishing a screen fewer people opened.
Much of the shutdown is a good model too: a clear date, a plain reason and a list of exactly what users will lose, though the short deadline for HIPAA customers shows how much notice matters. If you ever retire a feature, that list is the part customers remember. Telling people what to export, and by which date, saves everyone a stressful week.
Which of our screens would people stop opening if an agent did the job for them?
My Take
I admire a team that follows the evidence even when it means retiring something they built with care. The lesson I'd take is to measure outcomes rather than visits. What people really want is their email handled, and an inbox is only one way to get there.
LGTM 030 · By Elle Anderson · Published 22 June 2026 · Story From 22 June 2026 · 2 min read
The Short AnswerCloudflare is working with the makers of Chrome, Edge and Firefox on a privacy-preserving token that could replace many CAPTCHAs. It's an early proposal, but it hints at how the web may treat AI agents.
What Happened
On 22 June 2026, Cloudflare announced it was developing Private Access Control Tokens (PACT) with Google, Microsoft and Mozilla, with Shopify also involved. The idea is that a site with strong knowledge of a visitor issues an anonymous token, which the browser can show to other sites as evidence the session comes from a person or a legitimate agent. The partners say the tokens are designed so they can't be used to track people, and the work builds on the IETF's Privacy Pass standard. No deployment timeline has been announced, and The Register noted open questions about who decides what counts as legitimate traffic.
Why It Matters For Product People
Cloudflare's own data, cited by The Next Web, suggests automated systems now make more requests for web content than humans do. That's pushing sites towards blunt defences like CAPTCHAs and forced logins, which cost real customers time and, in shops, sales.
If your product is a website, or an AI agent that visits websites, this is worth watching. A shared way to vouch for good traffic could remove a lot of friction, but it could also become a gate your users or your agent need permission to pass.
How much friction do our genuine users hit because we can't tell them apart from bots?
My Take
I'm cautiously hopeful about this one. Fewer puzzles for real people would be a kindness, and I like that privacy is part of the brief from the start. I'd still keep an eye on who ends up holding the keys, because standards quietly decide who gets through the door.
LGTM 029 · By Elle Anderson · Published 15 June 2026 · Story From 10 June 2026 · 2 min read
The Short AnswerNew research from Writer found that memory features can nudge AI models into agreeing with a user's mistakes. In the tests, the more a model remembered about a user, the more it tended to agree with them.
What Happened
On 10 June 2026, the enterprise AI company Writer published two papers on how memory and personalisation affect sycophancy, a model's habit of agreeing with the user. Testing memory systems including Mem0 and Zep with several model families, the researchers found memory raised sycophancy in every condition they tried, by up to 25 times the baseline rate. In one test, a model that correctly judged a company's finances changed its answer to match a misconception the user had stored. The authors suggest saving the assistant's replies alongside user messages and summarising context before it's committed to memory. TechCrunch reported that the effect grew as user input took up more of the model's context window.
Why It Matters For Product People
Memory feels like an obvious win, because nobody enjoys repeating themselves. This research shows the trade-off: a remembered preference or a wrong assumption can quietly steer answers that have nothing to do with it, and the user won't see why.
If your product remembers things about people, treat what goes into memory as a design decision. Decide what's worth storing and let users see and edit it. Then test whether the model still disagrees when the facts call for it, and what happens when an old memory stops being true.
How would we know if our product's memory was making its answers worse?
My Take
I love features that make people feel known, but being known shouldn't mean being flattered. I'd want any memory feature tested for honesty as carefully as it's tested for delight. A good assistant, like a good colleague, will still tell you when you've got something wrong.
When A Support Chatbot Could Hand Over Your Account
Editor's Note, November 2026: Meta had in fact filed a breach notice with the Maine attorney general just before this note went out, and I missed it. The notice names up to 20,225 affected accounts and says the attacks began around 17 April 2026, so they ran for about seven weeks before Meta spotted them on 31 May. That makes the case for guarding account recovery even stronger. The Decoder: Instagram AI chatbot breach may have affected over 20,000 accounts, Meta discloses.
LGTM 028 · By Elle Anderson · Published 8 June 2026 · Story From 1 June 2026 · 2 min read
The Short AnswerAttackers persuaded Meta's AI support assistant to link Instagram accounts to email addresses they controlled, then reset the passwords. Meta says the issue has been fixed.
What Happened
Over the weekend of 30 and 31 May 2026, a number of Instagram users reported that their accounts had been taken over, including some high-profile ones. 404 Media and TechCrunch reported that attackers asked Meta's AI Support Assistant to add a new email address to someone else's account, then used the verification code sent there to reset the password. On 1 June, Meta spokesperson Andy Stone said the issue had been fixed, and by 3 June Instagram was sending security alerts and password resets to affected users. Meta hasn't said how many accounts were affected.
Why It Matters For Product People
An AI assistant that can change account details is part of your security system, whether or not anyone designed it that way. Account recovery is one of the most attacked flows in any product, and a helpful assistant that skips the checks a trained human would follow gives attackers a shortcut.
The practical fix is to give the assistant the same guardrails as every other route into an account. Sensitive actions like changing an email or resetting a password should need proof that only the real owner can provide, and a person should still be reachable when something goes wrong.
Which actions can our AI assistant take that we'd never let an unverified caller request?
My Take
To me, this is a reminder that helpfulness and safety have to be designed together, and that the riskiest step in a flow deserves the most care. When an assistant gains the power to act, it should inherit every rule a human agent would follow.
LGTM 027 · Tutorial · By Elle Anderson · Published 1 June 2026 · 3 min read
The Short AnswerFeature flags let your team switch a feature on for some people before everyone. Used well, they turn a risky launch into a calm, reversible one.
What You'll Learn
A feature flag is a switch in the code that turns a feature on or off without a new release. Combined with a staged rollout, it lets you start small, watch carefully and grow only when things look healthy. This tutorial shows how a product person can plan one with their team.
Ask Which Flags Already ExistAsk an engineer to show you the tool or settings where flags are managed. Notice how many there are, who can change them and how old some of them are. Old flags left in place are a quiet kind of tech debt.
Choose Your First AudienceDecide who sees the feature first. Common choices are your own team, a small share of customers or one region. Pick a group big enough to learn from and small enough that a problem stays contained.
Agree What Healthy Looks LikeBefore switching anything on, agree a few signals to watch, such as error rates, support tickets and the metric the feature should improve. Write down what would make you pause the rollout or switch the feature off completely.
Plan The Steps And The PausesSketch the stages, for example staff first, then a small slice, then half, then everyone. Leave time between stages to look at the signals. A rollout plan is mostly a list of moments to stop and check.
Plan The Cleanup TooOnce the feature is fully live, the flag should be removed from the code. Add a ticket for it at the start, so it doesn't linger. Your future team will be grateful.
If this feature misbehaves on day one, who can switch it off and how quickly?
My Take
I find feature flags take a lot of drama out of launches. They let me say yes to an ambitious idea while keeping a calm way back, which is a comfortable place for any team.
LGTM 026 · By Elle Anderson · Published 25 May 2026 · Story From 18 May 2026 · 2 min read
The Short AnswerDiscord says end-to-end encryption is now standard on its voice and video calls, with no setting for users to change. Getting there needed every app to support it.
What Happened
On 18 May 2026 Discord said end-to-end encryption is now standard for every voice and video call on the service, with no opt-in needed. It uses DAVE, an open-source protocol Discord introduced in September 2024 and had audited by the security firm Trail of Bits, and the migration across desktop, mobile, web, consoles and apps finished in early March. Stage channels, designed for broadcasting to large audiences, are excluded and text messages aren't covered. Discord also said it's removing the fallback code that allowed unencrypted calls.
Why It Matters For Product People
Every client, from games consoles to bots, had to support the new protocol before the old unencrypted path could be retired. That long tail of older versions is often the slowest and least glamorous part of any security upgrade, and it's worth planning for from day one. Removing the fallback is what turns a promise into a guarantee, because no call can quietly drop back to the weaker option.
Discord was also clear about the limits, explaining why large broadcast channels and text sit outside the scheme. It also published the protocol as open source and expanded its bug bounty to cover it. Being upfront about what a privacy feature doesn't cover helps people trust what it does.
Which old fallback path are we still keeping alive, and what would it take to retire it?
My Take
I admire migrations that actually finish, because so many stall at ninety per cent. Making the safe option the default meant nobody had to hunt for a setting. That's a kindness to users as much as a technical achievement. I'd love more products to treat privacy as the starting point rather than an extra.
Editor's Note, November 2026: When I wrote this, TechCrunch expected Workers to stay free until the end of August. In July Notion moved the start of billing to 15 October 2026, and since then Workers have used Notion credits. If you're trying them now, it's worth checking the cost in your workspace before anything important depends on them. PricingSaaS: Notion pushes back Workers beta credit billing by two months.
LGTM 025 · By Elle Anderson · Published 18 May 2026 · Story From 13 May 2026 · 2 min read
The Short AnswerNotion launched a developer platform that lets custom code and external AI agents work inside a shared workspace. It's a glimpse of to-do lists that people and agents share.
What Happened
On 13 May 2026 Notion launched its Developer Platform at a livestreamed event. It adds Workers, a sandboxed place to run custom code that can sync outside databases into Notion. It also lets external coding and support agents such as Cursor, Codex and Decagon be assigned work and have it tracked inside a workspace. Notion's help centre lists Workers and a new command-line tool in public beta and the External Agents API in private beta, with Workers needing a Business or Enterprise plan. TechCrunch reported that Workers would be free until the end of August so developers can experiment.
Why It Matters For Product People
Tools like Notion are becoming places where people and AI agents share the same task list. That raises fresh product questions about what an agent is allowed to read, what it can change and who reviews its work before anyone relies on it. Those are design decisions as much as technical ones, so they belong in product conversations early.
Betas and free periods are a great time to learn and a risky time to build something critical. If your team experiments, it's worth agreeing up front which data agents can see, where their actions are logged and how you'd pause them quickly.
If an agent changed something in our workspace overnight, how would we know?
My Take
I like that agents are getting a visible place to work, because hidden automation is the hardest kind to trust. Starting with one well-understood workflow usually teaches more than a grand rollout. Curiosity and caution can happily sit side by side. I'd also keep a person clearly responsible for each agent, the same way we'd name an owner for any other process.
LGTM 024 · By Elle Anderson · Published 11 May 2026 · Story From 5 May 2026 · 2 min read
The Short AnswerFor a little over three hours, many people couldn't reach websites ending in .de. The registry traced it to a routine security key change that produced signatures nobody could verify.
What Happened
On the evening of 5 May 2026, websites using Germany's .de domain became unreachable for many people for a little over three hours. DENIC, which runs the .de registry, said a routine DNSSEC key rollover produced signatures that couldn't be validated, so resolvers that check those signatures rejected answers for .de names. Because the faulty records affected how the whole zone is verified, even domains that don't use DNSSEC themselves were affected. DENIC said a fault in its in-house signing code wasn't covered by its tests, and that its three validation tools spotted the problem but the alerts weren't processed correctly.
Why It Matters For Product People
DNS is the internet's address book, and DNSSEC is a way of proving its entries haven't been altered. That's valuable, but it means a mistake in the signing step can make correct addresses look untrustworthy. Checks that fail safely protect people, and they can also turn a small error into a very visible outage.
The detail I'd share with any team is that the monitoring worked and the warning still didn't reach a person in time. Having alerts is only half the job. Someone needs to own them, test them and know how to respond when one fires.
When one of our alerts goes off at 10pm, who actually sees it?
My Take
Key rotations are the careful, routine work that rarely gets thanks, and DENIC explained what went wrong within days. I'd use this as a prompt to rehearse our own quiet maintenance tasks. The boring jobs deserve the same testing as the exciting ones. Some large resolver operators also switched off validation for .de temporarily, which shows how much the internet relies on quick, cooperative help.
How To Read An API's Documentation As A Product Person
LGTM 023 · Tutorial · By Elle Anderson · Published 4 May 2026 · 3 min read
The Short AnswerAPI documentation tells you what a service can and can't do long before your team builds anything. You can learn to skim it for the parts that shape your product.
What You'll Learn
An API is how one piece of software asks another to do something, and its documentation is the instruction manual. You won't need to write the code, but reading the docs helps you spot limits, costs and risks early. This tutorial uses a well documented payments API as a friendly example.
Find The Overview PageStart with the introduction or getting started page. It usually explains what the API is for, how it's organised and whether there's a safe test mode. A test mode means your team can try things without touching real customer data.
Look For The Main ObjectsMost APIs are built around a few nouns, such as customers, payments or orders. List them and what each one represents. That list often maps neatly onto your own product's language, and gaps between the two are worth noticing.
Check Authentication And LimitsLook for how requests are authorised and whether there are rate limits on how many requests you can send. Limits affect busy moments like a sale or a launch, so they're worth raising early.
Read The Errors SectionGood docs list the errors the API can return and what each one means. Each error is a moment your customer might experience. Ask your team how each important one will look on screen.
Find The Versioning And Change PolicyAPIs change over time. Look for how the provider announces changes and how long old versions keep working. That tells you how much upkeep the integration will need after launch.
Which errors from this API could a customer actually see, and what have we designed for each one?
My Take
I find API docs reassuring once I stop trying to read every line. A short skim for objects, limits, errors and change policy gives me better questions and fewer surprises later.
X Now Charges Twenty Times More For API Link Posts
LGTM 022 · By Elle Anderson · Published 27 April 2026 · Story From 20 April 2026 · 2 min read
The Short AnswerX raised the price of posting a link through its API from 1 cent to 20 cents. It's a timely reminder that someone else's price list can reshape your product very quickly.
What Happened
On 16 April 2026 X's developer account announced pricing changes to its pay-per-use API, and they took effect on 20 April. Creating a post that contains a link through the API rose from 1 cent to 20 cents, and X's developer documentation now lists $0.20 per request for a post with a URL. X said the changes were meant to curb spam and other misuse. Some services that post automatically, including the news aggregator Techmeme, stopped adding links to their posts in response.
Why It Matters For Product People
If part of your product posts, reads or syncs through someone else's platform, that price is a line in your costs that another company controls. A twentyfold rise on a single action can turn a cheap feature into an expensive one with only a few days' notice.
Platforms often change prices to shape behaviour, and X said as much here. It helps to know which of your features depend on one third-party call, so you can decide calmly whether to absorb the cost, change the design or step back from that channel. A short list of these dependencies, with an owner for each, is a cheap way to avoid a surprise.
If our most-used third-party API doubled its price next month, which feature would we change first?
My Take
Every platform has the right to set its own prices, and spam is a real problem for all of them. The part I'd focus on is our own exposure. Knowing what we rely on is far easier on a quiet day than a busy one. I'd much rather spot that risk in a planning session than in a monthly invoice.
Editor's Note, November 2026: When I wrote this, Vercel described a limited subset of affected customers. On 23 April 2026 it updated its incident page: it had found more compromised customer accounts, plus a separate and earlier compromise of a small number of accounts that may have come through social engineering or malware. The incident turned out wider than the first statement suggested, so the lesson about broad app permissions matters even more. TechCrunch: Vercel says some of its customers' data was stolen prior to its recent hack.
LGTM 021 · By Elle Anderson · Published 20 April 2026 · Story From 19 April 2026 · 2 min read
The Short AnswerVercel disclosed a security incident that began with a third-party AI tool an employee had linked to a work account. It's a clear case study in how broad app permissions can travel further than anyone intended.
What Happened
On 19 April 2026 Vercel said it had identified unauthorised access to some of its internal systems. According to Vercel and the AI start-up Context.ai, an attacker who had compromised Context.ai used an OAuth token to take over a Vercel employee's Google Workspace account, then reached environment variables that hadn't been marked as sensitive. Vercel said a limited subset of customers had credentials exposed, contacted them directly to recommend rotating keys and said its services stayed operational. Context.ai said it had detected and stopped unauthorised access to its own AWS environment in March.
Why It Matters For Product People
When someone clicks Allow on a new tool, they may be handing it much of the reach their own account has. Context.ai said the employee had granted broad permissions and that the company's settings allowed this. That's a common setup in plenty of workplaces, which is exactly why it's worth a calm look in yours.
The second lesson is about labels. Secrets marked as sensitive were treated differently from ones that weren't, so a small default choice shaped what was exposed. Product teams who design settings for developers can make the safer option the easy and obvious one.
Which third-party apps can act as our staff inside our work accounts today?
My Take
AI tools are arriving faster than most companies can review them, and that's understandable. I'd rather a team keep a simple, current list of what's connected than chase perfection. Small, regular checks tend to catch more than one big audit. None of this needs blame, just a little more care at the moment someone clicks Allow.
LGTM 020 · By Elle Anderson · Published 13 April 2026 · Story From 10 April 2026 · 2 min read
The Short AnswerBluesky's engineers published a frank write-up of an outage that hit about half its users. The root cause was a single endpoint with no limit on how much work it would take on at once.
What Happened
On 10 April 2026 Bluesky engineer Jim Calabro published a public post-mortem of an outage that peaked on Monday 6 April, when the service went down intermittently for roughly half of its users over about eight hours. Forbes reported more than 1,800 Downdetector reports that morning, with problems appearing resolved by 12:30pm Eastern time. The write-up explains that a new internal service sent a small number of very large requests to the one endpoint that had no concurrency limit, which used up available network connections and set off a cycle of crashes and restarts. A temporary workaround steadied things on the Monday and the real fix, adding the missing limit, shipped on Wednesday 8 April.
Why It Matters For Product People
Most systems guard against overload by capping how much work any single request can demand. Every endpoint here had that cap except one, and a new internal caller happened to find the gap. Changes like that, where a fresh consumer uses an old feature in an unusual way, rarely get the design review a brand new feature would.
The team also struggled to see what was happening, because their monitoring couldn't show which client was sending the heavy requests. Their follow-up plan centres on better visibility per client and on tracking requests that are few in number but huge in size. As the post puts it, "you should add extensive observability before you have an outage."
Which of our endpoints would happily accept a request a hundred times bigger than usual?
My Take
I'm always grateful when a team writes up an outage this openly, because the rest of us get the lesson for free. The fix itself was small, but finding it took days. That gap is exactly what good monitoring shortens. It's also why limits on every endpoint are worth treating as standard, even when the traffic looks gentle.
LGTM 019 · Tutorial · By Elle Anderson · Published 6 April 2026 · 3 min read
The Short AnswerSprint planning is where a team decides what it can realistically do next. Sitting in quietly and listening well will teach you how your team really thinks about effort.
What You'll Learn
Many teams start each sprint with a planning meeting, where they agree why the sprint matters, what they'll take on and roughly how they'll do it. If you're new to product work, watching one is a gentle way to see how ideas become tasks. This tutorial shows you how to be a helpful guest.
Ask To Join As An ObserverLet the team know you'd like to listen and learn, and that you won't be adding work. Most teams are happy to have a curious visitor. Read the backlog items beforehand so the names make sense. If you're the team's product owner, you'll take part rather than watch, but these listening habits still help.
Listen For The Sprint GoalA good planning session agrees one clear goal for the sprint. Notice how it's chosen and whether everyone can say it in a sentence. If it's unclear, that's a useful thing to raise gently afterwards.
Notice How Estimates Are MadeTeams often estimate in story points or rough sizes rather than hours. These are relative guesses about effort and uncertainty, not promises. Watch for the moments when estimates differ widely, because that's where hidden complexity usually sits.
Write Down The Questions The Team AsksEngineers will ask about edge cases, dependencies and unclear requirements. Each question is a hint about what your briefs could cover next time. Keep a quiet list for yourself.
Thank Them And Share One ObservationAfterwards, thank the team and share one thing you learned. Ask one question about something you didn't follow. It shows respect and makes the next invitation more likely.
When an estimate surprised you this sprint, what did you know that the brief didn't say?
My Take
I've found that the best way to understand a team's pace is to listen rather than push. Planning sessions show how much careful thinking goes into every estimate, and that makes me kinder when I write the next brief.
Reddit Will Label Helpful Bots And Check Suspicious Accounts
LGTM 018 · By Elle Anderson · Published 30 March 2026 · Story From 25 March 2026 · 2 min read
The Short AnswerReddit is adding labels for legitimate automated accounts and asking suspicious ones to prove a human is behind them. The aim is to confirm a person exists without learning who they are.
What Happened
On 25 March 2026, Reddit announced new measures against unwanted bots. Automated accounts that provide a service can register for an app label, which Reddit says will appear on account profiles from 31 March. Accounts that show signs of automation may be asked to confirm a human is behind them, using options such as passkeys or third-party checks, and those that fail may be restricted. Reddit says this isn't a sitewide requirement, and that it already removes around 100,000 accounts a day.
Why It Matters For Product People
Not all automation is bad. Labelling the helpful bots gives users context and gives honest developers a clear path, while making it easier to spot the accounts that hide what they are. Any product with reviews, messaging or user posts will face the same question as AI agents become more common.
The verification design is targeted rather than universal. Only accounts flagged by signals such as posting speed get asked, which keeps friction away from most people. Chief executive Steve Huffman said the goal is to "confirm there is a person behind the account, not who that person is", and that principle of asking for the minimum suits any team handling identity data.
Which of our checks ask users for more information than the decision really needs?
My Take
I'd love more products to treat friction as something you spend carefully, on the few cases that need it. Labelling good bots is a kind move too, because it rewards developers who play fair. The real test will be how gracefully Reddit handles the humans who get flagged by mistake.
What Cursor Learned From Not Naming Its Base Model
LGTM 017 · By Elle Anderson · Published 23 March 2026 · Story From 22 March 2026 · 2 min read
The Short AnswerCursor launched its Composer 2 coding model without saying it was built on Kimi K2.5, an open model from Moonshot AI. When a developer spotted it, Cursor quickly confirmed it and said it should have disclosed it from the start.
What Happened
Cursor, the AI coding company, launched Composer 2 in the week of 16 March 2026. A developer routing Cursor's traffic through a debugging proxy noticed a model ID that pointed to Kimi K2.5, an open source model released by Moonshot AI. On 22 March, Cursor's Lee Robinson confirmed the model started from an open source base and said about a quarter of the final model's compute came from that base, with the rest from Cursor's own training. Co-founder Aman Sanger said leaving it out of the launch post was a mistake, and Moonshot said the use came through an authorised commercial partnership with Fireworks AI.
Why It Matters For Product People
Many AI products are built on someone else's model, and that's perfectly normal. What customers increasingly want to know is which model, under what licence and with what data handling, because it shapes their own risk and compliance decisions. Being upfront costs very little at launch and saves a lot of trust later.
There's a quieter technical lesson too. The base model became visible because a model ID showed up in network traffic, and details often surface in ways teams don't expect. It's safer to assume anything your product sends can be seen, and to make the public story match it.
If a customer asked which models and vendors sit underneath our AI features, could we answer in one clear paragraph?
My Take
Building on an open model is a smart choice, and I'd be happy to say so loudly. What I take from this is how quickly a gracious correction can restore calm. When something's been left out, owning it plainly and fixing the process is the best response a team can give.
A New WordPress Workspace Runs Entirely In Your Browser
LGTM 016 · By Elle Anderson · Published 16 March 2026 · Story From 11 March 2026 · 2 min read
The Short AnswerA new service called my.WordPress.net lets anyone start a private WordPress site with no hosting, no domain and no sign-up. Everything lives in the browser, which brings real freedoms and real limits.
What Happened
On 11 March 2026, the WordPress project announced my.WordPress.net, a free workspace built on its open source WordPress Playground technology. The site runs in your browser and saves its content in the browser's own storage, so it's private by default and can't be reached from the public internet. Storage starts at roughly 100MB, a site can only be opened in the browser where it was created and the team recommends regular backups. Anyone who wants to publish can move a site to a normal WordPress host later.
Why It Matters For Product People
Most product people picture software as a front end in the browser talking to a back end hosted in the cloud. Here the whole application, including its stored content, sits on the user's own machine. That removes hosting costs and sign-up friction, which makes it a lovely way to learn, draft or prototype. It also shows how capable modern browsers have become as places to run real software.
The trade-offs deserve plain explanation for users. Data tied to one browser can be lost if someone clears their storage or switches device, and there's no server keeping a copy. If your team ever considers local-first features, backup, sync and clear warnings belong in the first version rather than a later one.
Which parts of our product truly need a server, and which could work on the user's device?
My Take
I love tools that let people try something before asking them to commit. A private, no-sign-up workspace is a generous way to learn. My only caution is to make the limits as visible as the freedoms, so nobody loses a month of writing by tidying up their browser.
Roblox Starts Rewriting Rude Chat Instead Of Hiding It
LGTM 015 · By Elle Anderson · Published 9 March 2026 · Story From 5 March 2026 · 2 min read
The Short AnswerRoblox has begun using AI to rephrase profanity in chat rather than replacing it with rows of hash marks. Everyone in the conversation is told when a message has been changed.
What Happened
On 5 March 2026, Roblox launched real-time AI rephrasing for filtered in-experience text chat, starting with profanity. For example, a hurried message containing a swear word can now come through as a plain 'Hurry up', and everyone in the chat sees a notice that it was rephrased. The feature covers chat between age-checked users in similar age groups and their Trusted Connections, and it works in every language Roblox auto-translates. Roblox says rephrased messages still count as violations and that serious behaviour is still handled by its wider safety systems.
Why It Matters For Product People
Blocking is the easy default for content filters, but hash marks break the flow of a conversation and leave people guessing what was said. Done well, this approach keeps the meaning while removing the harm, which is a more useful outcome for the reader. Transparency stays in place too, since the chat shows when a rewrite has happened.
Letting a model change someone's words is a big responsibility, and the guardrails here are worth copying. Tell people when it happens, start with one narrow use case and keep the existing rules in force. Roblox also says its specialised filters pass uncertain cases to larger models, which is a sensible way to balance cost and accuracy.
Where does our product block or hide something when a gentler correction would serve users better?
My Take
I like the way Roblox describes this, as a steering wheel rather than a stop sign. Most of us build stop signs because they're easier to design and test. This is a good prompt to look for places where a nudge would keep people moving, as long as we're honest whenever we've changed something.
LGTM 014 · Tutorial · By Elle Anderson · Published 2 March 2026 · 3 min read
The Short AnswerYou don't need a data science team to check whether an AI feature is any good. A small, honest evaluation with real examples will tell you more than any demo.
What You'll Learn
An AI evaluation, or eval, is a repeatable way of checking how well a model or prompt does the job you need. Product people are well placed to run the first one, because you know what good looks like for your customers. This learning path takes you from a blank page to a simple scorecard.
Write Down What Good Looks LikeBefore testing anything, describe in plain words what a great answer does and what a bad one does. Be specific to your product. Accurate, polite and short is a start, but name what accurate means in your context.
Collect Twenty To Fifty Real ExamplesGather real questions or tasks your customers bring, with any personal details removed. Include easy ones, awkward ones and a few that should be refused. Real examples usually teach you more than invented ones.
Score The Outputs By Hand FirstRun the examples through the feature and score each answer as pass or fail against your description. Note why each failure failed. This slow first pass is where most of the learning happens.
Group The Failures Into PatternsRead your notes and sort the failures into a few themes, such as made-up facts, wrong tone or missed instructions. A clear list of patterns gives your engineers something concrete to fix.
Rerun It Whenever Something ChangesKeep the examples and scores in a shared spreadsheet. Each time the prompt, model or data changes, run the same set again and compare. Your team may later automate parts of it, but the habit matters more than the tooling.
When we last changed the model or prompt, how did we know the answers got better rather than just different?
My Take
I find that a small evaluation done by hand is one of the most useful hours a product person can spend on an AI feature. It turns vague worries into specific patterns, and it gives the whole team a shared picture of quality.
How A Blank Value Took Cloudflare Customers Offline
LGTM 013 · By Elle Anderson · Published 23 February 2026 · Story From 20 February 2026 · 2 min read
The Short AnswerOn 20 February a cleanup job at Cloudflare sent a request with a missing value, and the system read it as 'all of them'. Some customers who bring their own IP addresses dropped off the internet for hours.
What Happened
On 20 February 2026, Cloudflare withdrew about 1,100 IP address ranges from the internet, including a quarter of those used by customers on its Bring Your Own IP service. Its postmortem explained that a new automated cleanup task queried an internal API for prefixes waiting to be deleted but sent the request without a value. The API read the empty value as a request for all of those customers' prefixes, and the task started removing them. Engineers stopped it about half an hour after the first alert, and full recovery took until 23:03 UTC. Cloudflare says no attack was involved.
Why It Matters For Product People
This is a classic API design trap. When a request leaves something out, the system has to decide what that means, and 'nothing' and 'everything' are very different answers. Safe defaults for missing input matter most on actions that delete or change things, so it's worth asking how your own APIs behave when a field is blank.
Cloudflare also said its tests missed the problem because the staging data didn't reflect real conditions. Its planned fixes include stricter API rules, health-checked staged rollouts and a circuit breaker that halts changes when too much disappears too quickly. Those ideas suit any product with bulk actions, from a batch email tool to an admin screen that archives records.
What happens in our system when a request to delete or update records arrives with its filter missing?
My Take
I appreciate postmortems like this one because they show how small the trigger usually is. Nobody decided to remove these customers' routes; a missing value and an optimistic default did it together. I'd treat 'what if this is empty?' as a standard product question for every destructive action.
Discord Plans To Start Every Account With Teen Settings
Editor's Note, October 2026: When I wrote this, Discord planned to start its global rollout in early March 2026. On 24 February it delayed that plan after a strong backlash and moved the launch to the second half of 2026. It promised more ways for adults to confirm their age, including a credit card, and said any facial age check would run entirely on the person's own device. The design lesson still stands, and the on-device promise is now firmer than the one I described. TechCrunch: Discord delays global rollout of age verification after backlash.
LGTM 012 · By Elle Anderson · Published 16 February 2026 · Story From 9 February 2026 · 2 min read
The Short AnswerDiscord says every account will get teen-appropriate settings by default from early March, unless the person behind it is confirmed as an adult. Where the age check happens is the most interesting design choice.
What Happened
On 9 February 2026, Discord announced a global rollout of what it calls teen-by-default settings, starting in phases from early March. Until an account is confirmed as an adult's, sensitive content stays blurred, messages from unknown people go to a separate inbox and age-restricted spaces stay closed. Adults can confirm their age with a video selfie that Discord says is analysed on the device, or with an ID sent to a vendor partner, which it says is usually deleted straight after the check. Discord also says a background model will recognise many adults from account signals without reading message content.
Why It Matters For Product People
Defaults do most of the work in any product. Making the safer setting the starting point, and asking people to opt out, changes who carries the effort. It also turns the age check into part of the core user journey, so its copy, speed and failure states deserve the same care as sign-up.
Discord's privacy promises rest on a front-end versus back-end choice. The selfie analysis runs on the person's own phone or computer, so the video shouldn't need to reach a server. Keeping sensitive processing on the device shrinks what a company stores and what could leak. Discord itself disclosed a security incident at a third-party vendor that handled age appeals last October, so the stakes are real.
Which sensitive checks in our product could run on the user's device instead of our servers?
My Take
Safety features often stumble on the details: the confusing prompt, the check that times out or the adult who gets wrongly flagged. I'd want to watch real people go through a flow like this before launch. Protecting younger users and respecting everyone's privacy can sit together, but only when a team designs for both from the start.
When A Trusted Update Button Pointed Somewhere Else
LGTM 011 · By Elle Anderson · Published 9 February 2026 · Story From 2 February 2026 · 2 min read
The Short AnswerNotepad++ has explained how attackers redirected some users' update requests to malicious servers for about six months. The fix makes the app check that every update is genuinely signed.
What Happened
On 2 February 2026, Notepad++ developer Don Ho published an account of how the popular free text editor's update traffic had been hijacked. Attackers compromised the shared hosting server behind the project's website in June 2025 and used weak update checks in older versions to send selected users to malicious downloads. The attackers lost direct access to the server in September 2025 but kept using service credentials that hadn't been changed, until the hosting provider shut them out in early December. Notepad++ has since moved to a new host, rotated credentials and added certificate and signature checks to its updater, and it's asking everyone to install the latest version.
Why It Matters For Product People
Most products have an update path, whether that's an app store release, a desktop updater or a script that pulls new code each night. If that path trusts whatever arrives, anyone who gets between you and your users can ship software in your name. Signing each update and checking the signature before installing means the app trusts who made a file, rather than the server that delivered it.
The other lesson sits in the timeline. The attackers lost direct access to the server in September but carried on because some internal service credentials hadn't been changed. Rotating secrets after any scare is dull work, and it's often the step that actually brings an incident to a close.
If someone took over the server that hosts our updates, what would stop our app installing whatever it was given?
My Take
I find this story reassuring as much as worrying, because the fix is so ordinary. Signed updates and fresh credentials aren't glamorous, but they're exactly the kind of quiet work a product team can champion. I'd rather ask the awkward question about our update path now than learn the answer from a security researcher.
LGTM 010 · Tutorial · By Elle Anderson · Published 2 February 2026 · 3 min read
The Short AnswerStatus pages and service level agreements tell you what your suppliers have actually promised. Knowing how to read them helps you set honest expectations with your own customers.
What You'll Learn
Almost every product leans on services run by someone else, such as payments, email or cloud hosting. Each one usually has a public status page and a written service level agreement. This tutorial shows you how to read both so you know what's promised and what isn't.
List The Services You Rely OnAsk an engineer to list the handful of outside services your product needs in order to run. Write them down with a link to each status page. Most teams find there are more than they first thought.
Learn The Three TermsAn SLA is the formal promise a supplier makes, often with service credits if they miss it. An SLO is the internal target a team sets itself. An SLI is the actual measurement, such as the share of requests that succeed.
Read What The Uptime Figure Really MeansA promise of 99.9 per cent uptime still allows roughly 43 minutes of downtime a month. Check how the supplier measures it, whether planned maintenance counts and what you get back if they fall short. Usually it's a credit rather than compensation for lost sales.
Browse The Incident HistoryMost status pages keep a record of past incidents. Scroll back a few months and notice how often problems happen, how quickly updates appear and how clearly they're written. That tells you a lot about the supplier's habits.
Compare Their Promise With YoursIf you promise your customers something, it can't be stronger than the weakest service underneath it. Sit down with your team and check that your own commitments are realistic, then agree what you'll say during a supplier outage.
If our most important supplier went down for an hour today, what would our customers see and who would tell them?
My Take
I used to think of status pages as something only engineers watched. Now I see them as part of the product promise, and reading them helps me make commitments I can keep.
What Gmail's Missing Spam Checks Teach About Failing Safely
LGTM 009 · By Elle Anderson · Published 26 January 2026 · Story From 24 January 2026 · 2 min read
The Short AnswerOn Saturday 24 January, Gmail's spam checks failed for several hours and inboxes filled with misfiled mail. Google kept delivering email and added a banner telling people those messages hadn't been scanned.
What Happened
On Saturday 24 January 2026, Gmail users began seeing promotional and social emails land in their Primary inbox, spam warnings on mail from known senders and some delayed messages. Google's Workspace status dashboard reported elevated failures in spam checking that began just after 5am Pacific time, with banners warning users that some messages hadn't been checked. Google later marked the issue resolved for all users, effective 9:55am Pacific, and said warnings could remain on mail received before the fix. It has promised to share an analysis once its internal investigation is complete.
Why It Matters For Product People
When a safety check fails, someone has to decide whether the product stops or carries on without it. Gmail kept delivering mail and told people plainly that a check was missing. That choice and the wording of the warning are product decisions as much as engineering ones, and they're best made calmly in advance rather than during an incident.
Delays matter more than they first appear. TechRadar noted that slower delivery also held back email-based two-factor codes, so a hiccup in one system can lock people out of entirely different services. Knowing which of your user journeys rely on a timely email or text helps you judge the real cost of a short delay.
If one of our checks failed tomorrow, would we pause the product or keep going and warn people?
My Take
I find it reassuring when a product admits what it can't do in the moment. A clear banner and regular status updates help keep trust intact while people wait for a fix. I'd much rather plan that message on a calm Tuesday than draft it during a busy Saturday.
Let's Encrypt Now Offers Certificates That Last Six Days
LGTM 008 · By Elle Anderson · Published 19 January 2026 · Story From 15 January 2026 · 2 min read
The Short AnswerOn 15 January Let's Encrypt made six-day certificates available to everyone. It's another sign that the security certificates behind websites are getting shorter lives.
What Happened
Let's Encrypt, the non-profit that issues free website certificates, announced on 15 January 2026 that short-lived certificates are generally available. They last 160 hours, just over six days, and site owners opt in by choosing the short-lived profile in their certificate software. It also made certificates for IP addresses generally available, and those must be short-lived. Short-lived certificates remain optional, while the default lifetime will fall from 90 days to 45 days over the next few years.
Why It Matters For Product People
Certificates are what let a browser trust that a site is genuine. When one expires, visitors see an alarming warning and many simply leave. Shorter lifetimes mean renewal has to be fully automatic, because no person can reliably renew something every few days. Certificate software does this through ACME, a standard protocol for requesting and renewing certificates. Anything still renewed by hand, or by a script nobody remembers writing, becomes a risk.
The security reasoning is sound. Revoking a compromised certificate doesn't work reliably, so a short life limits how long a stolen key stays useful. For product teams, the takeaway is to know who owns renewal for every domain you run and to have alerts well before anything expires. Certificates hiding in marketing sites, old campaigns and partner integrations are the usual surprises.
Do we know every domain we own and how each certificate gets renewed?
My Take
Certificate expiry is one of those problems that feels solved until a forgotten domain proves otherwise. I like changes that nudge teams towards automation, because automation is kinder to tired people. A short inventory exercise now could prevent a very public bad day later.
Bose Publishes Its Speaker API Before Switching Off The Cloud
LGTM 007 · By Elle Anderson · Published 12 January 2026 · Story From 7 January 2026 · 2 min read
The Short AnswerBose had planned to end cloud support for its older SoundTouch speakers in February. Instead it pushed the date to May and published technical details so developers can keep them useful.
What Happened
On 7 January 2026 Bose updated its plan for ending support for SoundTouch, a range of older Wi-Fi speakers and soundbars. Cloud support, first due to end on 18 February, will now end on 6 May 2026, when the app updates itself to a version that works locally. Streaming through AirPlay, Spotify Connect and Bluetooth will keep working, while features such as browsing music services inside the app will stop. Bose also published the SoundTouch API documentation so independent developers can build their own tools.
Why It Matters For Product People
Every connected product eventually reaches the end of its supported life, and customers remember how that ending is handled. Hardware that depends on a company's servers can lose much of its value overnight, so the plan for switching off deserves thought from the start. Bose's revised plan keeps everyday streaming working, which is a thoughtful place to begin.
Publishing an API is a generous and practical exit. It lets a community carry on where the company stops, and it costs far less than running servers indefinitely. Software products can borrow the idea through data exports, clear documentation and generous notice periods. Writing the sunset plan early also shows which parts of a product depend on you.
If we retired this product tomorrow, what would our customers still be able to do?
My Take
I'm always moved when a company treats an ending with care. Nobody throws a launch party for a graceful retirement, but customers notice and trust grows quietly. I'd love more roadmaps to include a sunset plan alongside the launch plan.
LGTM 006 · Tutorial · By Elle Anderson · Published 5 January 2026 · 3 min read
The Short AnswerAcceptance criteria are where a vague idea becomes something a team can build and test. A few simple habits make them clearer for everyone.
What You'll Learn
Acceptance criteria are the conditions a piece of work must meet before anyone can call it done. When they're clear, engineers and testers spend less time guessing and more time building. This tutorial walks you through writing them in a way your team will genuinely appreciate.
Start With The OutcomeBefore you list anything, write one sentence on what the customer should be able to do once this ships. Every criterion that follows should support that sentence, and anything that doesn't probably belongs in another ticket.
Write Each Criterion So It Can Be TestedA good criterion has a clear pass or fail. Instead of saying the page loads quickly, say what quickly means and on which kind of device. If a tester can't check it, rewrite it until they can.
Try The Given, When, Then ShapeMany teams like this format: given a starting situation, when the customer does something, then a specific result happens. It keeps each criterion small and makes edge cases easier to spot.
Cover The Unhappy PathsThink about what happens when things go wrong. What if the customer has no internet, enters the wrong details or presses the button twice? Engineers will ask these questions anyway, so answering them early saves a round trip.
Leave The How To The TeamDescribe what should happen, not how to build it. Naming a database or a framework in the criteria closes off better options. If you have a constraint, explain the reason so the team can weigh it fairly.
Read Them Aloud TogetherBefore work starts, spend ten minutes reading the criteria with an engineer and a tester. Questions that come up now are cheap. The same questions halfway through the sprint are much more expensive.
Which of my recent tickets left you guessing, and what would have made it clearer?
My Take
I've come to think of acceptance criteria as a kindness to the people building the thing. They help us all picture the same result before anyone writes a line of code.
Ubisoft Pauses Rainbow Six Siege To Undo Fraudulent Transactions
LGTM 005 · By Elle Anderson · Published 29 December 2025 · Story From 27 December 2025 · 2 min read
The Short AnswerAttackers flooded Rainbow Six Siege accounts with in-game currency and items on 27 December. Ubisoft switched off the game and its marketplace, then rolled the changes back.
What Happened
On Saturday 27 December 2025, players of Ubisoft's Rainbow Six Siege saw accounts receive around two billion in-game credits and rare cosmetic items, alongside reports of players being banned and unbanned. Ubisoft deliberately shut down the game and its in-game marketplace while it investigated. It said players wouldn't be punished for spending credits they'd been given and that it would roll back transactions made since that morning. By 29 December the game was open again with the rollback complete, though the marketplace stayed closed and Ubisoft hadn't said how the intrusion happened.
Why It Matters For Product People
Two things made recovery possible. Ubisoft could switch the whole service off quickly, and it had records good enough to undo a specific window of changes. Neither is glamorous, and both are easy to postpone until the day you need them. Ubisoft's posts on X also kept players informed while engineers worked.
Rollbacks have a human side too. Ubisoft told players early that they wouldn't be punished for spending what they'd been given, which likely helped keep a messy situation calmer. Deciding in advance how you'd treat customers caught up in an incident saves painful debates in the moment. It's worth writing that stance into your incident plan.
If bad data spread through our system for an hour, could we undo exactly that hour?
My Take
Holiday incidents are hard on everyone, and the people who restored this game deserve credit. What I notice is how much depends on preparation that nobody sees. A tested off switch and a clear record of changes can turn a disaster into a difficult weekend.
When A Genuine PayPal Email Carried A Scammer's Message
LGTM 004 · By Elle Anderson · Published 22 December 2025 · Story From 15 December 2025 · 2 min read
The Short AnswerScammers found a way to make PayPal's real notification emails carry fake purchase warnings. It's a useful reminder about every field where users can type.
What Happened
On 14 December 2025 BleepingComputer reported that scammers were abusing PayPal's subscriptions feature to send genuine emails from PayPal's own address. They put fake purchase text and a phone number in a subscription field meant for a customer service web address. PayPal's automatic notice about a stopped payment then carried their message, which they appear to have forwarded to targets through a mailing list. Because PayPal really sent the email, it passed the usual authenticity checks. PayPal said it was "actively mitigating this matter", and on 15 December Malwarebytes reported that the loophole had been closed.
Why It Matters For Product People
Any text a user can enter may end up somewhere you didn't expect, including messages your company sends with its full authority. Emails, receipts and notifications carry trust, so whatever flows into them needs the same care as anything on a public page. Customers have learned to trust your sender address, so misuse of it costs more than most bugs.
Validation is a product decision as much as a technical one. A field meant for a web address should only accept a web address, and it's worth checking what happens when someone types something else. Those rules are easier to agree early than to retrofit after misuse. A short list of allowed formats in each spec goes a long way.
Which fields that customers fill in can appear in emails we send?
My Take
I don't read this as a story about one company getting it wrong. Every product with user input carries the same risk, and people misusing it are patient and creative. A quiet review of where user text travels is one of the kindest things a team can do for its customers.
Rival AI Companies Give Agent Standards A Neutral Home
LGTM 003 · By Elle Anderson · Published 15 December 2025 · Story From 9 December 2025 · 2 min read
The Short AnswerOn 9 December the Linux Foundation launched the Agentic AI Foundation, a shared home for three early standards that many AI agents already use.
What Happened
The Linux Foundation announced the Agentic AI Foundation on 9 December 2025. Its founding projects are the Model Context Protocol, which connects AI models with tools and data, Block's open source agent framework goose and AGENTS.md, a simple file that tells coding agents how to work in a project. Until now each project was looked after by the company that created it. Platinum members include AWS, Bloomberg, Cloudflare, Google and Microsoft alongside the companies that contributed the projects. The foundation says technical steering committees set each project's roadmap, so no single member has unilateral control. Linux Foundation executive director Jim Zemlin said success would look like agents from many vendors adopting these shared standards.
Why It Matters For Product People
If your product connects with AI agents, or plans to, these are formats you'll probably meet. A neutral owner makes them a safer bet, because one vendor can't quietly change the rules or abandon them. It also signals which ideas the wider industry is backing, which helps when you're choosing a foundation.
Standards also shift the build or buy question. When tools speak a common language, you can swap models or agent platforms with less rework. That's leverage worth keeping before committing deeply to any one provider's way of doing things. Engineers will appreciate a product partner who asks about this early.
If we had to change AI provider next year, how much of our integration would we need to rebuild?
My Take
Standards are rarely exciting, but they're where long-term choices quietly get made. I'd encourage product people to learn the names here, even at a high level. Knowing what's shared and what's proprietary helps you spot lock-in before it becomes expensive.
An Urgent Security Fix Briefly Disrupted Cloudflare
LGTM 002 · By Elle Anderson · Published 8 December 2025 · Story From 5 December 2025 · 2 min read
The Short AnswerOn 5 December a change Cloudflare made to protect customers from a serious React flaw caused about 25 minutes of errors. The lesson is about how fast small changes travel.
What Happened
On 5 December 2025, from 08:47 to 09:12 UTC, around 28% of the web traffic Cloudflare serves returned errors. Cloudflare was protecting customers against React2Shell, a critical vulnerability in React Server Components, and had enlarged the buffer its firewall uses to inspect requests. A follow-up change switched off an internal testing tool through a global configuration system that reaches the whole network in seconds, and it triggered a long-dormant bug in older proxy code. Cloudflare said no cyber attack was involved and published a detailed postmortem the same day.
Why It Matters For Product People
The first change was rolled out gradually. The second one looked harmless, so it went through a system with no gradual rollout at all. Plenty of outages start this way: the risky-looking change gets care and the small tidy-up doesn't. The better question is whether a change can reach everything at once, whatever its size.
There's a real tension here. Security fixes need to move fast, and speed tempts teams to skip the usual checks. Cloudflare plans to bring staged rollouts, health checks and quick rollback to its fast-moving configuration too. It also wants systems to fall back to a known good state when something looks wrong. That's a pattern any team can borrow.
Which of our changes skip the gradual rollout, and who decides that they can?
My Take
I find this postmortem useful because it names the uncomfortable part plainly. Small changes deserve the same safety net as big ones, especially when everyone is moving quickly. I'd rather a team feel slightly slow than discover that their fastest path is also their least guarded.
How To Read An Incident Postmortem Without Getting Lost
LGTM 001 · Tutorial · By Elle Anderson · Published 1 December 2025 · 3 min read
The Short AnswerA postmortem is the clearest picture you'll ever get of how your product really behaves under stress. Here's how to read one calmly and come away with something useful.
What You'll Learn
When something breaks, good teams write a postmortem: a short document that explains what happened, why and what they'll change. You don't need to follow every technical detail to learn from one. This tutorial shows you what to look for and what to ask afterwards.
Ask For A Recent PostmortemAsk your engineering lead whether you can read a postmortem from the past few months. Pick one where customers noticed something, because the link to your product will be easier to see.
Start With The Summary And ImpactMost postmortems open with a few lines on what happened and who was affected. Note how long it lasted, which customers felt it and how the team first noticed it. Was it an alert, or a customer complaint?
Read The Timeline SlowlyThe timeline shows the order of events, often minute by minute. Look for the gap between the problem starting and someone noticing it, and between noticing and fixing. Those gaps usually teach more than the cause itself.
Look For Contributing Causes, Not CulpritsGood postmortems are blameless. They describe the conditions that let a mistake cause harm, such as a missing check or a confusing setting. If you catch yourself asking who did it, gently swap that for what made it easy.
Find The Actions And Their OwnersNear the end there's a list of follow-up actions. Check that each one has an owner and a date. Then ask whether any of them is waiting for roadmap time, because that's where you can really help.
Which follow-up action from our last postmortem is still open, and what would it take to finish it?
My Take
I find postmortems some of the most generous documents a team writes. Reading them with curiosity rather than judgement is a quiet way to earn trust, and it often shows me where the roadmap needs a little room for safety work.
Everything here works with a click. If you'd like to type, use the command line at the top and press RUN or Enter. These are the commands the Console understands.
help
Show this list.
ls
List every module in Computer Science For Product People.
open <module>
Open a module, for example open apis or open 3.
path
List the Learning Paths, or open one with path ai.
quiz <module>
Jump straight to a module's three questions, for example quiz security.
define <term>
Look up a term from any module, for example define latency.
lgtm
List my LGTM notes, then open one with read 1.
subscribe
Get new LGTM notes by email.
course
Open the AI Foundations course built on Stanford's free lectures.
report
Tell me about anything wrong or out of date.
whoami
A little about the person who built this.
home
Return to the start.
clear
Clear the command history.
exit
Leave the Console and return to andersonco.uk.
F1, F2, F3 and Esc
Open Help, the Guide or LGTM from your keyboard, or press Esc to return to the start. Press / to jump to the command line.
The date, time and weather in the ticker are your own, worked out from your approximate location by the site's host. Nothing is stored.
~/console/docs/report-an-issue
Report An Issue
Technology moves quickly, and I want every page here to be accurate. If you spot something wrong, out of date or unclear, please tell me. Your note comes straight to me, and corrections go into the next refresh.
I’ll use your email address only to reply about this report. It won’t be added to any newsletter, and you can ask me to delete it at any time. See the privacy notice.