When a product that works in demo falls over under load

The demo works. One user, a warm laptop, and a presenter who already knows which button to press. The first week of real traffic is when the same product waits, times out, or returns a stale answer. Buyers call that a scaling problem. It is usually a path problem: the request was designed for an audience of one.

KeenCraft's data and backend architecture work is the pass that makes that path survive users. It is caching, pipelines, and retrieval sized to a measured bar, not a rewrite proposed because a dashboard looked red.

Load is a tail, not an average

A product can look healthy on an average and still lose the people who hit the slow path. Voice callers hang up. Checkout users retry and double-charge themselves. Staff refresh a CRM and act on yesterday's stage. The number that matters is the slow tail, and whether the answer is still the right answer when it finally arrives.

• Speed: time to first useful response, not time until a spinner disappears.
• Correctness: a fast stale cache is a bug with better graphs.
• Cost: a path that calls a model or a vendor API on every keystroke will be expensive before it is slow.

The Rawk.ai pass was infrastructure, not a prompt

On Rawk.ai, a live voice-agent builder, KeenCraft led a latency pass that dropped end-to-end response time from roughly 900ms to roughly 320ms. Callers do not forgive a 900ms gap after every sentence. They talk over the agent, the agent talks over them, and the call collapses.

That drop did not come from a cleverer prompt. It came from the pipeline: streaming speech-to-text instead of waiting for a perfect transcript, streaming the model instead of buffering a paragraph, starting audio on the first clause, prefetching tool results, and colocating the services so extra regions were not added to every turn. The phone-specific version of that budget is in how we ship production voice agents. The same failure mode shows up in products that never place a call.

Where the time actually goes

Most slow products are a chain of reasonable steps that were never allowed to overlap.

1. Sequential I/O: the page waits for the user, then the permissions, then the list, then the counts. Each call is fine. The sum is the demo that died.
2. A cache that is either missing or lying: no Redis tier in front of a hot read, or a tier that never invalidates when the write happens.
3. Queries that scan because the access pattern was never the index. Postgres will answer. It will answer late.
4. Retrieval on every request: embeddings, vector search, and a model call for a question the last hundred users already asked.
5. Cross-region hops added because each vendor defaulted to a different cloud. Tens of milliseconds you cannot prompt your way out of.

Postgres and Redis are a tier, not a slogan

PostgreSQL stays the system of record. Redis sits in front of the reads that are hot, identical, and safe to serve twice. The design work is the boundary: which keys exist, how long they live, and which writes delete them. A cache without an invalidation rule is how a paid invoice stays on 'new lead' in the UI the staff trusts.

The same boundary shows up when the product retrieves. Pinecone or pgvector belongs in the path only when the answer depends on search, not as a default layer under every screen. If the fact is already a column, read the column.

Audit the live path before you propose a rebuild

KeenCraft sizes this as an audit when the product already has users. Measure the slow request, find the sequential calls, and change the smallest layer that moves the tail. A caching tier or a single pipeline is often the engagement. A platform rewrite is the proposal you write after that pass cannot hit the bar.

The deliverable includes the notes: an architecture overview and the API behavior your team will still need when the person who tuned it is not in the room. Undocumented speed is a demo you will fail to repeat.

1. Name the one request users feel. Time it on production-like data, including the slow tail.
2. List every downstream call on that request, in order, with what each one returns.
3. Mark which results are identical across users and which must never be shared.
4. Put a cache only on the identical, safe reads, with an explicit delete on write.
5. Colocate the services that talk on every request. Leave the rare admin job wherever it is.
6. Re-measure the same request. If the tail did not move, the cache was not the bottleneck.

This is sold as a fixed Statement of Work, with the latency or correctness bar written down. If the product already works for one user and fails for the next hundred, that is the engagement. See shipped work and book a call if you can name the request that is slow. A proposal typically lands in 2 to 5 business days.