Your first BFCM tests whether the app works. Your second one tests whether it works when a lot of people use it at once.
Our second BFCM ran heavier than our first, and that was the point of the exercise. Last year we launched six weeks before Black Friday and mostly hoped. This year we had a full year of merchants, more stores running campaigns, and meaningfully more orders flowing through the app at peak. Nothing fell over. But a few things strained, and a postmortem is only useful if it is honest about the strain and not just the survival. Here is what our second BFCM broke, and what we fixed.
A note on numbers: we are not going to quote a headline figure for the weekend. What matters for an engineering writeup is not a dollar total, it is the shape of the load. Our peak was concentrated in a few hours, orders per minute ran well above anything we had seen outside a sale, and the concurrency, many stores hitting checkout at the same moment, was the real test. That concentration is what surfaces the bugs a calmer week never would.
Why checkout-time code is unforgiving
Discount Prime runs its discount logic inside Shopify Functions, which means the calculation happens natively in Shopify's own checkout engine, in-line, at the moment a customer is deciding to pay. That is the right architecture and we would choose it again. It avoids the price-manipulation hacks that genuinely fall apart under load. But it also sets a hard bar: the function has to be fast and deterministic every single time, because there is no slow path that is acceptable when someone is mid-checkout on Black Friday. A feature can be late. A checkout cannot be slow.
That framing decided every fix below. When we had to choose between adding something and making the existing path more predictable, predictability won.
What strained, and what we did about it
Analytics ingestion lagged at peak. Our discount analytics, which shipped in September, records what each discount does so merchants can see which offers make money. During the busiest hours, the ingestion side, the processing that turns raw events into reporting, fell behind. Crucially, this never touched checkout: the discount calculation and the recording of it are decoupled by design, so a customer's cart was never waiting on a report. But dashboards lagged, and a merchant refreshing their numbers mid-sale saw stale figures for a while. We widened the buffer between recording and processing so ingestion can fall behind and catch up without any pressure reaching the checkout path, and without the merchant seeing gaps once it caught up.
A combination-resolution edge case appeared at high concurrency. When several discounts qualify for one cart, the order in which they resolve matters, and we found a narrow case where an unusual combination could resolve inconsistently under heavy concurrent load. It was rare and it was caught, but "rare during BFCM" still means real orders. We made combination resolution fully deterministic for that case, so the same cart produces the same, correct result regardless of how much traffic is hitting the app at that instant. This is the kind of bug that is nearly impossible to see in a quiet week and obvious in a busy hour.
A scheduling boundary behaved unexpectedly. A store scheduled a sale to end at a specific local time, and the behavior at the exact boundary, for carts already open when the clock flipped, was not what the merchant expected. Nobody was overcharged, but the transition was less clean than it should be. We tightened the timezone-aware scheduling so the start and end boundaries are unambiguous, including for carts that straddle the moment the sale changes state.
What held up
Plenty worked, and it is worth naming so this reads as a postmortem and not a confession. The core volume discount path, the one most of our stores lean on, stayed fast and correct through the peak. The Functions architecture kept latency predictable exactly when it mattered. And the observability we added earlier in the year, the ability to actually watch latency and errors in real time, is the only reason this postmortem has specifics instead of vague suspicions. You cannot fix what you could not see, and last year we could not see nearly this much.
What we are changing before next year
Three things go on the list for our third BFCM:
- Load-test against realistic peak concurrency, not average throughput. The bugs live at the concentration peaks, so that is what the test has to reproduce.
- Make ingestion backpressure a first-class design goal, so reporting can always lag safely without any path to the checkout.
- Expand the deterministic-combination test matrix, because the resolution edge cases are where correctness quietly slips under load.
Setting this up with Discount Prime
For merchants, the takeaway is simpler than the engineering: the app you run your sale on should treat checkout-time correctness as non-negotiable, because that is the moment you cannot afford to be slow or wrong. That is the standard we hold volume discounts and our analytics to, especially at peak. If you want the merchant-side version of a smooth sale, the two posts next to this one cover it: deep or wide but not both for the strategy, and last-minute BFCM fixes for the conflicts you can still catch before the next one.
