The Questions That Actually Separate Agencies From Agencies

If you're evaluating engineering vendors for a serious platform engagement, the RFP process most procurement teams run is optimized for the wrong signal. Case studies show you the best six months of a vendor's history, curated by their marketing team. Reference calls show you clients who agreed to be references, which selects for satisfaction. None of it tells you how the vendor behaves when the project is going badly, which is the only period that actually determines whether you'll regret the engagement.

Here's the checklist we'd want a buyer to run on any vendor, including us.

Ask who will actually write the code, by name, and ask to meet them before signing. Not "our senior team," not a bio page — the specific individuals. If the answer is vague, or if the people in the sales process aren't the people who'll be in your stand-ups, you've learned the most important fact about the engagement before it starts: you're buying a brand, not a team.

Ask what happens when the estimate is wrong. Every vendor's estimate will be wrong to some degree — that's not a red flag, it's a fact about software. What separates good vendors from bad ones is whether they have a defined process for surfacing and handling that divergence, or whether it gets absorbed silently into scope creep, quality shortcuts, or a change order ambush at the worst possible moment. Ask for an example of a project where the estimate was wrong and how they handled the conversation.

Ask to see a code review from a past engagement, redacted if necessary. This is uncomfortable to ask for and most vendors will resist, which is itself informative. A vendor with a genuine review culture will have something to show you — real comments, real disagreement, real evidence that more than one person's judgment shaped the code. A vendor that can't produce this either doesn't do meaningful review or doesn't want you to see what their review process actually looks like.

Ask what their on-call model is for the first ninety days post-launch. Launch is not the finish line for a serious platform — it's the point where real traffic and real edge cases start finding the gaps in your assumptions. A vendor's answer to "who gets paged and what's the SLA" tells you whether they're thinking about your platform as a product they're accountable for or a project they're closing out.

Ask what percentage of their engineering staff is subcontracted, and to whom. This isn't a question about offshore work being categorically bad — plenty of excellent engineers work anywhere in the world. It's a question about accountability chains. If your vendor subcontracts to a subcontractor, you have no direct relationship with the people writing your code, and no real leverage when something goes wrong beyond a contract dispute with your immediate vendor.

Ask for the failure story, not the success story. Any vendor worth hiring has had a project go sideways. Ask them to tell you about it, specifically — what broke, whose fault it was, what changed afterward. A vendor who claims a spotless record either hasn't done enough real work to have failed yet, or isn't being straight with you. The honest answer to this question is the single best predictor of how they'll treat you when your project has its own bad month.

None of these questions show up on a typical RFP scorecard, because they're harder to answer with a bullet point. That's exactly why they're the ones worth asking.