Practice question from Foundations of Modern AI Search
A colleague argues: 'We can skip building a labeled evaluation set — if users are happy and results look reasonable during testing, the system is working.' Based on the lesson's discussion of evaluation, what critical failure mode does this argument overlook?
Answer
!AI
Explanation
Qualitative inspection by developers tests only the queries developers think to try — a highly biased sample. Real users submit a far wider distribution of queries, including edge cases and vocabulary the developers did not anticipate. Without a labeled test set, there is no principled measurement of whether the system works on that real distribution, and no automated check to catch regressions when any component changes.