09.1
Define Tasks and Data Splits First
Evaluation begins with who needs help and what work they need to complete. Factual lookup, relationships across documents, thematic synthesis, and visual localization require different success criteria. One generic question answering score is insufficient. Each question should specify intent, acceptable answers, necessary evidence, forbidden inferences, and conditions for no answer. Annotators need these rules and should record reasons for disagreements. Open synthesis can be evaluated through required themes and preserved disagreements without assuming one uniquely correct paragraph. Evaluation surveys and RAGAs offer useful approaches, while local tasks still require adapted rubrics.[R25][R26]
Splits should prevent leakage inconsistent with deployment. New restaurant use can require restaurant based splits, future use temporal splits, and new entity generalization entity separated near duplicates. Training, prompt development, and final testing should be distinct. Questions generated from community summaries and evaluated using those same summaries may overreward their wording. Independently authored and real user questions should be included, with question origin retained as an analysis variable. The evaluation set defines the range of use supported by a conclusion. Transfer beyond that range requires further testing.
09.2
Match Retrieval Metrics to Relevance Definitions
Precision measures relevant items among returned results, while recall measures recovered items among all relevant evidence. Both require a defined relevance set. With incomplete annotation, an unjudged result should not automatically count as irrelevant. Graded relevance can use nDCG to assess whether more important evidence appears earlier. Its gain function, discount, and cutoff must be stated. The demonstration here uses exponential gain, two raised to the relevance grade minus one, and a base two logarithmic discount on rank plus one. The ideal ranking uses all judged candidates, not only a reordering of retrieved results. Scores with different definitions should not be combined in one comparison table.
Graph retrieval also requires path completeness. Finding both endpoints without connecting evidence does not solve a multihop question. Finding a path while omitting a critical negation does not provide complete support. Questions can have a minimum evidence set or several interchangeable sets, with success requiring coverage of at least one. Global themes additionally require source and viewpoint coverage so repeated high frequency passages do not occupy every top position. Duplicate rate, source diversity, and irrelevant expansion should accompany ranking measures. Metrics aligned with evidence needs explain why apparently extensive retrieval can still produce poor answers. Ranking is an intermediate objective; evidence must ultimately support the task.
09.3
Assess Answer Quality and Citation Quality Separately
An answer can be correct while citing the wrong material, or faithfully repeat a source that is itself mistaken. Evaluation should distinguish factual correctness, faithfulness to supplied evidence, question coverage, and citation support. Break answers into verifiable claims and classify each as directly supported, partially supported, contradicted, or unsupported. Citation precision asks whether cited material supports the associated claim. Citation coverage asks whether claims requiring evidence receive support. A working link alone does not establish support. Answers generated through summaries should remain traceable to original material rather than ending at another model summary.
Uncertainty handling also needs assessment. Admitting missing evidence may be better than giving a complete unsupported response. Conflicting sources may require separate statements with temporal scope rather than forced synthesis. Count justified abstention and excessive abstention separately so refusing everything does not earn a misleading quality score. Bilingual reports and cross language answers should compare numbers, negations, quantifiers, and conditions because fluent translation can change factual boundaries. An effective review sample includes answerable, partially answerable, conflicting, and unanswerable questions. These conditions reveal whether a system recognizes when inference should stop.
Original design
09.4
Demonstrate Multimodal Contribution through Controls
Multimodal evaluation must establish that questions actually require multiple modalities. Some nominally visual questions can be guessed from titles or text alone. High scores then do not prove image understanding. MultiModalQA and WebQA offer research examples of complex multimodal questions, while Visual-RAG examines visually knowledge intensive queries. Using these benchmarks still requires checking versions, tasks, and evidence annotations; a dataset name does not guarantee visual necessity.[R30][R31][R21]
Useful conditions include text alone, image alone, complete fusion, and incorrect pairings, with comparable generators and budgets. If fusion improves on individual modalities and incorrect pairings damage the relevant questions, there is stronger evidence of cross modal use. Masking answer regions, removing headers, and substituting similar images can test whether answers change appropriately. Graph controls can compare no graph, random edges, real edges, and entity names alone. Every ablation should distinguish executed components from planned ones. This report supplies experimental templates rather than scores for model ablations that were not run. Explicit hypotheses without results are more useful than invented performance bars.
Original design
09.5
Calibrate Automated Judges and Retain Human Review
Language model judges can reduce large scale review effort, but they are measurement instruments requiring validation. A judge may prefer longer, more fluent answers or wording resembling its own, and may be persuaded by citations that only look convincing. Human judgments on a sample should first establish a reference, followed by agreement analysis across error types rather than only overall correlation. Low agreement categories may need revised rubrics, better evidence locations, or continued human assessment. RAGAs supports automated feedback, but an automated score alone cannot prove final correctness.[R25]
Judge instructions should permit uncertainty, require supporting locations, and conceal system names or author identities. Pairwise comparisons can randomize answer order and test order effects. Human annotation is not absolute truth either. Independent judgments, disagreement discussions, and rule revision are especially important for importance judgments in open summaries. High impact errors should retain cases rather than only scores so teams can inspect how failures occur. Results can report automated measures, human sampling rates, and major disagreement types together. Readers can then assess measurement reliability and understand why one score change may not justify deployment or replacement.
09.6
Report Distributions and Differences beyond Averages
Results should be grouped by question type, source, entity frequency, and evidence length. Average gains can arise from easy questions while hiding regressions on important categories. Systems evaluated on the same questions should use paired differences and report sample size and uncertainty. Resampling intervals require an explicit sampling unit. Treating every question as independent when several share a document can underestimate uncertainty. Reporting only the best of many attempts also exaggerates stability. Model randomness, indexing randomness, and annotation differences should be considered separately.
Statistical significance and practical value are different judgments. A small stable score gain may not justify additional construction cost, while an uncertain average change may matter for a small set of valuable relational tasks. A decision table should combine effect size, cost, error severity, and maintenance burden. Insufficient data should lead to an explicit uncertainty statement and a more discriminating next experiment rather than elaborate statistics over weak samples. The demonstration ranking in this report explains calculation only and does not estimate system effectiveness. Business percentages also receive no invented intervals without original samples. Reporting these boundaries enables further verification.