A complete Research Evidence Graph implementation

13.1 Scope and fictional source collection

This chapter develops an original implementation design for a small research team. The aim is to answer evidence questions about retrieval methods while preserving provenance and allowing corrections. The initial collection contains three fictional documents and a small set of explicitly invented records. It is intentionally small enough to inspect by hand. The design can later scale, but the first objective is to make every transformation understandable and testable.

Document P1, dated February 2025, reports an evaluation of GraphLens and PathSearch on Dataset D version 1. Document P2, dated May 2025, repeats the comparison on Dataset D version 2 with a different outcome. Document P3, dated June 2025, corrects a metadata field in P1 and explains that one reported score used a different query subset. None of these documents represents an actual publication. The fictional names prevent the worked example from being mistaken for an empirical claim about a real method.

The competency questions are specific. Which results compare the two methods on the same dataset version and metric? Which claim depends on the corrected query subset? Which sources support a statement of improvement, and which use conditions that prevent direct comparison? The system should answer these questions with source references and should decline to produce an overall winner unless an aggregation policy and comparable evidence justify one.

13.2 Minimum schema and records

The minimum schema contains Document, SourceSpan, Person, Method, Dataset, Evaluation, and Claim. A Document has a local identifier, title, version, date, and content hash. A SourceSpan has a locator and a reference to its document version. An Evaluation connects the compared methods, dataset version, metric, and reported measurements. A Claim expresses the interpreted result and links to the relevant Evaluation and source evidence. Review status belongs to the claim record, while bibliographic identity belongs to the document.

The synthetic measurements use mean reciprocal rank on a stated query set. P1 reports 0.42 for GraphLens and 0.38 for PathSearch on D version 1. P2 reports 0.40 for GraphLens and 0.41 for PathSearch on D version 2. These invented values support two context specific comparisons. They do not justify averaging the four numbers without considering query sets, uncertainty, and evaluation protocols. The correction in P3 makes the comparability question even more important.

For P1, the graph can store an attributed claim that GraphLens had the higher reported score under the specified evaluation. It should not store the universal claim that GraphLens is better than PathSearch. For P2, it stores the corresponding result under the second dataset version. A comparison view displays both contexts and the correction notice. The graph's value is that it retains the structure needed to interpret the disagreement instead of forcing the documents into one unqualified edge.

13.3 Ingestion and validation

The ingestion process saves immutable source versions and extracts their document structure. It identifies candidate entities and links them to the local method and dataset registry. The extractor emits candidate evaluation records with source locators and explicit missing fields. It does not fill absent confidence intervals or query counts from general knowledge. A validator checks identifiers, numeric datatypes, relation directions, and the presence of required context.

The first human review focuses on the meaning of each comparison and the correction's scope. The reviewer confirms whether P3 changes the numerical value, the population to which it applies, or only a descriptive field. That distinction determines which records need revision. If the correction affects comparability but not the original measurement, the graph retains the original result and adds a qualification. Deleting the entire result would lose useful history and overstate the correction's effect.

The accepted release receives a snapshot identifier and a small manifest listing source versions, schema version, extraction configuration, and review decisions. The search index and any generated summaries record that snapshot. A later reprocessing run can propose changes without altering the currently accepted view. This makes model experimentation possible while preserving a stable basis for users' ongoing research.

13.4 Querying the accepted evidence

A bounded query can retrieve accepted claims associated with a selected dataset and their source passages. The following Cypher example illustrates the intended pattern for a property graph implementation. It assumes the schema just described and a bound dataset identifier parameter. It is a teaching example, not a command executed against the user's existing database. Database credentials, deployment paths, and private records are outside this report.

MATCH (c:Claim {status: 'accepted'})-[:BASED_ON]->(e:Evaluation)
      -[:USES_DATASET]->(d:Dataset {id: $dataset_id})
MATCH (c)-[:EVIDENCED_BY]->(s:SourceSpan)-[:IN_DOCUMENT]->(p:Document)
RETURN c.id, c.text, p.id, s.locator
ORDER BY c.id

The query returns evidence records rather than a free form conclusion. A rendering layer can display the claim, document, and locator together. If several source spans support one claim, the interface should group them without treating each span as a separate study. If access restrictions remove one source, the system must avoid implying that the remaining visible set is complete. The answer can state the scope of the accessible evidence and any resulting limitation.

A corresponding RDF implementation would express the same domain entities through IRIs and predicates and use SPARQL graph patterns. The semantic design should remain recognizable across both versions. The purpose of the example is to show that storage choice can change query syntax without changing the core evidence commitments. Translation should be tested for qualifiers, identity, and provenance rather than judged only by whether node counts match.

13.5 Agent interaction and correction

Suppose a user asks, “Does GraphLens outperform PathSearch?” The agent resolves the method identities and retrieves the two evaluations and correction. It should answer that the fictional evidence reports different outcomes under different conditions, then identify those conditions. It should explain that the available records do not establish a universal winner. A useful response can still be concise: it presents the two results, notes the correction, and identifies what additional comparison would be needed.

If the user asks for a graph update after providing P3, the agent prepares a proposed change containing the affected claim, evidence locator, qualification, and expected snapshot. The validator checks the proposal, and the configured review policy decides whether it can be accepted automatically. The write records the new state and its provenance. Any dependent summary is marked stale and regenerated before it is served as current evidence.

If an extraction run incorrectly links GraphLens to an unrelated product, the repair operates on the mention to entity mapping. The source remains intact. The system identifies claims and summaries that depended on the incorrect mapping, revises those records, and records the repair. This example demonstrates why reversible identity decisions are essential. A graph that had overwritten the source names and discarded mappings would make the same correction much harder.

13.6 Testing the implementation

The first acceptance test checks identity: P1's and P2's GraphLens mentions map to the intended method, while an unrelated same name entity remains separate. The second checks context: the dataset versions and query subsets remain distinct. The third checks evidence: every accepted comparison resolves to an exact source location. The fourth checks correction propagation: P3 updates the relevant view and invalidates its dependent summary without erasing unrelated results.

A retrieval test asks both a local factual question and an ambiguous global question. The local question should return the specified score and context. The global question should preserve disagreement and avoid an unsupported ranking. An access test removes permission to P2 and verifies that neither its direct result nor a derived summary leaks into the answer. A recovery test retries an interrupted update and confirms that the idempotency key prevents duplication.

Performance testing comes after semantic correctness. The team measures ingestion time, review effort, query latency, and evidence assembly size on representative documents. It can then decide whether to add indexing, caching, or more advanced retrieval. Starting with these measurements keeps optimization tied to observed bottlenecks. A small reliable system provides a better foundation for expansion than a large demonstration whose correctness cannot be inspected.

13.7 A phased delivery plan

The first phase establishes scope, identifiers, a minimal schema, and a manually checked seed collection. The second automates parsing and extraction into staging, with review and error analysis. The third adds bounded retrieval and evidence based answers. The fourth adds controlled updates, correction propagation, and monitoring. More autonomous planning is introduced only after the earlier capabilities are measured. These are dependency based phases rather than fixed calendar promises.

Each phase has an exit criterion. The seed graph must answer the competency questions. Extraction must meet a locally chosen quality threshold on held out documents. Retrieval must recover the evidence needed for representative questions. Updates must preserve provenance and recover from failure. This plan provides an implementation path for a local Neo4j project while remaining applicable to RDF or another backend. The central deliverable is a trustworthy evidence workflow, with the database serving that workflow.