Relation extraction and evidence preservation

4.1 Define what a relation asserts

Relation extraction asks whether a source expresses a particular relationship between identified arguments. The arguments may be entities, events, quantities, or claims. The task differs from simply finding two terms near each other. “GraphLens was compared with PathSearch” supports a comparison relation. It does not establish that either method outperformed the other. “The authors hope GraphLens will improve recall” expresses an intention or hypothesis, not an observed result. A graph construction system must preserve these distinctions if the graph will later support evidence based answers.

A relation inventory（关系集合）should define argument types, direction, meaning, and evidence requirements. USES_DATASET links an experiment to a dataset; AUTHORED_BY links a document to a person; REPORTS links a document to a claim. SUPPORTS and CONTRADICTS require more careful interpretation because support is usually conditional. A study can support one aspect of a claim while challenging another. The schema should either represent the relevant claim precisely or use a qualified assessment object that identifies the dimension being assessed.

Relationship direction is not a cosmetic choice. If a paper cites another paper, reversing the edge changes the interpretation of citation counts and traversal. If a company supplies another company, reversing the edge changes supply risk analysis. Extraction guidelines should include inverse formulations: “Dataset D was used by Experiment E” and “Experiment E used Dataset D” describe the same directed relation after normalization. Direction errors are easy to miss in a graph visualization because the connected entities still look plausible.

4.2 Sentence and document level extraction

A sentence level classifier predicts a relation for a pair of mentions within a sentence. It can exploit surrounding words and grammatical structure. This setting is useful when the relevant evidence is local and the schema is narrow. Its limitation appears when an argument is introduced elsewhere, a pronoun replaces an entity, or the relation depends on a sequence of statements. Treating every sentence independently then loses the information required for a correct interpretation.

DocRED was developed to support document level relation extraction, including cases requiring information across sentences [18]. The general lesson is that the unit of inference can be larger than the unit of mention detection. In the fictional example, one paragraph identifies GraphLens as the proposed method, another introduces Dataset D, and a results table reports the score. The accepted Evaluation record may therefore need evidence from several locations. A single sentence pointer would give a misleading impression of evidential completeness.

Joint extraction（联合抽取）models can share information between entity recognition and relation prediction. A likely USES_DATASET relation can help distinguish a dataset name from a method name, while entity types can constrain plausible relation labels. SpERT and DyGIE++ illustrate approaches that connect these tasks through contextual representations [16, 17]. This coupling can improve consistency, but it can also spread an early mistake. Evaluation should therefore inspect both the final relation and the argument identities on which it depends.

Generative extraction（生成式抽取）expresses relational outputs as a sequence. REBEL provides a notable example of relation extraction framed as language generation [19]. Such methods can produce several triples from a passage without a separate classifier for every candidate pair. The operational challenge is to distinguish valid structured output from faithful extraction. A syntactically correct triple may still add a relation absent from the source, omit a qualifier, or use a noncanonical entity label. Parsing is only the beginning of validation.

4.3 Negation modality and scope

Negation（否定）changes the meaning of a statement, but its scope can be difficult to determine. “The study did not find evidence that M improves recall” is not equivalent to “M reduces recall.” It also differs from “M never improves recall.” A graph that stores only a positive or negative edge loses these distinctions. The Research Evidence Graph represents the reported claim, its polarity, its modality, and the evaluation context. The accepted wording remains close enough to the source that a reviewer can check the interpretation.

Modality（情态）covers possibility, necessity, intention, expectation, and conditionality. “M may reduce cost” describes a possibility; “M must be run with parameter P” describes a requirement; “If the dataset is balanced, M improves precision” is conditional. An extractor that drops modal verbs can turn tentative language into a strong factual assertion. A practical test set should deliberately include these constructions rather than rewarding only simple affirmative sentences.

Quantifiers（量词）also matter. “Some studies report an improvement” and “All studies report an improvement” cannot be stored as the same relation. The first may be supported by a single study; the second depends on the defined population and completeness of the search. A claim about a proportion needs a denominator. If the source does not specify the population, the graph should preserve the phrase as a claim with unresolved scope rather than inventing a universal statement.

Temporal scope can create apparent contradictions. “Institute N hosts Project P” in an old annual report may coexist with a later report naming a different host. The extraction stage should capture dates and source context before a reconciliation policy chooses a current view. Otherwise, a later merge may erase the historical relationship or incorrectly classify one report as false. Both historical and current questions benefit from retaining the original statements.

4.4 Supervision and annotation strategies

Supervised extraction requires labeled examples that reflect the target domain. A useful annotation record identifies arguments, relation type, evidence span, qualifiers, and uncertainty. The annotation policy should distinguish “not expressed” from “expressed negatively” and from “cannot decide.” These outcomes may eventually be collapsed for a particular model, but preserving them in the reference data supports better error analysis. It also helps explain why human annotators disagree.

Distant supervision（远程监督）uses existing database relations to label text automatically. Mintz and colleagues developed an influential approach based on aligning known relations with textual occurrences [20]. The attraction is scale: existing structured knowledge can generate many training examples. The assumption is imperfect because a sentence mentioning both entities may discuss a different relationship. In an applied pipeline, such labels should be treated as noisy evidence for training rather than as verified gold annotations.

Weak supervision（弱监督）can combine rules, dictionaries, existing databases, and model predictions. A sensible project begins with high precision patterns for well defined relations, then identifies what those patterns miss. Human review can focus on examples where labeling sources disagree or where new relation types appear. This design makes annotation effort responsive to the actual failure modes of the system. It also avoids assuming that a large automatically labeled dataset necessarily provides broader conceptual coverage.

Active learning（主动学习）selects examples expected to improve a model or resolve consequential uncertainty. In the Research Evidence Graph, examples involving comparison language and multiple baselines might be especially informative because those errors affect the central application. Selection should not rely only on uncertainty scores, which can overlook confidently wrong patterns. A mixed sample containing uncertain, diverse, and randomly selected records gives a more informative review set.

4.5 A proposed acceptance contract

Before a candidate relation becomes an accepted graph assertion, the proposed system checks that its arguments exist, their types are compatible, the relation belongs to the schema, and the source locator resolves. It then checks whether the extracted wording preserves negation, time, and essential qualifiers. These checks do not all need one model. Deterministic validation can handle identifiers and datatypes, while a separate model or human reviewer can assess semantic faithfulness. The result records which checks ran and which remained unresolved.

For a comparative result, the contract is deliberately stricter. An accepted record needs the compared methods, dataset version, metric definition, and the reported outcome. If a source states only that a method is “better,” the system can store that as an attributed qualitative claim, but it should not manufacture a numeric improvement. If a table reports several metrics, the graph creates distinct metric observations tied to the same Evaluation rather than selecting the most favorable one.

The graph should preserve the difference between extraction and corroboration. Two documents can repeat the same unsupported assertion. Counting both as independent confirmation would exaggerate the evidence. Citation links, shared datasets, copied text, and common authorship can help identify dependence, but none alone provides a perfect independence test. The interface should therefore expose the evidence chain and allow a reviewer to examine whether apparently separate sources originate from the same study.

4.6 Measure useful extraction quality

Relation precision asks how many predicted relations are correct under the annotation policy. Recall asks how many reference relations were recovered. End to end scoring should require the correct argument identities, direction, relation label, and any qualifiers essential to the task. A separate diagnostic score can relax one requirement at a time. This helps distinguish a model that understands the relationship but misses an entity boundary from one that confidently reverses the result of an experiment.

Evidence quality deserves its own measure. A relation may happen to be true even though the cited passage does not support it. A research assistant must be evaluated on whether its evidence actually entails or justifies the reported statement. Human reviewers can label evidence sufficiency and identify missing context. For deployment, a smaller set of well supported relations can be more valuable than a large graph with impressive recall and weak evidential traceability.