Qualification test

Judge ten edited versions. You must answer every item. No comments are required.

• Judge only the edited version; the original reference may itself be inconsistent.
• Yellow text was added or changed. Struck-through text was deleted and is not part of the edited version.
• Consistent means there is at least one possible situation in which all statements are true. Inconsistent means no such situation exists. Use Cannot determine only if broken or unclear text makes a judgment impossible.
• For claims and rules, “either A or B” is inclusive OR: at least one must be true, and both may be true. An if–then rule is false only when its if-part is true and its then-part is false.

Passing requires at least 9 of 10 correct and at least 4 of 5 correct in each part.

Part A: Scene question–answer sets

Part B: Claims and rules

Qualification passed

You passed the qualification test. Next is the main annotation: five scene question–answer items and five claims-and-rules items.

How the labeling screen works

Each labeling item shows one original set of lines and one or more edited versions of that same set, labeled A, B, C, …

• The original set may already be inconsistent. It is a reference, not the thing you are scoring.
• Version letters are shuffled and do not name a method.
• If several methods produced the exact same edited text, they are shown as one version. Your judgment is applied to all methods that produced that text.
• Blue, green, and purple highlights separate statements within a line. Yellow highlighting marks text that changed in that version.
• Use View original in the bottom bar to open the original reference while keeping your place.
• Struck-through text labeled Deleted is missing from that version.
• Related lines are placed near each other and numbered 1, 2, 3, … in that reading order. The original set and all edited versions use the same order.

For each edited version, choose Consistent, Inconsistent, or Cannot determine. Use Cannot determine only when the text is too broken or unclear to judge.

Use highlighting to see what changed, but judge each edited set as a whole, including whether a changed line conflicts with an unchanged line.

Part 1 of 2

Scene questions and answers

You will first label sets of questions and answers about people, objects, and places in a scene.

Complete all five Part 1 items. Part 2 will begin on a separate transition page.

Tutorial 1: Scene questions and answers

These items are short questions and answers about people, objects, and places in a scene. Answers about the same person or object must agree.

Examples

Part 2 of 2

Claims and rules

You have completed the scene question–answer items. Part 2 uses everyday claims and logical rules, including “if–then” and “either–or” statements.

Complete all five Part 2 items.

Tutorial 2: Claims and rules

These items use everyday claims, including “if–then” and “either–or” lines. Some original sets also show short formulas such as (P₁).

Shared-symbol guide

• (P₁): this claim is true.
• (¬H₁): this claim is not true.
• (P₁ → H₁): if P₁ is true, then H₁ is true. This is equivalent to (¬P₁ ∨ H₁) (not P₁, or H₁, or both).
• (A₁ ∧ B₁): both A₁ and B₁ are true.
• (P₁ ∨ H₁): P₁ or H₁ is true (or both).

“Either A or B” uses inclusive OR. The formula (A ∨ B) means that at least one of A and B must be true. Both A and B may be true.

The same letter means the same claim in that example. Note that an “if–then” line does not say that the “if” part is already true.

Colored highlights mark the logical parts. For example, “Either A or B” separates the alternatives, and “If A, then B” separates the condition from its result. Yellow still marks edits. Formulas also appear beside unchanged lines in edited versions.

Examples

Labeling complete

Thank you — you finished all of the labeling items. Please answer two short questions before downloading your responses.