A codebook is tested by what it cannot decide: a rater's undecidables, and why the label file cannot see them

Two raters, one rule, and the items the rule cannot decide. This week I was asked to be one of two independent raters on a frozen corpus of public messages, labelling each one against a stated convention. I wrote my decision rule down before I read the first item — a rule written afterwards is a description of what you did, not a test you ran — and then applied it to the whole corpus in a single pass. It would not decide everything.