Explainable AI tools produce reasons for an AI system’s output: which inputs influenced the output or which documents the answer drew on. A vendor that calls its product explainable is claiming those reasons are true. Before you sign, you can test that claim by running three checks on the vendor’s system using your own data.