CLAIMBOUND / SOURCE-PUBLICATION EVIDENCE
AI benchmarks
Every card identifies what it proves, what it does not prove, its exact quote, locator, source hash, and local rerun command.
1. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 6
Narrow public claim
This draft document identifies practices for conducting automated benchmark1 evaluations of language models and similar general-purpose AI models that output text (herein “AI models”).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 6- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
This draft document identifies practices for conducting automated benchmark1 evaluations of language models and similar general-purpose AI models that output text (herein “AI models”).
2. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 6
Narrow public claim
Evaluations of these models, often embedded into systems capable of functioning as chatbots and AI agents, are increasingly common.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 6- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Evaluations of these models, often embedded into systems capable of functioning as chatbots and AI agents, are increasingly common.
3. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 6
Narrow public claim
This report supports NIST and CAISI efforts to develop guidelines and resources for Federal agencies to conduct evaluations of AI systems, as called for in the AI Action Plan [1].
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 6- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
This report supports NIST and CAISI efforts to develop guidelines and resources for Federal agencies to conduct evaluations of AI systems, as called for in the AI Action Plan [1].
4. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 6
Narrow public claim
It does not provide guidelines for roles, resources, responsibilities, or other enabling practices for performing AI evaluations within an organization.2 Future work may address benchmark development and practices for other types of AI evaluations.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 6- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
It does not provide guidelines for roles, resources, responsibilities, or other enabling practices for performing AI evaluations within an organization.2 Future work may address benchmark development and practices for other types of AI evaluations.
5. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 7
Narrow public claim
Such readers may include business decision-makers, technical integrators, end consumers, or procurement officers seeking information with which to select an AI model for integration or use.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 7- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Such readers may include business decision-makers, technical integrators, end consumers, or procurement officers seeking information with which to select an AI model for integration or use.
6. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 7
Narrow public claim
When AI evaluation organizations implement the practices provided in this report, this secondary audience may be able to better understand and use evaluation results.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 7- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
When AI evaluation organizations implement the practices provided in this report, this secondary audience may be able to better understand and use evaluation results.
7. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 8
Narrow public claim
The first stage in effectively evaluating AI capabilities is to select benchmarks3 that suit the evaluator’s purposes, divided here into two high-level practices: 5 1.1.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 8- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
The first stage in effectively evaluating AI capabilities is to select benchmarks3 that suit the evaluator’s purposes, divided here into two high-level practices: 5 1.1.
8. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 8
Narrow public claim
Based on how the evaluation will be used and what should be measured (Practice 1.1), the evaluator assesses what a candidate benchmark measures (Practice 1.2.1) and whether it relates to the evaluation objectives (Practice 1.2.2).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 8- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Based on how the evaluation will be used and what should be measured (Practice 1.1), the evaluator assesses what a candidate benchmark measures (Practice 1.2.1) and whether it relates to the evaluation objectives (Practice 1.2.2).
9. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 9
Narrow public claim
Every evaluation must be guided by clear objectives, which identify what the evaluation aims to measure in order to achieve the evaluation’s end goal (i.e., to support the intended use of evaluation results).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 9- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Every evaluation must be guided by clear objectives, which identify what the evaluation aims to measure in order to achieve the evaluation’s end goal (i.e., to support the intended use of evaluation results).
10. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 9
Narrow public claim
For example, a technical integrator choosing a model for a consumer chatbot application may conduct an evaluation with the objective of determining which of several models generates the most human-preferred responses to everyday questions.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 9- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t01_primary_claims.json --source-file <local-raw-pdf>Exact source quote
For example, a technical integrator choosing a model for a consumer chatbot application may conduct an evaluation with the objective of determining which of several models generates the most human-preferred responses to everyday questions.
11. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 10
Narrow public claim
Based on the intended use cases for the AI system or threat models for risk assessment, decide and document what concept should be measured 5 (the measurement construct).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 10- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Based on the intended use cases for the AI system or threat models for risk assessment, decide and document what concept should be measured 5 (the measurement construct).
12. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 10
Narrow public claim
Measuring average-, best-, or worst-case behavior. [Emerging Practice4] For example, high-stakes risk assessments may focus on best- or worst-case behavior; comparison shopping may focus on average-case behavior.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 10- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Measuring average-, best-, or worst-case behavior. [Emerging Practice4] For example, high-stakes risk assessments may focus on best- or worst-case behavior; comparison shopping may focus on average-case behavior.
13. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 10
Narrow public claim
For example, the GPQA benchmark [4] purports to measure accuracy at answering a selection of multiple-choice questions intended to be difficult to answer even with internet searches at the time of creation, developed and validated by PhD students or graduates in the fields of chemistry, biology, and physics.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 10- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
For example, the GPQA benchmark [4] purports to measure accuracy at answering a selection of multiple-choice questions intended to be difficult to answer even with internet searches at the time of creation, developed and validated by PhD students or graduates in the fields of chemistry, biology, and physics.
14. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 11
Narrow public claim
For example, the GPQA benchmark [4] may directly test graduate-level chemistry, biology, and physics multiple-choice question answering accuracy. i.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 11- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
For example, the GPQA benchmark [4] may directly test graduate-level chemistry, biology, and physics multiple-choice question answering accuracy. i.
15. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 11
Narrow public claim
Test items should have high coverage across the entire space of tasks that pertain to the evaluation objectives and few items should be irrelevant.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 11- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Test items should have high coverage across the entire space of tasks that pertain to the evaluation objectives and few items should be irrelevant.
16. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 12
Narrow public claim
If the objective is to compare model performance during development, then a benchmark that is either too hard or too easy to the point of saturating is not useful.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 12- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
If the objective is to compare model performance during development, then a benchmark that is either too hard or too easy to the point of saturating is not useful.
17. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 12
Narrow public claim
If the goal is to assess behavior relative to a threshold, many items should ideally be near that threshold to increase precision close to the boundary.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 12- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
If the goal is to assess behavior relative to a threshold, many items should ideally be near that threshold to increase precision close to the boundary.
18. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 13
Narrow public claim
Operational considerations include the amount of human labor required for setup and running, the computational cost to run the benchmark, and the benchmark’s compatibility with a wide-range of AI systems (e.g., agent architectures). b.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 13- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Operational considerations include the amount of human labor required for setup and running, the computational cost to run the benchmark, and the benchmark’s compatibility with a wide-range of AI systems (e.g., agent architectures). b.
19. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 13
Narrow public claim
If others have reported results for the same benchmark before, the benchmark may be a good candidate for validating and contextualizing the evaluation setup by comparing against previous evaluations 12 (also see Section 3).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 13- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
If others have reported results for the same benchmark before, the benchmark may be a good candidate for validating and contextualizing the evaluation setup by comparing against previous evaluations 12 (also see Section 3).
20. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 14
Narrow public claim
See Table 2.1 for some examples how these practices have been applied in past CAISI benchmark evaluations.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 14- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t02_primary_claims.json --source-file <local-raw-pdf>Exact source quote
See Table 2.1 for some examples how these practices have been applied in past CAISI benchmark evaluations.
21. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 14
Narrow public claim
Evaluation Protocol Selected Evaluation Relevant CAISI Debugging Log Benchmark Code Design Details of Protocol Principles CAISI [8] used Comparability Task settings inspect_ev Older models sometimes GPQA [4] Tools: none als/gpqa had trouble outputting among other Performance Instructions: asked model to the selected choice in science and optimization output selected choice, then an the correct format, knowledge explanation which was caught by benchmarks Submission attempts: one manual transcript for review.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 14- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Evaluation Protocol Selected Evaluation Relevant CAISI Debugging Log Benchmark Code Design Details of Protocol Principles CAISI [8] used Comparability Task settings inspect_ev Older models sometimes GPQA [4] Tools: none als/gpqa had trouble outputting among other Performance Instructions: asked model to the selected choice in science and optimization output selected choice, then an the correct format, knowledge explanation which was caught by benchmarks Submission attempts: one manual transcript for review.
22. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 15
Narrow public claim
7 2.1.1 Evaluation protocol design principles The most important principle of evaluation protocol design is that procedures should be designed to support the objectives of the evaluation (Practice 1.1).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 15- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
7 2.1.1 Evaluation protocol design principles The most important principle of evaluation protocol design is that procedures should be designed to support the objectives of the evaluation (Practice 1.1).
23. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 15
Narrow public claim
That said, there are common design principles that can help guide protocol design for a broad range of evaluation objectives.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 15- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
That said, there are common design principles that can help guide protocol design for a broad range of evaluation objectives.
24. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 16
Narrow public claim
For example, the evaluator may want to use a protocol that provides models with tools and information comparable to those used in existing baselines.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 16- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
For example, the evaluator may want to use a protocol that provides models with tools and information comparable to those used in existing baselines.
25. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 16
Narrow public claim
In many cases, the objective of an evaluation is to gain information on how a model or system will behave in certain external contexts (e.g., real-world, worstcase, best-case, hypothetical) different from that of the evaluation.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 16- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
In many cases, the objective of an evaluation is to gain information on how a model or system will behave in certain external contexts (e.g., real-world, worstcase, best-case, hypothetical) different from that of the evaluation.
26. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 17
Narrow public claim
Performance optimization. [Emerging Practice] For certain evaluation objectives, there may be a need to optimize aspects of an evaluation protocol.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 17- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Performance optimization. [Emerging Practice] For certain evaluation objectives, there may be a need to optimize aspects of an evaluation protocol.
27. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 17
Narrow public claim
26 2.1.2 Common evaluation protocol settings The previous sub-section listed some high-level design principles that can help guide the design of an evaluation protocol.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 17- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
26 2.1.2 Common evaluation protocol settings The previous sub-section listed some high-level design principles that can help guide the design of an evaluation protocol.
28. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 18
Narrow public claim
Task settings are those that determine how benchmark items are presented to models/systems and how models/systems can go about solving items.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 18- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Task settings are those that determine how benchmark items are presented to models/systems and how models/systems can go about solving items.
29. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 18
Narrow public claim
This table also serves as an evaluation design checklist, since making an improper choice for an applicable setting can significantly reduce an evaluation's relevance to its objectives.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 18- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
This table also serves as an evaluation design checklist, since making an improper choice for an applicable setting can significantly reduce an evaluation's relevance to its objectives.
30. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 19
Narrow public claim
Evaluation Setting Protocol Description Type(s) Setting Reasoning effort should generally be set based on the “cost control” design principle.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 19- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t03_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Evaluation Setting Protocol Description Type(s) Setting Reasoning effort should generally be set based on the “cost control” design principle.
31. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 19
Narrow public claim
Models can be served with intrinsic (e.g., weight-alteration based) or extrinsic (e.g., input/output classifier based) Inference safeguards that cause certain classes of tasks to be refused.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 19- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Models can be served with intrinsic (e.g., weight-alteration based) or extrinsic (e.g., input/output classifier based) Inference safeguards that cause certain classes of tasks to be refused.
32. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 20
Narrow public claim
Evaluation Setting Protocol Description Type(s) Setting A key sub-consideration when designing agents is whether and how to provide them with context-compression tools, which allow them to operate past their normal context limits.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 20- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Evaluation Setting Protocol Description Type(s) Setting A key sub-consideration when designing agents is whether and how to provide them with context-compression tools, which allow them to operate past their normal context limits.
33. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 20
Narrow public claim
Thus, unlike nonagentic evaluations where task items have a natural stopping point, for agentic evaluations a stopping condition must be explicitly defined.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 20- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Thus, unlike nonagentic evaluations where task items have a natural stopping point, for agentic evaluations a stopping condition must be explicitly defined.
34. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 21
Narrow public claim
Evaluation Setting Protocol Description Type(s) Setting The instructions presented to a model can impact what is being measured.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 21- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Evaluation Setting Protocol Description Type(s) Setting The instructions presented to a model can impact what is being measured.
35. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 21
Narrow public claim
For example, the level of detail in instructions can influence whether an evaluation is or is not measuring a model’s skill at resolving ambiguity and inferring intentions.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 21- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
For example, the level of detail in instructions can influence whether an evaluation is or is not measuring a model’s skill at resolving ambiguity and inferring intentions.
36. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 22
Narrow public claim
Evaluation Setting Protocol Description Type(s) Setting For example, this may be the case when the objective is to understand the utility provided by different tools.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 22- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Evaluation Setting Protocol Description Type(s) Setting For example, this may be the case when the objective is to understand the utility provided by different tools.
37. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 22
Narrow public claim
Note: Whether to provide internet search tools is a particularly consequential decision, as internet access may allow models to cheat by looking up answers online [9].
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 22- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Note: Whether to provide internet search tools is a particularly consequential decision, as internet access may allow models to cheat by looking up answers online [9].
38. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 23
Narrow public claim
Evaluation Setting Protocol Description Type(s) Setting Models are generally sampled nondeterministically (though in certain cases, it may be possible to directly measure performance without sampling, e.g., via next-token probabilities).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 23- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Evaluation Setting Protocol Description Type(s) Setting Models are generally sampled nondeterministically (though in certain cases, it may be possible to directly measure performance without sampling, e.g., via next-token probabilities).
39. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 23
Narrow public claim
Having models attempt each item multiple Number of times increases evaluation cost but can reduce uncertainty of trials per Scoring evaluation results and enable an evaluator to quantify what test item portion of the uncertainty stems from model sampling.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 23- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Having models attempt each item multiple Number of times increases evaluation cost but can reduce uncertainty of trials per Scoring evaluation results and enable an evaluator to quantify what test item portion of the uncertainty stems from model sampling.
40. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 24
Narrow public claim
Here, this sub-section describes practices for the micro-level implementation of an evaluation protocol as evaluation code: 3 1.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 24- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t04_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Here, this sub-section describes practices for the micro-level implementation of an evaluation protocol as evaluation code: 3 1.
41. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 24
Narrow public claim
Running evaluations can often be made easier by using evaluation frameworks, which provide software libraries for querying models, agent scaffolding and tools, error handling, and logging evaluation results instead of having to write those functions from scratch.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 24- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Running evaluations can often be made easier by using evaluation frameworks, which provide software libraries for querying models, agent scaffolding and tools, error handling, and logging evaluation results instead of having to write those functions from scratch.
42. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 25
Narrow public claim
18 2.4.1 Common bugs In this section, we list some common classes of bugs that one may encounter when performing automated benchmark evaluations.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 25- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
18 2.4.1 Common bugs In this section, we list some common classes of bugs that one may encounter when performing automated benchmark evaluations.
43. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 25
Narrow public claim
However, even commercial providers may serve models in a degraded state 28 (e.g. commercial providers often vary on what context length they serve a given model with).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 25- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
However, even commercial providers may serve models in a degraded state 28 (e.g. commercial providers often vary on what context length they serve a given model with).
44. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 26
Narrow public claim
If refusal behavior during evaluation may differ from refusal behavior under realistic usage,6 evaluators should check for the presence of refusals or other safeguards interventions in evaluation transcripts to determine whether and how they may be impacting evaluation results.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 26- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
If refusal behavior during evaluation may differ from refusal behavior under realistic usage,6 evaluators should check for the presence of refusals or other safeguards interventions in evaluation transcripts to determine whether and how they may be impacting evaluation results.
45. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 26
Narrow public claim
Evaluation awareness. [Emerging Practice] Recent work has shown that for certain models, model behavior can be influenced by cues that the input is part of an evaluation exercise [12][13][14].
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 26- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Evaluation awareness. [Emerging Practice] Recent work has shown that for certain models, model behavior can be influenced by cues that the input is part of an evaluation exercise [12][13][14].
46. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 26
Narrow public claim
Unfortunately, the absence of verbalized evaluation 6For example, a malicious user might be able to find jailbreaks that circumvent refusal.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 26- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Unfortunately, the absence of verbalized evaluation 6For example, a malicious user might be able to find jailbreaks that circumvent refusal.
47. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 27
Narrow public claim
5 2.4.2 Quality assurance techniques Quality assurance can diagnose issues in evaluation code and mistakes in evaluation protocols identified above.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 27- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
5 2.4.2 Quality assurance techniques Quality assurance can diagnose issues in evaluation code and mistakes in evaluation protocols identified above.
48. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 27
Narrow public claim
Automated transcript review. [Emerging Practice] Evaluators may be able to extend and augment manual transcript review processes using automated transcript review.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 27- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Automated transcript review. [Emerging Practice] Evaluators may be able to extend and augment manual transcript review processes using automated transcript review.
49. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 28
Narrow public claim
For evaluations that involve complex virtual environments, it can be useful to create and run deterministic solutions against the evaluator’s in-house setup of the environment.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 28- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
For evaluations that involve complex virtual environments, it can be useful to create and run deterministic solutions against the evaluator’s in-house setup of the environment.
50. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 28
Narrow public claim
Another assurance technique is to compare results to existing evidence such as other results on the same or closely related benchmarks.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 28- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t05_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Another assurance technique is to compare results to existing evidence such as other results on the same or closely related benchmarks.
51. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 29
Narrow public claim
Provides a description of statistical analysis and uncertainty reported (Practice 3.1), details on how the evaluation was performed (Practice 3.2), and how claims were qualified in the final report 20 (Practice 3.3).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 29- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Provides a description of statistical analysis and uncertainty reported (Practice 3.1), details on how the evaluation was performed (Practice 3.2), and how claims were qualified in the final report 20 (Practice 3.3).
52. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 29
Narrow public claim
Evaluation Statistical analysis Reported details Qualified claims CAISI [8] used CAISI reported CAISI reported CAISI reported that “U.S. models and DeepSeek’s GPQA [4] to average accuracy detailed models achieve similar performance on question evaluate across the full information on and answer-style science and knowledge models’ benchmark with a the evaluation benchmarks.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 29- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Evaluation Statistical analysis Reported details Qualified claims CAISI [8] used CAISI reported CAISI reported CAISI reported that “U.S. models and DeepSeek’s GPQA [4] to average accuracy detailed models achieve similar performance on question evaluate across the full information on and answer-style science and knowledge models’ benchmark with a the evaluation benchmarks.
53. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 30
Narrow public claim
It is good practice to define a statistical analysis procedure in advance of implementing and running the evaluation, in conjunction with the task setting specification practices described in Sections 1 and 2.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 30- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
It is good practice to define a statistical analysis procedure in advance of implementing and running the evaluation, in conjunction with the task setting specification practices described in Sections 1 and 2.
54. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 30
Narrow public claim
This includes any aggregation functions applied at a test item level (e.g., across trials, such as pass@k) as well as across items 18 (e.g., mean of item scores).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 30- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
This includes any aggregation functions applied at a test item level (e.g., across trials, such as pass@k) as well as across items 18 (e.g., mean of item scores).
55. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 30
Narrow public claim
For example, in many evaluations, two additive sources of variation in results that contribute to uncertainty of estimates are (1) variation due to nondeterministic sampling of model completions for each item and (2) variation stemming from the hypothetical sampling of test items into the benchmark from an item superpopulation.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 30- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
For example, in many evaluations, two additive sources of variation in results that contribute to uncertainty of estimates are (1) variation due to nondeterministic sampling of model completions for each item and (2) variation stemming from the hypothetical sampling of test items into the benchmark from an item superpopulation.
56. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 31
Narrow public claim
Report costs alongside performance. [Emerging Practice] When models and systems are not uniformly cost controlled, evaluation results should report both the performance of models and the costs-incurred to achieve such performance levels (see Section 2.1.1).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 31- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Report costs alongside performance. [Emerging Practice] When models and systems are not uniformly cost controlled, evaluation results should report both the performance of models and the costs-incurred to achieve such performance levels (see Section 2.1.1).
57. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 31
Narrow public claim
One possible way to present this data is to plot cost and performance of different systems on a two-dimensional graph (see Figure 3.2, for example).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 31- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
One possible way to present this data is to plot cost and performance of different systems on a two-dimensional graph (see Figure 3.2, for example).
58. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 32
Narrow public claim
Consider releasing transcripts. [Emerging Practice] Publishing full or representative transcripts can make it easier for other parties to interpret and reproduce the results of evaluations.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 32- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Consider releasing transcripts. [Emerging Practice] Publishing full or representative transcripts can make it easier for other parties to interpret and reproduce the results of evaluations.
59. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 32
Narrow public claim
Being able to review transcript-level data enables other researchers to more easily spot methodological decisions (or potential issues) that could impact their interpretation of the results, reducing the risks of drawing flawed conclusions and making it easier for the evaluation science community to collaborate on studying and resolving evaluation science issues. a.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 32- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Being able to review transcript-level data enables other researchers to more easily spot methodological decisions (or potential issues) that could impact their interpretation of the results, reducing the risks of drawing flawed conclusions and making it easier for the evaluation science community to collaborate on studying and resolving evaluation science issues. a.
60. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 33
Narrow public claim
Report assumptions or evidence for the relationship between performance on the evaluated benchmark and the intended measurement construct (see Practice 1.2).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 33- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t06_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Report assumptions or evidence for the relationship between performance on the evaluated benchmark and the intended measurement construct (see Practice 1.2).
61. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 33
Narrow public claim
If the benchmark is intended to be representative of real-world tasks, report the degree to which evaluation conditions resemble deployment conditions (see Practice 2.1). b.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 33- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
If the benchmark is intended to be representative of real-world tasks, report the degree to which evaluation conditions resemble deployment conditions (see Practice 2.1). b.
62. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 33
Narrow public claim
When comparing model behavior to baselines, provide context about baseline measure conditions (e.g., characteristics of baseline subjects, available tools, incentives, etc.) 20 (Practice 1.2).
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 33- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
When comparing model behavior to baselines, provide context about baseline measure conditions (e.g., characteristics of baseline subjects, available tools, incentives, etc.) 20 (Practice 1.2).
63. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 33
Narrow public claim
Use caution when making claims that generalize evaluation results beyond the scope of the intended measurements as established in Practice 1.2.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 33- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Use caution when making claims that generalize evaluation results beyond the scope of the intended measurements as established in Practice 1.2.
64. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 35
Narrow public claim
16 [17] Wallach H, Desai M, Cooper AF, Wang A, Atalla C, Barocas S, Blodgett SL, Chouldechova A, Corvi E, Dow PA, Garcia-Gathright J, Olteanu A, Pangakis N, Reed S, Sheng E, Vann D, Vaughan JW, Vogel M, Washington H, Jacobs AZ (2025) Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 35- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
16 [17] Wallach H, Desai M, Cooper AF, Wang A, Atalla C, Barocas S, Blodgett SL, Chouldechova A, Corvi E, Dow PA, Garcia-Gathright J, Olteanu A, Pangakis N, Reed S, Sheng E, Vann D, Vaughan JW, Vogel M, Washington H, Jacobs AZ (2025) Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge.
65. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 35
Narrow public claim
29 [21] Bengio Y, Mindermann S, Privitera D, Besiroglu T, Bommasani R, Casper S, Choi Y, Fox P, Garfinkel B, Goldfarb D, Heidari H, Ho A, Kapoor S, Khalatbari L, Longpre S, Manning S, Mavroudis V, Mazeika M, Michael J, … Zeng Y (2025) International AI Safety Report.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 35- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
29 [21] Bengio Y, Mindermann S, Privitera D, Besiroglu T, Bommasani R, Casper S, Choi Y, Fox P, Garfinkel B, Goldfarb D, Heidari H, Ho A, Kapoor S, Khalatbari L, Longpre S, Manning S, Mavroudis V, Mazeika M, Michael J, … Zeng Y (2025) International AI Safety Report.
66. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 37
Narrow public claim
Baseline Comparable reference measure of the behavior of a non-AI system, potentially including human individuals or teams, systems based on simple heuristics or rules, or systems based on random chance.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 37- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Baseline Comparable reference measure of the behavior of a non-AI system, potentially including human individuals or teams, systems based on simple heuristics or rules, or systems based on random chance.
67. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 37
Narrow public claim
Capability The range of tasks or functions that an AI system can (adapted from [21]) perform and how effectively it performs them.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 37- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Capability The range of tasks or functions that an AI system can (adapted from [21]) perform and how effectively it performs them.
68. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 38
Narrow public claim
External validity The extent to which measurements can also describe, or generalize to, conditions different from evaluation context.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 38- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
External validity The extent to which measurements can also describe, or generalize to, conditions different from evaluation context.
69. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 38
Narrow public claim
Measurement validity The degree to which accumulated evidence and theory [20] support a specific interpretation of test scores for a given use of a test.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 38- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
Measurement validity The degree to which accumulated evidence and theory [20] support a specific interpretation of test scores for a given use of a test.
70. NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems · PDF page 39
Narrow public claim
For example, LLM benchmark tasks are often represented by a set of multiple-choice questions with one or more correct answers.
What this card proves
The exact quoted statement occurred in the named official PDF fetched on 2026-08-04, and the complete PDF matched the recorded SHA-256.
What this card does not prove
It does not independently establish the underlying model capability, safety, comparison, or real-world truth. This is a retrospective single-maintainer source-publication audit.
- Primary source
- NIST AI 800-2 Initial Public Draft: Guidelines for Evaluating AI Systems
- Section locator
PDF page 39- Captured
- 2026-08-04
- Source SHA-256
5ae4244d58ed472803c7f62da92b310d7167a4afa18da19edd3d1dc8109b64f7- Reproduction
- Maintainer run only; no independent operator rerun is registered.
Repeat this exact check locally
Download the PDF to a local file, then run the committed verifier. A changed PDF is reported as drift rather than silently accepted.
python scripts/verify_primary_public_claim_group.py artifacts/cb7k_dom002_t07_primary_claims.json --source-file <local-raw-pdf>Exact source quote
For example, LLM benchmark tasks are often represented by a set of multiple-choice questions with one or more correct answers.