This is an additional reproducible analysis of how far raters agreed with one another about the quality of the descriptions in the ArchiveGPT study, and of what the reported single-rater intraclass correlation of ~0.11 does and does not establish.

It is accompanied by the technical note output/reliability_report.txt — every result in that note is produced by run_all.py.

Why this code starts from the raw exports

The published file analysis/data/clean_data/correctness_per_trial.csv assigns its trial column as rep(1:80, ...) — serial position (line 230 of 01_pre-processing.R). Presentation order was randomised per participant, so that column does not identify the stimulus. No item-level reliability analysis can be run from the published clean data alone.

Item identity survives only in the raw PsychoPy exports, in the image and desc_type columns. This package rebuilds it:

item = image + "|" + desc_type        # 40 images x 2 sources = 80 items

Each participant rated all 80 items across two blocks of 40, with the description source flipping between blocks. The design is fully crossed: 139 participants x 80 items = 11,120 ratings, one rating per cell, no missing data.

Fixing this in the release — adding image and desc_type to correctness_per_trial.csv — would make this reconstruction step unnecessary and close a genuine reproducibility gap.

Quick start

pip install -r requirements.txt

# 1. Point config.yaml at your copy of the extracted ArchiveGPT release,
#    or override it on the command line:
python run_all.py --data-root /path/to/archive_gpt

That writes three files to output/:

file	contents
`reliability_report.txt`	every table in the note, as plain text
`results.json`	the same numbers, machine-readable
`ratings_long.csv`	the rebuilt trial-level data with item identity

ratings_long.csv is the artefact worth keeping: it is what the published clean data would look like if it carried item identity.

Settings

Everything lives in config.yaml. The only value you normally change is:

paths:
  data_root: "D:/McDuf/Downloads/archive_gpt"

This must point at the extracted Zenodo release (record 15640284) — the folder containing analysis/data/raw_data/ and analysis/data/clean_data/. Forward slashes work on Windows. Use "." if you drop this package inside the release.

Other sections, all documented inline:

columns — column names in the raw exports. The two slider columns per measure (slider_q1.response, slider_q1_2.response) correspond to the two presentation blocks.
design — what an item is, how many to expect, which desc_type value means archivist-written vs model-generated, which expert_status value marks each panel.
analysis — primary_measure (accuracy or usefulness), alpha, Monte-Carlo homogeneity_sims, seed, and how many weak raters to drop in the final adjustment rung.
validation — whether to abort when the reconstruction disagrees with the published data. Leave enforce: true.

CLI flags override the file: --data-root, --output, --measure, --quick (200 simulations instead of 2000), --no-validate.

What gets checked before anything is computed

The whole analysis rests on having reassembled item identity correctly, so that is verified against the release before any coefficient is reported:

check	what it proves
fully crossed design	139 x 80 ratings, exactly one per participant x item
two balanced blocks	40 items per block per participant
counterbalancing exact	description source flips between blocks for every image
ratings match `correctness_per_trial.csv`	max abs difference 0.0
means match `accuracy_usefulness.csv`	max abs difference 1.4e-14
independent ICC cross-check (pingouin)	max discrepancy < 1e-15

The first comparison is on each participant's sorted ratings, because the published trial column carries no item identity to align on. Passing both means the reconstruction contains precisely the published ratings, and the only thing added is the item label.

The pingouin cross-check is skipped with a notice if pingouin is not installed. Everything else works without it.

Layout

archivegpt_reliability/
├── config.yaml              <- the only file you normally edit
├── requirements.txt
├── run_all.py               <- entry point
├── README.md
└── src/
    ├── config.py            loads and validates settings, resolves paths
    ├── build_dataset.py     rebuilds item identity from the raw exports
    ├── validate.py          checks the rebuild against the published data
    ├── icc.py               variance components + the whole ICC family + CIs
    ├── diagnostics.py       every Part II diagnostic
    └── report.py            renders the tables, writes the outputs

`icc.py`

Implements the two-way random-effects decomposition

X_ij = mu + t_i + r_j + e_ij

and every coefficient derived from it. Confidence intervals follow McGraw & Wong (1996) Table 7, and are validated against pingouin.

function	purpose
`variance_components`	sigma^2_item, sigma^2_rater, sigma^2_error + mean squares
`icc_estimates`	ICC(1,1), (A,1), (C,1) and their k-rater forms
`icc_confidence_intervals`	exact-F and Satterthwaite intervals
`single_rater_validity`	sqrt(ICC(C,1)) — one rater's correlation with truth
`descriptive_scale`	the components translated into rating points
`spearman_brown`, `raters_needed`	panel-size planning
`crosscheck_pingouin`	independent re-estimation

`diagnostics.py`

function	question it answers
`pairwise_stats`	distribution of rater-pair correlations
`homogeneity_test`	are raters interchangeable, or does quality genuinely differ?
`rater_validity`	each rater vs the rest, disattenuated
`severity_stats`	how differently raters use the scale
`cumulative_adjustments`	the coefficient as design effects are modelled
`block_effects`	first-block to second-block drift
`source_split`	agreement and true spread, by description source
`range_restriction_counterfactual`	what a lower ICC in a subset actually means
`cross_panel_agreement`	do the two panels rank the descriptions alike?
`substantive_effect`	the human-vs-model comparison, items fixed then random
`replication_available`	did anyone rate an item twice?
`image_taste_test`	is a rater's deviation stable across the same image?
`residual_split_consequences`	what turns on the unidentified residual split
`within_rater_measure_coherence`	is each rater internally self-consistent?

Three things the code is careful about

1. Which axis a "within-item SD" is taken over. data.std(axis=1) is the SD across raters within an item; axis=0 is the SD across items within a rater. They are different quantities and easy to confuse — an earlier draft of the note reported the second under the first's name. source_split now returns both, named explicitly.

2. Comparing like with like in the counterfactual. The range-restriction counterfactual must be built in the same ICC form as the observed value it is compared against. Substituting a consistency-form counterfactual for an absolute-agreement observation silently overstates the gap (0.171 instead of 0.105).

3. The block-order artifact in image_taste_test. The two descriptions of an image necessarily fall in different blocks, so the ~4-point block drift induces a spurious negative correlation if it is not removed first. Run with control_block=False to see the artifact (median r = -0.053, p = 0.014) and True for the real answer (median r = -0.017, p = 0.293). run_all.py prints both.

Known limitations

No replication. Exactly one rating per participant x item, so the residual confounds stable rater x item interaction with momentary noise. They are not identified, and the interpretation of the whole result turns on the split. residual_split_consequences shows the range; only a retest resolves it.
The two-way model has no block term. The ~4-point block drift is absorbed into the residual. cumulative_adjustments shows the effect of removing it (0.156 -> 0.165).
image_taste_test is a proxy, not a retest. It rules out stable taste for the depicted image; it says nothing about taste for the specific description, which is the component that matters.
Sample. 29 participants carry expert_status == "expert"; an external reanalysis reported 28. If you are comparing figures directly, resolve the one-participant discrepancy first.
"Expert" is heterogeneous: 14 archaeology or history students, 9 archive students, 4 archivists, 1 archaeologist, 1 hobby historian.

Requirements

Python 3.9+. numpy, pandas, scipy, PyYAML. Optionally pingouin for the independent ICC cross-check.

References

Shrout, P. E., & Fleiss, J. L. (1979). Psychological Bulletin, 86(2), 420–428.
McGraw, K. O., & Wong, S. P. (1996). Psychological Methods, 1(1), 30–46.
Koo, T. K., & Li, M. Y. (2016). Journal of Chiropractic Medicine, 15(2), 155–163.
Bornmann, L., Mutz, R., & Daniel, H.-D. (2010). PLoS ONE, 5(12), e14331.
Erosheva, E. A., Martinková, P., & Lee, C. J. (2021). JRSS-A, 184(3), 904–919.

Data

Abele, L., Anders, G., Aydın, T., Buder, J., Fischer, H., Kimmel, D., & Huff, M. ArchiveGPT. Zenodo, record 15640284. https://zenodo.org/records/15640284

README.md Unescape Escape

ArchiveGPT inter-rater reliability