input                  language model                  real    shuf    diff  restarts
vmsB                   Somali                         -2.63   -3.03    0.39  2/2
vmsB                   Romanian                       -2.15   -2.53    0.39  2/2
vmsB                   lm_cs_old                      -1.79   -2.11    0.33  2/2
vmsB                   Hungarian                      -2.33   -2.65    0.32  2/2
vmsB                   lm_de_mhg                      -1.77   -2.06    0.30  2/2
vmsB                   Shona                          -2.88   -3.14    0.26  2/2
vmsB                   Zulu-NT                        -2.68   -2.92    0.24  2/2
vmsB                   Estonian-PART                  -2.27   -2.51    0.24  2/2
vmsB                   Basque-NT                      -2.52   -2.75    0.24  2/2
vmsB                   Turkish                        -2.29   -2.52    0.23  2/2
vmsB                   Polish                         -2.33   -2.55    0.23  2/2
vmsB_line              lm_italian                     -1.63   -1.86    0.22  2/2
vmsB                   Czech                          -2.31   -2.53    0.22  2/2
vmsB_line              lm_latin_all                   -1.70   -1.91    0.21  2/2
vmsB                   Finnish                        -2.54   -2.75    0.21  2/2
vmsB                   Latin                          -2.56   -2.77    0.21  2/2
vmsB                   Spanish                        -2.64   -2.85    0.21  2/2
vmsB                   lm_latin_all                   -2.17   -2.37    0.20  4/4
vmsB                   Swahili-NT                     -2.57   -2.76    0.20  2/2
vmsB                   Quichua-NT                     -2.28   -2.46    0.18  2/2
vmsB                   Indonesian                     -2.55   -2.73    0.18  2/2
vmsA                   lm_de_mhg                      -1.98   -2.16    0.18  2/2
vmsB                   Vietnamese                     -2.47   -2.64    0.18  2/2
vmsB                   Tagalog                        -2.53   -2.71    0.17  2/2
vmsB                   Italian                        -2.46   -2.62    0.16  2/2
vmsB                   Lithuanian                     -2.58   -2.73    0.16  2/2
bioB2                  lm_italian                     -2.10   -2.25    0.15  2/2
vmsB                   Greek                          -2.59   -2.74    0.15  2/2
vmsB                   me                             -2.14   -2.29    0.15  2/2
vmsB                   Nahuatl-NT                     -2.45   -2.58    0.13  2/2
bioB2                  lm_latin_all                   -2.21   -2.34    0.13  2/2
vmsB                   French                         -2.69   -2.82    0.13  2/2
vmsB                   English                        -2.78   -2.90    0.12  2/2
vmsA                   lm_cs_old                      -2.01   -2.13    0.12  2/2
vmsB                   Romani-NT                      -2.45   -2.56    0.12  2/2
vmsB                   lm_de_early                    -2.19   -2.30    0.11  2/2
vmsB                   Maori                          -2.31   -2.41    0.10  2/2
vmsB                   Farsi                          -1.64   -1.75    0.10  2/2
vmsB_rev               lm_latin_all                   -2.23   -2.33    0.10  2/2
vmsB                   Esperanto                      -2.81   -2.91    0.10  2/2
vmsB                   Cebuano                        -2.59   -2.69    0.10  2/2
vmsB                   lm_italian                     -2.09   -2.18    0.09  4/4
vmsB_rev               lm_italian                     -2.17   -2.25    0.08  2/2
vmsB                   Albanian                       -2.51   -2.60    0.08  2/2
vmsB                   Hebrew                         -1.39   -1.46    0.08  2/2
starsB3                lm_latin_all                   -2.34   -2.40    0.06  2/2
vmsB                   Malagasy                       -2.77   -2.82    0.05  2/2
vmsB                   Portuguese                     -2.56   -2.60    0.05  2/2
vmsB                   German                         -2.83   -2.87    0.04  2/2
vmsB                   Arabic                         -1.45   -1.49    0.04  2/2
vmsA                   lm_de_early                    -2.44   -2.47    0.03  2/2
starsB3                lm_italian                     -2.23   -2.25    0.02  2/2
vmsA                   me                             -2.35   -2.32   -0.02  2/2
herbalA1               lm_latin_all                   -2.45   -2.42   -0.03  2/2
vmsA                   lm_latin_all                   -2.45   -2.40   -0.04  4/4
herbalA1               lm_italian                     -2.43   -2.35   -0.08  2/2
vmsA                   lm_italian                     -2.39   -2.28   -0.11  4/4
controls:
  ctrl_gart              de_early                     heldout   1.23 acc 0.9905628197839682
  ctrl_ME                me                           heldout  -0.06 acc 0.9306424104604889
  ctrl11_shuffled        lat                          heldout  -2.82 acc 0.02183058555997726
  ctrl_English_11k       bible                        heldout  -2.81 acc 0.14394542353610007
  ctrl_English_11k       bible_r4                     heldout  -2.62 acc 0.25969300739056284
  ctrl_Italian_11k       bible                        heldout   0.85 acc 0.9928368391131325
  ctrl_Spanish_11k       bible                        heldout  -2.02 acc 0.20932347924957362
  ctrl_Spanish_11k       bible_r4                     heldout   1.28 acc 0.9860147811256396
