Skip to content

Cole Simmons

I'm an engineer and researcher.

I used to be a San Francisco-based technical startup founder.

Now I'm at Oxford, working on the corpus of 500,000+ cuneiform texts.

Honors thesis, Stanford, 2024

Babbage in Babylon: Pioneering Deep Learning Approaches to Sumerian Cuneiform

More than 120,000 Sumerian texts have been excavated. Few people can read them, and few have been translated.

Reading a tablet takes three steps: recognising its signs, deciding how each is read in context, and translating the result. This thesis set each step up as a machine-learning task. What had held that back was data, not models. Editions online record each reading but not the sign behind it, so nothing paired the two.

I built an open dataset for each step and trained two models on the largest. One restores masked signs with 80% accuracy. The other transliterates tablets it has not seen at 97.54 chrF, where choosing readings by frequency alone scores 61.22.

Read the thesis in the Stanford Digital Repository.

research

degree B.S. with Honors, Symbolic Systems
university Stanford
advisor Dan Jurafsky
second readers Ian Morris, Niek Veldhuis
submitted June 2024
award Firestone Medal
licence CC BY-SA 4.0

Three steps to reading a tablet

A specialist reading a tablet identifies its signs, settles how each is read, and translates. Each step maps one representation of the text to the next, and each became a dataset of pairs. The splits agree across all three: a tablet held out for testing in one is held out in every other, so models for successive steps can be chained without leaking test tablets.

  1. Step 1

    Recognise the signs

    image to 𒃻𒃶𒅅

    dataset SumTablets_Photos
    pairs 79,633 images
    model not yet trained

    Photographs (42,742) and line drawings (36,891) from CDLI, each paired with the signs of its tablet: the ground for cuneiform OCR.

  2. Step 2

    Transliterate

    𒃻𒃶𒅅 to ninda he₂-ŋal₂

    dataset SumTablets
    pairs 91,606 tablets
    model 97.54 chrF

    Unicode cuneiform paired with Oracc transliterations, 6,970,407 signs in all. Both models below are trained on it.

  3. Step 3

    Translate

    ninda he₂-ŋal₂ to ‘If there was food,’

    dataset SumTablets_English
    pairs 2,127 tablets
    model not yet trained

    Transliterations paired with the English translations in CDLI. Too few to train a model from scratch, but a start for pretrained models and back-translation.

One tablet, three ways

UET 6/2 320, a school tablet from Ur with a proverb from Proverb Collection 1. Old Babylonian; CDLI P254888. It is in all three datasets, as a line drawing in SumTablets_Photos, and held out for testing in each. Translation from CDLI.
  1. 1 GAR GAN IG ninda he₂-ŋal₂ ‘If there was food,’
  2. 2 AN NIN PEŠ₂ TUM KA×GAR ᵈnin-ka₆ ib₂-gu₇ ‘a mongoose ate it.’
  3. 3 GAR MU GAN IG ninda-ŋu₁₀ he₂-ŋal₂ ‘If my food was present’
  4. 4 LU₂ PAP TUM KA×GAR lu₂-kur₂ ib₂-gu₇ ‘A stranger ate it.’

Signs and transliteration make a SumTablets pair; transliteration and translation, a SumTablets_English pair. A transliteration model sees only the signs, with no word division, and chooses every reading. GAR is ninda ‘food’ both times; TUM is the prefix ib₂- and KA×GAR the verb gu₇ ‘to eat’; AN is not dingir ‘god’ but the determinative written before nin-ka₆ ‘mongoose’.

The signs are rebuilt from the edition’s readings, not read off the clay, so the data is only as exact as the edition. Oracc’s ka₆ maps to the sign PEŠ₂; CDLI’s edition of this tablet records the sign written there as U.PEŠ₂.

Why the data came first

Oracc and CDLI hold transliterations for most of the Sumerian corpus, made by dozens of projects over decades. As they stand, they are not training data. Conventions change from project to project, the text is threaded with editorial annotation, and nothing records the signs: an editor reads from the tablet or a photograph and publishes only the reading.

The signs can be recovered because, although a sign has many readings, each reading belongs to only one sign: ka, dug₄, and inim are all readings of KA. The ePSD2 and Oracc sign lists map every reading to its sign and every sign to Unicode. Surfaces, columns, rulings, line breaks, and breakage are kept as tokens, and photographs and translations from CDLI join on each tablet’s ID.

The full construction is on the SumTablets page, the paper that published this dataset.

Learning the signs

The first question was whether a multilingual model could learn Sumerian signs at all. Sumerian is an isolate, but XLM-R was pretrained on languages that share its features: ergative alignment with Basque, agglutination with Turkish and Japanese, verb-final order with Korean.

A new tokenizer gives each of the 621 signs in the data its own token. The model learns by restoring one sign in ten, masked at random, with the network unfrozen in stages from its embeddings and output layer down to every layer. With its earlier layers still frozen it reached nearly 70%; fully trained, 80%, alike on training, validation, and test tablets.

That is not enough to restore broken passages. A break usually takes out several neighbouring signs at once, and a few signs are so common that the model over-predicts them. The likelier use is flagging signs that look improbable in context, for an editor to check.

Transliteration

The transliteration model joins that encoder to an XLM-R decoder that writes readings, trained first with the encoder frozen and then end to end. The floor is a dictionary: weighted by frequency, a sign has 22.17 readings, and the baseline samples one in proportion to how often it occurs.

  • All periods 4,577 tablets

    dictionary baseline
    61.22
    neural model
    97.54
  • Ur III 3,951 tablets

    dictionary baseline
    62.89
    neural model
    98.46
  • Old Akkadian 265 tablets

    dictionary baseline
    64.52
    neural model
    94.03
  • Early Dynastic IIIb 192 tablets

    dictionary baseline
    62.51
    neural model
    97.08
  • Old Babylonian 73 tablets

    dictionary baseline
    37.70
    neural model
    90.38
  • Lagaš II 44 tablets

    dictionary baseline
    58.55
    neural model
    93.97
  • Early Dynastic IIIa 42 tablets

    dictionary baseline
    67.85
    neural model
    95.02
chrF on held-out tablets, by period, out of 100. Periods with fewer than forty test tablets are left out. By genre the spread is wider: literary texts score 90.67 and letters 90.99 (see SumTablets).

Where it goes wrong

  • Genre drives the gaps. Administrative texts dominate the training data and score highest; letters, literature, and liturgy differ in style and vocabulary, and are hard for specialists too. The Old Babylonian gap is mostly genre: Sumerian was no longer spoken, and 53 of its 73 test tablets are literary.
  • Conventions split the evidence. saŋ and sag write the same word; editors differ only in how they render the nasal. The model sees two patterns where there is one.
  • Names are hardest. Many errors are valid readings of a sign inside a personal name, just not the editor’s. Whether they are less plausible takes a specialist to judge.

What it leaves open

  • Two of the three steps. The photograph and translation datasets are released, but no model was trained on either.
  • The editions’ errors. The signs are derived from transliterations, so an edition’s mistakes and normalisations pass into the data and into any score measured against it.
  • An administrative corpus. More than nine in ten tablets are administrative records, most from the Ur III state, and the model is best at what it has seen most.
  • A control. Whether the multilingual starting point helps has not been tested against a model trained on Sumerian alone.

What comes next

The thesis ends on a loop rather than a model. A specialist corrects a generated transliteration instead of writing one from scratch. Corrected editions retrain the model, and the masked-sign model points to the likeliest errors in what is already published. Each round starts from better data. With models for sign recognition and translation, the same loop can run over signs and translations.

SumTablets

The dataset and the transliteration baselines were published at the ML4AL workshop at ACL 2024, with Richard Diehl Martinez and Dan Jurafsky. The paper has its own page.

Towards a Spatial Grammar

Every dataset here treats a tablet as a sequence of signs. My MPhil thesis takes up part of what a sequence leaves out: in early cuneiform, where each sign sits inside its cell. The thesis has its own page.

Cite

Simmons, Cole. 2024. Babbage in Babylon: Pioneering Deep Learning Approaches to Sumerian Cuneiform. Honors thesis, Stanford University. Stanford Digital Repository. doi:10.25740/mn578sb9212

@thesis{simmons-2024-babbage,
    title = {Babbage in Babylon: Pioneering Deep Learning Approaches to {S}umerian Cuneiform},
    author = {Simmons, Cole},
    type = {Honors thesis},
    institution = {Stanford University},
    year = {2024},
    month = jun,
    doi = {10.25740/mn578sb9212},
    url = {https://purl.stanford.edu/mn578sb9212}
}