Skip to content

Cole Simmons

I'm an engineer and researcher.

I used to be a San Francisco-based technical startup founder.

Now I'm at Oxford, working on the corpus of 500,000+ cuneiform texts.

Work, 2016–2022

AbstractCRE

Commercial property owners run on the terms inside their leases: rent, dates, areas, options. Those terms are usually copied by hand into spreadsheets, where nobody can see which page a number came from or whether it is right.

AbstractCRE turns leases, loan documents, and income statements into structured data, with every value linked to the words that support it. Models propose values; a person confirms each one against the page.

I left Stanford to co-found the company and was its CTO for six years, and its only engineer for most of them. I led the engineering: the product, the API, the document pipeline, and, with an ML collaborator, the models.

work

role Co-founder & CTO
years 2016–2022
documents Leases, loan documents, income statements
customers Commercial property owners
company abstractcre.com

A lease and its abstract

leasep. 1

  1. 1. Parties. This Lease is made between Harbor Point Holdings LLC (“Landlord”) and Juniper & Vale Coffee Co.1 (“Tenant”).
  2. 2. Premises. Landlord leases to Tenant approximately 2,450 rentable square feet2 on the ground floor of the Building.
  3. 3. Term. The Term begins on March 1, 20213 (the “Commencement Date”) and ends on the last day of the sixtieth full month thereafter.
  4. 4. Base Rent. Tenant shall pay Base Rent of $42.50 per rentable square foot per year4, in equal monthly installments in advance.
  5. 5. Renewal. Tenant shall have one option to extend the Term for five years, on at least nine months’ prior written notice to Landlord, at the then-prevailing market rate. 5
An abstract and the lease it came from. Every value keeps a pointer to the words that support it, so a reviewer checks it against the page rather than taking it on trust. Hover over or select a field to see its source. The lease is invented.

Pages, not PDFs

A reviewer checking a value needs its page now, not after the browser has downloaded and drawn the whole document.

Early on, opening an abstract loaded the entire PDF into the browser, and some ran to 100 MB. I asked Gradescope, which shows graders scanned student work, how they handled it, and took their approach: turn every page into an image ahead of time, then fetch only the pages in view.

The first pipeline was a task queue running beside the API in Kubernetes. Under load it slowed the API down, and it failed often. We replaced it with functions chained by Pub/Sub messages, fanned out so that every page of a document was rendered, and its OCR formatted, in parallel.

At enough volume, storage errors that strike one request in a thousand become routine, so any document could be reset and run again from the start. One join stayed hacky: Cloud Vision writes nothing for a blank page, so no formatting function could tell that it was the last, and indexing waited a fixed interval after OCR finished instead.

upload A PDF lands in storage The upload triggers a function that checks the file and publishes two messages, one for each branch.

text

  1. Read the whole PDF Cloud Vision OCRs the original file, not the compressed page images.
  2. Format each page A compact form drives the highlights for in-document search; the full response goes to cold storage. once per page, in parallel
  3. Index the text The document’s text goes into Elasticsearch, and the document is marked indexed.

images

  1. Count the pages One message is published for every page.
  2. Render each page Each page becomes an image, and its record in the database is marked ready. once per page, in parallel
  3. Mark the document converted When the last page lands, the whole document is ready to view.
viewer Load only what is on screen The viewer fetches the page in view and a few on either side, through a CDN, instead of the whole file.

Three small models

Each model is fitted to the shape of the answer it looks for. The numbers refer to the fields above.

We chose LSTMs, an older architecture, on the advice that with a dataset as small as ours they often do as well as much larger language models, and train far faster. They followed a first attempt, contracted out, that tried a CRF and then BERT without reaching the quality we needed.

Our ML collaborator built the masked token classifier; once it worked, I built the other two on its pattern. When I left, a fourth was nearly finished: one that predicts which fields a lease simply does not have, so reviewers stop searching for them.

Around the models I built the plumbing: preprocessing on Apache Beam and Dataflow; a tool for mapping abstract templates onto shared generic templates; the models served as separate App Engine services; and predictions queued through Cloud Tasks, so a burst of uploads never swamped them.

  1. masked token classifier fields 2, 3, 4 Dates, amounts, and areas Only tokens that match a date, currency, or number format are candidates. The model labels each candidate with the field it fills, if any.
  2. token classifier fields 1 Names and addresses The same approach without the mask, for short spans of free text such as the tenant’s name or the property’s address.
  3. paragraph classifier fields 5 Clauses Long text fields, such as a renewal option, are found a paragraph at a time.

Measuring what a reviewer feels

The number that mattered was time to abstract: how long a person takes to produce a finished, checked abstract.

Recall, up to a point

With a reviewer confirming every value, a missed field costs a search and a wrong suggestion costs a glance, so we leaned toward recall. Past some point, though, reviewers spend more time dismissing bad suggestions than the good ones save them.

Labels that told the truth

An abstract linked each field to one place in the lease, though a lease may state its area five times. The model was marked wrong for finding the other four, which hurt training and made its precision look worse than it was. We changed the data model to allow many links per field and filled in old abstracts; the model’s own suggestions made full coverage quicker to label.

Evaluation we owned

The first attempt, contracted out, taught us to keep evaluation in our own hands: precision and recall mean little without code you can rerun yourself.

What changed, and why

  1. api Go on App Engine became Python on Kubernetes One language across the API and the machine learning, and an easier one to hire for.
  2. database Cloud Datastore became PostgreSQL I had chosen a document store for customer-defined field types before I knew Postgres handled JSON. Postgres should be the default unless there is a real reason otherwise.
  3. fields A JSON blob per abstract became A typed row per field Anything could end up in a blob, and one bad write could erase every field in an abstract. Typed tables made some queries harder; the safety was worth it.
  4. contracts Flask with hand-written OpenAPI became FastAPI and Pydantic The spec is generated from typed request and response models, which removed much of the hand-written validation and a class of bugs with it. Response times dropped too.
  5. documents A task queue beside the API became Cloud Functions on Pub/Sub The queue competed with the API for resources and failed often. Functions scaled with the page count and stayed off the API’s machines.
  6. search Algolia for records, Elasticsearch for text Algolia is fast and low-maintenance for finding abstracts and properties by name, but it bills per record and its records are small, so the full text of every document would have cost far too much.
  7. auth In-house became Auth0 Enterprise customers needed single sign-on, and running our own authentication was a risk we no longer had to carry.
  8. front end Webpack became Vite Production builds went from minutes to seconds.

Looking back

We first targeted investment-sales brokers, and found that most of their problems started earlier, in how owners kept data while they held a property. So we moved to the owners.

I left in 2022 to finish my degree, and wrote my successor a guide to the whole system: every repository and cloud service, how a document moves through the pipeline, how the models were built, what was half-finished, and what I would do next. The company is still operating, at abstractcre.com.

Stack

front end React, Redux, Vite
api Python, FastAPI, PostgreSQL
models PyTorch, Apache Beam
documents Cloud Vision, Cloud Functions, Pub/Sub
search Elasticsearch, Algolia
platform Google Cloud: GKE, App Engine, Cloud CDN