Co-Geeking

Boosters of large language models (LLMs) and other kinds of so-called artificial intelligence make big claims about what the technology can do for us, sometimes referencing the benefits brought by other inventions like the Internet or mass production. I rarely find such arguments convincing when applied to my field, history. An experience from my graduate student days may help illustrate why.

When I was a graduate student in the early 2000s, I wanted to write about the Greek historian Polybius and his idea of what constituted Italy. Polybius lived and wrote in a time when the Roman state had fought a century of wars to conquer and defend the Italian peninsula. The idea of Italy as a single thing that could be defined and found on a map was still somewhat new and up for debate. I was interested in seeing where the boundaries of what Polybius called Italy lay, as a reflection of how the Roman elite whose society he moved among thought about their empire and its place in the world.

The obvious place to start was to search the text of Polybius’ Histories for references to Italy, but remember that this was the early 2000s. Search engines for the internet were still in their adolescence, and while there were some projects under way to digitize Classical texts and make them searchable, they still had their limitations. To get the information I needed, I went to the library and found a concordance of Polybius.

A concordance is a type of scholarly reference work that was common in the days before texts became searchable. It is a list of every word used in a particular text (such as Polybius’ Histories) and the context in which it is used. To find every instance when Polybius used the word Italy (or Ἰταλία in Greek), I just had to open the volume to the letter iota, scan down to Ἰταλία, and start going through the references to find which ones were worth looking up in my copy of Polybius and which ones were not useful for my research.

The work that went into creating such a concordance was enormous. The surviving text of Polybius’ Histories runs into the hundreds of pages in a modern printing. Someone had to go through the Greek text and catalog every single word (not to mention dealing with the issues of differing texts in different manuscript traditions, scribal errors, and emendations), then compile all those references into one enormous volume. All of this work was done by hand in the days before computers. The book that I laid out on the table in front of me when I was writing that paper represented thousands of work-hours, a significant chunk of some previous scholar’s working life. (I was lucky to have chosen a research question about a well-known author whose work had been concordanced by scholars of past generations. If I had wanted to check the work of some more obscure author or uncatalogued fragments, I would have had to sit down and scan every page myself.)

If I wanted to research the same question today, I could simply load a copy of the Greek text, type the word into a search box, and have the results in seconds.

Technologies like searchable electronic text have not only changed what questions scholars are able to ask, they have changed the meaning of scholarly work altogether. The kind of rote mechanical labor that went into creating something like a concordance of Polybius used to be a staple of an academic historian’s life. While scholars have always aimed to make new discoveries and present new interpretations of the evidence, up to the late twentieth century it was understood that as a working historian, you would spend a significant amount of your productive life just reading through texts and assembling data a piece at a time, either for your own research or to make a tool for others to use.

These days, although there are still times when searching doesn’t help and you still need to just go through the text line by line, a significant amount of what historians used to do is now automated. Indexed, searchable texts with good metadata have taken the place of a lot of the more cumbersome old scholarly tools in much the same way that electronic databases have replaced the old card catalog system.

This is a change I fully approve of. I have no nostalgia for the old days. I am not shaking my cane at the clouds complaining about kids these days who don’t have to use a concordance in the snow uphill both ways. Making basic information more readily available and easier to probe in new and unexpected ways leads to better questions and more interesting arguments about history, and both scholarly and non-specialist audiences benefit from the wealth of new research that modern tools have made possible.

Now, some have tried to present artificial intelligence as a new revolution in scholarship parallel to the development of searchable catalogs and texts. Just as searchable texts allow us to skip the tedious and unrewarding work of slogging through sources word by word gathering references by hand, so an LLM can save us the tedious work of reading through the existing literature finding the answers to questions so we can spend more time focusing on our own research interest. I find this argument unpersuasive for two reasons.

First, the LLM services which currently exist and promise to perform this kind of operation are not up to the task. They may have scanned all the relevant literature that I would want to consult in my research (and there is a good chance that they have not, but let us suppose for a moment that they have), but they have no understanding of it. They do not know how to separate different threads of argument, how to weigh different theoretical approaches or contrast older and newer scholarship, or how to critically assess evidence. They do not actually know anything, they just slap together text in a way that fits the models they’ve been fed. A search engine may produce wrong results, depending on how well the text it’s searching has been coded or how accurate a search term one uses, but these errors at least point to specific data points that can be checked. An LLM produces authoritative-sounding nonsense with as much facility as truth. It saves no time or effort to use an LLM for research, since everything it produces is suspect, and it does not present its sources for checking.

Second, the tedious work of reading through existing literature is a vital part of scholarship. We have to understand the arguments made by scholars in the past and the bases on which they made them if we are going to do any better at tackling the same questions ourselves. Historical research depends on extensive reading of sources and prior scholarship, not just as a way of assembling data but in order to actually understand our subjects. It is not the same as the rote work of compiling all the words used in a text. There is no royal road to historical understanding, and this part of the research process cannot be automated away.

No one makes concordances any more, and hardly anyone uses them. Search technology saves us labor and frees up scholars’ time to do the more interesting and more important work of engaging with evidence and contemplating new questions. The human work that searches replaced was work that we could well do without. The work that LLMs promise to replace is essential, and they can’t do it for us effectively anyway.

Here there be opinions!