Subjective Computation

Run payroll for fifty thousand people, and nobody rechecks the arithmetic.

The system was tested and validated; now it runs without anyone checking each result. The same is true of a query that totals last quarter’s revenue, code that routes a payment, or a formula copied down ten thousand rows of a spreadsheet. We trust these systems to handle real business operations without reviewing every execution. That trust is justified when the logic is correct and the inputs meet the assumptions the system was built for. Under those conditions, the same inputs produce the same results.

Now ask a language model to summarize a contract, draft a customer email, or identify risks in a vendor agreement.

Each output needs review. Someone has to read it, check whether the claims are sound, and decide whether it is fit for use. Approving yesterday's output tells you nothing about the quality of today's.

The difference is structural. The two systems operate on different principles, and improving a language model doesn't eliminate that difference. One can execute validated logic without reviewing every result. The other requires evaluation of its outputs. This article focuses on the latter: subjective computation. The deterministic side of the comparison is covered in Deterministic Computation.

Computation by Prediction

When you run SUM over a column, the answer is fixed by the data and the definition of addition. There is no judgment in it and no room to go two ways. Alan Turing described this kind of machine in 1936^[9]: a fixed rule that reads the current state and decides, with no latitude, what happens next.

Every classical program your company runs sits on that principle.

Given some text, a language model assigns probabilities to possible next tokens, selects one, and repeats the process until it reaches the end of the output.^[10] The model is effectively navigating a high-dimensional probability space learned from its training data.^[7] At each step, it selects what is statistically likely to come next, not what is necessarily true. A long sequence of locally likely choices can produce fluent, confident prose without the model having a mechanism that distinguishes a true statement from a false one.

The previous article in this series, What Do Large Language Models Do Then?, covers this mechanism in depth.

A good engineer will point out that you can reduce the randomness. Set the model to select its most likely token at every step and, given the same inputs and implementation, you can make the output reproducible. Set temperature to zero and you get determinism. You don't get correctness.

The model is still ranking possible tokens by how well they fit the patterns it learned from its training data. It isn't evaluating whether the resulting statement is true. Reducing variance changes the consistency of the output, not the basis on which the output is produced.

Determinism can make a language model repeat the same mistake reliably; it cannot give the model a mechanism for measuring truth.

Meaning Comes From Context

The model learns statistical relationships between words across its training corpus—what words tend to occur together and in what contexts. That is what makes the output easy to mistake for understanding: the model can reproduce the language of a subject convincingly without actually representing the subject itself.

A model learns the statistical relationships between words by observing which words appear together across a vast corpus of text. “Invoice” appears alongside “payment,” “due,” and “net 30,” so the model learns how the word is typically used. The linguist Zellig Harris described this principle in 1954^[3]: words with similar meanings tend to occur in similar contexts, and differences in meaning show up as differences in where words occur.

But usage is not reference. The model can learn how people use the word “invoice” without having any connection to the thing the word refers to. It has processed “invoice” billions of times without ever seeing an invoice, being owed money, or waiting for a late payment.

Stevan Harnad identified the underlying problem in 1990.^[2] Symbols in a formal system do not acquire meaning from the system itself; their meaning has to be grounded outside it, in the world those symbols describe. Inside a language model, words are connected to other words, but there is no underlying connection to the things those words refer to.

The earlier article in this series, Do Large Language Models Reason?, examines the same problem through the four components of reasoning and reaches a similar conclusion from a different direction.

The meaning of a model's output is supplied by the reader. The reader brings the context, the stakes, and the knowledge needed to understand what the words refer to. Two people can read the same model-generated summary and come away with different interpretations because each supplies context the text itself does not contain.

A SUM query produces the same result regardless of who reads it. A model generates text that the reader must interpret and validate.

That dependence on the reader is what makes this computation subjective.

Plausibly Correct

You can trust a billing system because its correctness is established once and then carried forward by every subsequent run. Test it, work through the edge cases, review the implementation, and the same logic applies every time.

After that, oversight becomes monitoring. You watch for a changed tax rule or a bad data feed, not for the arithmetic itself. The calculation does not change from one execution to the next.

Language models work differently, and that difference changes the nature of the oversight required.

A language model does not necessarily reproduce the same output, so the usual debugging model breaks down. There is no stable result to diff against and no single failing case to isolate. Change a prompt without materially changing its intent, and the output can change.

There is also no equivalent of a known-correct result to compare it with.

“Is this contract summary correct?” does not have a fixed answer in the way a column total does.

Correctness depends on the purpose of the summary, the relevant facts, and the judgment of the person reviewing it. Evaluating the output can require nearly as much domain expertise as producing it. A plausible wrong answer can look the same as a correct one until someone qualified examines it.

There is no cheap verification step for something that looks "plausibly correct".

The cost of verification does not amortize. A billing system can be thoroughly tested once, then trusted across millions of executions. A model output requires evaluation each time.

Kahneman and Klein (2009)^[4] describe when fast, intuitive judgment can be trusted: the environment has to be regular enough to be predictable, and the person needs prolonged practice with rapid, unambiguous feedback. Someone reviewing open-ended model output usually has neither. The cues to whether an answer is correct are weak, and feedback on whether it was right comes late or not at all. Each output has to be checked deliberately. As usage increases, the cost of oversight increases with it.

Better models reduce the problem without removing it. Kalai and Vempala (2024)^[5] proved that a pretrained language model that is calibrated, meaning its confidence matches how often it is right, must hallucinate at some rate on arbitrary facts: facts that can't be worked out from patterns in the training data. That rate is roughly the share of such facts that appear only once in training. The bound comes from statistics, not from a particular model architecture.

The authors are explicit about the limits of the result. It doesn't apply to systematic facts like arithmetic, and post-training or looking facts up in a database at generation time can lower the hallucination rate, at the cost of calibration. Where a fact can be checked against a fixed source, the reliable fix is to move that check onto deterministic software.

Classical computing has its own fundamental limits: some questions about programs cannot be resolved by any algorithm, such as whether an arbitrary program will ever produce a given output.^[9][8] Language models have a different limit. A model built to predict plausible text has no built-in check on whether that text is grounded in fact. Engineering can reduce how often that matters, but the check still has to come from outside the model.

The Case for Language Models

This is a different kind of machine, with a different strength.

A database operates within a defined structure: typed data, fixed schemas, and a known set of operations. But much of the information a business depends on does not fit that structure. Contracts, emails, support tickets, research, and even the half-formed question a customer asks are expressed in open-ended language that does not map neatly to a schema.

Classical software could store, index, and retrieve that text. It could not reliably work with the meaning expressed in it.

Language models operate on this layer. You can give them contracts, emails, support tickets, and research notes and get back a useful summary, an answer to a question, or an analysis of the material. That is a real capability, and it explains much of the market's interest in the technology.

The trade-off is straightforward. A database gives you predictable results within a defined schema. A language model can work across unstructured text, where the information doesn't fit neatly into predefined fields and relationships. Much of the potential business value comes from being able to analyze information that conventional software could store and search but not interpret.

The companion article, What Do Large Language Models Do Then?, explores how this works in practice.

When Plausibility Passes for Correctness

Both systems produce outputs that look convincing, and that resemblance is where teams get into trouble. A language model returns an answer that is fluent, well-structured, and confident.

It looks like something worth trusting.

Joseph Weizenbaum, who built ELIZA, one of the first chatbots, in the 1960s, observed that users attributed understanding to his program because its responses sounded appropriate.^[11][12] People tend to judge text that is easy to process as more credible.^[1] A language model produces fluent text whether its answer is accurate or not.

Don't treat model output as if it carries a guarantee of correctness. A summary that looks finished has not necessarily been verified. A language model uses the same underlying process to generate correct and incorrect answers, and both can sound equally convincing.

The risk is greatest when the output looks polished enough that people stop checking it. In Mata v. Avianca (2023), lawyers filed a brief citing court decisions that ChatGPT had invented.^[6] The citations were properly formatted and sounded credible. Opposing counsel caught them by trying to look the cases up and failing to find them; the judge then confirmed the cases did not exist.

Building reliable systems with language models means identifying which parts of a task can be made deterministic and which require judgment.

Use conventional software for the parts that can be specified precisely. When a model needs to perform arithmetic or apply strict logic, have it generate code and execute it rather than generate the answer directly. The result can then be checked against defined rules.

The remaining work—interpreting the result, evaluating its relevance, and deciding whether it is correct for the situation—requires human judgment. That division of responsibility isn't a temporary workaround until better models arrive. Deterministic computation can be verified against a defined standard. Language model outputs must be evaluated in context. Good system design accounts for that difference rather than pretending it will disappear.

When Computation Needs Judgment

Two kinds of computation, with different guarantees. Deterministic computation produces exact, repeatable results under defined rules and data constraints. Language models operate on the ambiguity of language and meaning, where there is no single correct output guaranteed by the computation itself. Each result has to be evaluated on its merits.

Nearly every system worth building today combines both. The engineering challenge is deciding which parts of the work can be made deterministic and where a language model is necessary. Model outputs still need human oversight because the computation itself cannot establish whether a result is correct.

---

References

1. Alter, A. L., & Oppenheimer, D. M. (2009). Uniting the tribes of fluency to form a metacognitive nation. Personality and Social Psychology Review, 13(3), 219–235. doi:10.1177/1088868309341564
2. Harnad, S. (1990). The symbol grounding problem. Physica D, 42(1–3), 335–346. doi:10.1016/0167-2789(90)90087-6
3. Harris, Z. S. (1954). Distributional structure. Word, 10(2–3), 146–162.
4. Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515–526.
5. Kalai, A. T., & Vempala, S. S. (2024). Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing (STOC 2024). arXiv:2311.14648.
6. Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023).
7. Quattrociocchi, W., Capraro, V., & Perc, M. (2025). Epistemological fault lines between human and artificial intelligence. arXiv:2512.19466.
8. Rice, H. G. (1953). Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society, 74(2), 358–366.
9. Turing, A. M. (1936). On computable numbers, with an application to the Entscheidungsproblem. Proceedings of the London Mathematical Society, Series 2, 42, 230–265. doi:10.1112/plms/s2-42.1.230
10. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems 30.
11. Weizenbaum, J. (1966). ELIZA—a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36–45. doi:10.1145/365153.365168
12. Weizenbaum, J. (1976). Computer Power and Human Reason: From Judgment to Calculation. W. H. Freeman.