How I think
I start with how it works.
In a complex dispute, I need to understand the detail well enough to explain what it changes for the client. An interesting legal argument may make little difference to the decision. A missing document, an assumption about funding or the cost of the next procedural step may change it completely. Part of my job is working out which of those things deserves the client's attention.
1Explaining the decision
Explaining that clearly is also how I test my own understanding. I organise a problem around relationships: which facts support the argument, what remains uncertain, and how a change in one part affects the rest. If I cannot explain why a detail matters without taking someone through the entire file, I usually need to do more work on the explanation.
That habit has shaped my approach to technology. When a legal AI tool produces an answer, I want to understand what happened before the words appeared: which documents it received, how they were selected, what was excluded and which checks actually ran. Building systems myself has made those questions concrete. I have had to make the choices that an interface can conceal.
It also puts me in a useful position when talking to a firm or a client. I can examine a technical decision and explain its practical consequence: whether the tool can answer the question they need answered, what a lawyer must still check, and whether the time saved justifies the work and risk involved.
2How I test an answer
I start with the process that produced it. Who selected the documents? What question were they asked? What were they trying to achieve? A witness statement, a disclosure list and an AI answer all reflect choices that may be invisible in the finished text.
I also distinguish an instruction from a control. Asking a model to check citations does not establish that it has done so. Software can block an export when a citation has not passed a defined check, but the check itself needs testing. Finding a real case does not establish that it supports the proposition attributed to it.
Finding a real case does not establish that it supports the proposition attributed to it.
I want tests that expose plausible mistakes. A high average score is little comfort if the tool fails on the class of documents a team relies on. A long report is not necessarily a useful one. I look at what was missed, what was wrongly flagged and how much work the reviewer must do to find out.
There is a cost to checking, too. The aim is to reduce a material risk. When another review adds no useful assurance, it is time to stop.
3What building revealed
When I started using language models, I wrote detailed protocols: cite the source, state the other side's position fairly, flag missing information, never invent an authority. They helped, but the model could repeat a rule and then breach it in the next answer. An apology gave me no reason to trust the next attempt.
To answer the questions I had about document selection and omissions, I needed to understand the steps before the model received its instructions. I learned how retrieval systems were built and built one myself.
The choices behind the answer
Building one exposed decisions I could not see from a chat window. How should a document be divided into passages? Would a clause remain attached to its definitions and qualifications? How would the search rank the passages, how many would be returned, and how much could fit into the model's context? Each choice affected the material available for the answer.
A limit on the number of retrieved passages can look like a minor configuration setting until you consider the question a lawyer might ask. Finding a particular indemnity and finding every indemnity in a four-hundred-page bundle are different tasks. A search that returns enough material for the first may leave the second incomplete. The model could accurately describe every passage it received and still miss the provision that matters.
From the outside, both tasks can look like asking a question about an uploaded bundle. Once I was building the system, I had to decide what the search would return and how I would know whether it had missed something. The apparent simplicity of the interface concealed decisions about scope, context and completeness that mattered to the legal work.
An illustrative retrieval limit, not a description of every tool.
- 01The bundle
Passages are indexed for search.
- 02The selection
Suppose only eight ranked passages are returned.
- 03The answer
The model can use those passages; another relevant clause may be absent.
Check coverage separately when the task requires an exhaustive answer.
This is where that way of thinking led me: a concern about an answer became an investigation of the process that produced it, then a tool I could inspect and change. It also changed what I wanted to teach. Lawyers need enough understanding of the system to recognise when the question they are asking requires more than the tool has been shown to do.
What I could enforce
Building software also made me more exact about controls. Calculations, required fields and source-reference checks can be handled in code. Assessing an argument needs judgement. Both need testing, and the lawyer remains responsible for deciding whether the result can be used.
I build games as well as legal tools. Playing a game is a useful way to catch a model claiming success because something looks finished: the world may render, yet the character cannot get through the door. Legal tools have their equivalents. A report can contain all the required headings and still misstate the evidence.
My approach is to retain source references through each stage, distinguish a failed check from one that could not be completed, and test against examples with known errors. I still have to read the outputs. Code makes some requirements enforceable; it does not make the whole system correct.
4What I want to change
LegalQuants gave me a group of lawyers with whom I could build, compare results and discuss failures openly. Being able to show someone a tool and have them tell me specifically why it does not work has been more useful than general encouragement. That is the kind of teaching I want to offer through the LegalQuants AI Residency.
I want clients to receive better work at a lower cost, and lawyers to spend less of their time on tasks a well-tested tool can help with. There is a personal reason too: I have a large family and would like to see more of them.
Hourly billing can make time savings commercially awkward. Under a fixed fee, the same saving may improve the firm's margin. Neither arrangement settles how much time a tool really saves once checking and correction are included, or how the benefit should be shared with the client.
Those decisions need to be part of adoption. A firm needs to consider its fees, supervision and allocation of work alongside the software. Otherwise, faster drafting may simply mean more drafts to review, and saved hours may turn into a higher workload.