What a Language Model Finds When It Reads Your Constitution

There are two conversations about AI and the law. The first is about legal reasoning: can a large language model interpret an ambiguous statute, assess the strength of competing legal arguments, predict what a court will decide? This is the interesting conversation. It is also mostly a distraction, because the answer is "sometimes, with caveats, in ways that require expert oversight to be reliable" — and because it misses the category of legal AI work where the answer is already clearly yes.

The second conversation is about legal compliance checking: can a large language model read a document against a specific checklist and tell you whether the document satisfies the checklist? This is not a question of legal reasoning. It is a question of structured pattern matching. The model does not need to exercise judgment about what the law means. It needs to determine whether the document contains language that satisfies a requirement whose terms are already specified.

For this task, the answer is yes. Not with caveats. Not with expert oversight as a prerequisite. With reasonable reliability, at scale, in a way that is already useful.

PBOMaster's constitution analyser is built on this distinction. Understanding what it does — and what it does not do — matters for the broader argument this essay is making about where AI creates genuine value in the legal compliance domain.

The Checklist Problem

Return to the Second Schedule of the PBO Act. The requirements for a compliant constitution are, to use the technical term, enumerable. They constitute a list of specific things that must be present. Each item on the list can be checked: either the document contains language that satisfies the requirement, or it does not. Some items require the checking of the language itself — does the conflict of interest clause prohibit participation in conflicted decisions, or merely require disclosure? — but even these involve checking against a defined standard, not exercising open-ended legal judgment.

This is the task that large language models are well-designed for. The academic literature on LLM performance in legal tasks has developed rapidly since 2023. Katz et al.'s evaluation of GPT-4 on the Multistate Bar Examination found performance at or above the passing threshold across all subject matter categories.¹ Choi et al., evaluating ChatGPT on law school examination questions, found it capable of passing with grades in the C+ range across multiple law school courses.² These results are relevant, but they describe performance on the hard problem — the one involving legal reasoning and interpretation.

The compliance-checking problem is easier. It does not require the model to reason about what the law means. It requires the model to determine whether a specific provision is present in a document, and whether the language of that provision satisfies a defined standard. On this task, performance is higher and consistency is more reliable than on open-ended legal reasoning. The model does not need to be a lawyer. It needs to be a thorough reader with a complete checklist.

What the Analyser Finds

In the course of building PBOMaster's constitution analyser and testing it against Kenyan NGO and CBO constitutions, a pattern emerged that was consistent enough to be worth documenting here. The following gaps appear across constitutions of different ages, from organisations of different sizes and types.

Conflict of interest provisions are absent or insufficient. This is the most consistent gap. Most constitutions older than 2013 were drafted before conflict of interest governance became a standard expectation in Kenya's civil society sector. Many constitutions that have a conflict of interest clause have one that requires disclosure but does not specify the consequence: board members are expected to declare interests, but the constitution does not prohibit participation in conflicted decisions. The PBO Act requires both.

Dissolution clauses contain the wrong asset-reversion language. Many constitutions have dissolution clauses but specify that assets revert to "members" or are "distributed according to board resolution" rather than reverting to another public benefit organisation or to a public fund. This was acceptable under the old framework. Under the PBO Act, it is a disqualifying defect.

Board term limits are absent or non-compliant. The PBO Act's governance requirements include provisions designed to prevent board capture — indefinite board tenure by founding members being a particular concern in long-running CBOs. Constitutions drafted before the Act often have no term limits, or term limits written in a way that allows indefinite renewal.

Meeting provisions use ambiguous language. Quorum requirements specified as "a majority of members" are not specific enough — the Act requires numerical clarity. Meeting frequency specified as "as required" rather than at defined intervals does not satisfy the minimum meeting frequency requirements.

The public benefit objectives clause is too general. Many constitutions state objectives at a level of generality — "to promote community development and welfare" — that does not map clearly onto the First Schedule's categories of qualifying public benefit activities. The constitution needs to be specific enough to make the regulatory case.

These are not obscure defects. They are structural, common, and invisible to an organisation reading its own constitution without a compliance checklist in hand.

The Two Kinds of Legal AI

Legal AI is often discussed as though "can AI do legal work?" were a single question with a single answer. It is not. The question has very different answers depending on which legal task you are asking about.

At one end: Can AI predict how a judge will rule on a novel question of constitutional interpretation? No, not reliably. This involves reasoning about ambiguous text in the context of a specific factual record, anticipating how a particular judicial temperament will weigh competing considerations, and exercising the kind of integrative judgment that comes from years of legal practice. AI can assist with components of this task. It cannot do the task.

At the other end: Can AI determine whether an organisation's constitution contains a conflict of interest clause that prohibits participation in conflicted decisions? Yes, reliably. This involves finding text, reading it against a defined standard, and reporting the result. It does not require legal judgment. It requires thoroughness and a good checklist.

PBOMaster is an instance of the second kind of legal AI. The constitution analyser does not reason about the law. It reads documents against a checklist the legislation provides. The value it creates is not in the interpretation — the interpretation is already done, by the Act and its schedules. The value is in the systematic application of that interpretation to every document that needs to be checked, faster and more completely than any individual reviewer would do it.

The Generalisation

The broader argument is this: the discourse around AI and professional services has been captured by the hard cases — the ones that require genuine reasoning, judgment, and professional accountability. These are real and interesting. But they have obscured the category of work where AI is already reliable, already scalable, and already underused: the work of checking documents and processes against specific, enumerable standards.

Regulatory compliance is disproportionately populated with this kind of work. Checking a constitution against the PBO Act's Second Schedule. Checking a financial report against the Companies Act's reporting requirements. Checking a board composition against the governance rules of a specific funding framework. For every one of these tasks, the compliance requirement is already specified. The work of encoding it has already been done by the regulator. The remaining work — the work AI can do — is applying that encoding systematically to the documents that need to be checked.

This is the category of work PBOMaster is built for. It is also the category of work that has the highest return on AI investment: not because the AI is doing something sophisticated, but because the alternative — doing it manually — is slow, expensive, and inconsistent. The next essay examines what happens when the urgency of doing it shifts from theoretical to immediate.