Artificial Intelligence – warnings from a litigator

The purpose of this article isn’t to pass judgement on whether AI is good or bad: there are already plenty of people who will tell you it is a ‘very good thing’; and plenty who will tell you it is a ‘very bad thing’.

Either way, there is no doubting that it is at least a ‘thing’ – a tool that is now built into our phones, computer software, and for many, our ways of working. At Archor we have seen its use become more commonplace over the last few years, to the extent that we now regularly see (for example) clients using it to collate and summarise documents, other parties using it to write submissions, and (anecdotally at least) it playing a part even in adjudicator’s decisions. And in the wider world, there have been numerous reports of AI hallucinations and other horror stories.

So, with the use of AI not realistically about to stop anytime soon, this article considers two of the biggest dangers from the perspective of a litigator.

Hallucination

AI can come in many forms, but in this context we are talking about Large Language Models such as ChatGPT and Claude. LLMs generate responses by predicting what comes next in a particular context, based on patterns identified from masses of training data.

A judgment from the President of the King’s Bench Division of the High Court last year (R (Ayinde) v. London Borough of Haringey [2025]) cast this warning about the use of tools like ChatGPT in the legal world: “Such tools can produce apparently coherent and plausible responses to prompts, but those coherent and plausible responses may turn out to be entirely incorrect. The responses may make confident assertions that are simply untrue. They may cite sources that do not exist. They may purport to quote passages from a genuine source that do not appear in that source”.

These issues are generally termed ‘hallucinations’, but that is a euphemism: what the AI is actually doing is making things up.

Most AI systems are trained to be helpful, so want to give users the answers they look for, but the data from which they are trained is not always complete (or, the answer they’re looking for doesn’t exist). Rather than disappoint its user, some AI systems will ‘fill the gaps’ itself – or ‘hallucinate’.

One example comes from the Haringey case. The claimant brought proceedings against the Council due to a failure to provide interim accommodation. In the grounds for judicial review, five cases were cited by his lawyers (Haringey Law Centre) which simply did not exist. When the Council asked for copies of these cases, they could not be provided (because they did not exist). The cases looked real – they were in the format one would expect and contained citations again in the correct format (which in one case was real – just for a different case) – but fundamentally they didn’t exist.

Another comes more recently with a case from this year involving solicitors Pinsent Masons. In Malcolm Cork & Anor v. Smith [2026], as part of an application Pinsent Masons referred to a power within the Insolvency Act 1986 that did not exist. The judge queried this, and asked Pinsent Masons for more information. In response they asserted a specific provision of the Insolvency Rules gave the relevant power, which the judge said “came as a surprise to me”. When he looked himself, no such power existed, and he demanded an explanation.

Pinsent Mason’s response was to claim that they had not argued there was a specific power and nothing was intended as a ‘direct quotations’. The judge said “I was astonished by this reply” with the explanation “impossible to accept”. Understandably irate, the judge called for Pinsent Masons to give full evidence of what had happened, through which it transpired that a fee earner had used their custom AI software to draft the two letters. 59 pages of AI prompts were disclosed in which – in fairness to the AI – it had warned that it could not be relied on and presciently observed that “The last thing you want is to cite a provision to the court with inaccurate wording”.
Both these cases involved legally qualified individuals who obviously should have known better – in the Haringey case the judge described the lawyer’s actions as ‘improper, unreasonable and negligent’. And both involved ‘cover ups’ where the response to the errors clearly made matters worse.

But they are far from the only cases – Haringey lists numerous other cases of hallucinations both domestically and overseas, and that is only those that have been spotted. How many other hallucinations have been referred to in correspondence, in non-court submissions (such as adjudication), or simply not picked up on?

The lesson should be clear: LLMs like ChatGPT are not reliable. As the judge in Haringey put it, they “are not capable of conducting reliable legal research”. They may tell you the answer you want, or at least give you an answer, but without checking underlying source material you cannot be sure that what you are being told is accurate or relevant.

Confidentiality and disclosure

A lot of LLMs, the free and consumer versions of ChatGPT included, are not confidential. They may store conversations, and they may use the contents of discussions to train AI models. The courts have confirmed that placing information “into an open source AI tool, such as ChatGPT, is to place this information on the internet in the public domain” (R (Munir) v. Secretary of State for the Home Department [2026]).

Confidentiality is key for most businesses. Asking questions about any confidential business or financial transactions risks that information becoming known to third parties, with commercial consequences.

In a legal context, confidentiality can become even more important. As part of litigation, parties are generally required to disclose relevant, non-privileged documents to other parties.

‘Documents’ has a wide meaning in that context and includes electronic records. It is fairly clear that records of prompts to AI software, and the results of those prompts (including work product from the LLM), would come within that definition.

As such, all conversations with AI may ultimately fall to be disclosed – indeed, AI prompts have formed part of the evidence in a number of the cases referred to above. It is unlikely that such prompts would be exempt from disclosure, such as through privilege, because of the lack of confidentiality and the fact that AI discussions are inherently not with a solicitor (so legal professional advice privilege will not apply).

Consider, then, what this means in practice. Potentially all discussions with AI, and its responses, may end up being seen by your opponent. You may ask AI about weaknesses in your case, drawing attention to those. You may make comments that imply an admission about certain matters. Or you may make other comments that come back to haunt you.

One recent example comes from America and a case concerning an explosion in 2020 at a manufacturing plant in Houston which killed three people and destroyed around 200 homes. Through ‘discovery’ (the US version of disclosure), it transpired that an expert retained by 3M, Josh Autenrieth, had asked ChatGPT to “create an exceptional expert witness report defending the standard of care at 3M” which he said should “show how 3M is 0% at fault for the explosion”. Analysis showed that Mr Autenrieth’s report was 85%+ created by ChatGPT – despite him being paid around $90,000 for it. The revelation clearly significantly undermined Mr Autenrieth’s credentials as an independent expert – in England, experts owe a duty to the court to present impartial evidence, and the suggestion that the report should show his instructing client to be ‘0% at fault’ would undoubtedly have been catastrophic. Although this related to an expert, it’s also not difficult to imagine a case where a factual witness statement might be found to have used AI in its drafting, which could be similarly damaging.

While the interplay between AI and disclosure is relatively untested at this point – it has simply not been round long enough for there to have been significant litigation about it – this will undoubtedly happen over the next few years. Our warning is simple: be aware of the risk, and if you choose to have discussions with an AI engine about a legal case, know that it may not be confidential and could all be disclosable.

Conclusion

AI “is a tool to be used with caution” and “has the potential to be wholly unreliable”. So said the judge in the Pinsent Masons case, and similar sentiments have been made on now frequent occasions by other parts of the judiciary in England and Wales and beyond.

While it can undoubtedly be a helpful research tool, AI is a relatively new and untested tool where the consequences of getting it wrong can be dramatic, as the examples in this article show. Even leaving aside potential contempts of court or regulatory actions for professionals, the practical effect of using AI wrongly will generally be to undermine confidence in a party’s case, or through disclosure of non-confidential AI prompts to give an opponent an easy cross-examination point.

Next time you consider asking an AI for the answer, or to draft something for you, ask yourself whether it’s worth the risk, and make sure you understand the inherent weaknesses of whatever it produces.