Should You Use Generic AI for Legal? Probably Not.

TL;DR AI is, like everything else, a tool. Off-the-shelf AI systems have several pitfalls that make them less accurate at legal tasks, but they can still be used with limitations. Lawyers who want to use AI across their practice should really consider a legal-specific AI tool.

Many lawyers have began incorporating AI into their law practices, but many are still cautious for good reason. Asking Claude, ChatGPT, or Gemini about any moderately complex law task will frequently give false or not-quite-correct information, and hallucinations are rampant. Just one and a half weeks ago, four lawyers were penalized for using AI in court. Should you use these generic systems? Almost certainly not, and there's a lot of reasons.

Still Hallucinating

With over 1,000 documented court submissions of hallucinated material, this is what most lawyers are (and should be) worried about. A 2024 study by Stanford University found a 58-82% hallucination rate for general models, and although models have improved dramatically over the past two years, legal still sits as the category with the most hallucinations. Without proper safeguards in place, most will have no problem citing nonexistent cases or making assumptions.

One of the several primary issues is that legal information moves quickly, and most models have information cutoffs around 2024 or 2025. Further, legal documents are often longer and more complex than other sources of information, making them inherently difficult for models to learn from.

Eager to Please

Most generic AI is made for to be pleasant in conversation, and as such will avoid things that might make the user unhappy. It will almost always speak authoritatively, and will do its best to agree with the user no matter what; this is called sycophancy.

In the past, AI sycophancy has led to financial, medical, and self harm in pursuit of reinforcing what the user was saying. Systems without proper safeguards in place will go out of their way to agree with the user, often ignoring evidence or facts that state the contrary.

Lost in The Context

One of the most frequent uses for AI in law is for document processing. Whether it's reviewing large contracts, parsing through vast amounts of data, or comparing and contrasting, AI can save a lot of time. But don't trust it too much, especially for longer tasks.

We tested flagship AI models with industry-standard document processing (the kind that happens when you use, say, ChatGPT), and found a high variability in where data was and how well it was understood.

In over 300 questions across 25 M&A agreements, we found generic systems' accuracy dropped nearly 20% when questions pertained to information in the middle of the context versus the beginning or end. If you have a long chat history or more than 5 large documents, the quality of your answer could change dramatically just based on how far back the relevant information is.

So What Should You Use?

Generic AI systems are still great for basic, non-expertise-heavy tasks. Summarizing short documents, doing preliminary web searching, or bouncing ideas off the wall. Understanding the issues and limitations helps lower the risk of using it since you'll be able to notice patterns like sycophancy.

However, if you want to accelerate work across your firm and have more reliable legal AI, consider a legal-specific AI tool like Compact. Compact is built by people who understand the shortcomings of AI in law, and has designed custom systems and guardrails for the specific areas where it fails the most. If you want a custom tailored legal solution, we can do that too.