Increasingly, court filings are citing legal cases that don’t actually exist

Terry Gerton We’ve all been warned about AI hallucinations, but it’s showing up in legal cases. In fact, recently a federal judge criticized some Justice Department attorneys after one of their filings cited a non-existent case that appeared to be generated by AI. Why is this starting to show up more and more? What’s going on here?

Crawford Appleby Yeah, I think that the reason you see this so often is that AI is creating a strong presumption for people to cut corners. And as you’re noting, it can happen to anybody. It can happen government lawyers, it could happen to big firm lawyers, small firm lawyers. The reason is, is that AI quickly provides confident sounding, polished answers to even the most complex legal questions. And the words it writes are very impressive at first glance. And so when you combine that with how much pressure attorneys are under to produce quality work as fast as possible. Lawyers are sometimes going to fail to verify what AI gives them, and then you’re going to see fake cases ending up in court documents.

Terry Gerton This is starting to be a problem across the legal profession, private firms and government lawyers. We’re hearing about it in protest cases that GAO hears. In fact, they’re warning protesters about using hallucinated cases there. Why are experienced attorneys continuing to get caught by this problem?

Crawford Appleby I think because basically they don’t have systems in place to catch these situations. They’re not prepared and I think part of it also is a failure to understand what the technology is. Now, if you haven’t used AI before or if you’re unfamiliar with what it is exactly, it can appear like magic to you if you are going online, you just start having a conversation and it’s giving you information. But I think it’s important for any attorney who’s using AI in their practice to understand the technology, at least at a basic level. And so a lot of the technology we see up until now, the AI that we’re familiar with are known as large language models. And what that means is they are AI that was trained on an immense amount of data that was created by human beings, text and images and everything else. And what it’s designed to do is essentially guess what the next word in a sentence would be if it was written by a person. It’s not necessarily designed to be accurate. And that’s why you see it making up case law. It has a gap in its data. If you ask it a question and it doesn’t know the answer, oftentimes it will try to invent that answer because it’s trying to guess what the next words would be. And so it’ll create a fake case or it’ll reference a real case for something that it doesn’t’ say, or try to quote a real case for something it doesn’t say or even just basically misrepresent a real case and things that are said in a real-case.

Terry Gerton You mentioned at the beginning the pressure that attorneys are under just in terms of workload volume. And AI seems like it’s a, I don’t know, a research assistant, a buddy, a partner there that’s able to help reduce that workload. As AI becomes embedded in legal work though, are you seeing any assumptions about the system beginning to change in terms of responsibility or process flow or who’s got to check what?

Crawford Appleby Yeah, I think that the technology is improving and there are tools that are being added all the time. For example, many of the better-known AI tools are incorporating something called deep research, which is, I have found through my personal use with it, to be more accurate in terms of pulling up relevant case law when you’re doing legal research, and so that can be a useful tool. But again, I believe right now that the cure for AI hallucinations is human verification. So what every attorney needs to do is, if they are going to work with AI incorporated into their practice, they need to make sure that all the output they’re receiving, facts, law, anything at all, is gonna be verified every time they get something and they’re planning to use it.

Terry Gerton When you talk about human verification, are you seeing places, firms, practices where there’s a protocol that might be transferable that people could follow to say, oh, I’ve got to remember to check these sorts of things?

Crawford Appleby Yeah, I can’t speak to other firms generally about what their internal policies are. I have been to an AI conference recently in New York City where some firms were sharing how they dealt with AI and their knowledge management space. But I will say that at our firm, we have a policy about what AI tools are approved for use. We have firm accounts, and we have an AI policy that everyone’s familiar with. And so essentially, what I like to say about it is this, you know, it sounds counterintuitive. If someone says, oh, well, if AI is going to produce something for me that I can’t trust, then what’s the point in using it? I’m not saving any time. Well, the truth is, is that if it were to take you 30 minutes to write a paragraph for some legal document, and AI can produce something similar in a couple of minutes, well, how long is it going to take to verify all the law and facts in that paragraph? Maybe five, 10 minutes, right? If you’re familiar with the case, you’ll know what you’re doing. You just saved yourself 20 minutes. So the real good use of AI at the moment in law practice is a long game. It’s about shaving minutes off an hour, hours off a day, hours off week.

Terry Gerton Crawford Appleby is a partner at Wisner Baum. Crawford, do you think the trend here is that we rolled out these new tools, nobody was quite sure what they were gonna generate, we saw a lot of these hallucinations, people have been warned now, are you starting to see a return to normalcy here as people get into the verification habit?

Crawford Appleby I hope so. I don’t know if we’re seeing any less often hallucinated cases showing up as we see it in that database that’s available online. It’s over 1,800 at the moment. In terms of what’s been caught and what’s being noted, I’m sure there’s a fair amount of situations that have not been caught or noted, which actually is more frightening to me because they could end up in a court order if the court’s not catching them and no one else is. And so that’s an increased danger. But I’m hoping that folks will start to really accept verification and really start to understand the technology so that this will start to decrease and not increase.

Terry Gerton You mentioned the courts. Perhaps the thinking has been along these lines is like, well, if a lawyer makes a mistake in filing a case, at least the courts will catch it. Do the courts have that capacity? Can they be a reliable backstop here?

Crawford Appleby I would say that we all hope so, that the courts would be held to a higher standard than the attorneys. But I don’t think we can be 100% sure that that’s always going to happen. I think the courts and their staff, judges and their staff can make mistakes just like anybody else. No one’s immune to AI hallucinations.

Terry Gerton One of the other challenges here is that the AI, especially when it comes to legal matters, can be seen as such a time saver for small businesses who may not have a staff attorney and they might use AI to support some of their activities. They don’t have that check and balance capacity. Are there tools that you’d want them to be thinking about to make sure that as they use AI kind of as that staff augmentation, they’re not generating hallucinations?

Crawford Appleby So the first thing I would say is that I really would strongly recommend that small businesses and folks who are planning to use AI don’t try to use it for anything legal related. There’s two reasons for that. The first one is that it may give you very polished, confident answers that are 100% wrong. And there’s no way for you to know because you don’t have access to, say, legitimate legal database information like a Westlaw or a Lexus usually. That’s usually something a firm is going to have access and not a small business because it’s expensive. The other reason is that folks don’t realize that when they are having a conversation with AI and it’s about law, and if they were having that conversation with an attorney, that would be covered by the attorney-client privilege and it would be confidential. But if you are having that with an AI chatbot, everything, all the data that you’re giving and all the information you’re sharing, that’s not protected in any way. And we’ve already seen courts rule on that and tell folks that in some orders not long ago.

Terry Gerton As you think about this going forward, do you see the legal profession creating new standards around research drafting and case drafting to help address this issue of AI hallucination?

Crawford Appleby Yeah, I wouldn’t be surprised if you see some model ethical rules come out in the next year or two that are about AI use specifically. There’s already a duty of competency that attorneys are subject to that covers technology. But I think that this is a special thing that’s happening. And in fact, you’re already seeing courts issue what are called local rules, which means in my courtroom, this is how it goes, judges have a certain amount of leeway to do that. And they’re issuing their own AI rules where they, for example, ask attorneys to verify whether or not they’ve used AI in drafting documents, whether or not they verified what’s in their legal documents that they have used AI. You may even see some situations where judges might try to ban AI. But I think the problem with that, and this goes for law firms as well, is that when you try to band something that’s inevitable, you’re just going to drive it underground. And at that point, you are going to lose the ability to supervise and regulate something. Really does need to be supervised and regulated highly.

Terry Gerton So it sounds like in the near term, the responsibility resolves back to the submitting attorney to make sure that whatever is in that case is accurate and correct.

Crawford Appleby That’s correct. The attorneys, they’re subject to an ethical rule called the duty of candor to the tribunal. And essentially what that means is you can’t make things up that you tell the court, and that goes for facts and that goes for law as well. And so in some of the orders that you’ve seen with attorneys being, you know, getting in trouble with the court for filing fake cases and information, that’s been brought up as being violated in that situation. And so every time you file something, whether it’s with the Court at the federal or the state level, You’re going to be subject to that duty. In federal court, you could be subject to Rule 11 sanctions for making something up. And so there’s risk there.