The report, published March 4, was on a study that involved 127 law school students at the University of Minnesota and the University of Michigan. It was conducted to test two emerging software innovations related to artificial intelligence: retrieval augmented generation, or RAG, which grounds analysis in legal sources, and reasoning models, which seemingly analyze data more thoroughly than previous chatbots.
"These findings suggest that integrating domain-specific RAG capabilities with reasoning models could yield synergistic improvements, shaping the next generation of AI-powered legal tools and the future of lawyering more generally," the study argued.
The 89-page report, posted to the Social Science Research Network, was authored by Daniel Schwarcz and David R. Cleveland of the University of Minnesota Law School; J.J. Prescott and Patrick Barry of the University of Michigan Law School; Sam Manning from the nonprofit Centre for the Governance of AI; and Beverly Rich, practice innovation counsel at the law firm Ogletree Deakins.
"This new study provides the first empirical evidence, to our knowledge, that AI tools can consistently and significantly enhance the quality of human lawyers' work across various realistic legal assignments," the paper said.
Legal stakeholders at schools and firms have conducted experiments with AI in the past, with varying results. A Suffolk University professor earlier this year posted an SSRN paper primarily produced by OpenAI's ChatGPT, which said it would redefine legal scholarship, while Magic Circle firm Linklaters LLP released a report last month recommending that users of OpenAI o1 and Google Gemini 2.0 avoid using the programs in English law advice without human supervision.
A Law360 Pulse survey, also published March 4, found that more attorneys seem to be using generative AI tools and view them positively compared with last year, but lawyers are still concerned about legal ethics and client confidentiality when it comes to the technology.
The Minnesota-Michigan study included 127 participants recruited in September among more than 250 second-year, third-year and master of law students, with the experiment taking place throughout October. The students were randomly assigned six lawyering assignments, with each participant completing two with no AI software, two with OpenAI o1 and two with Vincent AI. The students were trained on the use of AI models beforehand, and the six assignments, which reflected typical work for a first- or second-year associate, were developed by at least one co-author and a practice attorney. Grading occurred blindly with lawyers who were involved in data collection or analysis.
The tasks were to draft an email for a client, a legal memo for a partner, a nondisclosure agreement, a motion to consolidate and a persuasive letter addressing the enforceability of a covenant not to compete, as well as to analyze a complaint and draft a written analysis.
"Our most significant finding is that access to both o1-preview and Vincent AI led to statistically significant and meaningful improvements in overall quality of work across four of the six assignments tested — with [OpenAI o1] producing larger and more statistically significant gains than Vincent AI."
For Vincent AI, graders observed improvements in clarity, organization and professionalism, though the tool's impact on accuracy was mixed. Overall accuracy scores did not improve much in any assignment where participants used the software, and even reduced the accuracy of submitted work in one particular task. However, Vincent AI produced far fewer made-up facts or "hallucinations" (three) than OpenAI o1 (11), and one fewer than work completed without the programs (four).
For OpenAI o1, the study found stronger and more widespread improvements in the quality of legal work compared to Vincent AI. The team also observed "statistically significant and substantial improvements in the quality of the legal analysis contained in three of the six assignments tested."
Both tools improved the quality of three common assignments, according to the study: the legal memo, complaint analysis and motion to consolidate. OpenAI o1 improved the submitted work for the persuasive letter, while Vincent AI improved the writing of a client email. Overall, participants completed tasks more quickly using the two AI models, though neither program consistently led to statistically significant improvements in the accuracy of the work.
"Each AI system appears to enhance legal work through distinct mechanisms, which can be and already are being combined with one another in updated legal technology tools," the report said. "Although this integration may result in additive benefits, it may also produce multiplicative benefits."
The researchers also noted that among the participants who answered a post-experiment questionnaire, most felt their experience in the study increased their likelihood of using similar tools in the future.
"Interestingly, participants perceived [OpenAI o1] as more effective for improving speed and Vincent AI as more helpful for enhancing quality," the study said. "These subjective impressions diverge somewhat from the actual results."
AI remained a hot topic throughout 2024 and continued to be front of mind for law firm leaders, legal tech companies, general counsel, legal ethics professors and professional conduct leaders heading into the new year.
And the technology has secured a spot in some law schools. Institutions like Washington University in St. Louis and Arizona State University have added AI courses to their curricula, while professors have hypothesized about using ChatGPT to quiz students more effectively.
Last year, legal technology companies raised about $4.98 billion in funding, with AI being a significant driving force behind the new capital.
--Editing by Robert Rudinger.
Social Science Research Network is owned by Elsevier, a division of RELX Group, which also owns Law360.
For a reprint of this article, please contact reprints@law360.com.
