A report titled "Law Professors Prefer AI Over Colleagues' Answers," co-authored by Professor Julian Nyarko from Stanford University Law School, was published in early June. He and his colleagues conducted the study to determine if large language models could effectively assist in contract law courses.
The survey involved 16 professors from prestigious US law schools. They developed 40 typical contract law questions, similar to those students often ask after class or during private discussions with instructors.
The professors wrote their own answers and then evaluated responses, unaware if they were compiled by AI or their colleagues. The process was strictly controlled to ensure objectivity.
As a result, they scored AI higher than other colleagues in 75% of nearly 3,000 comparisons. The AI system also performed on par with the excellent instructors in the research group.
This discrepancy surprised the research team. According to Nyarko, these were not simple questions with readily available answers. Instead, they required respondents to synthesize complex material, apply it to real-world situations, and explain legal concepts to help students develop independent analytical skills.
He also stated that the research team chose the legal field because it demands judgmental thinking, sharp reasoning, and the ability to handle ambiguous situations, rather than merely memorizing facts.
Professor Sarath Sanga from Yale University Law School, a co-author of the study, said that in the legal field, two opposing arguments can both be valid and compelling, unlike many other fields with only one correct answer. They wanted to know if AI could meet the implicit professional standards lawyers use to evaluate each other's arguments. The answer is yes, according to this study.
The most surprising detail was that professors flagged only 3.5% of AI's answers as "pedagogically harmful," while this rate was 12% for human-written responses.
 |
A corner of the Stanford University Law School campus, US. Photo: School Fanpage |
A corner of the Stanford University Law School campus, US. Photo: School Fanpage
This discovery is predicted to fundamentally change legal education in the future, as many US law schools grapple with how to integrate AI tools into teaching. They are concerned about potential risks such as AI "hallucinating" (providing misleading but plausible information) and students becoming overly reliant on AI, leading to an erosion of critical thinking.
Professor Alejandro Salinas from Stanford University, one of the study's authors, believes that if evaluated by legal education experts, AI teaching assistants could provide students with high-quality, immediate support, complementing classroom instruction.
The capabilities of artificial intelligence are becoming increasingly challenging. In late May, many were surprised when OpenAI's internal AI successfully solved a unit distance problem on a plane that had existed since 1946, refuting the old solution.
Khanh Linh (According to Forbes, Stanford Law School)
