ChatGPT Is Passing the Tests Required for Medical Licenses and Business Degrees

ChatGPT, the viral chatbot that has raised concerns from teachers and academics over its ability to cheat on essays and exams, has now passed a Wharton MBA final exam, the United States Medical Licensing Exam, and components of the bar exam. 

A Wharton Professor conducted a study in which he used OpenAI’s GPT-3, the language model on which ChatGPT is built, to take a final exam of a core MBA course. He concluded that GPT-3 would have received a B to B- on the exam. 

The professor, Christian Terwiesch, found that GPT-3 performs the best “at basic operations management and process[ing] analysis questions.” For these, the chatbot provided both correct answers and excellent explanations as to why an answer was selected. In the paper’s summary, Tewriesch acknowledged that GPT-3 is by no means perfect. At times the bot made mistakes in simple mathematical calculations and wasn’t able to handle more advanced process analysis questions. 

This study further fueled the conversation that academics are now having as a result of ChatGPT’s advanced writing skills, regarding exam policies and what it means to utilize ChatGPT as a resource rather than a replacement for certain skills. 

Researchers have also tested ChatGPT on the United States Medical Licensing Exam (USMLE), which is a three-part exam that is required for medical licensure in the U.S. by all medical school graduates. They found that ChatGPT performed near or at the passing threshold for the three exams and demonstrated high levels of consistency and insight in its explanations. 

The researchers concluded that large language models like ChatGPT have the potential to assist with medical education and decision-making. The paper states that clinicians within a virtual clinic called Ansible Health have already begun to experiment with ChatGPT in assisting with writing tasks, such as composing appeal letters and simplifying complicated medical reports to help patients better understand their conditions. 

Furthering its range of expertise, ChatGPT scored a 50 percent accuracy rate on the multiple-choice component of the Bar Exam, or the Multistate Bar Examination (MBE). The Bar Exam is the test that law school graduates need to pass in order to officially practice law and is composed of three parts, with the MBE being the first. According to a study, GPT-3.5’s score of 50 percent is higher than the baseline guessing rate of 25 percent, which is the likelihood of a correct answer when randomly picking from four multiple-choice options. GPT-3.5 reached the average passing rate for Torts and Evidence, which are two of the seven subject areas. The researchers concluded that due to these results, a large language model such as GPT will be able to “pass the MBE component of the Bar Exam in the near future.”

Alex Tabarrok, a Professor of Economics at George Mason University, tested GPT competitor Claude AI on a GMU law and economics exam, announcing that the AI earned a “marginal pass” on the exam. 

As large language models such as ChatGPT and Claude AI surpass a number of humans in their ability to take high-level exams without years of studying and obtaining degrees, they are beginning to destabilize the measurements we use to determine academic success. 

While ChatGPT is unlikely to make higher education obsolete, many people are looking to safeguard against the uninhibited use of the tool. A 22-year-old college student named Edward Tian is currently developing an app that can detect ChatGPT-generated text. “Everyone deserves to reap the benefits of AI, so we need safeguards so that these new technologies are not abused,” Tian told Motherboard. 

Though ChatGPT is still in beta, the tool has been used by millions of people for a number of purposes, from negotiating internet bills to creating a virtual “wife.” On Monday, it was announced that Microsoft is making a $10 billion investment in OpenAI, the parent company behind ChatGPT and the text-to-image generator DALL-E. This is a significant increase in its initial $1 billion investment in the company made in 2019 and comes after the company announced that it will be laying off 10,000 employees last Wednesday.