Microsoft's MAI-DxO Diagnosis Study: What the 83% Score Does and Does Not Show
Microsoft says its AI diagnostic system beat primary care doctors on hard cases. The comparison group, the cost data and the base model all matter.
In this story 9 sections
Microsoft's MAI-DxO system was correct 83% of the time on complex New England Journal of Medicine cases, versus 36% for primary care doctors and general internists working alone with reference materials. The study is a strong benchmark result. It does not show that the system beats specialist teams, and much of the gain comes from the underlying models.
A big score on a hard test is not the same as a safer clinic. Evaluation methodology is where claims like this are won or lost, and Microsoft's latest study, published in Science, is a useful case. Emergent Wire covers the gap between published AI results and real-world performance, and this explainer is for readers who want to know what the numbers actually support.
What Is MAI-DxO and How Does It Work?
MAI-DxO is an orchestrator, not a single model. Microsoft Research describes it as a model-agnostic system that simulates a panel of physicians, proposes differential diagnoses and chooses high-value, cost-effective tests. In practice, several AI agents question a patient, order tests and debate a diagnosis.
The test is built on the Sequential Diagnosis Benchmark. It converts 304 diagnostically challenging NEJM clinicopathological conference cases into stepwise encounters. The system starts with a short abstract and must ask a gatekeeper model for each finding, which reveals results only when asked.
That design matters. It is closer to real diagnosis than a multiple-choice quiz, because a wrong test order costs money in the scoring. The benchmark rewards asking the right question at the right time.
What Did the New Study Actually Find?
The headline numbers come from the newest paper. According to TechCrunch, MAI-DxO was correct 83% of the time on complex NEJM cases in simulation. Primary care doctors and general internists working alone with reference materials were correct 36% of the time.
The earlier 2025 version of the work reported 80% accuracy with OpenAI's o3 versus a 20% average for generalist physicians, so the new paper shows both stronger AI and a stronger human baseline. That is progress in the design of the test, not only in the model.
Mustafa Suleyman, Microsoft AI's chief executive, framed the work as a step toward what he calls "medical superintelligence." That is a goal statement, not a finding. The study measures accuracy on a defined set of cases.
Who Was the AI Compared Against?
The comparison group is the first thing to check, and here it is narrow. The 36% figure comes from primary care doctors and general internists. TechCrunch notes the comparison excluded specialists and physician teams, who usually handle cases this complex.
That is not a flaw by itself. It is a limit on what the result means. Saying the system beats generalists working alone on hard puzzle cases is accurate. Saying it beats doctors is broader than the evidence.
The doctors were also working in an artificial setting. In real care they can examine a patient, watch how symptoms change over days and consult colleagues. A benchmark that strips those advantages out measures a narrower skill.
How Much of the Gain Comes From the Base Model?
Most of it. The study's own results show how much the underlying model matters. According to TechCrunch, Microsoft's best results used OpenAI's GPT-5 and GPT-5.5, although it tested models from Google, Anthropic, Meta and DeepSeek too.
GPT-5 alone scored 78.9% on complex cases at about $8,400 per case. MAI-DxO cut the cost by about $3,000 but raised accuracy by only about four points. With GPT-5.5, MAI-DxO raised complex-case accuracy from 72% to nearly 87% at a slightly lower cost.
The weakest model shows the pattern clearly. With Gemini 2.5 Flash, the orchestrator lifted accuracy from 46.4% to nearly 69%. The gap between weaker and stronger models was still larger than the boost from MAI-DxO.
The table below summarizes the results as reported.
| Setup | Complex-case accuracy | Notes |
|---|---|---|
| GPT-5 alone | 78.9% | About $8,400 per case |
| MAI-DxO with GPT-5 | About 83% | Cost cut by about $3,000 |
| GPT-5.5 alone | 72% | Baseline |
| MAI-DxO with GPT-5.5 | Nearly 87% | Slightly lower cost |
| Gemini 2.5 Flash alone | 46.4% | Weaker baseline |
| MAI-DxO with Gemini 2.5 Flash | Nearly 69% | Orchestration helps most here |
The practical reading: the orchestrator helps most when the base model is weak, and adds less as base models improve. That is a common pattern in AI systems, and it is worth knowing before crediting a wrapper with the whole score.
Why Did Emergency Cases Show Little Benefit?
Emergency cases are a different kind of problem. TechCrunch reports that on emergency department cases, MAI-DxO added little accuracy or cost savings for either model, and both GPT-5.5 setups scored about 85%.
That fits how emergency medicine works. Cases are often more common, the signals are clearer and the decisions are faster. A panel of agents debating a diagnosis helps most when the puzzle is rare and ambiguous.
It is also a useful reminder that a benchmark average hides variation. A system can shine on one type of case and add nothing on another. Any claim about medical AI should say which case type it was tested on.
What Do Benchmark Results Like This Leave Out?
Benchmarks leave out the messy parts of care. They use written cases with clean data, while real patients give incomplete histories, change their story and arrive with several conditions at once. Our explainer on what AI benchmark scores actually mean covers the general problem.
They also leave out error types. An AI system can be right 83% of the time and still make a dangerous mistake in the other 17%. The cost of a wrong answer depends on what the answer is. Our guide to AI hallucinations and why models make things up explains why confident wrong answers are the hardest to catch.
Finally, they leave out workflow. A doctor using the system still has to decide whether to trust it, and the system still has to fit into a busy clinic. Orchestrated agents are also harder to audit, which is a point our look at AI agents that use tools raises more generally.
What Happens Next for MAI-DxO?
The next step is the one that matters most: a real-world test. TechCrunch reports that Microsoft will study MAI-DxO with Beth Israel Deaconess Medical Center in Boston, comparing its recommendations with those of human doctors. Microsoft is also working with the Mayo Clinic on a healthcare foundation model planned for release next year.
The 2025 research page lists the tool as a research system and says Microsoft plans clinical testing for safety and accuracy. Neither the 2025 nor the 2026 coverage describes the tool as available for patient care.
A well-designed clinical study would compare the system against specialists, track errors and measure outcomes, not only accuracy on written cases. Until then, the 83% figure is a promising signal, not proof of benefit.
There is a second open question about cost. The study reports dollar figures per case, such as about $8,400 for GPT-5 alone on complex cases. Those are simulated costs of tests and visits, not hospital bills. Real savings would depend on how a clinic orders tests today and whether doctors would follow the system's suggestions in practice.
Regulators will also have a say before any tool reaches patients. A system that suggests diagnoses and tests affects care decisions, so it will face scrutiny over safety, bias across patient groups and accountability when it is wrong. Those questions rarely appear in a benchmark paper, yet they decide whether a research result ever becomes a product in a hospital.
How Should You Read Claims About Medical AI?
Use a short checklist before you accept any headline number.
- Check who the comparison group was and whether specialists or teams were included.
- Ask whether the test used real patients or written cases.
- Look at how much comes from the base model versus the added system.
- Look for results by case type, not only an average.
- Check whether anyone outside the company has replicated the result.
None of this is cynicism. A tool that helps a general practitioner reason through a hard case could be valuable. The point is to match the claim to the evidence, which is the standard we apply to every model announcement at Emergent Wire.
The Bottom Line on MAI-DxO
MAI-DxO is a serious piece of research, and the 83% result is strong. It also compares the AI with a narrow group of doctors in a simulated setting, and most of the improvement traces back to the base model. The honest takeaway is progress on a benchmark, with clinical proof still to come.
Our editors will keep following the Beth Israel study and any outside replications. Your next step, if you work in health or buy AI tools, is to ask any vendor for results by case type and by comparison group, not just a single headline accuracy.