Capabilities

Microsoft's MAI-DxO Diagnosis Study: What the 83% Score Does and Does Not Show

Microsoft says its AI diagnostic system beat primary care doctors on hard cases. The comparison group, the cost data and the base model all matter.

Elena Vasquez

Former ML Researcher, Industry Analysis Lead

Published 7 min read
Medical professional reviewing brain scans with an elderly man in a clinic.
In this story 9 sections

Microsoft's MAI-DxO system was correct 83% of the time on complex New England Journal of Medicine cases, versus 36% for primary care doctors and general internists working alone with reference materials. The study is a strong benchmark result. It does not show that the system beats specialist teams, and much of the gain comes from the underlying models.

A big score on a hard test is not the same as a safer clinic. Evaluation methodology is where claims like this are won or lost, and Microsoft's latest study, published in Science, is a useful case. Emergent Wire covers the gap between published AI results and real-world performance, and this explainer is for readers who want to know what the numbers actually support.

What Is MAI-DxO and How Does It Work?

Doctors and nurses interacting in a hospital hallway, showcasing teamwork and professionalism.

MAI-DxO is an orchestrator, not a single model. Microsoft Research describes it as a model-agnostic system that simulates a panel of physicians, proposes differential diagnoses and chooses high-value, cost-effective tests. In practice, several AI agents question a patient, order tests and debate a diagnosis.

The test is built on the Sequential Diagnosis Benchmark. It converts 304 diagnostically challenging NEJM clinicopathological conference cases into stepwise encounters. The system starts with a short abstract and must ask a gatekeeper model for each finding, which reveals results only when asked.

That design matters. It is closer to real diagnosis than a multiple-choice quiz, because a wrong test order costs money in the scoring. The benchmark rewards asking the right question at the right time.

What Did the New Study Actually Find?

The headline numbers come from the newest paper. According to TechCrunch, MAI-DxO was correct 83% of the time on complex NEJM cases in simulation. Primary care doctors and general internists working alone with reference materials were correct 36% of the time.

The earlier 2025 version of the work reported 80% accuracy with OpenAI's o3 versus a 20% average for generalist physicians, so the new paper shows both stronger AI and a stronger human baseline. That is progress in the design of the test, not only in the model.

Mustafa Suleyman, Microsoft AI's chief executive, framed the work as a step toward what he calls "medical superintelligence." That is a goal statement, not a finding. The study measures accuracy on a defined set of cases.

Who Was the AI Compared Against?

Medical stethoscope and laptop on a white desk, symbolizing digital health solutions.

The comparison group is the first thing to check, and here it is narrow. The 36% figure comes from primary care doctors and general internists. TechCrunch notes the comparison excluded specialists and physician teams, who usually handle cases this complex.

That is not a flaw by itself. It is a limit on what the result means. Saying the system beats generalists working alone on hard puzzle cases is accurate. Saying it beats doctors is broader than the evidence.

The doctors were also working in an artificial setting. In real care they can examine a patient, watch how symptoms change over days and consult colleagues. A benchmark that strips those advantages out measures a narrower skill.

How Much of the Gain Comes From the Base Model?

Most of it. The study's own results show how much the underlying model matters. According to TechCrunch, Microsoft's best results used OpenAI's GPT-5 and GPT-5.5, although it tested models from Google, Anthropic, Meta and DeepSeek too.

GPT-5 alone scored 78.9% on complex cases at about $8,400 per case. MAI-DxO cut the cost by about $3,000 but raised accuracy by only about four points. With GPT-5.5, MAI-DxO raised complex-case accuracy from 72% to nearly 87% at a slightly lower cost.

The weakest model shows the pattern clearly. With Gemini 2.5 Flash, the orchestrator lifted accuracy from 46.4% to nearly 69%. The gap between weaker and stronger models was still larger than the boost from MAI-DxO.

The table below summarizes the results as reported.

How Much of the Gain Comes From the Base Model?
SetupComplex-case accuracyNotes
GPT-5 alone78.9%About $8,400 per case
MAI-DxO with GPT-5About 83%Cost cut by about $3,000
GPT-5.5 alone72%Baseline
MAI-DxO with GPT-5.5Nearly 87%Slightly lower cost
Gemini 2.5 Flash alone46.4%Weaker baseline
MAI-DxO with Gemini 2.5 FlashNearly 69%Orchestration helps most here

The practical reading: the orchestrator helps most when the base model is weak, and adds less as base models improve. That is a common pattern in AI systems, and it is worth knowing before crediting a wrapper with the whole score.

Why Did Emergency Cases Show Little Benefit?

Professional meeting in a healthcare office setting with medical staff discussion.

Emergency cases are a different kind of problem. TechCrunch reports that on emergency department cases, MAI-DxO added little accuracy or cost savings for either model, and both GPT-5.5 setups scored about 85%.

That fits how emergency medicine works. Cases are often more common, the signals are clearer and the decisions are faster. A panel of agents debating a diagnosis helps most when the puzzle is rare and ambiguous.

It is also a useful reminder that a benchmark average hides variation. A system can shine on one type of case and add nothing on another. Any claim about medical AI should say which case type it was tested on.

What Do Benchmark Results Like This Leave Out?

Benchmarks leave out the messy parts of care. They use written cases with clean data, while real patients give incomplete histories, change their story and arrive with several conditions at once. Our explainer on what AI benchmark scores actually mean covers the general problem.

They also leave out error types. An AI system can be right 83% of the time and still make a dangerous mistake in the other 17%. The cost of a wrong answer depends on what the answer is. Our guide to AI hallucinations and why models make things up explains why confident wrong answers are the hardest to catch.

Finally, they leave out workflow. A doctor using the system still has to decide whether to trust it, and the system still has to fit into a busy clinic. Orchestrated agents are also harder to audit, which is a point our look at AI agents that use tools raises more generally.

What Happens Next for MAI-DxO?

View of University Hospitals building in Cleveland, OH against a clear sky.

The next step is the one that matters most: a real-world test. TechCrunch reports that Microsoft will study MAI-DxO with Beth Israel Deaconess Medical Center in Boston, comparing its recommendations with those of human doctors. Microsoft is also working with the Mayo Clinic on a healthcare foundation model planned for release next year.

The 2025 research page lists the tool as a research system and says Microsoft plans clinical testing for safety and accuracy. Neither the 2025 nor the 2026 coverage describes the tool as available for patient care.

A well-designed clinical study would compare the system against specialists, track errors and measure outcomes, not only accuracy on written cases. Until then, the 83% figure is a promising signal, not proof of benefit.

There is a second open question about cost. The study reports dollar figures per case, such as about $8,400 for GPT-5 alone on complex cases. Those are simulated costs of tests and visits, not hospital bills. Real savings would depend on how a clinic orders tests today and whether doctors would follow the system's suggestions in practice.

Regulators will also have a say before any tool reaches patients. A system that suggests diagnoses and tests affects care decisions, so it will face scrutiny over safety, bias across patient groups and accountability when it is wrong. Those questions rarely appear in a benchmark paper, yet they decide whether a research result ever becomes a product in a hospital.

How Should You Read Claims About Medical AI?

Use a short checklist before you accept any headline number.

  1. Check who the comparison group was and whether specialists or teams were included.
  2. Ask whether the test used real patients or written cases.
  3. Look at how much comes from the base model versus the added system.
  4. Look for results by case type, not only an average.
  5. Check whether anyone outside the company has replicated the result.

None of this is cynicism. A tool that helps a general practitioner reason through a hard case could be valuable. The point is to match the claim to the evidence, which is the standard we apply to every model announcement at Emergent Wire.

The Bottom Line on MAI-DxO

MAI-DxO is a serious piece of research, and the 83% result is strong. It also compares the AI with a narrow group of doctors in a simulated setting, and most of the improvement traces back to the base model. The honest takeaway is progress on a benchmark, with clinical proof still to come.

Our editors will keep following the Beth Israel study and any outside replications. Your next step, if you work in health or buy AI tools, is to ask any vendor for results by case type and by comparison group, not just a single headline accuracy.

What is Microsoft MAI-DxO?
MAI-DxO is a Microsoft system that coordinates several AI agents acting like a panel of doctors. They question a patient, order tests and debate a diagnosis. It works with different underlying AI models and is evaluated on a benchmark built from complex New England Journal of Medicine cases. It is a research system, not a clinical product.
Did MAI-DxO really beat doctors at diagnosis?
Only in a narrow comparison. In simulation it scored 83% on complex cases versus 36% for primary care doctors and general internists working alone with reference materials. Specialists and physician teams were excluded, so the result does not show it outperforms the doctors who usually handle such cases.
How much does the base model matter in MAI-DxO results?
It matters a great deal. Microsoft's best results used OpenAI's GPT-5 and GPT-5.5, and the gap between stronger and weaker models was larger than MAI-DxO's boost. With a weak model it added about 22 points, but with a strong one it added only a few.
Is MAI-DxO available for patients to use?
No. Microsoft describes it as research and plans a real-world study with Beth Israel Deaconess Medical Center in Boston, comparing its recommendations with human doctors. Clinical testing for safety and accuracy is still ahead, so no coverage describes the system as ready for patient care today.

Written by

Elena Vasquez

Former ML Researcher, Industry Analysis Lead

9+ years experience

Elena Vasquez holds a PhD in computer science with a research focus on model evaluation methodology, and spent seven years split between an academic lab and a mid-size AI startup's research team before moving into journalism full time. She joined Emergent Wire to bring a practitioner's skepticism to industry coverage — she has built and broken enough benchmarks herself to recognize when a vendor's claimed number is doing more marketing work than measurement work. Her beat covers model capability claims, evaluation methodology, and the gap between published benchmarks and production performance.

Credentials

  • PhD, Computer Science (Machine Learning)
  • Former research scientist, applied AI startup (4 years)
  • Published research on model evaluation methodology