Microsoft Says Its AI Beats Doctors. Here Is What The Science Paper Actually Measured

🔍 Read the full analysis: Microsoft Says Its AI Beats Doctors. Here Is What The Science Paper Actually Measured on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft reported that its MAI-DxO diagnostic system scored 83.2% on 304 complex medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, in a peer-reviewed paper published in Science on October 22. The results concern a text-based simulation using historical cases, not diagnosis or treatment of real patients, so they do not establish that the system outperforms doctors in clinical practice.

The paper, titled Sequential diagnosis with language models, describes a benchmark in which each case begins with a brief summary. A participant must request additional information, such as test results, from a simulated gatekeeper that holds the fuller case record. The benchmark assesses both the final diagnosis and the tests requested along the way. Microsoft also reported scores of 80.3% for MAI-DxO and 46.8% for physicians on a separate set of 325 emergency-department cases.

The physician comparison was with generalists working individually, rather than specialist teams. In the earlier preprint’s account of the study, 21 U.S. and U.K. primary-care doctors and hospital generalists took part; they could not use search engines or language models. The final paper adds an open-book physician result of 36.1% on the complex cases, while the score without reference materials was 22.3%. Microsoft’s published results are its own study findings; publication after peer review does not mean outside researchers have independently reproduced them.

The paper also reports that GPT-5 alone scored 78.9% on the complex cases, close to MAI-DxO’s 83.2%. Microsoft’s system organizes a language model into a panel of virtual physicians that propose diagnoses, request further information and critique each other. The company says this orchestration improved performance and reduced estimated testing costs. The cost figures are modelled estimates, not bills from real patients: the preprint describes mapping tests to billing codes and using a U.S. hospital price list, alongside a fixed physician-visit cost.

Why the Test Setting Matters

The results are relevant because diagnosis often involves deciding which question or test to pursue next, not merely selecting an answer after all the facts are supplied. MAI-DxO’s benchmark attempts to measure that sequence as well as the final diagnosis, making it different from a conventional medical question-and-answer test.

But a high score on selected difficult cases is not the same as safe performance in a clinic or emergency department. The cases were historical and text-only, with no physical examination, real-time imaging or direct interaction with patients. The study also compared a coordinated AI system with individual generalists, rather than the specialists or teams who may handle unusually difficult cases in practice.

The case mix limits what the results can show about harms. The source material says the NEJM cases did not include healthy patients or harmless explanations, and the authors noted that false alarms could not be measured in that set. A system that tests aggressively for rare conditions may perform well on a collection focused on difficult diagnoses; this study does not establish how often it might recommend unnecessary tests or raise false alarms in a broader patient population.

From Preprint to Science Paper

An earlier version appeared on arXiv in June 2025 as a non-peer-reviewed preprint. The source material says that version reported an 80% AI score against 20% for doctors, using a prior model. The Science paper, published October 22, 2026, reports updated results and includes an open-book physician comparison. The paper was received in April and accepted in August, according to the supplied account.

The authors are listed as Microsoft-affiliated, including Microsoft AI chief executive Mustafa Suleyman. Microsoft’s blog promoted the findings using comparisons such as “more than twice” the accuracy of physicians with reference materials on complex cases. Those descriptions refer to the particular benchmark and comparison groups in the paper; they should not be read as a general finding that the system is twice as effective as doctors in everyday care.

Some methodological details cited in the source material come from the preprint because the full Science text is behind a paywall. Those details include the model-based scoring of diagnoses, the generation of synthetic results when a requested test was absent from a case record, and the estimated test-pricing method. They may differ from the final paper, so they should not be assumed to describe every detail of the published version.

“The company described MAI-DxO as “1.7 times as accurate” as physicians on emergency-department cases and “more than twice” as accurate on complex cases when physicians could use reference materials.”

— Microsoft, in its blog post, “Building Medical Superintelligence”

Clinical Performance Still Untested

The study does not show how MAI-DxO performs with real patients, live clinical workflows or physical examinations. It also does not establish whether the system’s proposed diagnoses and testing decisions would improve outcomes, save money or avoid delays when used alongside clinicians. The supplied material gives no results from independent replication, and does not establish whether the final paper retained every methodological detail reported in the preprint.

Other open questions include how the system handles common, less complex complaints; how often it recommends unnecessary tests; and how its performance compares with specialists or multidisciplinary teams. The benchmark’s estimated costs are not evidence of actual savings, and the reported diagnostic scores do not by themselves measure patient safety. Those questions require evaluation in settings that reflect ordinary care, with outcomes and potential harms tracked.

Patient Study Planned in Boston

The source material says Microsoft plans a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their usual care. That design could provide evidence about how MAI-DxO works in a clinical setting, but no results are available in the supplied material.

Key details remain to be reported, including when the study will begin, how many patients will take part, what conditions it will cover and how researchers will measure safety and diagnostic accuracy. Until those findings are published, the Science results should be understood as a benchmark comparison rather than proof that MAI-DxO can replace or outperform physicians in patient care.

Key Questions

Did Microsoft show that its AI is better than doctors in hospitals?

No. The paper reports higher scores for MAI-DxO on simulated historical case records. It did not test whether the system provides better care or outcomes for real patients in hospitals.

What were the main diagnostic scores?

Microsoft reported 83.2% for MAI-DxO and 36.1% for generalist physicians using textbooks and the internet on 304 complex cases. Without those references, physicians scored 22.3%. On 325 emergency-department cases, the reported scores were 80.3% and 46.8%, respectively.

Who were the doctors in the comparison?

The comparison was with generalist physicians working alone, not specialist teams. The supplied account of the preprint describes 21 U.S. and U.K. primary-care doctors and hospital generalists.

Has the system been tested with patients?

Not in the results described here. The source says a prospective study at Beth Israel Deaconess Medical Center is planned, with consenting patients interacting with the system alongside their usual care.

Source: ThorstenMeyerAI.com