Microsoft diagnostic AI outscored generalist doctors on complex test cases
Microsoft says its multi-agent diagnostic system identified more complex case diagnoses than individual generalist doctors in a simulation, but hospital testing is still ahead.
Microsoft's Diagnostic Orchestrator correctly diagnosed about 83% of complex medical cases in a retrospective test reported on October 22, compared with 36% for individual generalist physicians using reference materials. The result, described in a study published in Science, offers a benchmark for diagnostic AI, but its performance with patients in clinical care remains unknown.
The cases came from reports published in The New England Journal of Medicine. Microsoft built a benchmark called SDBench from the cases and plans to make it open source, The Verge reported. The company designed the system, also called MAI-DxO, to coordinate several AI agents that ask questions, consider tests and work toward a diagnosis in a process modeled on a panel of doctors.
How the doctor comparison was made
The physicians in the comparison were primary care doctors and general internists working individually, TechCrunch reported. It noted that cases this complex would ordinarily be handled by specialists or physician teams, neither of which was included in the comparison. The 83% result therefore describes performance against a particular group of doctors under test conditions, rather than a comparison with the teams that would usually manage such cases.
Microsoft also measured the estimated cost of running the AI on each case, according to TechCrunch. In one comparison using GPT-5, the orchestrated system cost about $3,000 less per complex case than a short, basic prompt to GPT-5 alone, while improving accuracy by about four percentage points. Those figures describe the test setup and estimated AI costs, not the cost of treating a patient.
Hospital evaluation is planned
Mustafa Suleyman, Microsoft AI's chief executive, told The Verge the system needs clinical validation. The Verge reported that Microsoft plans to start testing it in hospital settings with Beth Israel Deaconess Medical Center in Boston. Dr. Dominic King, Microsoft's AI health lead and a study coauthor, said the work would likely begin in the next couple of months, but neither he nor Suleyman gave a firm date.
TechCrunch described the Beth Israel project as an evaluation of the system's clinical reasoning across different hospital settings. Suleyman also told TechCrunch that Microsoft is working with Mayo Clinic on a separate healthcare foundation model, which the company plans to release next year. That model is distinct from the diagnostic orchestrator tested on the published cases.
Read in depth: AI
NorForge finds AI agents used in campaign targeting more than 1,200 websites
Researchers observed exposed Cairn panels used in attempted intrusions, while Reuters identified companies affected by one campaign. The extent of any data theft remains unclear.
YouTube Music starts labeling songs made with generative AI
Listeners can tap a label on the player page to see how AI contributed to a track. YouTube says it will use partner disclosures and its own detection tools.
Sources and context
- Microsoft says its AI tool is better than doctors at diagnosing diseasesThe Verge
- Microsoft’s Mustafa Suleyman says its AI outperforms doctors on hard cases as he chases ‘medical superintelligence’TechCrunch
AI-assisted article checked against the listed sources. NewsJaws did not conduct interviews or attend the reported events.