Fighting disease with machine learning and statistics

The previous post about the potential for treating a subset of patients differently based on data mined from patient information and outcomes raises the issue of the paucity of such data today. Extracting patterns from large data sets is one of the things that machine learning is really good at -- if only we could get at the data needed to draw sophisticated conclusions from the silos it is currently hidden away in (or from the ether into which it vanishes, unrecorded). Truly remarkable things would result in terms of treatment if reliable statistics about outcomes, treatment methods, symptoms and best practices were kept. Even more could be done someday when DNA records are attached (as was used in the breast cancer research). Although little attention is paid, many are aware that mistakes could be avoided using these kinds of statistics, but there is even less awareness of the potential new treatment options that would arise -- as in the breast cancer study, where high risk patients with cancers much more likely to metastasize could be treated differently. Making this a national priority could have more impact on health than any comparable undertaking -- as evidenced by the incredible advances made in anesthesiology when accurate statistics were tracked. Those advances were primarily aimed at mistakes, not breakthroughs in treatment and, if nothing else, keeping these statistics would allow us to fairly evaluate hospitals, doctors and their differing techniques -- current raw metrics, like death rates, as is often noted, ignore the severity of cases. Machine learning techniques could take all recorded factors into consideration to estimate the impact of choosing a particular hospital or doctor. Today information about the benefits of small differences, such as the provision (or not) of hand sanitizers requires considerable investment (can't find the reference right now), but could be done statistically if data was routinely collected. For more information about mistakes in medicine, including the impact when anesthesiologists decided to put patient's health ahead of their own desire to avoid facing their mistakes, I suggest the entertaining and sometimes dramatic book Complications, by Atul Gawande