Three Principles for Collaborative Benchmarking Challenges

By Jiaxin Zheng and James Eddy 

At Sage Bionetworks, we strive to speed the translation of science to medicine by embracing open practices. 

Benchmarking reliable methods is one of the ways we deliver on that mission. Through our work with DREAM Challenges, we’ve pioneered the development of infrastructure and tools to objectively evaluate algorithms across a broad spectrum of biomedical domains, including bioinformatics, biomedical informatics, and predictive modeling of clinical outcomes. With algorithms playing an increasing role in biomedical analysis, crowd-sourced perspectives can shape more objective method evaluation and mitigate the self-assessment bias.

As the Challenge platform provider for the RSNA-ASNR-MICCAI Brats 2021 Segmentation Task, here are some of the principles our technology embraces to empower the benchmarking ecosystem. 

Collaboration: Synapse, Sage’s open-source research platform that allows teams to share data and track analyses, provided a centralized workspace for Challenge participants to collaborate. In addition to being able to access data, participants could post questions, find potential teammates, and submit models. From the wiki page to the evaluation workflow, we partnered with organizers to customize the space to meet their needs.

 

Portability: Sage developed the model-to-data approach where containerized algorithms rather than predictions are submitted for assessment on hidden data. These containers will be made available after the Challenge, promoting scientific reproducibility and reusability for the broader BraTS community. The model-to-data approach also eliminates the requirement for direct dissemination of validation data, reducing data transfer costs and enhancing security for Challenge organizers.

 

Innovation: To best serve the changes in the dynamic imaging space, we have augmented our infrastructure to integrate graphical processing unit (GPU) capabilities. These efforts enable easier exploration of large complex datasets and quicker model training, streamlining both development and evaluation. Our new GPU capability will be used for future imaging data inference competitions, and help stimulate algorithm development at the cutting edge of image-based learning. 

The BraTS community combined with the RSNA, ASNR, and MICCAI research networks has led to an impressive global turnout of Challenge participants, with more than 1,200 submissions from five continents. But this is only the beginning. Future phases of the BraTS Challenge will provide a sustainable cloud-based platform for open and continuous benchmarking of image analysis tools. We also plan to integrate with the DREAM Challenges community of solvers, and include genomics data in addition to images and new challenge tasks to address questions related to both.

Task 1 of the BraTS-RSNA-ASNR-MICCAI 2021 Challenge is the result of a collaboration by Sage Bionetworks, Perelman School of Medicine at the University of Pennsylvania, Radiological Society of North America, American Society of Neuroradiology, the Medical Image Computing and Computer Assisted Intervention Society, and sponsorship by Intel, RSNA, and Neosoma. We look forward to continued dialogue on how we can guide future algorithm development in order to best serve the broader biomedical community. 

The Michael J. Fox Foundation and Sage Bionetworks Announce Winners of the $25,000 BEAT-PD DREAM Challenge

NEW YORK (June 17, 2020) – The Michael J. Fox Foundation for Parkinson’s Research (MJFF) and Sage Bionetworks announce the winners of the BEAT-PD DREAM Challenge. BEAT-PD (Biomarker and Endpoint Assessment to Track Parkinson’s Disease) is a data challenge designed to benchmark new methods to predict Parkinson’s disease severity in patients in their homes. MJFF and Sage partnered with Evidation Health, Northwestern University, Radboud University Medical Center, and the BRAIN Commons to host the BEAT-PD Challenge.

Forty-three teams participated in the Challenge with data hosted by the BRAIN Commons. The teams received access to raw sensor (accelerometer and gyroscope) time-series data, which they used to predict individual medication state and symptom severity. Winners from the Challenge share a $25,000 prize.

The winners of the BEAT-PD Challenge are:

  • dbmi (Yidi Huang, Brett Beaulieu-Jones, Mark Keller, Mohammed Saqib) from Harvard Medical School, Department of Biomedical Informatics
  • ROC BEATPD (Alex Page, Monica Javidnia, Greta Smith, Robbie Zielinski, and Charles Venuto) from the University of Rochester Medical Center
  • Yuanfang Guan from University of Michigan
  • HaProzdor (Ayala Matzner, Yuval El-Hanany, Izhar Bar-Gad) from the Gonda Brain Research Center at Bar Ilan University

“We congratulate all the winners. The Foundation has supported research into sensors and other digital tools for Parkinson’s for many years,” says Mark Frasier, PhD, Senior Vice President, Research Programs at MJFF. “The BEAT-PD projects are unlocking the potential of data collected by digital devices to help people with Parkinson’s, their physicians, and researchers. Now, more than ever, we understand the critical importance of remote monitoring for the safe and effective delivery of healthcare and the progress of clinical research.”

In a previous data challenge, teams proved that disease status and symptom severity could be predicted using data collected during the completion of specific tasks while monitored by a physician. The BEAT-PD Challenge built on this to determine whether disease severity can be assessed from passive sensor data from consumer electronics, collected during daily life, not pre-set tasks, which will bring us closer to the promise of at-home monitoring of disease progression.

Three of the teams (dbmi, ROC BEATPD and HaProzdor) approached the problem by applying signal processing methods to the smartwatch and smartphone sensor data, the results of which were then used in machine learning models which allowed for patient-specific characteristics. The fourth solution, by Yuanfang Guan, applied a deep-learning model incorporating spatial and temporal data augmentation of the sensor data.

BEAT-PD data used in the DREAM Challenge is available on the BRAIN Commons. For more information and to apply for access to these data, please visit: https://www.braincommons.org/beat-pd-data-release/.

The winning teams have been invited to collaborate to improve upon their individual models, as well as to test them against clinician-validated symptom severity ratings and to co-author a manuscript based on their findings.

Learn more about the BEAT-PD Challenge: www.synapse.org/beatpdchallenge

Contacts: 

Kristina Magana
The Michael J. Fox Foundation
kmagana@michaeljfox.org

Hsiao-Ching Chou
Sage Bionetworks
chou@sagebionetworks.org


ABOUT THE MICHAEL J. FOX FOUNDATION

As the world’s largest nonprofit funder of Parkinson’s research, The Michael J. Fox Foundation is dedicated to accelerating a cure for Parkinson’s disease and improved therapies for those living with the condition today. The Foundation pursues its goals through an aggressively funded, highly targeted research program coupled with active global engagement of scientists, Parkinson’s patients, business leaders, clinical trial participants, donors and volunteers. In addition to funding more than $900 million in research to date, the Foundation has fundamentally altered the trajectory of progress toward a cure. Operating at the hub of worldwide Parkinson’s research, the Foundation forges groundbreaking collaborations with industry leaders, academic scientists and government research funders; increases the flow of participants into Parkinson’s disease clinical trials with its online tool, Fox Trial Finder; promotes Parkinson’s awareness through high-profile advocacy, events and outreach; and coordinates the grassroots involvement of thousands of Team Fox members around the world. For more information, visit us at michaeljfox.org, on Facebook or Twitter.

ABOUT SAGE BIONETWORKS

Sage Bionetworks is a nonprofit biomedical research and technology development organization that was founded in Seattle in 2009. Our focus is to develop and apply open practices to data-driven research for the advancement of human health. Our interdisciplinary team of scientists and engineers work together to provide researchers access to technology tools and scientific approaches to share data, benchmark methods, and explore collective insights, all backed by Sage’s gold-standard governance protocols and commitment to user-centered design. Sage is a 501c3 and is supported through a portfolio of competitive research grants, commercial partnerships, and philanthropic contributions.

ABOUT DREAM CHALLENGES

DREAM (Dialogue on Reverse Engineering and Assessment Methods) Challenges emerged in 2006 to leverage the wisdom of the multidisciplinary scientific community to solve fundamental and difficult questions in biomedical research. DREAM’s methodology is based on crowd-sourcing scientific Challenges, fostering open and collaborative research, and promoting data sharing. In 2013, DREAM partnered with Sage Bionetworks, which developed and administers the technology platform that underpins DREAM Challenges.

ABOUT BRAIN COMMONS

The BRAIN Commons is a scalable cloud based platform for computational discovery designed for the brain health community. The BRAIN Commons empowers the global research community by providing access to multi-modal data, state-of-the-art analytic tools and a secure interoperable system for data sharing. The BRAIN Commons is spearheaded by Cohen Veterans Bioscience, a non-profit research biotech dedicated to advancing brain health through data driven science. In partnership with The Michael J. Fox Foundation, the BRAIN Commons hosts the DREAM challenge data. www.braincommons.org

COVID-19 DREAM Challenge Has Launched

The COVID-19 EHR DREAM Challenge is now open for registration. Help identify COVID-19 risk factors and predict who might be positive. The Challenge initially will use UW Medicine clinical data sets. If successful, we expect to expand this benchmarking effort to include methods for predicting trajectory and mortality as well as data from additional sites.

This is a research project being supported by many including Sage Bionetworks, the University of Washington School of Medicine Departments (Anesthesiology and Pain Medicine, Radiology and Biomedical Informatics and Medical Education), UW Medicine Information Technology Services, the Institute of Translational Health Sciences, the Center for Data To Health, the CLEAR Center and many faculty. We thank the National Center for Advancing Translational Sciences (NCATS) for support and for providing cloud infrastructure for facilitating synthetic data access to data scientists for the purpose of validating models.

LEARN MORE

 

Press Release: DM Dream Challenge

Combining Artificial Intelligence with Assessments from Radiologists Could Help Improve Accuracy of Mammography Screenings

SEATTLE, March 2, 2020 – In a study published today in the journal JAMA Network Open, researchers demonstrated that machine-learning algorithms could help improve the accuracy of breast cancer screenings when used in combination with assessments from radiologists. The study was based on results from the Digital Mammography (DM) DREAM Challenge, a crowd-sourced competition that kicked off in 2016 to engage a broad, international scientific community to assess whether artificial intelligence (AI) algorithms could meet or beat radiologist interpretive accuracy.

“This DREAM Challenge allowed for a rigorous, apples-to-apples assessment of dozens of state-of-the-art deep learning algorithms in two independent datasets,” said Dr. Justin Guinney, VP of Computational Oncology at Sage Bionetworks and Chair of DREAM Challenges. “This is a much-needed comparison effort given the importance and activity of AI research in this field.”

Conducted by IBM Research, Sage Bionetworks, Kaiser Permanente Washington Health Research Institute, and the University of Washington School of Medicine, the Digital Mammography DREAM Challenge determined that, while no single algorithm outperformed radiologists, a combination of methods in addition to assessments by radiologists helped improve overall accuracy of screenings. The research was conducted using hundreds of thousands of de-identified mammograms and clinical data from Kaiser Permanente Washington (KPWA) and the Karolinska Institute (KI) in Sweden, without releasing the data to participants.

“Our study suggests that an algorithmic combination of AI and radiologist interpretations could provide a mechanism for significantly reducing unnecessary diagnostic work-ups in the U.S. alone,” said Dr. Gustavo Stolovitzky, the Director of the IBM Translational Systems Biology and Nanobiotechnology Program at IBM’s Thomas J. Watson Research Center, and Founder of the DREAM Challenges.

To help protect data privacy and prevent participants from downloading sensitive mammography data, study organizers applied the model-to-data approach, which avoids the distribution of data to participants and mitigates the risk of sensitive patient data being released. Participants were invited to submit their algorithms to the study organizers who developed a system that automatically ran the models on the data.

“The concerns that patients feel about the use of medical images is always first in our minds. The novel model-to-data approach for data sharing is particularly innovative and essential to preserving privacy, because it allows participants to contribute innovations which might actually improve the standard of care, without receiving access to the underlying data,” said Dr. Diana Buist, of Kaiser Permanente Washington Health Research Institute and co-first author of the paper. “Also, the inclusion of data from two different countries with differing mammography screening practices highlights important translational differences in how AI could be used in different populations.”

Mammography screening is commonly used for early detection of breast cancer. While this detection tool has generally been effective, mammograms must be assessed and interpreted by a radiologist, using human visual perception to identify signs of cancer. This has led to false-positive results in an estimated 10 percent of the 40 million women who receive routine annual breast cancer screenings in the U.S.

“Based on our findings, adding AI to radiologists’ interpretation could potentially prevent hundreds of thousands of unnecessary diagnostic workups each year in the United States. Robust clinical validation is necessary, however, before any AI algorithm can be adopted broadly,” said Dr. Christoph Lee, professor of radiology at the University of Washington School of Medicine. He was the lead radiologist for the Challenge and co-first author of the paper.

Media Contacts:


RELATED CONTENT

Blog post from Dr. Diana Buist: https://www.kpwashingtonresearch.org/news-and-events/blog/2020/artificial-intelligence-aiding-not-replacing-radiologists


ABOUT IBM RESEARCH

For more than seven decades, IBM Research has defined the future of information technology, with more than 3,000 researchers in 12 labs located across six continents. Scientists from IBM Research have been awarded six Nobel prizes, a U.S. Presidential Medal of Freedom, ten U.S. National Medals of Technology, five U.S. National Medals of Science, and six Turing Awards. The teams have also included 19 inductees into the U.S. National Academy of Sciences and 20 inductees into the U.S. National Inventors Hall of Fame. For more information about IBM Research, visit www.ibm.com/research.

ABOUT SAGE BIONETWORKS

Sage Bionetworks is a nonprofit biomedical research and technology development organization that was founded in Seattle in 2009. Our focus is to develop and apply open practices to data-driven research for the advancement of human health. Our interdisciplinary team of scientists and engineers work together to provide researchers access to technology tools and scientific approaches to share data, benchmark methods, and explore collective insights, all backed by Sage’s gold-standard governance protocols and commitment to user-centered design. Sage is a 501c3 and is supported through a portfolio of competitive research grants, commercial partnerships, and philanthropic contributions.

ABOUT KAISER PERMANENTE WASHINGTON HEALTH RESEARCH INSTITUTE

Kaiser Permanente Washington Health Research Institute (KPWHRI) improves the health and health care of Kaiser Permanente members and the public. The Institute has conducted nonproprietary public-interest research on preventing, diagnosing, and treating major health problems since 1983. Government and private research grants provide our main funding.

ABOUT THE UNIVERSITY OF WASHINGTON SCHOOL OF MEDICINE

The University of Washington School of Medicine is part of the UW Medicine health system. The school educates the next generation of physicians and scientists, leads the community-based WWAMI Program that serves Washington, Wyoming, Alaska, Montana and Idaho, and was second in the nation (FY 2018) in biomedical research funding with $923.1 million in total revenue, according to the Association of American Medical Colleges.

ABOUT DREAM CHALLENGES

DREAM (Dialogue on Reverse Engineering and Assessment Methods) Challenges emerged in 2006 to leverage the wisdom of the multidisciplinary scientific community to solve fundamental and difficult questions in biomedical research. DREAM’s methodology is based on crowd-sourcing scientific Challenges, fostering open and collaborative research, and promoting data sharing. In 2013, DREAM partnered with Sage Bionetworks, which developed and administers the technology platform that underpins DREAM Challenges.

Sage Joins Open Wearables Initiative

“We are excited to partner with OWEAR and look forward to bring together OWEARs open initiative with Sage Bionetworks efforts for open and collaborative benchmarking. Open wearables first benchmarking project is focusing on the use of wearable sensors to measure gait,” said Dr. Larsson Omberg, vice president, systems biology at Sage Bionetworks.

Shimmer Research announced today that Sage Bionetworks has joined the OWEAR Working Group. OWEAR is a collaboration designed to promote the effective use of high-quality, sensor-generated measures of health in clinical research through the open sharing and benchmarking of algorithms and datasets.

Read the press release…


Related media:

Feb. 14, 2020 – https://mhealthintelligence.com/news/new-project-eyes-an-open-platform-for-data-from-mhealth-wearables

The Michael J. Fox Foundation and Sage Bionetworks Launch $25,000 BEAT-PD DREAM Challenge

FOR IMMEDIATE RELEASE

NEW YORK (Jan. 13, 2020) – The Michael J. Fox Foundation for Parkinson’s Research (MJFF) and Sage Bionetworks have partnered with Evidation Health, Northwestern University, Radboud University Medical Center, and BRAIN Commons to launch the BEAT-PD (Biomarker and Endpoint Assessment to Track Parkinson’s Disease) DREAM Challenge. BEAT-PD is a data challenge designed to benchmark new methods to predict Parkinson’s disease severity. Winners from the Challenge will share a $25,000 prize.

Recent advances in digital health have demonstrated the potential of sensor-based technologies for quantitative, remote monitoring of health – particularly for conditions affecting motor function such as Parkinson’s disease, a chronic neurological disorder that affects more than one million people in the U.S. alone. Devices, including smartphones, watches, and fitness trackers, can measure symptoms related to Parkinson’s in more detail and at a higher frequency than in-clinic assessments. The barrier is the lack of standardized methods to convert sensor-based data into digital biomarkers for patients whose symptoms can vary.

“Sensor data holds potential for helping us understand the daily experience of Parkinson’s patients and how disease progresses over time,” says Mark Frasier, PhD, senior vice president of research programs at MJFF. “We look forward to seeing what the teams can accomplish with the data and to further develop biomarkers in Parkinson’s.”

In a previous data challenge, teams proved that disease status and symptom severity could be predicted using data collected during the completion of specific tasks. The BEAT-PD Challenge builds on this by attempting to determine whether disease severity can be assessed from passive sensor data from consumer electronics, collected during daily life, not pre-set tasks, which will bring us closer to the promise of at-home monitoring of disease progression. Teams participating in the Challenge will have access to raw sensor (accelerometer and gyroscope) time-series data that can be used to predict individual medication state and symptom severity.

“By focusing on data collected in the home environment without supervision, we are targeting the harder problem of understanding the personalized variation in disease burden,” says Larsson Omberg, PhD, vice president of systems biology at Sage Bionetworks. “Ideally, insights derived from the challenge can aid in the development of digital biomarkers for PD.”

Submissions to the BEAT-PD Challenge are due by April 22, 2020. For more information, interested applicants are invited to attend a webinar on Feb. 4, 2020, at 8 a.m. PST. The winning team and a runner-up will be announced in May 2020.

Register for the challenge: www.synapse.org/beatpdchallenge

 

Contacts: 

Kristina Magana
The Michael J. Fox Foundation
kmagana@michaeljfox.org

 

Hsiao-Ching Chou
Sage Bionetworks
chou@sagebionetworks.org

 

ABOUT SAGE BIONETWORKS

Sage Bionetworks is a nonprofit biomedical research and technology development organization that was founded in Seattle in 2009. Our focus is to develop and apply open practices to data-driven research for the advancement of human health. Our interdisciplinary team of scientists and engineers work together to provide researchers access to technology tools and scientific approaches to share data, benchmark methods, and explore collective insights, all backed by Sage’s gold-standard governance protocols and commitment to user-centered design. Sage is a 501c3 and is supported through a portfolio of competitive research grants, commercial partnerships, and philanthropic contributions.

ABOUT THE MICHAEL J. FOX FOUNDATION

As the world’s largest nonprofit funder of Parkinson’s research, The Michael J. Fox Foundation is dedicated to accelerating a cure for Parkinson’s disease and improved therapies for those living with the condition today. The Foundation pursues its goals through an aggressively funded, highly targeted research program coupled with active global engagement of scientists, Parkinson’s patients, business leaders, clinical trial participants, donors and volunteers. In addition to funding more than $900 million in research to date, the Foundation has fundamentally altered the trajectory of progress toward a cure. Operating at the hub of worldwide Parkinson’s research, the Foundation forges groundbreaking collaborations with industry leaders, academic scientists and government research funders; increases the flow of participants into Parkinson’s disease clinical trials with its online tool, Fox Trial Finder; promotes Parkinson’s awareness through high-profile advocacy, events and outreach; and coordinates the grassroots involvement of thousands of Team Fox members around the world. For more information, visit us at michaeljfox.org, on Facebook or Twitter.

ABOUT DREAM CHALLENGES

DREAM (Dialogue on Reverse Engineering and Assessment Methods) Challenges emerged in 2006 to leverage the wisdom of the multidisciplinary scientific community to solve fundamental and difficult questions in biomedical research. DREAM’s methodology is based on crowd-sourcing scientific Challenges, fostering open and collaborative research, and promoting data sharing. In 2013, DREAM partnered with Sage Bionetworks, which developed and administers the technology platform that underpins DREAM Challenges.

ABOUT BRAIN COMMONS

The BRAIN Commons is a scalable cloud based platform for computational discovery designed for the brain health community. The BRAIN Commons empowers the global research community by providing access to multi-modal data, state-of-the-art analytic tools and a secure interoperable system for data sharing. Spearheaded by Cohen Veterans Bioscience, a leading global brain health research non-profit foundation, the BRAIN Commons is co-developed with the Center for Data Intensive Science at the University of Chicago and the Open Commons Consortium, which have successfully launched generations of data commons platforms. In partnership with The Michael J. Fox Foundation, the BRAIN Commons will host the DREAM challenge data. www.braincommons.org

New Papers: Remote Retention in Digital Health Studies, Machine Learning, Reproducible Benchmarking

Detecting the impact of subject characteristics on machine learning-based diagnostic applications

Journal: NPJ Digital Medicine

Authors: Elias Chaibub Neto, Abhishek Pratap, Thanneer M. Perumal, Meghasyam Tummalacherla, Phil Snyder, Brian M. Bot, Andrew D. Trister, Stephen H. Friend, Lara Mangravite and Larsson Omberg

Read the paper…


Indicators of retention in remote digital health studies: A cross-study evaluation of 100,000 participants

Preprint: arXiv:1910.01165 [stat.AP]

Authors: Abhishek Pratap, Elias Chaibub Neto, Phil Snyder, Carl Stepnowsky, Noémie Elhadad, Daniel Grant, Matthew H. Mohebbi, Sean Mooney, Christine Suver, John Wilbanks, Lara Mangravite, Patrick Heagerty, Pat Arean, Larsson Omberg

Read the paper…


Reproducible biomedical benchmarking in the cloud: lessons from crowd-sourced data challenges

Journal: Genome Biology

Authors: Kyle Ellrott, Alex Buchanan, Allison Creason, Michael Mason, Thomas Schaffter, Bruce Hoff, James Eddy, John M. Chilton, Thomas Yu, Joshua M. Stuart, Julio Saez-Rodriguez, Gustavo Stolovitzky, Paul C. Boutros, Justin Guinney

Read the paper…

DREAM Challenges Announces Changes in Its Leadership That Support a Rich Future in Open Benchmarking 

FOR IMMEDIATE RELEASE

SEATTLE – DREAM Challenges has announced changes to its governance and leadership to reflect its vision for future growth. DREAM is a volunteer organization of more than 30,000 solvers focused on crowd-sourced challenges to benchmark informatic algorithms in biomedicine. The peer-based governance process has been formalized by charter and will be led by a volunteer board of directors, with a rotating chair position. Dr. Justin Guinney, VP of Computational Oncology at Sage Bionetworks, is the new chair.

Sage Bionetworks has partnered with DREAM since 2013, providing the technology platform, governance protocols, and data science expertise that underpin DREAM Challenges. The DREAM methodology has already been adopted by other groups that host challenges. By assuming a key role in the charter, Sage is making a clear commitment to upholding and growing DREAM’s mission to enable collaborative science.

DREAM was founded in 2006 by Dr. Gustavo Stolovitzky, who announced this week at the 14th DREAM Challenges Conference in New York that he would be stepping back from actively leading the organization, though he will remain involved. Dr. Stolovitzky, the Director of the IBM Translational Systems Biology and Nanobiotechnology Program at IBM’s Thomas J. Watson Research Center, will continue to support the mission of DREAM as Chair Emeritus and DREAM Board Director.

“I’m proud of the community we’ve created with the DREAM Challenges,” said Dr. Stolovitzky. “Our approach has confirmed that data sharing and collaborative science is a powerful methodology not only for scientific research but also for the rigorous evaluation of algorithmic performance that is necessary for reproducible science. There is much to discover yet and it will be a pleasure to watch DREAM evolve under Justin Guinney’s creative leadership.”

Dr. Guinney, who has served on the DREAM board since in 2013, will work closely with Dr. Stolovitzky and the board to develop a vision for how to extend the reach of DREAM Challenges to untapped domains that are poised to benefit from crowd-sourced solutions. Dr. Guinney’s contributions to DREAM have focused on advanced predictive modeling in diagnostic mammography screening and drug combination therapy; prognostic modeling in breast and prostate cancer and multiple myeloma; and, recently, a benchmarking challenge using electronic healthcare records.

“The power of DREAM is the power to connect,” said Dr. Guinney, whose work at Sage Bionetworks focuses on integrative data analysis for prognostic and predictive modeling of cancer outcomes and response to therapy. “Building on the foundation of our past successes, DREAM will continue to serve its mission of connecting individuals and teams with data and algorithms to tackle complex biomedical questions.”

Under Dr. Stolovitzky’s leadership, DREAM evolved into a model of how to use crowdsourcing and open science in biomedicine, with first-of-its-kind Challenges that led to scientific insights in a wide range of research topics including biological network inference, cancer genomics and pharmacogenomics, predicting disease progression, and medical imaging among others. DREAM has launched more than 60 challenges that have engaged and touched the careers of thousands of scientists from dozens of institutions across the public and private sectors around the world.

“Having been involved with DREAM all these years, it’s an honor to help usher in the next generation of Challenges,” said Dr. Guinney.

In addition to Drs. Guinney and Stolovitzky, other board directors include: Dr. Laura Heiser, Oregon Health Sciences University; Dr. Paul Boutros, University of California, Los Angeles; Dr. James Costello, University of Colorado; Dr. Pablo Meyer, IBM; and Dr. Julio Saez-Rodriguez, Heidelberg University. More information about the DREAM leadership and governance can be found at www.dreamchallenges/people.


ABOUT DREAM CHALLENGES

DREAM (Dialogue on Reverse Engineering and Assessment Methods) Challenges emerged in 2006 to leverage the wisdom of the multidisciplinary scientific community to solve fundamental and difficult questions in biomedical research. DREAM’s methodology is based on crowd-sourcing scientific Challenges, fostering open and collaborative research, and  promoting data sharing. In 2013, DREAM partnered with Sage Bionetworks, which developed and administers the technology platform that underpins DREAM Challenges.

ABOUT SAGE BIONETWORKS

Sage Bionetworks is a nonprofit biomedical research and technology development organization that was founded in Seattle in 2009. Our focus is to develop and apply open practices to data-driven research for the advancement of human health. Our interdisciplinary team of scientists and engineers work together to provide researchers access to technology tools and scientific approaches to share data, benchmark methods, and explore collective insights, all backed by Sage’s gold-standard governance protocols and commitment to user-centered design. Sage is a 501c3 and is supported through a portfolio of competitive research grants, commercial partnerships, and philanthropic contributions.

Contact: Hsiao-Ching Chou
chou@sagebionetworks.org

Toward A Distributed, Benchmarking Ecosystem Using Electronic Healthcare Data

The deployment and use of artificial intelligence (AI) in healthcare is regarded by many as a fait accompli, despite few examples of success. As with most technological movements, hype precedes results, and this certainly holds true for clinical AI. However, given the enormous inefficiencies and costs within today’s healthcare system, the question is not whether AI will impact healthcare, but when.

Part of the allure of AI is the desire of healthcare institutions to move away from a rules-based approach to clinical care toward a data-driven model. However, in the translation of tools and algorithms into the clinic, the ability to rapidly and robustly verify the performance characteristics of an algorithm is a major bottleneck. Several barriers – among many – that stand out are areas that we at Sage Bionetworks frequently encounter and have developed strategies to address:

  1. Inability to access data. Many clinical algorithms are developed within a host sites’ EHR system, using site-specific data. Given the sensitivity of electronic healthcare data, it is difficult to gain access to verify performance in other contexts, thereby limiting the ability to assess the generalizability of an algorithm.
  2. Bias in evaluation. In most cases, the evaluation of algorithms is carried out by the same person who developed the algorithm, a circumstance known as the “self-assessment trap.” This tends to produce highly biased performance measurements and leads to results that often do not hold up under closer scrutiny.
  3. Irreproducibility. Many clinical algorithms are not well-documented and are hard to use by third parties. The problem of reproducibility in biomedical science has been well-described and presents an especially acute problem when translating research into clinical practice.

New DREAM Challenge

To overcome these challenges, Sage Bionetworks has partnered with NCATS, National Center for Data to Health (CD2H), and the University of Washington to launch the Electronic Healthcare Record (EHR) DREAM Challenge. This Challenge represents a first-of-its-kind demonstration for using electronic healthcare patient data to prospectively benchmark AI algorithms in a community challenge. As healthcare institutions move increasingly toward quantitative, data-driven decision making, there is a concomitant need to objectively assess and report the performance and generalizability of clinical algorithms. In the EHR DREAM Challenge, we intend to show how governance, technology, and community engagement can be combined to robustly assess AI within the healthcare system.

This Challenge represents a first-of-its-kind demonstration for using electronic healthcare patient data to prospectively benchmark AI algorithms in a community challenge.

A graphic with an outline of the United States. There are icons and a network that describes the process of the EHR DREAM Challenge.
FIG. 1: 1. Participants will be able to build and submit their prediction models to these challenges. 2. These models will be distributed to many healthcare institutions across the country that have been on-boarded into the evaluation and benchmarking network. 3. They will be trained and evaluated against the host site’s EHR repository. 4. The results from each of the sites will be returned to the Synapse leaderboard. 5. Model submitter reviews. As we continue to evaluate models, the best performing ones will move the top of the leaderboards.observations, and 221 million measurements. We expect to incorporate a second EHR database by the end of the Challenge through collaboration with another large healthcare provider.

In this first of a series of EHR-related Challenges, we are asking participants to predict patient mortality within six months of their last hospital visit. The data host for this Challenge is the University of Washington Medical System, which has prepared a curated dataset from their EHR enterprise data warehouse. The data collected span 10 years (2009-2019), with 1.3 million patients, 22 million visits, 33 million procedures, 5 million drug exposure records, 48 million condition records, 10 million

Given the highly sensitive nature of EHR data and associated risks of re-identifiability, we are using the model-to-data approach for receiving and evaluating submitted algorithms. This means participants will never have direct access to the data, thereby preserving the integrity, privacy, and security of the data. Participants must submit their algorithm in the form of a Docker container, which will be executed on their behalf within a secure and private cloud. Performance results of the algorithms will be reported back to participants via a leaderboard, providing an objective benchmarking and reporting of results to the research community.

Broader Vision

We are approaching this Challenge as the first step toward a larger goal: developing an ecosystem for evaluating algorithms on EHR data across a secure and distributed network of healthcare providers (Fig 1). We are extending the model-to-data framework for simpler deployment in multiple cloud and on-premises compute environments. In Challenges, we will ask participants to predict patient or population outcomes.

While we are starting with patient mortality, planning is already under way for future EHR Challenges that will address pressing clinical issues such as sepsis, cardiovascular disease, and patient re-admission. We intend on expanding to additional data modalities, such as genetics and imaging, as we demonstrated with the Digital Mammography DREAM Challenge (in partnership with Kaiser Permanente).

In developing a robust evaluation network, we are hopeful that this framework will accelerate the use and – more importantly – the benefits of AI to patients. The EHR DREAM Challenge is a first step toward this goal.


Learn more about the EHR DREAM Challenge:

REGISTER NOW

 

 

Wellcome Trust partners with Sage Bionetworks to launch Wellcome Data Re-use Prizes

 

To learn more and submit your idea, visit the contest page.

 

A photo illustration of a person who's staring at a reflected screen of data and graphics.
Credit: istock. Prize submissions are likely to be a piece of code or analysis, plus a short description of the work.

Data re-use can generate new insights that in turn lead to vital health benefits. To stimulate and celebrate the innovative re-use of data, the Wellcome Trust has announced the launch of the Wellcome Data Re-use Prizes. With the support of Sage Bionetworks and its Synapse platform, the Wellcome Data Re-use Prizes will present two awards, each based on a topic of strategic importance to Wellcome: AMR surveillance and malaria.

The challenges create incentives for re-use and help form a community around the data.

“We are excited to partner with the Wellcome Trust to demonstrate how the re-use of public datasets can lead to better scientific insights,” said Dr. Larsson Omberg, Vice President for Systems Biology at Sage Bionetworks. “In addition to judging the reproducibility of the methods, we are encouraging people to work together by awarding extra points for new collaborations.”

Call for Entries

Wellcome is fully committed to ensuring that research outputs are made available to accelerate health benefits. This means that the researchers we support must maximize the availability of their data and other outputs with as few restrictions as possible.

Making data available in a timely and responsible way ensures other researchers can verify it, build on it, and use it to advance knowledge and make health improvements. But we don’t want to encourage data sharing for its own sake – we want the data that is shared by our researchers to be re-used by others to generate new insights and tools.

Wellcome Data Re-use Prizes

Entries have to generate a new insight, tool or health application from data available in an open data resource of Wellcome’s choosing.

There are two prizes, each based on a topic of strategic importance to Wellcome: AMR surveillance and malaria. The winner of each prize will get £15,000 (approx. $19,000 USD). Two runners-up will get £5,000 (approx. $6,400 USD).

 

To learn more and submit your idea, visit the contest page.

 


About Wellcome Trust: Wellcome exists to improve health for everyone by helping great ideas to thrive. We’re a global charitable foundation, both politically and financially independent. We support scientists and researchers, take on big problems, fuel imaginations, and spark debate. We remain true to the vision and values of our founder, Sir Henry Wellcome, a medical entrepreneur, collector and philanthropist. Our work today reflects the amazing breadth of Henry’s interests, and his belief that science and research expand knowledge by testing and investigating ideas.

About Sage Bionetworks: Sage Bionetworks is a nonprofit research organization that believes that open practices can help accelerate biomedical research. Founded in 2009, our interdisciplinary team of scientists and engineers work together to provide researchers access to technology tools and scientific approaches to share data, benchmark methods, and explore collective insights, all backed by Sage’s gold-standard governance protocols and commitment to user-centered design. Located in Seattle, Sage is supported through a portfolio of competitive research grants, commercial partnerships, and philanthropic contributions. Learn more at www.sagebionetworks.org.