Challenges & Benchmarking
The rapid growth and availability of biomedical and molecular data has motivated the computational community to develop many novel tools and methods. With algorithms playing an increasing role in biomedical analysis and patient care, there’s a need to earn trust with a rigorous and transparent framework for evaluating the real-world capabilities of these algorithms. Public benchmarking provides the research community with a way to objectively assess the relative merits and limitations of different methodologies. Community Challenges are a proven approach to focus attention with deadlines and incentives, encouraging new and diverse contributions from researchers.
Sage Bionetworks has pioneered the development of concepts and infrastructure to objectively evaluate algorithms across a broad spectrum of biomedical domains, including bioinformatics, biomedical informatics, medical imaging, and clinical trials. As a trusted partner in hosting challenges and benchmarking initiatives, Sage serves as an honest broker between data generators, owners, and modelers.
We combine technology with biomedical and data science expertise to facilitate the objective assessment of algorithms on critical data sets, including proprietary data and algorithms that cannot be broadly shared. These assessments serve the biomedical community in defining standards and benchmarks, and in guiding future algorithm development.
Current Projects
DREAM Challenges powered by Sage Bionetworks
A collection of community competitions on fundamental questions about systems biology and translational medicine, with the aim of advancing computational methods
NLPSandbox.io
NLPSandbox.io is one of the first tool-benchmarking platforms that securely connects developers to healthcare data providers
Registry of Open Community Challenges (ROCC)
A Challenge Registry which provides a central portal for discovery of open biomedical data challenges, currently in development.
BraTS Continuous Evaluation
In collaboration with MICCAI and University of Pennsylvania, this benchmarking event seeks to identify the current, state-of-the-art segmentation algorithms for brain diffuse glioma patients and their sub-regions. Snapshots of benchmarks will be captured annually.