Our Challenge Platform is an open-science, collaborative competition framework for evaluating and comparing computational algorithms. We use this platform to establish methodological benchmarks, to determine assessment metrics for developing algorithms, and to promote best-practices in bioinformatics. Using this platform, we administer challenges, serve data, interact with cloud computing servers and provide a collaborative framework for research communities to exchange ideas, data and code. Groups can use this platform to organize challenges and broadcast them to participants. The platform includes a web client where challenge organizers can register participants, broadcast a detailed challenge description, distribute data, collect submissions, post results and communicate with participants through forums and list-serves.
The platform can support challenges where participants are asked to submit algorithms in place of solutions. In these cases, the platform runs the algorithms on the data and returns results to participants. This allows organizers to perform evaluations using data that cannot be made openly available to challenge participants due to legal or ethical restrictions. This infrastructure permits researchers outside of the challenge to access and run their pipelines for their own research purposes, thereby facilitating a clearinghouse for tools and methods.
Our Challenge Platform is used to power DREAM Challenges, which are focused exclusively on biomedicine with a strong emphasis on cancer, and has an explicit mandate for transparency, openness, and collaboration. These principles enable DREAM to focus on the overarching goal of providing a community resource for the evaluation of emerging methodologies based on practical applications. For more information – and an extended list of current and past DREAM challenges – please visit dreamchallenges.org. In addition, we have helped to launch several independent Challenges using our platform.
To explore opportunities for partnering with Sage experts and using this platform to address scientific questions and foster open and collaborative science, please contact info@sagebase.org.
