By Jiaxin Zheng and James Eddy 

At Sage Bionetworks, we strive to speed the translation of science to medicine by embracing open practices. 

Benchmarking reliable methods is one of the ways we deliver on that mission. Through our work with DREAM Challenges, we’ve pioneered the development of infrastructure and tools to objectively evaluate algorithms across a broad spectrum of biomedical domains, including bioinformatics, biomedical informatics, and predictive modeling of clinical outcomes. With algorithms playing an increasing role in biomedical analysis, crowd-sourced perspectives can shape more objective method evaluation and mitigate the self-assessment bias.

As the Challenge platform provider for the RSNA-ASNR-MICCAI Brats 2021 Segmentation Task, here are some of the principles our technology embraces to empower the benchmarking ecosystem. 

Collaboration: Synapse, Sage’s open-source research platform that allows teams to share data and track analyses, provided a centralized workspace for Challenge participants to collaborate. In addition to being able to access data, participants could post questions, find potential teammates, and submit models. From the wiki page to the evaluation workflow, we partnered with organizers to customize the space to meet their needs.

Portability: Sage developed the model-to-data approach where containerized algorithms rather than predictions are submitted for assessment on hidden data. These containers will be made available after the Challenge, promoting scientific reproducibility and reusability for the broader BraTS community. The model-to-data approach also eliminates the requirement for direct dissemination of validation data, reducing data transfer costs and enhancing security for Challenge organizers.

Innovation: To best serve the changes in the dynamic imaging space, we have augmented our infrastructure to integrate graphical processing unit (GPU) capabilities. These efforts enable easier exploration of large complex datasets and quicker model training, streamlining both development and evaluation. Our new GPU capability will be used for future imaging data inference competitions, and help stimulate algorithm development at the cutting edge of image-based learning. 

The BraTS community combined with the RSNA, ASNR, and MICCAI research networks has led to an impressive global turnout of Challenge participants, with more than 1,200 submissions from five continents. But this is only the beginning. Future phases of the BraTS Challenge will provide a sustainable cloud-based platform for open and continuous benchmarking of image analysis tools. We also plan to integrate with the DREAM Challenges community of solvers, and include genomics data in addition to images and new challenge tasks to address questions related to both.

Task 1 of the BraTS-RSNA-ASNR-MICCAI 2021 Challenge is the result of a collaboration by Sage Bionetworks, Perelman School of Medicine at the University of Pennsylvania, Radiological Society of North America, American Society of Neuroradiology, the Medical Image Computing and Computer Assisted Intervention Society, and sponsorship by Intel, RSNA, and Neosoma. We look forward to continued dialogue on how we can guide future algorithm development in order to best serve the broader biomedical community.