-
New paper – Optimizing Disease Outbreak Forecast Ensembles
The Fox lab had our first “lab” paper come out this year that was started and finished during my tenure here at UGA! In the paper, myself and my collaborators (Lauren Meyers from UT, and Evan, Nick, and Minsu from UMass Amherst) investigated how the size of ensembles impacted their performance using data from all of the recent collaborative forecast hub data: https://wwwnc.cdc.gov/eid/article/30/9/24-0026-f2
Background
The background for this project came as Nick, Evan, and I were chatting at the CSTE Forecast conference in 2022 about forecasting. There was a question that came up repeatedly that year if we really needed dozens of teams to be producing weekly forecasts for COVID-19, because in the end we tend to only broadcast a single ensemble forecast. Since all of the collaborative hubs have publicly available data, we mapped out a project to actually investigate it.
Methods
We gathered all of the forecasts for all recent collaborative forecast efforts (millions and millions of data points), constructed random ensembles of varying sizes, and scored their performance. A key limitation is that we chose to run the analysis with a subset of models that had regularly forecasted during the time periods we chosen, meaning that our sample size wasn’t that large for many of the hubs AND that our analysis likely focuses on some of the better forecast models (many models that don’t perform well come in and out, so the consistent models tend to be some of the better ones submitted).
Results
We have three results from our primary analysis: (1) adding models always improves and reduces variability of the average forecast performance, (2) performance gains level off after 4-5 models are included, and (3) you want to include at least four models in your ensemble to beat baseline forecasts. All of these conclusions can be seen in Figure 2 of the manuscript (Panel B of this version):
After running these analyses, one can see that there are some randomly assembled ensembles for almost all of these efforts that beat the published ensemble that is composed of all submitted forecasts (some of the grey shaded regions are below the horizontal purple line and lower numbers are better in these). We set out to see if we could develop a simple way to select these models and found that generally ensembles of models that perform well in one season perform well in others (The Ensemble rank line in this figure):
While I think this is a really amazing finding, I didn’t feel comfortable having this as the key finding in the paper, for two reasons:
- We didn’t investigate more sophisticated ensemble weighting methodologies that likely will outperform these ones
- This approach will not work if the submitted models change season to season. For example, this past season the Flusion model performed amazingly, but would not have been included in this ensemble, because we were basing everything on performance from the previous season. I would love to pursue a method to make a real-time version of this ensemble that would address this issue and see if it outperforms other approaches.
Conclusions
Hub organizers should target a minimum of 4 validated forecast models to ensure robust performance compared with baseline models. Adding more models reduces the variability in expected ensemble performance but might come with diminishing returns in average forecast skill. As public health officials and researchers look to expand collaborative forecast efforts, and as funding agencies allocate budgets across methodological and applied forecast efforts, our results can be used to identify target participation rates, assemble appropriate forecast models, and further improve ensemble forecast performance.
The project was largely built on infrastructure made available through the hubverse ecosystem, and I think these data have a lot of potential for project you’re interested in the topic or methodology, you can find the code to recreate the analysis here: https://github.com/sjfox/ensemble-size
-
Hiring in Fall 2024!
We’re hiring! We’re looking for an additional postdoc to join our group!! Are you interested? Here are some quick hits about it:
- The position and project focus are flexible. I want you to be excited about the projects you’re working on, so the interview process will involve learning about your interests and discussing possible projects that would make sense based on our overlap. Currently our lab focuses on forecasting infectious disease dynamics and a wide-range of infectious disease modeling projects. Check out some of our latest publications and reach out to hear more about what we’re working on: https://scholar.google.com/citations?hl=en&user=qZ0DYksAAAAJ&view_op=list_works&sortby=pubdate
- The position will pay $57-67k/year for two years, and I will have additional funds that we can use to support conference travel, equipment, and other scientific expenses.
- The UGA benefits are good! Healthcare, dental, vision options. Retirement matching, subsidized child care, etc. For more information see here: https://hr.uga.edu/Current_Employees/Benefits/benefits/
- I am very flexible about the work arrangement and would be happy for the postdoc to be in-person or fully remote. This flexibility also extends to start dates, vacations, etc.
The link to the job advertisement can be found here, but feel free to email me directly with any questions or concerns before applying: https://www.ugajobsearch.com/postings/392752
Please don’t hesitate to reach out if you have any questions about the position or our group!
A photo of the State Botanical Garden that is just south of the main UGA campus, one of the great outdoors spaces nearby! -
The Fox lab, est. 2022
This summer I excitedly accepted a position at the University of Georgia as an Assistant Professor jointly appointed in the Department of Epidemiology & Biostatistics and the Institute of Bioinformatics. I also have an appointment with the Center for the Ecology of Infectious Diseases and am hoping to receive a courtesy appointment with the Odum School of Ecology (tbd).
I’m sad to leave my position as Associate Director of the UT COVID-19 Modeling Consortium, but I’m excited (and honestly a bit anxious overall) about starting my own research lab at UGA. Luckily we were already working over Zoom, so my collaboration with Lauren and team at UT is just moving time zones!
I’ll be looking for PhD students and postdoctoral researchers soon, so stay tuned!
Leroy filled to the brim ready to drive across the country to Georgia 
Entrance to the health sciences campus commonly seen after grabbing a coffee from Sip’s 
New office, with a lovely door sign given to me as a parting gift from the UT COVID-19 Modeling Consortium 
My new work home (Miller Hall) has an awesome porch 
Office dog visits help break up the day Some photos of travel and the new office space at UGA
The Fox Lab 