This release brings a range of enhancements, new features, and bug fixes, reflecting the effort of a large number of contributors. It has a single breaking change, which adjusts the default max_delay parameter in enw_process_data() to be the maximum observed delay in the input data. This change aims to encourage users to tailor this setting to their specific datasets and to give them a more reasonable default if they do not.
The package infrastructure has also had significant updates, including improved search functionality on the pkgdown website, the adoption of an organization-level pkgdown theme, the ability to cache Stan models across R sessions, and additional continuous integration tests.
Model enhancements include updated internal handling of PMF discretization and support for non-parametric reference date models, alongside documentation improvements that provide clearer guidance and examples for users.
A range of bug fixes have been implemented, including a fix for a bug in the enw_expectation() module that was causing issues with models containing multiple time series.
Full details on the changes in this release can be found in the following sections or in the GitHub release notes. To see the development timeline of this release see the 0.3.0 project.
@jamesmbaazam, @medewitt, @sbfnk, @adrian-lison, @kathsherratt, @natemcintosh, @Bisaloo and @seabbs contributed code to this release.
@jamesmbaazam, @adrian-lison, @sbfnk, @bisaloo, @pearsonca, @natemcintosh, and @seabbs reviewed pull requests for this release.
@jbracher, @medewitt, @kathsherratt, @jamesmbaazam, @zsusswein, @TimTaylor, @sbfnk, @natemcintosh, @pearsonca, @bisaloo, @parksw3, @adrian-lison, and @seabbs reported bugs, made suggestions, or contributed to discussions that led to improvements in this release.
max_delay in enw_process_data() has been changed to be the maximum observed delay in the input data rather than being 20 days. When this default is used a warning is now thrown and in general users should be setting this based on their data and application. See the documentation of enw_preprocess_data() for more details. See #224 by @adrianlison and reviewed by @seabbs.enw_expectation() module was not appropriately defining initial conditions when multiple groups were present. This issue was related to recent changes in cmdstan 2.32.1 and is required in order to use versions of cmdstan beyond 2.32.0 with models that contain multiple time series. See #282 by @seabbs and self-reviewed.enw_preprocess_data()) were defined based on report vs reference date. This won't have impacted most usage but was a problem when trying to fit a model to retrospective (and so completely reported) data. See #312 by @seabbs and self-reviewed.data.table passed to enw_quantile_to_long() could throw an error. See #324 by @natemcintosh and reviewed by @pearsonca.enw_aggregate_cumulative() initialised its time step from the first reference date + 1 day rather than the first reference date. See #336 by @seabbs and self-reviewed.enw_filter_reference_dates when using remove_days on data with missing reference dates. See #351 by @adrian-lison and reviewed by @seabbs.%in% with a scale on the right hand side and other similar issues. See #382 by @seabbs and reviewed by @pearsonca and @Bisaloo.pkgdown theming elements have moved to an organization-level pkgdown theme to increase re-usability and DRY-ness across the organization. See #419 by @Bisaloo and reviewed by @pearsonca and @seabbs.lintr checks are now run also on the tests/ directory. See #418 by @Bisaloo and reviewed by @seabbs.single-timeseries-rt-estimation.Rmd. The WORDLIST used by spelling has also been updated to eliminate false positives. Future typos will now generate an error in the continuous integration check so that we can catch them as early as possible. See #341 by @Bisaloo and reviewed by @seabbs.enw_expectation() module is appropriately defining initial conditions when multiple groups are present. See #282 by @seabbs and self-reviewed.epinowcast() to check models with multiple time series can be fit as expected on example data. See #282 by @seabbs and reviewed by @adrian-lison.{touchstone} benchmark that includes multiple time-series to ensure that this functionality is appropriately tested. See #282 by @seabbs and reviewed by @adrian-lison.merge_group option to all required GitHub Actions. This enables the use of a merge queue for pull requests. See #300 by @seabbs and self-reviewed.check_group_date_unique() function which ensures that user supplied groups result in unique combinations of group and dates. This function is used in enw_preprocess_data() and enw_complete_dates() to ensure that the user supplied groups are valid. See #295 by @adrian-lison and reviewed by @seabbs.simulate_double_censored_pmf() which helps users define "correct" probability mass functions for double censored delays based on work in epidist by @parksw3 and @seabbs. Note this function is likely to be spun out into its own package in the near future. See #312 by @seabbs and self-reviewed.min_reference_date argument to enw_aggregate_cumulative() to allow users to specify the minimum reference date to include in the output. This is useful when users want to aggregate to a timestep with a specified initialisation date that is not the default. For example if users data is already reported with a weekly cadence they would use min(data$report_date) + 1 to preserve that timestep. See #340 by @seabbs and reviewed by @natemcintosh.enw_complete_dates() for min_date and max_date arguments. These arguments allow users to specify the minimum and maximum dates to include in the output. This may be useful to users who want to ensure that their data is complete for a specified time period. See #340 by @seabbs and reviewed by @natemcintosh.enw_one_hot_encode_feature() for one hot encoding variables and binding them into the original data. This is useful when users want to include parts of variables in their models as binary indicators - for example giving a specific delay its own effect. See #348 by @seabbs and self-reviewed.threads_per_chain = 1 which should offer a small performance improvement. See #366 by @sbfnk and reviewed by @seabbs.cmdstan model can be compiled and has the correct syntax. This runs on pull requests whenever stan code is changed, when code is merged onto main with altered stan code, and on a weekly schedule against the latest main branch. See #386 by @seabbs.{cli} package for all package messaging in order to have modern and pretty notifications. See #188 by @nikosbosse and @seabbs reviewed by @pearsonca.check_max_delay() which allows to obtain coverage statistics for the assumed maximum delay based on the observed data. Enhanced postprocessing functions to accept a different max_delay than used in the model, by adding artificial samples/summaries for not-modeled dates. Further improved documentation and warnings around max_delay. See #224 by @adrian-lison and @seabbs and reviewed by @seabbs.enw_stan_to_r() to the user. This function is used for testing and in development to expose {epinowcast} stan code in R. Users may find this function useful as it allows them to explore the stan code used in {epinowcast} models more easily. Note that this functionality is known to be unstable when {rstan} is loaded in the same R session. See #431 by @seabbs and reviewed by @sbfnk.extract_sparse_matrix() to allow us to drop our {rstan} dependency. See #431 by @seabbs and reviewed by @sbfnk.enw_cache_location, which can be set using the set_enw_cache() function. See #407 by @medewitt and @seabbs and reviewed by @sbfnk and @pearsonca.observation_indicator argument to enw_obs(). Support was also added to enw_complete_dates() to flag missing data and as part of this new helper functions (enw_flag_observed_observations() and enw_impute_na_observations()) were also added. This support is likely most useful when used in conjunction to a known reporting structure. See #327 by @seabbs and self-reviewed.inst/examples/germany_max_delay_greater_than_data.R, has been added to demonstrate this feature. See #346 by @seabbs and reviewed by @adrian-lison.<epinowcast> object. The object returned by epinowcast() now has a variable called priors and can be accessed for inspection and downstream analyses. See #399 by @jamesmbaazam and reviewed by @pearsonca and @seabbs.citation() function in the README so that the command is shown to users and the output is treated like normal text. See #272 by @seabbs and self-reviewed.bookdown for pkgdown vignettes and moved to the flatly theme for pkgdown rather than the preferably theme. See #312 by @seabbs and self-reviewed.CITATION file to automatically pull relevant citation fields from the DESCRIPTION file. Also added a GitHub Actions workflow to auto-generate a citation.cff file whenever CITATION or DESCRIPTION change. This way, all three files will always be up to date. See #369 by @jamesmbazam and reviewed by @seabbs.epinowcast. See #405 by @medewitt and reviewed by @seabbs, @zsusswein, and @pearsonca.enw_delay_filter(): Deprecated with a warning in favour of enw_filter_delay(). This renaming is to better reflect the function's purpose. See #365 by @kathsherratt and reviewed by @seabbs.This is a minor release that fixes a bug in the handling of optional initial conditions that was introduced by a recent change in cmdstan 2.32.1. Upgrading is recommended for all users who wish to use versions of cmdstan beyond 2.32.0. In addition to fixing this issue, the release also includes some minor documentation and vignette improvements, along with enhancements in input checking.
@sbfnk and @seabbs contributed code to this release.
@seabbs reviewed pull requests for this release.
@sbfnk and @seabbs reported bugs, made suggestions, or contributed to discussions that led to improvements in this release.
cmdstan 2.32.1. This fix is required in order to use versions of cmdstan beyond 2.32.0. See #276 by @seabbs and self-reviewed.max_delay in enw_preprocess_data() to ensure that the maximum delay is greater than or equal to 1 and that it can be coerced to be an integer. See #274 by @sbfnk and reviewed by @seabbs.max_delay in enw_preprocess_data() and fixed a typo in the same documentation. See #274 by @sbfnk and reviewed by @seabbs.In this release, we focused on improving the internal code structure, documentation, and development infrastructure of the package to make it easier to maintain and extend functionality in the future. We also fixed a number of bugs and made some minor improvements to the interface. These changes included extending test and documentation coverage across all package functions, improving internal data checking and internalization, and removing some deprecated functions.
While these changes are not expected to impact most users, we recommend that all users upgrade to this version. We also suggest that users who have fitted models with both random effects and random walks should refit these models and compare the output to previous fits in order to understand the impact of a bug in the specification of these models that was fixed in this release.
This release lays the groundwork for planned features in 0.3.0 and 0.4.0 including: support for non-parametric delays, non-daily data with a non-daily process model (i.e. weekly data with a weekly process model), additional flexibility specifying generation times and latent reporting delays, improved case studies, and adding support for forecasting.
Full details on the changes in this release can be found in the following sections or in the GitHub release notes. To see the development timeline of this release see the 0.2.1 project.
@adrian-lison, @Bisaloo, @pearsonca, @FelixGuenther, @Lnrivas, @seabbs, @sbfnk, and @jhellewell14 made code contributions to this release.
@pearsonca, @Bisaloo, @adrian-lison, and @seabbs reviewed pull requests for this release.
@Gulfa, @WardBrian, @parkws3, @adrian-lison, @Bisaloo, @pearsonca, @FelixGuenther, @Lnrivas, @seabbs, @sbfnk and @jhellewell14 reported bugs, made suggestions, or contributed to discussions that led to improvements in this release.
enw_add_pooling_effect(): replaced string argument with ... argument, to enable passing arbitrary arguments to the finder_fn argument. The same general usage is supported, but now e.g. the default argument to supply is prefix = "somevalue" vs string = "somevalue" and argument positions have changed. This function is primarily for internal use and we expect only a small subset of advanced users who are creating models outside the currently supported formula interface to be impacted See #222 by @pearsonca and reviewed by @seabbs.enw_dates_to_factors(): Deprecated and removed as no longer needed. We expect this function had little to no external use and so there should be little impact on users. See #216 by @seabbs and reviewed by @adrian-lison.enw_formula() so that their variances parameters were not shared between the correct parameters when used together. This only impacts models that used formulas with both random effects and random walks and for these models appears to have led to increased run-times, fitting issues, and potentially unreliable posterior estimates but to have had a less significant impact on actual nowcasts. We suggest refitting these models and comparing the output to previous fits in order to understand the impact on your usage. See #228 by @seabbs and self-reviewed.enw_replace_priors() where the function could not deal with epinowcast summarised posterior estimates due to the new use of the pillar class. Added tests to catch if this issue reoccurs in the future. See #228 by @seabbs and self-reviewed.scoringutils. For an unknown reason our example data contained pillar classes (likely due to an upstream change). This caused an issue with internal scoringutils that was using implicit type conversion (see here). See #201 by @seabbs and reviewed by @pearsonca.enw_plot_quantiles() where the documented default for log was FALSE but the actual default was TRUE. See #209 by @seabbs and self-reviewed.enw_expectation() where when models were specified with zero intercept a initial condition was still being specified for the intercept of the growth rate (expr_r_int, #246). This was not flagged as an issue by cmdstan 2.31.0 but as of cmdstan 2.32.0, due to improvements in how initial conditions were being read in (stan-dev/stan#3182), it throws an error causing models to fail. Solution suggested by @WardBrian, implemented in #255 by @seabbs, and reviewed by @pearsonca.enw_incidence_to_cumulative(): Deprecated with a warning in favour of enw_add_cumulative(). This renaming is to better reflect the function's purpose. enw_incidence_to_cumulative() will be removed in 0.3.0. See #247 by @seabbs and reviewed by @pearsonca.enw_cumulative_to_incidence(): Deprecated with a warning in favour of enw_add_incidence(). This renaming is to better reflect the function's purpose. enw_cumulative_to_incidence() will be removed in 0.3.0. See #247 by @seabbs and reviewed by @pearsonca.README.md, NEWS.md, the model.Rmd vignette and convolution_matrix() documentation. The WORDLIST used by spelling has also been updated by eliminate false positives. See #221 by @Bisaloo and reviewed by @seabbs and @adrian-lison..lintr configuration file. This file is used when lintr::lint_package() is run or in the new lint-changed-files.yaml GitHub Actions workflow. See #220 by @Bisaloo and reviewed by @pearsonca and @seabbs.lint-changed-files.yaml GitHub Actions workflow instead of the regular lint.yaml to avoid annotations unrelated to the changes made in the PR. See #220 by @Bisaloo and reviewed by @pearsonca and @seabbs.summary.epinowcast() and plot.epinowcast() methods. See #209 by @seabbs and reviewed by @pearsonca.enw_plot_obs() where not otherwise covered by plot.epinowcast() tests. See #209 by @seabbs and reviewed by @pearsonca.coerce_dt(), addressing issues #242, #241, #214, and #149. This eliminates the need for add_group(), check_by(), and check_dates() (and associated documentation, tests - some of these were intermediate capabilities introduced within this minor version; see #208) which have all been removed. Also starts to enable internal versus external use of exposed methods with the copy = ... argument. See #239 by @pearsonca, reviewed by @seabbs.check_ functions used to check inputs. See #217 by @seabbs and reviewed by @adrian-lison.target_date in enw_metadata() as flagged in #212 by @pearsonca using formals() to instead detect the default values from the function specification. See #232 by @seabbs and self-reviewed.return() of stop() increases the number of branches in the code, which makes it harder to read. It also translates into a higher cyclomatic complexity. We have removed all else statements after return() and stop() in the package. See #229 by @Bisaloo and reviewed by @seabbs.no_contrasts in enw_formula() as this was unused. Identified by @bisaloo in #220 and raised in #223. See #228 by @seabbs and self-reviewed.enw_replace_priors() to check that it can handle epinowcast summarised posterior estimates. See #228 by @seabbs and self-reviewed.rw__) in enw_formula() and construct_rw() to indicate when a random effect variance is a random walk versus a random effect. See #228 by @seabbs and reviewed by.construct_rw() when a random walk is specified for a variable that is not a numeric variable. See #228 by @seabbs and self-reviewed.touchstone based on the implementation in EpiNow2 by @sbfnk. See #200 by @seabbs, @adrian-lison, @sbfnk, and self-reviewed.enw_linelist_to_incidence() which previously took several steps. See #247 by @seabbs and @jhellewell14 and reviewed by @pearsonca.develop branch for development versions of the package. This change was discussed in #250 with the major motivator being that since the introduction of release only builds to R Universe we no longer need to have a stable main branch of GitHub to control our releases. See #256 by @seabbs and reviewed by @Bisaloo and @pearsonca.summary.epinowcast() and plot.epinowcast() methods to the documentation. See #209 by @seabbs and reviewed by @pearsonca.enw_replace_priors() showing how to use a nowcast posterior to update the default priors. See #228 by @seabbs and self-reviewed.0.2.1 release and to use the recommended bibentry() approach. See #236 and #237 by @seabbs and reviewed by @Bisaloo.STYLE_GUIDE.md) to document the style conventions used in the package. See #64 by @seabbs and reviewed by @pearsonca and @Bisaloo.This release adds several extensions to our modelling framework, including modelling of missing data, flexible modelling of the generative process underlying case counts, an optional renewal equation-based generative process (enabling direct estimation of the effective reproduction number), and convolution-based latent reporting delays (enabling the modelling of both directly observed and unobserved delays as well as partial ascertainment). Much of the methodology used in these extensions is based on work done by Adrian Lison and is currently being evaluated.
On top of model extensions this release also adds a range of quality of life features, such as a helper functions for constructing convolution matrices and combining probability mass functions. It also comes with improved computational efficiency, thanks to a refactoring of the hazard model computations to the log scale and extended parallelisation of the likelihood that is optimised for the structure of the input data. We have also extended the package documentation and streamlined the contribution process.
As a large-scale project, the package remains in an experimental state, though it is sufficiently stable for both research and production usage. More core development is needed to improve post-processing, pre-processing, documentation coverage, and evaluate optimal configurations in different settings) please see our community site, contributing guide, and list of issues/proposed features if interested in being involved (any scale of contribution is warmly welcomed including user feedback, requests to extend our functionality to cover your setting, and evaluating the package for your context). This is a community project that needs support from its users in order to provide improved tools for real-time infectious disease surveillance.
We thank @adrian-lison, @choi-hannah, @sbfnk, @Bisaloo, @seabbs, @pearsonca, and @pratikunterwegs for code contributions to this release. We also thank all community members for their contributions including @jhellewell14, @FelixGuenther, @parksw3, and @jbracher.
Full details on the changes in this release can be found in the following sections.
.Rhistory to the .gitignore file. See #132 by @choi-hannah.DESCRIPTION file. See #132 by @choi-hannah.enw_new_reports() to enw_cumulative_to_incidence() and added the reverse function enw_incidence_to_cumulative() both functions use a by argument to allow specification of variable groupings. See #157 by @seabbs.inherits(x, "class") rather than class(x) %in% "class". See #155 by @Bisaloo.enw_add_metaobs_features() interface to have holidays argument as
a series of dates. Changed interface of enw_preprocess_data() to pass ... to enw_add_metaobs_features(). Interface changes come with internal rewrite and unit tests. As part of internal rewrite, introduces coerce_date() to R/utils.R, which wraps data.table::as.IDate() with error handling. See #151 by @pearsonca.match.arg for validating inputs. Briefly, the preference is now to define options via function arguments and validate with automatic match.arg idiom with corresponding enumerated documentation of the options. For this idiom, the first item in the definition is the default. This approach only applies to string-based arguments; different types of arguments cannot be matched this way, nor can arguments that allow for vector-valued options (e.g., if somearg = c("option1", "option2") were a legal argument indicating to use both options). See #162 by @pearsonca addressing issue #156 by @Bisaloo.cmdstan locally to improve the developer/contributor experience. See #147 by @seabbs and @adrian-lison.cpp_options to cmdstanr::cmdstan_model(). See #182 by @seabbs.convolution_matrix() for constructing convolution matrices. See #183 by @seabbs.enw_model() to write_stan_files_no_profile() for the target_dir argument. This allows users to compile the model once and then share the compiled model across sessions rather than having to recompile each time the temporary directory is cleared. See #185 by @seabbs.add_pmfs(), to sum probability mass functions into a new probability mass function. Initial implementation by @seabbs in #183, refactored by @pratikunterwegs in #187, following a suggestion in issue #186 by @pearsonca.convert_cmdstan_to_rstan. See #192 by @sbfnk.inst/stan/epinowcast.stan to increase modularity and clarity. See #140 by @seabbs.delay_snap_lmpf and delay_group_lmpf. These stratify by either snapshots or groups. This is helpful for some models (such as the missingness module). The ability to choose which function is used has been exposed to the user in enw_fit_opts() via the likelihood_aggregation argument. Both of these functions rely on a newly added expected_obs_from_snaps function which vectorises expected_obs_from_index. See #138 by @seabbs and @adrian-lison.delay_group_lmpf to support modelling observations missing reference dates. Also updated the generated quantities to support this mode. See #147 by @seabbs and @adrian-lison based on #64 by @adrian-lison.NEWS.md file. See #132 by @choi-hannah.enw_missing() model module. See #138 by @seabbs and @adrian-lison.ref_as_p in the stan code. There was an additional issue in that the enw_report() module currently self-declares as on regardless of it is or not. This bug had no impact on results but would have increased runtimes for simple models. Both of these issues were fixed in #142 by @seabbs.enw_example() pointing at an old file name when type="script". By @pearsonca.This is a major release focusing on improving the user experience, and preparing for future package extensions, with an increase in modularity, development of a flexible and full-featured formula interface, and hopefully future-proofing as far as possible. This prepares the ground for future model extensions which will allow a broad range of real-time infectious disease questions to be better answered. These extensions include:
EpiNow2, epidemia, etc.).If interested in contributing to these features, or other aspects of package development (for example improving post-processing, the coverage of documentation, or contributing case studies) please see our contributing guide and/or just reach out. This is a community project that needs support from its users in order to provide improved tools for real-time infectious disease surveillance.
This release contains multiple breaking changes. If needing the old interface please install 0.0.7 from GitHub. For ease, we have stratified changes below into interface, package, documentation, and model changes. Note the package is still flagged as experimental but is in regular use by the authors.
@adrian-lison, @sbfnk, and @seabbs contributed to this release.
lme4 random effects, and random walks. See #27 by @seabbs.epinowcast() with a particular focus on improving the modularity of the model components (described as modules in the documentation). All of the package documentation and vignettes have been updated to reflect this new interface. See #112 by @seabbs.enw_delay_metadata() has been added. This produces metadata about the delay distribution vector that may be helpful in future modelling. This prepares the way for #4 where this data.frame will be combined with the reference metadata in order to build non-parametric hazard reference and delay-based models. In addition to adding this function, it has also been added to the output of enw_preprocess_data() in order to make the metadata readily available to end-users. See #80 by @seabbs.enw_filter_reference_dates() and enw_filter_report_dates() have been added. These replace enw_retrospective_data() but allow users to similarly construct retrospective data. Splitting these functions out into components also allows for additional use cases that were not previously possible. Note that by definition it is assumed that a report date for a given reference date must be equal or greater (i.e a report cannot happen before the event being reported occurs). See #82 by @sbfnk and @seabbs.enw_preprocess_data() that highlights this information to the user (alternatively can be accessed by users using enw_missing_reference()). In addition, these missing observations have been setup to be passed to stan in order to allow their use in modelling. This feature is in preparation of adding full support for missing observations (see #43). See
#106 by @adrian-lison and @seabbs.cmdstan 2.30.0 where passing optimisation flags to stanc_options by default was causing a compilation error by not passing these flags by default. See #117 by @sbfnk and @seabbs.epinowcast() and enw_preprocess_data() with convergence checking for several example nowcasting models. Lower level tests for model tools and model modules have also been added. See #112 by @seabbs.cumulative_sum implementation suggested by @adrian-lison in #98. See #103 by @seabbs.expose_stan_fns() (which itself depends on rstan), additional functionality has been added to parse stan code in this function. See #74, #85, and #93 by @sbfnk and @seabbs.max_confirm) of notifications is available in every row (for both cumulative and incidence notifications) and the cumulative and daily empirical proportion reported are calculated for the user during pre-processing (see #62 by @seabbs).0.0.6 and previous versions notifications beyond the maximum delay were silently dropped. In 0.0.7 this is now optional behaviour (set using max_delay_strat in enw_preprocess_data()) and the default is instead to add these notifications to the last included delay were present. This should produce more accurate long-term nowcasts when data is available but means that reported notifications for the maximum delay need to be interpreted with this in mind. See #62 by @seabbs.neg_binomial_2_log family of functions (over their natural scale counterparts). See #65 by @seabbsdistribution parameter with both the Lognormal and Gamma families being tested to work. Note that both parameterisations use their standard parameterisations as given in the stan manual (see #42 by @adrian-lison and @seabbs)include_paths (see #41 and #54 by @adrian-lison).hazard_to_prob has been optimised using vectorisation (see #53 by @adrian-lison and @seabbs).prob_to_hazard has been optimised so that only required cumulative probabilities are calculated (see #53 by @adrian-lison and @seabbs).inv_sqrt stan function (see #60 by @seabbs).scoringutils 1.0.0 (see #61 by @seabbs).enw_example(), to power examples and tests based on work done in forecast.vocs (see #61 by @seabbs).IDate when utilising enw_retrospective_data to solve esoteric error.include_paths to avoid console chatterstanc_options argument to enw_model() and specifies a new default of list("01") which enables simple pre-compilation optimisations. See here of these optimisation for details.inv_logit and logit as may instead use base R plogit and qlogit.enw_model().enw_summarise_samples() returned duplicate samples.scoringutils).