Sequential Scoring Rule Evaluation for Forecast Method Selection
Authors:
David T. Frazier,
Donald S. Poskitt
Abstract:
This paper shows that sequential statistical analysis techniques can be generalised to the problem of selecting between alternative forecasting methods using scoring rules. A return to basic principles is necessary in order to show that ideas and concepts from sequential statistical methods can be adapted and applied to sequential scoring rule evaluation (SSRE). One key technical contribution of t…
▽ More
This paper shows that sequential statistical analysis techniques can be generalised to the problem of selecting between alternative forecasting methods using scoring rules. A return to basic principles is necessary in order to show that ideas and concepts from sequential statistical methods can be adapted and applied to sequential scoring rule evaluation (SSRE). One key technical contribution of this paper is the development of a large deviations type result for SSRE schemes using a change of measure that parallels a traditional exponential tilting form. Further, we also show that SSRE will terminate in finite time with probability one, and that the moments of the SSRE stopping time exist. A second key contribution is to show that the exponential tilting form underlying our large deviations result allows us to cast SSRE within the framework of generalised e-values. Relying on this formulation, we devise sequential testing approaches that are both powerful and maintain control on error probabilities underlying the analysis. Through several simulated examples, we demonstrate that our e-values based SSRE approach delivers reliable results that are more powerful than more commonly applied testing methods precisely in the situations where these commonly applied methods can be expected to fail.
△ Less
Submitted 13 May, 2025;
originally announced May 2025.
Issues in the Estimation of Mis-Specified Models of Fractionally Integrated Processes
Authors:
Gael M. Martin,
Kanchana Nadarajah,
D. S. Poskitt
Abstract:
We provide a comprehensive set of new results on the impact of mis-specifying the short run dynamics in fractionally integrated processes. We show that four alternative parametric estimators - frequency domain maximum likelihood, Whittle, time domain maximum likelihood and conditional sum of squares - converge to the same pseudo-true value under common mis-specification, and that they possess a co…
▽ More
We provide a comprehensive set of new results on the impact of mis-specifying the short run dynamics in fractionally integrated processes. We show that four alternative parametric estimators - frequency domain maximum likelihood, Whittle, time domain maximum likelihood and conditional sum of squares - converge to the same pseudo-true value under common mis-specification, and that they possess a common asymptotic distribution. The results are derived assuming a completely general parametric specification for the short run dynamics of the estimated (mis-specified) fractional model, and with long memory, short memory and antipersistence in both the model and the true data generating process accommodated. As well as providing new theoretical insights, we undertake an extensive set of numerical explorations, beginning with the numerical evaluation, and implementation, of the (common) asymptotic distribution that holds under the most extreme form of mis-specification. Simulation experiments are then conducted to assess the relative finite sample performance of all four mis-specified estimators, initially under the assumption of a known mean, as accords with the theoretical derivations. The importance of the known mean assumption is illustrated via the production of an alternative set of bias and mean squared error results, in which the estimators are applied to demeaned data. The paper concludes with a discussion of open problems.
△ Less
Submitted 19 October, 2018; v1 submitted 4 July, 2014;
originally announced July 2014.
