Multilevel Quality Indicators (MQI): Methodology and Monte Carlo evidence
Authors:
Martin Roessler,
Claudia Schulte,
Uwe Repschläger,
Dagmar Hertle,
Danny Wende
Abstract:
Background: Quality indicators are frequently used to assess the performance of healthcare providers, in particular hospitals. Established approaches to the design of such indicators are subject to distortions due to indirect standardization and high variance of estimators. Indicators for geographical regions are rarely considered.
Objectives: To develop and evaluate a methodology of Multilevel…
▽ More
Background: Quality indicators are frequently used to assess the performance of healthcare providers, in particular hospitals. Established approaches to the design of such indicators are subject to distortions due to indirect standardization and high variance of estimators. Indicators for geographical regions are rarely considered.
Objectives: To develop and evaluate a methodology of Multilevel Quality Indicators (MQI) for both healthcare providers and geographical regions.
Research Design: We formally derived MQI from a statistical multilevel model, which may include characteristics of patients, providers, and regions. We used Monte Carlo simulation to assess the performance of MQI relative to established approaches based on the standardized mortality/morbidity ratio (SMR) and the risk-standardized mortality rate (RSMR).
Measures: Rank correlation between true provider/region effects and quality indicator estimates; shares of the 10% best and 10% worst providers identified by the quality indicators.
Results: The proposed MQI are 1) standardized hospital outcome rate (SHOR), 2) regional SHOR (RSHOR), and 3) regional standardized patient outcome rate (RSPOR). Monte Carlo simulations indicated that the SHOR provides substantially better estimates of provider performance than the SMR and RSMR in almost all scenarios. RSPOR was slightly more stable than the regional SMR. We also found that modeling of regional characteristics generally improves the adequacy of provider-level estimates.
Conclusions: MQI methodology facilitates adequate and efficient estimation of quality indicators for both healthcare providers and geographical regions.
△ Less
Submitted 27 June, 2022;
originally announced June 2022.
Can we trust the standardized mortality ratio? A formal analysis and evaluation based on axiomatic requirements
Authors:
Martin Roessler,
Jochen Schmitt,
Olaf Schoffer
Abstract:
Background: The standardized mortality ratio (SMR) is often used to assess and compare hospital performance. While it has been recognized that hospitals may differ in their SMRs due to differences in patient composition, there is a lack of rigorous analysis of this and other - largely unrecognized - properties of the SMR. Methods: This paper proposes five axiomatic requirements for adequate standa…
▽ More
Background: The standardized mortality ratio (SMR) is often used to assess and compare hospital performance. While it has been recognized that hospitals may differ in their SMRs due to differences in patient composition, there is a lack of rigorous analysis of this and other - largely unrecognized - properties of the SMR. Methods: This paper proposes five axiomatic requirements for adequate standardized mortality measures: strict monotonicity, case-mix insensitivity, scale insensitivity, equivalence principle, and dominance principle. Given these axiomatic requirements, effects of variations in patient composition, hospital size, and actual and expected mortality rates on the SMR were examined using basic algebra and calculus. In this regard, we distinguished between standardization using expected mortality rates derived from a different dataset (external standardization) and standardization based on a dataset including the considered hospitals (internal standardization). Results: Under external standardization, the SMR fulfills the axiomatic requirements of strict monotonicity and scale insensitivity but violates the requirement of case-mix insensitivity, the equivalence principle, and the dominance principle. All axiomatic requirements not fulfilled under external standardization are also not fulfilled under internal standardization. In addition, the SMR under internal standardization is scale sensitive and violates the axiomatic requirement of strict monotonicity. Conclusions: The SMR fulfills only two (none) out of the five proposed axiomatic requirements under external (internal) standardization. Generally, the SMRs of hospitals are differently affected by variations in case mix and actual and expected mortality rates unless the hospitals are identical in these characteristics. These properties hamper valid assessment and comparison of hospital performance based on the SMR.
△ Less
Submitted 8 September, 2020;
originally announced September 2020.
