Title:How to analyze data in a factorial design? An extensive simulation study

Abstract:Factorial designs are frequently used in different fields of science, e.g. psychological, medical or biometric studies. Standard approaches, as the ANOVA $F$-test, make different assumptions on the distribution of the error terms, the variances or the sample sizes in the different groups. Because of time constraints or a lack of statistical background, many users do not check these assumptions; enhancing the risk of potentially inflated type-$I$ error rates or a substantial loss of power. It is the aim of the present paper, to give an overview of different methods without such restrictive assumptions and to identify situations in which one method is superior compared to others. In particular, after summarizing their underlying assumptions, the different approaches are compared within extensive simulations. To also address the current discussion about redefining the statistical significance level, we also included simulations for the 0.5\% level.