20: Model Evaluation and Generalisation

Author

Affiliation

Smit, A. J.

University of the Western Cape

1 Introduction

Once we have several plausible, competing biological hypotheses formulated as statistical models, we need a principled way to evaluate and compare them. This chapter covers two critical, related aspects of model evaluation: model selection and generalisation.

First, we introduce information criteria (like AIC) as a powerful tool for comparing different models, representing competing hypotheses. Second, we address the common problem of overfitting, where a model fits our current data perfectly but fails to predict new data. This introduces the fundamental trade-off between model complexity and its ability to generalise beyond the data it was built on.

2 Key Concepts

• AIC and AICc: Understanding the Akaike Information Criterion (and its small-sample correction, AICc) as a measure of a model’s relative quality, balancing goodness-of-fit against complexity.
• Comparing Competing Hypotheses: Using AIC to weigh the evidence for different models that represent distinct biological ideas, rather than just searching for the “best” statistical fit.
• The Problem with Stepwise Methods: Explaining why automated, stepwise model selection procedures are statistically flawed and often lead to unreliable and uninterpretable models.
• The Bias–Variance Trade-off: A core concept explaining the trade-off between a model’s ability to fit the training data (low bias) and its ability to perform well on new data (low variance).
• Overfitting: Identifying when a model is too complex and has learned the noise in the data, not just the signal.
• Cross-Validation: Introducing k-fold cross-validation as a robust method for estimating a model’s predictive performance on unseen data.
• Training vs. Test Error: Emphasizing the critical distinction between how well a model fits the data it was built with and how well it predicts new data.

This chapter is a placeholder. Content will be developed further.