Why overfitting is bad for scienc: Lessons from psycology

Many published research findings in psychology cannot be replicated. Even formerly “wellestablished” effects such as power-posing and implicit priming have failed to replicate. The crisis is not limited to psychology – replication issues abound across numerous fields, including neuroscience and biomedical sciences (e.g. Button et al., 2013; Ioannidis, 2005). The main causes of the replication crisis are thought to be inadequate statistical literacy and questionable research practices, such as p-hacking and “HARKing” (Hypothesizing After Results are Known). However, there may also be a less well-appreciated contributor to replication crisis – overfitting (Yarkoni & Westfall, 2017). Overfitting occurs when an overly complex model provides a good fit to the data it was trained on, but fails to accurately predict new samples. The goal of the classical statistical frameworks used in psychology, such as OLS and maximum likelihood methods, is to provide inference by finding the best fit to the data at hand. As such, these methods are liable to overfitting, especially when used alongside automatic variable selection methods such as forward, backward, and stepwise regression. Conversely, the goal of more recent statistical and machine learning methods is to maximize prediction accuracy in new samples and guard against overfitting directly. As such, psychologists and other scientific researchers may benefit from incorporating newer statistical and machine learning methods into their research in order to improve its replicability. To this end, more user-friendly open-source machine learning software packages are now being developed, such as the recent R package PredPsych and machine learning module for JASP. The proliferation of convenient digital tools for machine learning may lead to more replicable and reliable research, in psychology and in experimental science in general.