In Praise of “False” Models and Rich Data

ABSTRACT

The authors argue that “true” models that aim at faithfully mimicking or reproducing every property of the sensorimotor system cannot be compact as they need many free parameters. Consequently, most scientists in motor control use what are called “false” models—models that derive from well-defined approximations. The authors conceptualize these models as a priori limited in scope and approximate. As such, they argue that a quantitative characterization of the deviations between the system and the model, more than the mere act of falsifying, allows scientists to make progress in understanding the sensorimotor system. Ultimately, this process should result in models that explain as much data variance as possible. The authors conclude by arguing that progress in that direction could strongly benefit from databases of experimental results and collections of models.

Keywords:

• Bayesianism
• falsifiability
• motor control

ACKNOWLEDGMENTS

Hugo L. Fernandes is supported by the PhD Program in Computational Biology of the Instituto Gulbenkian de Ciência, Portugal (sponsored by Fundação Calouste Gulbenkian, Siemens SA, and Fundação para a Ciencia e Tecnologia; SFRH/BD/33525/2008). The authors thank the NIH (R01NS057814, R01NS063399) and the Chicago Community Trust for financial support. They also want to thank Bruno Martins for helpful discussions and an anonymous referee for inspiration. Last, the authors want to thank the 2009 NCM satellite workshop on falsifiability in motor control organized by Dagmar Sternad and Robert Ajemian without which we certainly would not have written the article. They want to thank John Krakauer, Opher Donchin, and Mark Latash for outstandingly fruitful discussions.

Notes

1. We need to emphasize that we are scientists studying motor control and have no formal training in the philosophy or sociology of science. As such, we want to make clear that this article is work in progress and only reflects our present thinking behind the practical studies we perform in the area of motor control. We also want to stress that there are a good numbers of attempts at comparing physics and biology (e.g., Schrodinger, 1946) and that we only use a superficial comparison as a starting point of our discussion.

2. It is hard to define what a model is. In the context of this article we simply consider it an approximation to reality.

3. Compactness is a concept that is difficult to define. Intuitively compactness is the number of book pages we would need to fully write out the model, including all parameters. A more mathematical specification would be the length of the shortest Turing program to implement the model, a concept equivalent to Kolmogorov complexity (Li & Vitanyi, 1997).

4. In this article, we give a lot of reasons for our believe in this hypothesis, but obviously we could be wrong. Our hypothesis could be falsified by someone coming up with a compact model that precisely explains the bulk of data on motor control.

5. We focus on data variance in this paper because no model explains all the data and the world is probabilistic. Although we talk about data variance we acknowledge the structure of noise and ask which model makes the data most likely.

6. The question if a model should describe some data very well or a wide range of data acceptably is hard to answer and outside of the scope of this article.

7. In fact, one of the authors of this article, Konrad Kording, spent most of his PhD thesis building biologically realistic models of cortical function and thus models that fall squarely into this class.

8. We want to emphasize that although “true” models need to be bottom-up, there are “false” bottom-up models. For example, a bottom-up model may assume random synaptic weights—clearly not meant to be true—but which may be a sufficient assumption for say modeling network dynamics.

9. One possibility of reducing the number of free parameters is to build hybrid models in which parameters are determined by implicit top-down models. Such models have some properties of “true” models and some properties of “false” models and may be feasible. However, as this approach is not formalized yet in a clear way, we do not discuss it in detail in the present article.

10. In fact, it may be argued that a central difference between physics and neuroscience is the nature and usefulness of approximations. For example, many approximate models in physics approach reality arbitrarily well in certain limits. For example, ideal gas equations become arbitrarily precise for low density and high temperature gases. Analogous approximations in neuroscience (e.g., mean field or random connectivity) do not seem to have such properties. This may render models in the motor control domain trivially falsifiable.

11. For answering certain scientific questions, certain aspects of the nervous system may not be relevant. For example, the exact pattern of neural connections may not be relevant if we want to predict the patterns of epilepsy (Traub & Wong, 1982). Those models may well be compact—however, the underlying approximations make these models fall into the “false” category.

12. Incidentally, this classification does not apply to physics. Physical systems do not generally have a purpose but certainly many systems in biology are best understood by assigning purpose to them. Marr Level 1 is explicitly concerned with computational purposes.

13. The Marr levels are a model of models. As such, we do not know how complete the Marr levels are and there may well be models to which the Marr levels cannot be gainfully applied.

14. Although experimental falsification per se is not very interesting, we want to stress that it is absolutely vital for models to be falsifiable (Popper, 1959). If a model only makes nontestable hypotheses it does not help scientific progress. From our perspective it is of primary importance for a model to predict data.

15. In the following we use the word falsify to refer to the process of analyzing how a model is false.