Better data ≫ Bigger data

Dear Sir

We read Ellaway et al.’s (2014) article on Big Data in health professions education with great interest. We share the authors’ excitement and thank them for starting the conversation in our field. Here we stress two key Big Data concerns: while analytics have undeniable benefits for hypothesis generation, we can’t eschew broader questions of scientific design and analysis.

First, Big Data is not objective data. Just as with small, purposeful datasets, large datasets are defined by the assumptions, questions, tools, and interpretations that underpin them. Our understanding of health professions education may regress if we ignore issues of design, construct selection and validation of measurements. Large or small, purposefully collected datasets wrestle with these issues upfront; datasets of convenience rarely do.

Second, not all data analysis – no matter how large the dataset – constitutes science. Exploration of the signals (and noise) in large datasets without adequate conceptual frameworks can be misleading if not dangerous. Secondary data analysis is a useful but inherently limited scientific tool as it cannot robustly infer causation. It is only when data collection and analysis are informed by theory that robust results are possible.

The scientific method was developed to navigate the complex challenges of making meaning from data. In this endeavor, better data will always trump bigger data. Without proper design and analytic rigor, Big Data could easily make us aggrandize spurious results and lead us astray.

Others fields have navigated these challenges and used theory to guide Big Data. For example, Shwed and Bearman (2010) used Latour’s ‘Black Box’ theory to model scientific consensus formation. They analyzed citation networks from about 30,000 publications and 124,000 citations to shed light on controversies such as the carcinogenicity of tobacco and the autism/MMR vaccine connection. In medical education, Asch and colleagues (2009) tracked maternal complication rates for 4000 obstetricians who collectively performed 4.9 million deliveries over 15 years. The authors showed the effects of training program, experience, and individual ability on clinical performance, thereby testing and confirming theories developed by experimental studies.

These studies suggest that we as a community of scholars can use Big Data to serve research, rather than have Big Data dictate it. Meaningful knowledge comes only from scientifically informed design and analysis. Ultimately, it is not about the size of the dataset.