Using databases in medical education research: AMEE Guide No. 77

Figures & data

Picture 1. A paper-based database.

Table

Practice points

A database is a tool for storing, managing and retrieving data, so it can be interpreted and used for various (in this case, for medical education research) purposes. Databases can be used to manage and analyse numerical and word-based data from quantitative and qualitative research projects. Deciding the nature of your research, understanding the nature of your data and how to classify data types are crucial to setting up your database. Practical considerations such as cost, available resources and support usually need to be taken into consideration, as well as the data management and analysis requirements of the project. Be aware of the local ethics requirements and international guidance for carrying out medical education research, and adhere to these. Being able to show how you addressed any issues of risk will help if you want to publish your work in a journal.

Table 1.  The difference between information and data

Paddy	Yes
Susan	No	No
Rama	Yes
Angelo	Yes
By adding headings, the data becomes information
Student Name	Attended ward round	Absence pre-authorised
Paddy	Yes
Susan	No	No
Rama	Yes
Angelo	Yes

Table 2.  Different types of categorical data

Nominal (no inherent order in categories)	Eye colour, ethnicity, diagnosis
Ordinal (categories have inherent order)	Job grade, age groups, course (e.g. Year 1 course, Year 2 course)
Binary (where there are only two categories)	Gender, grade for course (e.g. pass/fail)

Figure 1. Different types of data.

Table 3.  Examples of quantitative data and categorical data

Quantitative data	Categorical data
Height	Gender
Weight	Religion
Income	Method of treatment
University size	Type of teaching approach
Group size	Marital status
Self-efficacy test score	Qualifications
Percentage of lectures attended	Native language
Clinical skills performance	Type of instruction
Number of errors	Problem solving strategy used
Age	Social class

Box 1 Research paradigms

Research methodology falls into two main camps: quantitative and qualitative methodology. Quantitative approaches are used to test a hypothesis, to answer questions about ‘how much’ or ‘how many’. Qualitative approaches seek answers to the ‘what’, ‘how’ or ‘why’ of a phenomenon, exploring people's attitudes, behaviours, and so on. Quantitative methods tend to generate numerical data, while qualitative methods tend to generate language data (written or oral). In quantitative research, the research question can be exploratory or explanatory, while qualitative research is always exploratory.

There is an ongoing ‘paradigm’ war (Gage, 1989) in medical education. Clinicians working in medical education have been taught and trained in the scientific model. This is based on quantitative research - gathering empirical and measurable evidence through experiments, hypothesis testing (e.g., ‘is this treatment or intervention effective?’), controlling variables other than the one(s) under study so that any statistically significant difference must be a consequence of the treatment/intervention, and producing objective data which can be verified by other scientists reproducing the study. Randomised controlled trials (RCTs) are the gold standard research methodology of this paradigm. If one believes science is the only reliable source of knowledge, then the positivist, or scientific, model is therefore the appropriate model for all research, including educational research. However, social scientists working in medical education have been taught and trained in quite a different model, one that focuses on the human social life and which requires quite a different approach to understand what people do and why. In methodological terms, social scientists argue that we cannot understand why people do what they do, or why particular institutions exist and operate in characteristic ways, without grasping how those involved interpret and make sense of their world: in other words without understanding the distinctive nature of their perceptions, beliefs, attitudes. There is no ‘gold standard’ methodology; rather social scientists tend to use qualitative methods to explore the phenomenon under question.

Table 4.  An overview of some of the strengths and weaknesses of quantitative and qualitative research methods

Quantitative research methods		Qualitative research methods
Strengths	Weaknesses	Strengths	Weaknesses
Precision - through quantitative and reliable measurement.	Because of the complexity of human experience it is difficult to rule out or control all the variables.	Qualitative descriptions can play the important role of suggesting possible relationships, causes, effects and dynamic processes.	Contexts, situations, events, conditions and interactions cannot be replicated to any extent nor can generalisations be made to a wider context than the one studied with any confidence.
Control - through sampling and design.	Because of human agency people do not all respond in the same ways as inert matter in the physical science.	Because of close researcher involvement, the researcher gains an insider's view of the field. This allows the researcher to find issues that are often missed (such as subtleties and complexities) by the scientific, more positivistic enquiries.	Researcher's presence has a profound effect on the subjects of study.
Ability to produce causality statements, through the use of controlled experiments	It leads to the assumption that facts are true and the same for all people all of the time.	Qualitative research adds flesh and blood to analysis.	Issues of anonymity and confidentiality present problems when selecting findings.
Statistical techniques allow for sophisticated analyses.	It is not totally objective because the researcher is subjectively involved in the very choice of a problem as worthy of investigation and in the interpretation of the results.		The viewpoints of both researcher and participants have to be identified and elucidated because of issues of bias.

Box 2 Choosing a database

The choice of database depends on a number of factors. The main considerations are:

• What is available? Some software packages may require an expensive licence fee.

• Who will be using the database and how experienced are they in using the chosen software? The general ease of use of a database and possible requirement for training is important.

• What will the database be used for?

• What resources (e.g., staff time, money) can be spent developing it?

• Who will enter or check the data? What skills and experience do they have? Will there be more than one person entering data at the same time?

Box 3 Handy hints

Always document your variable and value labels – don’t rely on memory or assume it's obvious.

Be consistent with your value labels – keep No and Yes the same (e.g., 0 and 1) wherever these are in the database.

Codes 0 and 1 are often preferred for value labels, rather than 1 and 2, as this makes the interpretation of some types of statistical analysis easier.

Figure 2. Example of records/cases and variables.

Figure 3. Two ways of displaying the same data: long format (top) or intermediate format (bottom).

Figure 4. Example of a coding sheet (data dictionary).

Box 4 Handy hint: Never mix text and numeric data

However tempting, never mix text and numeric data as this will be time-consuming to sort out. Write notes in separate text variables. Do not write ‘N/A’ or ‘Left exam due to sickness’ in the middle of a numeric variable. This will cause problems when you attempt to analyse your data (see below).

Box 5 ‘Rubbish in, rubbish out’ (RIRO)

If rubbish (garbage) is entered into the database, this will obviously have a great effect on what comes out. If this happens and data is not double entered or checked, the likelihood is that the whole procedure will have wasted time, effort and money.

Making data entry as simple and as straightforward as possible is the easiest way to prevent bad data being entered. Data checking or validation is also good practice.

Figure 5. Example of a questionnaire format incorporating coding that can be used for data entry.

Box 6 Eyeballing your data

In a study of student admission, where the variable name represented the offer of a place, students offered a place were coded ‘1’, those not offered a place were coded ‘0’. Eyeballing the data indicated a couple of other numbers in the column – a 3 and a 9. This was not possible given there were only two coded categories for that variable. Checking against the raw (paper data) indicated that these were data error entries, which were then corrected.

Figure 6. The results of two student tests. (a) The best data descriptors will be mean and standard deviation as the distribution is symmetrical; (b) use median and range as the distribution is skewed.

Box 7 The difference between a bar chart and a histogram

Table 5.  Common uses for databases in undergraduate medical education

Student records – personal data, home address, age, next-of-kin, etc

Admissions data

Attendance at teaching events

Submission of work

Examination results

Course evaluation/feedback data

Progression (including time out of programme, resits)

etc.

Box 8 Example of a quantitative research study (Cleland et al., 2008a)

Cohort study on predicting grades: is performance on early MB ChB assessments predictive of later undergraduate grades?

This study aimed to identify whether poor performance in degree assessments early in a UK medical degree course predicts poor performance in later medical degree programme assessments. (The research question.)

The study subjects were University of Aberdeen undergraduate MB ChB students graduating in 2003–07 (five cohorts of students). Data on gender, age, funding status (home or overseas), previous qualifications (graduate or not) and marks were routinely collected during the selection and degree assessment processes. (The data/information.)

SPSS for Windows Version 15.0 (SPSS Inc., Chicago, IL, USA) was used for data storage and analysis. (The database.)

After statistically adjustment for cohort, age, gender, funding source, intercalation and graduate status, [poor] performance (fail and borderline pass) in the Year 2 first semester written examination Principles of Medicine II was found to be a statistically significant predictor of [poor] performance in all subsequent written examinations (all P < 0.001). Poor performance in the Year 3 objective structured clinical examination (OSCE) was a significant predictor of poor performance in Year 4 and 5 OSCEs, but there was no statistically significant relationship between essay-based summative assessments and poor performance. Males had statistically significantly poorer performance than females. (The statistical analysis of the data.)

We concluded that Examinations taken as early as mid-Year 2 can be used to identify medical students who would benefit from intervention and support. (The conclusion.)

Box 9 An example of transcribed data (from Cleland et al., 2008b)

Focus group participant 1: And um predictably the chap was upset, he burst into tears and he was distraught cause this was new feedback for him you know the issue had never been addressed at an earlier stage and that was where the difficulty and the tension came in, uh that it was very hard for us to uh identify for this individual that in our explanation um his performance was substandard in spite of the impression he’d been given over a number of years that he was he was um uh functioning adequately and that maybe a reflection on what uh we’ve just seen on the slides that poorly performing individuals do not accept feedback readily and it's possible that he had had some sort of feedback that hadn’t registered at the requisite level to change his his uh self-appraisal.

Interviewer: Yeah. What were the difficulties for you as a medical educator or for your team of medical educators?

Participant 1: The main one was just the um that's not the right word really but you know the kind of emotional difficulty that there is in approaching that kind of scenario that you know you’re going to you’re going to have to disappoint somebody and that's a thing that's uh very uncomfortable so there's a tension in that, you know you’ve got to do it because that's your responsibility but uh you don’t like um hurting people's feelings.

Participant 2: And I think that's true at an undergraduate level on the basis that they have already invested huge amounts of time and effort to particularly in the later stages of their undergraduate career and I was going to lead onto just follow on from what xxxxx (participant's name) said in terms of what the problem with the trainee that you alluded to is probably attitudinal rather than intellectual.

Participant 1: Absolutely

Figure 7. An example of a coding page (Moffat et al. 2007).

Box 10 Example of a qualitative research study (Cleland et al., 2008b)

Is it me or is it them? Factors that influence the passing of underperforming students

This study aimed to explore if ‘failure to fail’ is an issue for medical educators in the UK, and, if so, what are its determinants? (The research question)

The study subjects were medical educators (general practitioners, hospital doctors and non-clinical tutors) from two different UK medical schools who took part in focus group discussions which were recorded and transcribed for analysis (The data/information).

The qualitative data analysis software Atlas-ti ™ (Atlas-ti Scientific Software Development GmbH, Berlin, Germany) (http://www.atlasti.com/index.html) was used for data storage and analysis. (The database).

Using both theory and data-driven framework analysis, we identified six main themes relevant to the integrative model of behavioural prediction. These were: tutor attitudes towards an individual student; tutor attitudes towards failing a student; normative beliefs and motivation to comply; efficacy beliefs (self-efficacy); skills and knowledge; and environmental constraints (The data analysis).

We concluded that many different factors impact on medical educators’ failure to report underperformance in students. There are conflicts between these factors and the need to report competence accurately (The conclusion).

Gage NL. The paradigm wars and their aftermath. Educ Res 1989; 18: 4–10

 Google Scholar

Cleland JA, Knight L, Rees C, Tracey S, Bond CB. ‘Is it me or is it them?’ Factors influencing assessors’ failure to report underperformance in medical students. Med Educ 2008b; 42: 800–809

 PubMed Web of Science ®Google Scholar

Moffat M, Cleland J, van der Molen T, Price D. Poor communication may impair optimal asthma care: A qualitative study. Fam Pract 2007; 24: 65–70.

 PubMed Web of Science ®Google Scholar