Comparison of Beginning R Students’ Perceptions of Peer-Made Plots Created in Two Plotting Systems: A Randomized Experiment

Figures & data

Table 1 Student demographics in the Reproducible Research course.

Characteristics	Shares
Gender	Male: 76%
	Female: 24%
Student status	Nondegree student: 68%
	Full-time student: 24%
	Part-time student: 8%
Education	College (no degree): 4%
	Bachelor’s degree: 34%
	Master’s degree: 46%
	Doctorate degree: 11%
	Other: 5%
Employment status	Full-time: 68%
	Part-time: 4%
	Unemployed (looking for work): 16%
	Other: 12%
Language	English: 89%
	Chinese: 3%
	Other: 8%
Country	United States: 36%
	India: 12%
	Great Britain: 4%
	Canada: 3%
	China: 3%
	Germany: 3%
	Australia: 2%
	Brazil: 2%
	Spain: 2%
	France: 2%
	The Netherlands: 2%
	Russia: 2%
	Singapore: 2%
	Other countries: 25%

Fig. 1 Sample student submissions. The top panels show example student submissions for the simple plot that are of low (a) and high (b) quality. The bottom panels show example submissions for the complex plot that are of low (c) and high (d) quality. The left-hand plots in each section were made in base R, and the right-hand plots were made in ggplot2. For privacy reasons, none of these figures were made by students. These figures are recreations that show general types of figures that were commonly made by students.

Table 2 Peer review rubric.

Plot aspect		Prompt/answer choices
	Plot 1	Plot 2
Presence/compliance	Did they upload a plot? No Yes Yes, and they made it with (base R)(ggplot2).	Did they upload a plot? No Yes
Overall clarity	Does the plot clearly show the relationship between mean covered charges (Average.Covered.Charges) and mean total payments (Average.Total.Payments) in New York? No Yes	Does the plot clearly show the relationship between mean covered charges (Average.Covered.Charges) and mean total payments (Average.Total.Payments) vary by medical condition (DRG.Definition) and the state in which care was received (Provider.State)? No Yes
General aesthetics	Is the plot visually pleasing? No Somewhat Yes
Standalone clarity	Can the plot be understood without a figure caption? No Yes
Legend and label clarity	Are the legends and labels sufficient to explain what the plot is showing? No Yes
Text size	Are the plot text and labels large enough to read? No Yes
Full words	Do the plot text and labels use full words instead of abbreviations? No Yes

NOTE: Words in italics below indicate forced-choice options available on the online peer review form within the course.

Table 3 Annotated features of plots.

Feature	Plot	Description
Correct system	S + C	Is the plot made in the assigned system?
Is scatterplot	S + C	Is the plot a scatterplot? If “no,” the plot type was documented.
Shows points	S + C	If the plot is set up to be a scatterplot, does it show the actual data points and not just a smooth trend?
Both x and y axis labels	S + C	Do both the x and y axes have labels?
Non-default axis labels	S + C	Do the axis labels use descriptive text beyond the original variable names (e.g., Mean.Covered.Charges)?
Has units	S + C	Does the plot labeling indicate units?
Informative title	S	Does the plot have a title that describes the quantitative variables, indicates that the relationship is specific to New York, or gives an interpretation?
Trend line	S + C	Does the plot shows a smooth trend line? (Typically, linear fit or local regression.)
Guide line	S + C	Does the plot show other line(s) to guide interpretation? e.g., Reference line at y = x.
Lines labeled	S + C	If trend or guide lines are shown, are they all labeled via a legend or indicated otherwise?
*Number of panels	C	How many panels are in the plot? Common values: 1, 2, 6, 12, and 36.
*Color	C	Were points colored according to some variable?
Legend present	C	Is a legend present?
No labels cut off	C	Is all of the labeling visible and not cut off?
Medical condition clear	C	If numeric codes for medical conditions are shown, are these numbers explained?
Panel labels complete	C	Are panel labels present, and do they adequately describe what information is in the panel?
Fixed axis scales	C	Do both the x and y axes have fixed scales for better comparability between panels?

NOTE: Some features were only annotated in the simple (S) or in the complex (C) plot.

* Number of panels and color are combined into one feature (type of plot) later in our analysis.

All of these features are annotated as being “Present” or “Absent” except for “Number of panels,” which is annotated with an integer value.

Table 4 Compliance rates.

Arm	Plot	# plots	# annotated	% compliant
Base R	Simple	385	385	92.7% (357/385)
	Complex	400	399	87.5% (349/399)
ggplot2	Simple	565	561	99.5% (558/561)
	Complex	587	584	99.7% (582/584)

NOTE: The third column gives the total number of plot submissions. The fourth column gives the number of submissions that could be viewed and annotated. The last column gives the compliance rate as a percentage of the submissions that could be annotated.

Table 5 Comparison of peer review responses in the base R and ggplot2 arms (all student submissions).

Plot	Prompt	Response	Base R	ggplot2	ggplot2 – base R
Simple	Is the plot visually pleasing?	Somewhat	23.1% (288/1246)	18.3% (261/1428)	–4.8% (–8%, –1.7%)
	Is the plot visually pleasing?	Yes	73.7% (918/1246)	80.5% (1150/1428)	6.9% (3.6%, 10.1%)
	Clearly shows relationship?	Yes	86.2% (1074/1246)	89.7% (1281/1428)	3.5% (1%, 6.1%)
	Understandable without caption?	Yes	90.9% (1132/1246)	91% (1299/1428)	0.1% (–2.1%, 2.4%)
	Legends and labels sufficient?	Yes	89.4% (1114/1246)	90.8% (1296/1428)	1.4% (–1%, 3.7%)
	Text and labels large enough?	Yes	97.8% (1219/1246)	99% (1414/1428)	1.2% (0.2%, 2.2%)
	Use full words vs. abbreviations?	Yes	95.4% (1189/1246)	96.1% (1373/1428)	0.7% (–0.9%, 2.3%)
Complex	Is the plot visually pleasing?	Somewhat	30% (374/1245)	30.8% (439/1425)	0.8% (–2.8%, 4.3%)
	Is the plot visually pleasing?	Yes	59.5% (741/1245)	60.6% (863/1425)	1% (–2.8%, 4.8%)
	Clearly shows relationships?	Yes	72.3% (900/1245)	83.6% (1192/1425)	11.4% (8.1%, 14.6%)
	Understandable without caption?	Yes	76.8% (956/1245)	81.5% (1161/1425)	4.7% (1.5%, 7.9%)
	Legends and labels sufficient?	Yes	77.9% (970/1245)	82.4% (1174/1425)	4.5% (1.4%, 7.6%)
	Text and labels large enough?	Yes	89.8% (1118/1245)	86.3% (1230/1425)	–3.5% (–6%, –1%)
	Use full words vs. abbreviations?	Yes	83.6% (1041/1245)	85.4% (1217/1425)	1.8% (–1%, 4.6%)

NOTE: For each rubric item and response, the percentage of reviews indicating that response is shown. The last column gives the difference between the ggplot2 and base R arms and the 95% confidence interval for that difference.

Table 6 Agreement in expert and student responses to the review rubric.

Prompt	Among experts	and experts
Is the plot visually pleasing?	63.3% (76/120)	81.7% (98/120)
Clearly shows relationship?	57.5% (69/120)	69.2% (83/120)
Understandable without caption?	30% (36/120)	69.2% (83/120)
Legends and labels sufficient?	37.5% (45/120)	69.2% (83/120)
Text and labels large enough?	74.2% (89/120)	80% (96/120)
Use full words vs. abbreviations?	60% (72/120)	67.5% (81/120)

NOTE: The sample size is 120 plot submissions. “Among experts” gives the rate of perfect agreement among all 3 experts. “Between students and experts” gives the rate of agreement between summary responses from students and experts.

Table 7 Presence of plot features in student submissions.

	Simple		Complex
Plot feature	Base R	ggplot2	Base R	ggplot2
Is scatterplot	95.3% (343/360)	96.6% (566/586)	92.9% (326/351)	95.4% (601/630)
Shows points	100% (345/345)	99.6% (568/570)	97.3% (319/328)	94.5% (572/605)
Has x and y axis labels	97.8% (352/360)	99.7% (584/586)	83.2% (292/351)	99.8% (629/630)
Non-default axis labels	81.9% (290/354)	78.3% (459/586)	83.5% (253/303)	73.3% (461/629)
Has units	34.3% (122/356)	37.5% (220/586)	31.4% (95/303)	36.4% (229/629)
Has informative title	97.9% (330/337)	100% (518/518)
Has trend line	61.7% (222/360)	79.5% (466/586)	46.4% (163/351)	67.5% (425/630)
Has guide line	2.8% (10/360)	1.4% (8/586)	1.4% (5/351)	0.5% (3/630)
Lines labeled	17.8% (41/230)	4.9% (23/470)	6% (10/168)	1.9% (8/427)
Good plot sizing	98.3% (354/360)	99.7% (584/586)	99.1% (348/351)	98.6% (622/631)
Legend present			86.3% (164/190)	100% (274/274)
No labels cut off			86.3% (303/351)	64.3% (405/630)
Medical code is clear			84.3% (279/331)	94.7% (593/626)
Panel labels are complete			96% (336/350)	99.8% (628/629)
Fixed x and y axis scales			38.7% (127/328)	92.1% (572/621)

NOTE: Sample sizes reflect the number of plots that could be annotated for the given feature. Sometimes a feature was not applicable or the information needed to annotate a feature was missing or cut off.

Fig. 2 Student perceptions of clarity in plots with different features. The denominators of these fractions are the two colored numbers to the left of the dots. They represent the number of reviews for the plots that had the indicated feature (blue) and that lacked the indicated feature (red). Dots represent the fraction of those reviews indicating that the submitted plot was clear. Certain features were annotated only in the simple plots or only in the complex plots (Table 3). When the dots are close together, students’ perceptions of clarity (column questions) were not very dependent on the presence of the plot feature.

Fig. 3 Examples of types of complex plots. (a) A typical 36 panel plot. (b) A typical 12 panel plot with coloring has panels for the 6 states colored by medical condition and panels for the 6 medical conditions colored by state. A typical 6 panel plot would show one of these two rows. (c) A typical 2 panel plot has one panel colored by the 6 states and a second panel colored by the 6 medical conditions.

Table 8 Types of complex plots.

Plot type	Base R	ggplot2	ggplot2 – base R
36 panels	31.9% (112/351)	54.4% (343/631)	22.4% (16%, 28.9%)
12 panels, no color	8.8% (31/351)	2.7% (17/631)	–6.1% (–9.6%, –2.7%)
12 panels, color	1.7% (6/351)	0.2% (1/631)	–1.6% (–3.2%, 0.1%)
6 panels, no color	4% (14/351)	1.6% (10/631)	–2.4% (–4.9%, 0.1%)
6 panels, color	41.6% (146/351)	37.1% (234/631)	–4.5% (–11.1%, 2.1%)
2 panels, color	5.1% (18/351)	1.9% (12/631)	–3.2% (–6%, –0.5%)

NOTE: For the 6 most common plot types, the percentage of submissions in which that type of plot was made is shown. The last column gives the difference between the ggplot2 and base R arms and the 95% confidence interval for that difference.

Table 9 Clarity of the different types of complex plots.

Plot type	Base R	ggplot2	ggplot2 – base R
36 panels	82.7% (283/342)	88.8% (728/820)	6% (1.3%, 10.8%)
12 panels, no color	60.7% (54/89)	71.9% (23/32)	11.2% (–9.5%, 31.9%)
12 panels, color	87.5% (7/8)	100% (2/2)	12.5%
6 panels, no color	52.1% (25/48)	52.4% (11/21)	0.3% (–25.6%, 26.2%)
6 panels, color	79.3% (384/484)	78.9% (476/603)	–0.4% (–5.4%, 4.6%)
2 panels, color	45.9% (28/61)	56.4% (22/39)	10.5% (–11.6%, 32.6%)

NOTE: For the 6 most common plot types, we show the percentage of reviews for submissions of each type that indicated that the submission clearly shows the intended relationship. Numbers in parentheses indicate sample sizes. The last column gives the difference between the ggplot2 and base R arms and the 95% confidence interval for that difference. We exclude the confidence interval for “12 panels, color” due to the small sample sizes.