Dual Mode Delivery in an Introductory Statistics Course: Design and Evaluation

Figures & data

Table 1 Summary of Studies in Statistics courses – Satisfaction and Performance (F2F vs. Online).

	Learning	Academic
Study	satisfaction	performance
Jones (1999)	Not assessed	No difference
Wang and Newlin (2000)	Not assessed	F2F Higher
Gunnarsson (2001)	Not assessed	No difference
Hulburt (2001)	Not assessed	No difference
McLaren (2004)	Not assessed	No difference
Dutton and Dutton (2005)	Not assessed	Online higher
McGready (2006)	Not assessed	No difference
Suanpang and Petocz (2006)	Not assessed	Online higher
Evans et al. (2007)	Not assessed	No difference
Lu and Lemonde (2012)	Not assessed	No difference^1
Schutte (1998)	Online higher	Online higher
Yablon and Katz (2001)	Online higher	No difference
Katz and Yablon (2003)	Online higher	Online higher
Summers et al. (2005)	F2F Higher	No difference

Fig. 1 Dual mode course structure.

Table 2 Student evaluation of course (SEC).

								z-test value
	F2F			Online			t-test	(Wilcoxon Rank
	Mean	Std Dev	n	Mean	Std Dev	n	value	Sum Test)
Question
1. The course was well organized	4.737	0.446	38	4.483	0.986	29	1.290	0.519
2. The assessment was clear and fair	4.500	0.688	38	4.517	0.688	29	–0.107	–0.101
3. I received helpful feedback on my assessment	4.158	0.886	38	4.069	1.067	29	0.363	0.127
4. This course engaged me in learning	4.368	0.714	38	4.276	0.922	29	0.448	0.611
5. The teaching (instructors, tutors, online, etc. on this course was effective in helping me to learn	4.763	0.431	38	4.448	0.910	29	${1.722}^{**}$	0.981
6. Overall I am satisfied with the quality of this course	4.605	0.547	38	4.379	0.942	29	1.152	0.525
7. The blend of F2F and online teaching was effective for my learning in this course	4.739	0.541	23	4.095	1.179	21	${2.292}^{**}$	${1.727}^{**}$
8. The recorded lectures that I could download assisted my learning	4.652	0.714	23	4.619	0.805	21	0.144	0.094
9. This course effectively used online assessment (e.g., quiz) to help me to learn	4.783	0.518	23	4.667	0.913	21	0.512	–0.094

NOTE: ${}^{*}$Statistically significant at the.10 level; ${}^{**}$statistically significant at the.05 level; ${}^{***}$statistically significant at the.01 level.

H0: No difference amongst the two groups; Ha: F2F students have a better course experience, t critical = 1.69 at the 5% significance level; z critical (Wilcoxon test) = 1.65 at the 5% significance level.

Table 3 Student evaluation of teaching (SET).

								z-test value
	F2F			Online			t-test	(Wilcoxon Rank
	Mean	Std Dev	n	Mean	Std Dev	n	value	Sum Test)
Question
1. This staff member presented material in a clearly organized way.	4.778	0.422	36	4.593	0.931	27	0.963	0.167
2. This staff member presented material in an interesting way.	4.778	0.422	36	4.519	0.975	27	1.293	0.611
3. This staff member treated students with respect.	4.778	0.422	36	4.778	0.424	27	0.000	0.000
4. This staff member showed a good knowledge of the subject matter.	4.889	0.398	36	4.852	0.362	27	0.385	0.410
5. Overall I am satisfied with the teaching of this staff member.	4.806	0.401	36	4.692	0.618	26	0.818	0.342
6. The staff member was effective in using innovative methods for design and delivery of the course materials.	4.762	0.539	21	4.571	0.978	21	0.781	0.314
7. The staff member communicated his/her enthusiasm for the subject.	4.952	0.218	21	4.857	0.359	21	1.040	0.528
8. This staff member stimulated our interest and kept students engaged in learning.	4.667	0.658	21	4.714	0.644	21	0.407	–0.239

NOTE: ${}^{*}$Statistically significant at the.10 level; ${}^{**}$statistically significant at the.05 level; ${}^{***}$statistically significant at the.01 level.

H0: No difference among the two groups; Ha: F2F students have a better teaching experience, t critical = 1.69 at the 5% significance level; z critical (Wilcoxon test) = 1.65 at the 5% significance level.

Table 4 Tests for academic performance.

	F2F			Online			Difference	t-test
	Mean	Std. Dev	n	Mean	Std. Dev	n	in Means	value
Online Quizzes	15.786	3.459	75	15.121	3.857	96	0.665	1.171
Comp. Assignment	16.133	3.923	75	15.130	4.381	96	1.003	1.555
Final Exam	36.007	13.254	75	35.234	13.765	96	0.772	0.370
Semester Total	67.926	17.463	75	65.485	19.452	96	2.441	0.851

NOTE: ${}^{*}$Statistically significant at the.10 level; ${}^{**}$statistically significant at the.05 level; ${}^{***}$statistically significant at the.01 level.

The Semester Total is out of 100 and represents the sum of the students mark in the three assessment items. The cutoff total marks of 85, 75, 65, and 50 refer to the grades of high distinction, distinction, credit and pass, respectively. A mark below 50 is a fail.

t-test H0: No difference amongst the two groups; Ha: F2F students perform higher than online students, t critical = 1.65 at the 5% significance level.

Table 5 Estimations of Academic Performance, OLS vs PSM.

	OLS			PSM ATT
	Pooled	2014	2015	Pooled	2014	2015
Online Quizzes	–0.992	–1.220	–0.721	–0.420	0.338	–0.798
	(0.609)	(0.979)	(0.812)	(1.158)	(1.457)	(1.514)
Comp. Assignment	–0.612	–0.633	–0.498	–0.026	1.000	0.138
	(0.671)	(1.033)	(0.909)	(0.996)	(2.083)	(1.282)
Final Exam	–0.936	–5.834	2.302	2.400	–3.793	5.615
	(2.219)	(3.270)	(2.960)	(3.398)	(5.511)	(3.641)
Semester Total	0.884	–3.498	3.561	6.492	7.310	10.015
	(1.860)	(2.598)	(2.551)	(4.196)	(7.194)	(5.116)
No. Obs.	171	67	104	170	67	102

NOTE: Bootstrapped standard errors (500 reps) reported in parentheses for the ATT.

${}^{*}$Statistically significant at the 0.10 level; ${}^{**}$satistically significant at the.05 level; ${}^{***}$satistically significant at the.01 level. Each coefficient is a separate estimation of the effect of studying online. Students are matched on age, age squared, gender, and bachelor type and year level of study for the ‘online quizzes’, ‘assignment’ and ‘final exam’. ‘Total’ also uses the online quizzes and assignment as a matching covariate. The ATT was estimated using Stata14 and the ‘psmatch2’ command. Nearest neighbor match with replacement was used.

Fig. A1 An example of online workshops using Blackboard Collaborate.

Fig. B1 Histogram of semester total marks (F2F vs online).

Table C1 Two one sided test equivalence using the Wilcoxon rank sum method for the student evaluation of course (SEC).

	Lower		Upper
	z-stat	p-value	z-stat	p-value	Conclusion
1. The course was well organized.	1.340	0.090	2.660	0.004	Indeterminate${}^{*}$
2. The assessment was clear and fair.	2.117	0.017	1.883	0.030	Equivalent
3. I received helpful feedback on my assessment.	1.865	0.031	2.135	0.016	Equivalent
4. This course engaged me in learning.	1.826	0.034	2.174	0.015	Equivalent
5. The teaching (instructors, tutors, online, etc. on this course was effective in helping me to learn.	0.756	0.225	3.244	0.001	Indeterminate
6. Overall I am satisfied with the quality of this course.	1.388	0.083	2.612	0.004	Indeterminate${}^{*}$
7. The blend of F2F and online teaching was effective for my learning in this course.	–0.055	0.522	4.055	0.000	Relevant Difference
8. The recorded lectures that I could download assisted my learning.	1.872	0.031	2.128	0.017	Equivalent
9. This course effectively used online assessment (e.g., quiz) to help me to learn.	2.148	0.016	1.852	0.032	Equivalent

NOTE: Ho1: ε - difference $<=0$, Ha1: -$\epsilon <$ difference (lower tail); Ho2: ε + difference $<=0$, Ha2: difference $<\epsilon$ (upper tail) where ε is the equivalence level $=1$ (expressed in the units of the z distribution). Conclusions marked with ${}^{*}$ are considered equivalent when using a 10% significance level.

Table C2 Two one sided test equivalence using the Wilcoxon rank sum method for the student evaluation of teaching (SET).

	Lower		Upper
	z-stat	p-value	z-stat	p-value	Conclusion
1. This staff member presented material in a clearly organized way.	1.770	0.038	2.230	0.013	Equivalent
2. This staff member presented material in an interesting way.	1.193	0.116	2.807	0.003	Indeterminate
3. This staff member treated students with respect.	2.000	0.023	2.000	0.023	Equivalent
4. This staff member showed a good knowledge of the subject matter.	1.248	0.106	2.752	0.003	Indeterminate
5. Overall I am satisfied with the teaching of this staff member.	1.516	0.065	2.484	0.007	Indeterminate${}^{*}$
6. The staff member was effective in using innovative methods for design and delivery of the course materials.	1.561	0.059	2.439	0.007	Indeterminate${}^{*}$
7. The staff member communicated his/her enthusiasm for the subject.	0.961	0.168	3.039	0.001	Indeterminate
8. This staff member stimulated our interest and kept students engaged in learning.	2.334	0.010	1.666	0.048	Equivalent

NOTE: Ho1: ε - difference $<=0$, Ha1: -$\epsilon <$ difference (lower tail); Ho2: ε + difference $<=0$, Ha2: difference $<\epsilon$ (upper tail) where ε is the equivalence level $=1$ (expressed in the units of the z distribution). Conclusions marked with ${}^{*}$ are considered equivalent when using a 10% significance level.

Table C3 Two one sided t-test of equivalence for academic performance.

	Lower		Upper
	t-test value	p-value	t-test value	p-value	Conclusion
Online Quizzes	2.103	0.019	4.476	0.000	Equivalent
Computing Assignment	1.723	0.043	4.876	0.000	Equivalent
Final Exam	2.877	0.002	3.621	0.000	Equivalent
Semester Total	2.424	0.008	4.149	0.000	Equivalent

NOTE: Two one sided test (TOST) conducted in Stata15. Ho1: ε - difference $>=0$, Ha1: ε - difference < 0 (lower tail); Ho2: difference $+\epsilon <=0$ Ha2: difference $+\epsilon >0$ (upper tail) where ε is the equivalence level. The equivalence level is the appropriate allowable difference for the analysis. Equivalence level used for the displayed results utilises half the pooled standard deviation. A robustness check was conducted using 15% of the mean for the F2F group as the equivalence level. The conclusions do not change (selected based on the available discussion in the literature in Campbell and Gustafson (2018), and Mara and Cribbie (2012)). P values are reported as t-critical changes depending on the specification

Wang, A. Y., and Newlin, M. H. (2000), “Characteristics of Students who enroll and succeed in Psychology Web-based Classes,” Journal of Educational Psychology, 92, 137–143. DOI:10.1037/0022-0663.92.1.137.

 Web of Science ®Google Scholar

Dutton, J., and Dutton, M. (2005), “Characteristics and Performance of Students in an Online Section of Business Statistics,” Journal of Statistics Education, 13. Available at https://amstat.tandfonline.com/doi/abs/10.1080/10691898.2005.11910564#.XN4WdFIzbRZ

 Google Scholar

Suanpang, P., and Petocz, P. (2006), “E-learning in Thailand: An Analysis and Case Study,” International Journal of E-Learning, 5, 415–438.

 Google Scholar

Evans, S. R., Wang, R., Yeh, T., Anderson, J., Haija, R., McBratney-Owen, P., Peeples, L., Sinha, S., Xanthakis, V., Rajicic, N., and Zhang, J. (2007), “Evaluation of Distance Learning in an Introduction to Biostatistics Class: A Case Study,” Statistics Education Research Journal, 6, 59–77.

 Google Scholar

Lu, F., and Lemonde, M. (2012), “A Comparison of Online versus Face-to-face Teaching Delivery in Statistics Instruction for Undergraduate Health Science Students,” Advances in Health Science Education, 18, 963–973. DOI:10.1007/s10459-012-9435-3.

 PubMed Web of Science ®Google Scholar

Yablon, Y. B., and Katz, Y. J. (2001), “Statistics through the Medium of the Internet: What Students Think and Achieve,” Academic Exchange Quarterly, 5, 17–22.

 Google Scholar

Katz, Y. J., and Yablon, Y. B. (2003), “Locus of Control, Self-esteem, Motivation and Satisfaction, Teaching and Learning through an Internet-based and Traditional “Introduction to Statistics” Course,” in Proceedings of European Distance Education Network, Rhodes: EDEN., 183–197.

 Google Scholar

Summers, J. J., Waigandt, A., and Whittaker, T. A. (2005), “A Comparison of Student Achievement and Satisfaction in an Online versus a Traditional Face-to-face Statistics Class,” Innovative Higher Education, 29, 233–250. DOI:10.1007/s10755-005-1938-x.

 Google Scholar

Campbell, H., and Gustafson, P. (2018), “Conditional Equivalence testing: An Alternative Remedy for Publication Bias,” PloS One, 13.

 Google Scholar

Mara, C. A., and Cribbie, R. A. (2012), “Paired-Samples Tests of Equivalence,” Communications in Statistics-Simulation and Computation, 41, 1928–1943. DOI:10.1080/03610918.2011.626545.

 Web of Science ®Google Scholar