Pre/post-Test Evaluations of Agricultural Biosecurity Curriculum

ABSTRACT

Objective

In this combined study, the efficacy of different educational methods in enhancing students’ agricultural biosecurity knowledge, which is vital for public health, food security, and agricultural safety, was assessed.

Methods

In-person and virtual learning methods, utilizing educational content developed for the Gearing Up for Safety program, were investigated with 692 total participants. In the in-person learning method, 459 students aged 12–20 received a 50-minute in-person training session using the Gearing Up for Safety lesson on agricultural biosecurity. A 20 question pre- and post-test gauged knowledge gain. In the virtual learning method, 223 students, aged 12–20 received two, 40-minute virtual training sessions using the same content.

Results

Results of knowledge gain from the in-person method improved significantly from 55% to 77% (t(458) = -22.37, P < .001). Chi-squared analysis showed 19 out of the 20 questions significantly contributed to learning. Results of knowledge gain from the virtual method also showed a notable knowledge gain increase from 53% to 66% (t(222) = -11.86,  P< .001). However, a chi-squared analysis revealed that only 10 of the 20 questions significantly contributed to learning. In-person learning effectively supported better understanding of general biosecurity.

Conclusions

Virtual education improved general understanding but required enhanced focus on topics like vaccination, pest management, and community biosecurity preparation for comprehensive knowledge. Both in-person and virtual learning resulted in an increase in agricultural biosecurity knowledge, with in-person learning exhibiting a higher proportion of significant knowledge gains. These findings emphasize the importance of diverse educational methods in preparing future agricultural workers to ensure safety and health in agricultural settings. Results were used to enhance the curricula content being made available as part of the Gearing Up for Safety program for young and beginning workers.

KEYWORDS:

• Biosecurity curricula
• biosecurity training
• small/large farm
• biosecurity plan/training
• biosecurity resources
• and agrosecurity

Introduction

A review of literature regarding instruction in agricultural education programs related to biosecurity threats found a gap in curricula content.¹ A gap was also identified in evaluation methodology for delivery; no evaluations for agricultural biosecurity were identified on the scale of over 500 students. To address this gap, a set of instructional resources was developed to educate youth and beginning workers, ages 12–20, about the importance of enhancing biosecurity measures on agricultural production operations. The aim of this research was to evaluate the effectiveness of the developed content using both in-person and virtual content delivery methods in a formal agricultural education setting. This research was conducted in response to a review of curricula and standards during the development of a biosecurity-specific lesson for the nationally available Gearing Up for Safety curriculum.¹ Contents focused on how to identify and categorize farm pests, hosts, and environments to better understand prevention and the significance of early detection, swift eradication, and mitigation of biosecurity threats.

Agricultural biosecurity relates to the knowledge and practices associated with maintaining and protecting the health and well-being of both agricultural workers and livestock from biological threats. Farm biosecurity is crucial for maintaining public health, food supply chain safety, and quality of plant and animal products, as it reduces the risk of diseases and pests that can have severe economic and ecological impacts.² Typical procedures that have traditionally involved biosecurity practices include frequent hand washing, equipment washing, “shower in and shower out” procedures, or changing of clothes when entering different areas of a farm. Some commercial farming operations involve training in proper biosecurity techniques and disease detection. Educating young and beginning workers about biosecurity can equip them with the necessary knowledge and skills to minimize the threats and manage disease and pest outbreaks effectively, enabling them to become the next generation of skilled agricultural professionals.^3,4

Effective communication through instructional means is crucial in motivating and equipping individuals to engage in good biosecurity practices and prevent the spread of harmful diseases.⁵ Biosecurity-based socio-scientific issues are the most effective way to bridge the gap between experiential and disciplinary knowledge, making it relevant and meaningful for learners.⁶ In-person instructional methodologies can improve public knowledge and understanding of biosecurity, particularly among young people who may lack awareness due to a top-down approach to managing biosecurity.⁶ Previous research focused on virtual learning for biosecurity found that incorporating interactive elements, facilitates heightened engagement among students and leads to enhanced information retention compared to traditional instructional formats.⁷ Videos serve as an effective medium for disseminating essential biosecurity information, offering visual and interactive content that enhances understanding and encourages widespread adoption of best practices.⁸

The lack of knowledge among the public, particularly young and beginning workers with limited agricultural experience in biosecurity, is a concern.⁹ The lack of a trained workforce can cause considerable economic losses, decrease biodiversity, and potentially pose health hazards to humans and animals, potentially disrupting natural ecosystems.¹⁰ For example, a young worker who fails to recognize the potential disease risks their personal backyard poultry flock presents when they are employed at a commercial laying operation can lead to a biological disaster, resulting in the depopulation of entire flocks of birds. Additionally, the distribution of resources from stakeholders and organizations varies greatly and can affect what information a young worker might be exposed to.¹¹

In-person hands-on learning lessons and experiences can provide interactive opportunities for youth to engage with biosecurity practices, observe the impact of invasive species, and learn about their responsibilities in preventing the spread of harmful diseases and/or pests. Building upon the identified needs to equip youth and beginning workers with the knowledge and skills essential for safeguarding agricultural operations, this study evaluated the effectiveness of a developed instructional resource through both in-person and virtual content delivery methods.

Methods

With support from the Gearing Up for Safety agricultural design team, research was conducted to design, test, implement, and evaluate the efficacy of a biosecurity-specific lesson. Gearing Up for Safety aims to equip educators, parents, employers, and supervisors with the essential resources to organize and implement local production agriculture safety training programs for young and beginning farmers (aged 12–20), while providing parents, employers, and supervisors the tools for effective on-the-job safety training. This lesson has become nationally available as part of the Gearing Up for Safety online program (agsafety4youth.info).

A group of interdisciplinary experts participated in focus groups to prioritize learning objectives, content, and competencies for biosecurity lesson content. The resulting 15 core competencies and associated content were derived from a review of published research outcomes and state education standards related to agricultural biosecurity.¹

The study included two distinct groups of students: one enrolled in a virtual learning environment and the other engaged in traditional, in-person classroom settings. A mixed-methods approach was employed to comprehensively investigate differences in study environments between the virtual and in-person student groups. A set of 20 questions, aligned with the competencies, was developed, pilot-tested, and administered to measure knowledge gained from the training. All questions were either multiple choice (of four possible answers) or a “true” or “false” question. Every question, for both delivery methods and pre-/post-, had an “I don’t know’ choice that could be selected as a possible answer.

An a-priori power analysis was conducted using R. Studio to test the difference between the two independent groups means using a two-tailed test, a medium effect size (d = .80), and an alpha of .05. Results showed that a total sample of 70 participants with two equal-sized groups of n = 35 was required to achieve a power of 0.95.

In the case of participants attending the in-person lesson, traditional feedback mechanisms, such as interactions with instructors and opportunities for asking questions were available. These classroom observations were an important component of safety to gather insight into the dynamics of the in-person learning environment. For virtual participants, the analysis concentrated on the use of the videos and knowledge gain. Teachers at the education sites who disseminated the virtual lesson were incorporating it as part of their routine academic instruction. Hands-on focused activities incorporated as part of the lesson were shown in the video with no opportunity for discussion. During the online session, the instructor would pause when the PowerPoint activity was displayed and discuss the correct answers after some time had passed.

For the in-person testing, the curriculum was administered to 504 students in 21 classrooms at 10 secondary and post-secondary education sites. For the virtual lesson, the curriculum was administered to 229 students in nine classrooms at five secondary and post-secondary education sites. The classroom presentation for the in-person lesson consisted of a 50-minute lecture that included hands-on student-focused activities. The same presenter, an expert in the content area, delivered all lessons. The pre-test was administered 0–2 days before the classroom presentation, and the post-test administered 1–2 days after the classroom presentation. For the virtual lesson, the curriculum was implemented virtually and included two 20-minute lectures and instructional activities using the same content and presenter as the in-person lesson, with pre-tests conducted 0–2 days prior to the lecture, and post-tests administered 1–2 days after the lecture.

Participants were given a random number of pre-tests which aligned with the post-test. Of the 504 tests administered for the in-person testing, 459 tests matched the given randomized pre- and post-test number. Of the 229 tests administered for the virtual testing, 223 tests matched the randomized pre- and post-test number.

The statistical analysis was conducted using a two-sided unpaired t-test for the comparison between the two teaching methods (Table 1) and a paired t-test for each specific teaching method (Table 2 in person and Table 3 virtual) and a chi-square test for the statistical relevance of each of the 20 questions for each of the testing methods (Table 4 in-person and Table 5 virtual). These methods are commonly used for testing and question-specific analysis.¹² A p-value at the 5% significance level and 95% power was calculated for the following: pre-test average, post-test average, and question comparison. Table 2 identifies an average of the pre- and post-test scores, along with a confusion index (confusion index = (post “I don’t know” score-pretest “I don’t know” score)/(total questions)), and a learned index (learned index = (post-test score – pre-test score)/(total score – pre-test score)).

Table 1. Difference in delivery method, comparing knowledge gain.

	In-Person Method	Virtual Method
Mean	0.227	0.134
Variance	0.047	0.029
Standard Deviation	0.217	0.169
Observations	459	223
Coehns-d	0.46 (medium)
df	550
t Stat	6.11
P(T≤t) two-tail	1.88E–09

Table 2. Comparing knowledge gain (pre- and post-test) scores for in-person testing.

	Pre-Test	Post-Test
Mean	0.552	0.779
Variance	0.049	0.017
Standard Deviation	0.222	0.13
Observations	459	459
Pearson Correlation	0.33
Cohen’s d estimate:	1.25 (large)
df	458
t Stat	−22.37
P(T≤t) two-tail	1.89E–75

Table 3. Comparing knowledge gain (pre- and post-test) scores for virtual testing.

	Pre-Test	Post-Test
Mean	0.53	0.664
Variance	0.032	0.034
Standard Deviation	0.178	0.185
Observations	223	223
Pearson Correlation	0.57
Cohen’s d estimate:	0.74 (medium)
df	222
t Stat	−11.86
P(T≤t) two-tail	1.917E–25

Table 4. Learned index, confusion matrix, chi-squared P-value, and chi-squared test statistic for each of the 20 questions for in-person testing.

Question	Learned Index	Confusion Index	Chi-squared P-value	Chi-squared test statistic
1	68%	−12%	6.66E–30	1.29E + 02
2	56%	−13%	9.25E–19	7.82E + 01
3	83%	−7%	2.35E–15	6.27E + 01
4	94%	−14%	1.73E–49	2.19E + 02
5	6%	−11%	2.89E–01	1.13E + 00
6	61%	−15%	3.12E–17	7.13E + 01
7	65%	−12%	2.44E–16	6.72E + 01
8	86%	−19%	3.62E–21	8.92E + 01
9	76%	−8%	1.34E–12	5.03E + 01
10	62%	−9%	3.12E–13	5.31E + 01
11	55%	−5%	7.58E–08	2.89E + 01
12	39%	−8%	9.54E–07	2.40E + 01
13	59%	−10%	2.37E–08	3.12E + 01
14	49%	−13%	4.66E–11	4.33E + 01
15	82%	−18%	1.12E–48	2.15E + 02
16	49%	−21%	6.69E–11	4.26E + 01
17	28%	−21%	3.15E–11	4.41E + 01
18	13%	−13%	5.78E–06	2.06E + 01
19	53%	−12%	1.40E–24	1.05E + 02
20	75%	−11%	1.16E–12	5.06E + 01

Table 5. Learned index, confusion matrix, chi-squared P-value, and chi-squared test statistic for each of the 20 questions for virtual testing.

Question	Learned Index	Confusion Index	Chi-squared P-value	Chi-squared test statistic
1	65%	−6%	5.473E–18	7.470E + 01
2	33%	−4%	3.073E–05	1.737E + 01
3	28%	−1%	8.512E–02	2.964E + 00
4	38%	−3%	9.273E–05	1.528E + 01
5	18%	−6%	1.573E–02	5.832E + 00
6	46%	−5%	2.766E–04	1.322E + 01
7	19%	−1%	1.280E–01	2.316E + 00
8	30%	−10%	1.139E–01	2.499E + 00
9	19%	−1%	2.493E–01	1.327E + 00
10	34%	−4%	6.632E–05	1.591E + 01
11	10%	0%	3.808E–01	7.682E–01
12	13%	−3%	2.459E–01	1.346E + 00
13	24%	−5%	6.072E–02	3.518E + 00
14	6%	−1%	5.678E–01	3.263E–01
15	28%	−3%	9.108E–04	1.100E + 01
16	42%	−15%	1.125E–06	2.370E + 01
17	24%	−10%	1.053E–04	1.504E + 01
18	22%	−8%	1.593E–06	2.303E + 01
19	44%	−6%	1.334E–06	2.337E + 01
20	8%	−6%	5.944E–01	2.835E–01

This study, approved by the Purdue Institutional Review Board (IRB), examined pre- and post-test scores at schools located in Indiana. No incentives or pressure were applied for completing the testing and writing instruments (i.e., pencils and pens) were provided as needed. While the classroom instruction and content were conducted by a single instructor, there may have been variations in content emphasis for in-person delivery. The content delivered in the virtual setting was unmodified for all testing sites and included the same videos. It is worth noting that while most participants represented primarily rural students, some may not have had experience with agricultural biosecurity practices, while others may have come from commercial livestock operations with implemented biosecurity policies and procedures. For example, the use of certain biosecurity-related technical terms, such as “vector” or “quarantine,” may have caused confusion if the participant was not familiar with the meaning prior to testing.

Results

Results for the curriculum showed a difference in knowledge gain, based on the delivery method, for all participants (t(550) = 6.11, P < .001), as described in Table 1. The results from the in-person pre- (M = 55%, SD = 0.22) and post- (M = 77.9%, SD = 0.13) biosecurity tests, as described in Table 2, indicate an improvement in knowledge gain (t(458) =-22.37, P < .001). The degree of the difference between means was deemed as large (Cohen’s d = 1.17). The positive Pearson Correlation of 0.33 value measured positive strength and direction of the linear relationship, as hypothesized. Additionally, the results from the virtual pre- (M = 53%, SD = 0.17) and post- (M = 66%, SD = 0.18) biosecurity tests, as described in Table 3, indicate an improvement in knowledge gain (t(222) = −11.86, P < .001). The degree of the difference between means was deemed as medium (Cohen’s d = 0.74). The positive Pearson Correlation of 0.57 value also measured the positive strength and direction of the linear relationship, as hypothesized.

Tables 4 and 5 provide a summary of the item analysis for each of the 20 questions for in-person and virtual lesson, respectively, while Table 6 provides the text of all the questions. This includes pre-test average, post-test average, confusion index, learned index, and chi-square test P-value, comparing pre- and post-test correct answer rate. The distribution of knowledge gain for in-person and virtual learning is included in supplemental Figures S1 and S2, respectively.

Table 6. Questions used in the pre- and post-test.

Question Number	Question Text
1	What are the best two practices when implementing new livestock on to a farm?
2	Why are feed stocks and water sources an area of concern for livestock biosecurity?
3	What is a vector carrier for a disease?
4	What are the three sections of the disease triangle, or, what needs to be present for a disease to survive?
5	What is not a method used in crop systems to improve biosecurity?
6	What is the main benefit of using integrated pest management on a farm?
7	Identify which is not a biosecurity practice for farm visitors and staff.
8	T/F: Zoned areas reduce exposure and access to farmstead and improve biosecurity to visitors and staff.
9	When should sanitation and cleaning be used on a farm?
10	Identify what could be a consequence of a break in biosecurity?
11	What is the goal of vaccinations on a farm?
12	When should vaccinations be used on a farm?
13	T/F. Following biosecurity protocols will always eliminate any chance of a disease outbreak on a farm
14	How long should you quarantine a sick or ill animals?
15	How long should you quarantine new animals on a farm?
16	T/F: PPE is a last resort for biosecurity measures and should only be used in extreme situations
17	What is not a correct technique of Intergraded Pest Management?
18	Which of the following operations are not concerned with regulations of agricultural biosecurity?
19	Identify the best resource(s) for biosecurity information in the event of an emergency of outbreak?
20	T/F: Most diseases have very similar symptoms and it’s a waste of time to try to identify what disease a livestock may have?

For the in-person testing, the responses to questions 5 and 18 showed the weakest increase (<15%). with a minimal confusion (>-10%). These questions focused on operations concerned with biosecurity and biosecurity use in crop systems (Table 6). For all questions, except question 5, a statistically significant difference in the rate of correct responses was observed. For virtual testing, the responses to questions 14, and 20 showed the weakest learning increase (<15%) with a minimal confusion (>-10%). These questions focused on general biosecurity operations with quarantining sick animals and how diseases are different from one another (reference Table 6 for exact question). Twenty (20) questions from in-person and 19 questions from virtual delivery method showed a negative confusion matrix, meaning there was less confusion on the correct answer after receiving the curricula. One question on the virtual delivery showed a 0% confusion matrix as a result.

A chi-square test of independence was performed (testing with a P-value significance of 0.0025) to assess the relationship between the pre- and post-test for each of the 20 questions. Results, as shown in Table 4, showed that 19 of the 20 questions had statistically significant knowledge gain for the in-person lesson. Question 5 did not show statistically significant results with the chi-square test, X2 (1, N = 459) = 1.125, P = 2.89E–01). The responses to questions 4 and 15 showed the strongest learning increase (>70%) with minimal confusion (<-14%). Those questions focused on aspects of the disease triangle and how long it takes to quarantine new animals (reference Table 6 for exact questions).

For the virtual lesson, the same chi-square test of independence was performed (testing with a p value significance of 0.0025) to assess the relationship between the pre- and post-test for each of the 20 questions. Table 5 presents a set of questions along with the corresponding statistical measures. Only 10 out of the 20 questions had statistically significant knowledge gain. The topics that had significant knowledge gained consisted of parts of the disease triangle, emergency management of an outbreak, and general biosecurity protocols. The learned index ranged from 6% to 65%, representing the proportion of correct answers. The chi-squared p-values varied from 5.47E–18 to 5.94E–01, and the associated chi-squared values ranged from 0.28 to 74.70, reflecting the significance and strength of associations between question variables.

Discussion

Conducting these curricula evaluations in 30 classrooms across 15 secondary and post-secondary education sites provided an opportunity to assess the current “culture” of biosecurity content within correct agricultural education programs. Analysis of the results of each of the 20 questions provided a better understanding of the participants’ initial knowledge level, and what topics needed reduced or increased emphasis to achieve the greatest impact. The efficacy of in-person lessons for youth, as demonstrated in our study within the biosecurity context, resonates with broader agricultural education literature. Younger students, in general, seem to benefit from the developmental alignment provided by in-person interactions, fostering cognitive, social, and emotional growth.¹³

This research led to the following observations: 1), youth participants in the both in-person and virtual testing showed significant knowledge gain, with in-person showing the greatest difference in post-test scores, with the average increasing from 55% to 77%.; 2), the implementation of biosecurity curricula resulted in a substantial improvement in participants’ knowledge as demonstrated by the significant increase in post-test scores; and 3), questions focusing on the disease triangle and quarantine duration for new animals showed the highest learning gains, while questions related to biosecurity in crop systems and specific operations had relatively weaker improvement.

In in-person settings, research has shown that teachers experienced more dynamic interactions, fostering greater engagement.¹³ This was observed with the instructor having more interaction with in-person than virtual. The same study acknowledged the benefits of both virtual and traditional methods, emphasizing the importance of maintaining face-to-face interactions and challenges in student responsiveness in both settings. For biosecurity education, balanced approach, leveraging the strengths of both virtual and traditional methods, could be beneficial if there was a focus on teacher–student interactions and engagement.

Item analysis reveals specific questions that posed challenges for participants, such as questions 5 and 18 in the in-person setting and questions 1 and 16 in the virtual setting (Table 6 provides the text of all the questions). The confusion index and chi-square test results further highlight areas of strength and weakness, shedding light on topics that significantly contributed to learning and those that did not. The identification of content focus areas, such as aspects of the disease triangle and quarantine practices, presents opportunities for refining instructional strategies and curriculum development. Some of these topics and certainly aspects of all biosecurity awareness were included in the activities, which, depending on the education delivery format, could have played a vital part in why there were differences in learning for the different content areas.

In-person testing seemed to demonstrate a greater average in knowledge gain, learned indexes, and effect size (Cohen’s d). The Cohen’s d test score was important in assessing the impact of the curriculum in an in-person to virtual setting as it provided an additional quantitative measure of the magnitude of the knowledge gain. The results of the curriculum assessment indicated a significant difference in knowledge gain based on the delivery method, with in-person lessons demonstrating a higher improvement in knowledge compared to virtual lessons (t(550) = 6.11, P < .001). In the in-person setting, both pre- (M = 55%, SD = 0.22) and post- (M = 77.9%, SD = 0.13) biosecurity tests exhibited a substantial improvement in knowledge gain (t(458) = −22.37,  P< .001, Cohen’s d = 1.17).

Conclusion

The review of the literature on agricultural education programs addressing biosecurity threats identified deficiencies in curricula content and evaluation methodology. In response to this gap, a set of instructional resources was created to educate individuals aged 12–20, focusing on the significance of improving biosecurity measures in agricultural production operations. Two instructional methods, in-person, and virtual learning, utilizing a lesson on biosecurity designed for the Gearing Up for Safety program, were used. The first phase involved 459 students aged 12–20 undergoing a 50-minute in-person session, resulting in a significant improvement in post-test scores from 55% to 77% (t(458) = −22.37, P < .001). The second phase involved 223 students aged 12–20 undergoing a 50-minute virtual lesson and also resulted in significant improvement from the post-test scores from 53% to 66% (t(222) = −11.86, P < .001).

As results indicated, the virtual biosecurity curricula effectively enhanced participants’ understanding of biosecurity principles and practices. This significance was due to 10 of the 20 questions compared to 19 of the 20 questions being significant in creating a change in knowledge for the in-person learning methodology, as indicated by the chi-squared test. For biosecurity testing, the in-person learning method was more effective for this population of young adults. Virtual education enhanced general understanding but necessitates a stronger focus on specific topics like vaccination, pest management, and community biosecurity preparation for comprehensive knowledge. Both in-person and virtual learning led to increased agricultural biosecurity knowledge, with in-person learning exhibiting a higher level of significant knowledge gains.

These findings underscore the importance of incorporating targeted educational interventions to promote biosecurity awareness and knowledge. Delivery methods or teaching biosecurity and similar content, for this population, are understudied. By identifying areas of strength and weakness, instructors can tailor their teaching strategies to maximize the impact of educational programs. This type of content, both in-person and virtual, can ensure the development of a well-informed and capable community in managing biosecurity challenges on or off a farm.

Acknowledgments

The contents of these earlier efforts were used as a basis for the development and validation of a new curriculum with support from a USDA/NIFA grant (2021-67037-33375).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/1059924X.2024.2329153

Additional information

Funding

The authors recieved support for this research though a USDA/NIFA grant (2021-67037-33375).