Are farmers’ minds really too elusive to capture? A novel evaluation framework for the knowledge transfer initiatives in agriculture

ABSTRACT

Knowledge transfer initiatives aim to improve farmers’ knowledge and skills for tackling the growing economic, environmental and social challenges in agriculture. However, their evaluation is often inadequate due to institutional barriers and a lack of comprehensive methodology for assessing normative and behavioural shifts within this social group. To address this gap, we developed an extended evaluation framework for the knowledge transfer activities within the European Union's Common Agricultural Policy (CAP), including three indicators levels: output, result and impact. The indicators and survey instruments were based on the Theory of Planned Behaviour and piloted in a selected EU Member State for evaluating knowledge transfer measures in the programming period 2014–2022. The results indicate a promising potential of the proposed framework for capturing the multifaceted outcomes of knowledge transfer activities in agriculture. The usefulness and feasibility of the evaluation framework were further assessed through interviews with AKIS actors. They highlighted that evaluation activities at the Member State level may be hindered by inconsistent long-term data collection, insufficient response and involvement of farmers, deviations from the agreed protocols and a lack of dedicated resources. To ensure the uptake of the proposed evaluation framework, evaluation culture within the managing authorities and among stakeholders should be systematically fostered at the Member State level. This process can be enhanced by adopting a more elaborated CAP monitoring framework and methodology guidelines and by providing targeted training on evidence-driven policy-making.

KEYWORDS:

• Knowledge transfer in agriculture
• evaluation framework
• common agricultural policy
• theory-based evaluation
• theory of planned behaviour

1. Introduction

Agriculture faces an increasing range of challenges arising from social, environmental, political, economic and technological changes. To recognize and effectively respond to these challenges, farmers must be equipped with the latest knowledge and novel skills, which often go beyond the scope of primary agricultural production (Faure et al., 2012). Many governments are thus providing information-sharing and advisory support through various agricultural policy instruments. For example, the European Union's (EU) Common Agricultural Policy (CAP) provides funding for research, innovation, advisory services and training for farmers (ENRD, 2019). Similarly, the United States Department of Agriculture (USDA) funds knowledge-sharing initiatives through the Cooperative Extension System (CES) (Wang, 2014), and Australia's Rural Research and Development Corporations have established public-private partnerships to promote innovation and knowledge transfer in the agricultural sector (Paschen et al., 2017). Moreover, new approaches to farmer training are being developed, emphasizing the importance of practical skills, field demonstrations and participatory approaches for farmers’ successful adoption of sustainable agricultural practices (Bourne et al., 2021; Knook & Turner, 2020).

However, the assessment of knowledge transfer initiatives within the CAP remains limited, especially compared to other more prominent and financially extensive policy interventions, such as investment support and agri-environmental measures (ECA, 2015; Hill et al., 2017). This is a drawback as evaluation is a crucial component of the policy cycle, enabling the assessment of the policy's effectiveness and informing future decision-making (Cairney, 2019). Various approaches and theories have been developed to guide the design of evaluations for specific contexts (Fitzpatrick et al., 2011). One such is theory-based evaluation (Chen, 1990; Weiss, 1997), which assesses the effectiveness of interventions, policies or programs by focusing on their underlying theories of change and explaining the logical model between their objectives and measures. Furthermore, it aims to analyse the expediency of the actions taken to achieve the objectives and the intended outcomes (Fitzpatrick et al., 2011).

The EU's structural policies, such as the CAP and cohesion policy, consistently use the intervention logic approach to illustrate the relationships between proposed interventions and their intended outcomes (Gaffey, 2013). The policy interventions are illustrated through a logic model (Grinnell et al., 2016), which identifies the relationships between inputs, activities, outputs and immediate outcomes, as well as longer-term impacts. The assumed cause-impact relationships are measured by the indicators, which assess whether the program is achieving its intended objectives and doing so as expected based on the underlying theory (Harmelink et al., 2008). The EU's intervention logic uses a three-level indicator framework, which includes output, result and impact indicators. Output indicators refer to the direct goods and services delivered by the policy activities, while result indicators assess the immediate effects of the policy on the target population or beneficiaries. On the other hand, impact indicators measure the broader and long-term effects on society and the environment (EC, 2017).

In practice, intervention logic models and evaluation frameworks often lack clarity and quality in their design (Donaldson et al., 2019; Harmelink et al., 2008; Šumrada et al., 2020). For example, within the CAP evaluation framework, knowledge transfer is currently relatively weakly defined (EC, 2022; ECA, 2015), offering a limited list of indicators, which primarily focus on measuring interventions’ descriptive and quantitative inputs and outputs, such as the number of trainings implemented and the number of participating farmers. However, higher-level indicators that should capture normative and behavioural outcomes of knowledge transfer initiatives, such as farmers’ values and knowledge, technology adoption and agricultural performance, have been rarely considered (EC, 2022; ECA, 2015; Prager et al., 2017). Furthermore, the involvement of beneficiaries, i.e. farmers, in the evaluation process is becoming imperative (ECA, 2015) as their engagement can enhance the quality of policy planning procedures and foster a stronger sense of ownership, which can lead to increased adoption rates and improved efficiency of measures (Luján Soto et al., 2021; Masset & Haddad, 2015).

For a theory-based evaluation approach, it is fundamental to align policy theory with established behavioural or social theories (Stewart I. Donaldson 2007). The Theory of Planned Behaviour (Ajzen, 1991) is frequently employed in educational research (e.g. Hollett et al., 2020; Skoglund et al., 2020) and for explaining farmers’ decision-making processes and behaviour (e.g. Rezaei et al., 2019). It can also serve as a conceptual framework for evaluating policy interventions in agriculture (Wauters et al., 2010), where it can be used to analyse their effects and to identify the reasons for their success or failure (Ajzen, 2011). However, despite its relatively frequent use in the studies of farmers’ behaviour (Sok et al., 2021), it has not yet been widely applied to evaluate knowledge transfer in agriculture (Hall et al., 2019).

Scientific research on knowledge transfer in agriculture is relatively well-established in developing countries (Danso-Abbeam, 2022; Giulivi et al., 2023; Rajalahti & Swanson, 2010). However, such evaluations are relatively rare in developed countries and Europe (Faure et al., 2012; OECD, 2015). The main focus of these studies is on the impacts of knowledge transfer on farm management and performance (Cawley et al., 2018; Dinar et al., 2007) and on the uptake of specific farming practices (Alif et al., 2024; Ingram et al., 2022; Nordin & Höjgård, 2017). Some evaluations also focused on assessing the governance structures, capacity, management and methods involved in the knowledge transfer system (Lioutas et al., 2019; Prager et al., 2017). However, there continues to be a lack of comprehensive theory-based evaluations of knowledge transfer policies, which would connect inputs and outputs of the interventions with higher-level indicators assessing the immediate and long-term effects on beneficiaries.

Our study aims to develop a systematic evaluation framework and methodology to assess the effectiveness of knowledge transfer interventions in agriculture. It expands the current CAP monitoring and evaluation system by specifically focusing on result and impact indicators and engaging farmers in the evaluation process. We build on the Theory-based approach (Chen, 1990; Weiss, 1997) and the Theory of Planned Behaviour (Ajzen, 1991) to develop a comprehensive set of indicators and suitable survey instruments and pilot them in Slovenia, a Central European EU Member State. Finally, we explore the feasibility and potential challenges that may arise while implementing the developed framework in practice by interviewing representatives of the Slovenian Agricultural Knowledge and Innovations System (AKIS).

2. Case study description

Slovenia is a Central European country and an EU Member State since 2004. Given its predominantly hilly or karst landscape and alpine and continental climate, permanent grasslands cover 57% of agricultural land. Livestock farming, particularly cattle, is a prevailing farming sector, with 66% of agricultural holdings engaging in it (Bedrač et al., 2023). Despite accelerated structural changes in recent decades, Slovenian agriculture still grapples with relatively low productivity compared to other EU countries and a predominantly small farm structure with an average farm size of 7 ha of agricultural land (Bedrač et al., 2023). The average age of Slovenian farmholders is 58 years (SURS, 2020). Additionally, the educational structure of Slovenian farmers is rather unfavourable, as more than half of farmholders (52%) possess only practical experience in agriculture, while only 10% have formal agricultural education at the secondary or higher level (SURS, 2020).

In Slovenia, the key systemic support for knowledge transfer is implemented through the Public Agricultural Advisory Service, a central organization for agricultural advisory operating at the regional and local levels (JSKS, 2023). It offers advisory support to farm holdings in various fields, including agricultural production, agri-environment, legal and tax issues, and CAP subsidy and investment support. In addition, knowledge transfer activities are supported as part of the CAP's Rural Development Policy (RDP). During the 2014–2022 programming period, these were promoted through training programs for farmers enrolled in Agri-environmental-climate measures (AECM), Organic farming (OF) and Animal welfare (AW), as well as Cooperation measures, which funded pilot and European Innovation Partnership (EIP) projects aimed at fostering cooperation between different actors in agriculture and related sectors (MKGP, 2021).

3. Methodology

The study followed a three-phase structure (Figure 1). The first phase involved the design of a comprehensive evaluation framework, which included indicators and survey instruments to assess outputs, results, and long-term impacts of knowledge transfer interventions in agriculture. Next, the evaluation framework was piloted in the case of knowledge transfer initiatives in Slovenian agricultural policy. The final phase included a qualitative assessment of the suitability and feasibility of the proposed framework by conducting interviews with the representatives of the Slovenian AKIS.

Figure 1. Research design for the development of an evaluation framework for the knowledge transfer initiatives within the EU's Common Agricultural Policy (CAP).

3.1. Design of the evaluation framework and survey instruments

The EU's monitoring and evaluation framework for CAP interventions, including those aimed at facilitating knowledge and innovation transfer, establishes standardized indicators across output, result, and impact levels (EC, 2017; Regulation 2021/2115). Our evaluation framework aimed to align with and build upon these guidelines, thus, our indicator development process began by reviewing those for 2014–2022 (Regulation 834/2014) and the 2023–2027 CAP programming period (Regulation, 2021/2115), along with relevant literature assessing knowledge transfer and advisory programs (e.g. ECA, 2015; Knierim et al., 2017; Prager et al., 2017).

Further, the framework and survey instruments for measuring indicators on the result and impact level were formulated and verified through several focus group sessions. The first session in June 2021 involved three representatives from the Slovenian Ministry of Agriculture and seven from the Public Agricultural Advisory Service (JSKS). We discussed the goals of the Slovenian knowledge transfer system and the needs, challenges and suggestions for designing its evaluation framework. Next, throughout the summer of 2021, we organized four sessions to design the survey instruments with six experts in knowledge transfer and agricultural economics. This step involved formulating constructs with associated statements for each envisaged indicator. In September 2021, we reconvened representatives from the ministry and JSKS to review the final set of indicators and survey instruments. In addition, five agricultural advisors with good knowledge of the sampling population were consulted to ensure the overall quality of the survey instruments regarding statement wording and clarifications. Finally, we pre-tested the survey for assessing result indicators with 15 farmers and the survey for impact indicators with 29 farmers.

The evaluation framework in the presented pilot study was applied to the knowledge transfer initiatives promoting agri-environmental practices. However, when designing it, we aimed to make it applicable with minor adjustments across diverse knowledge transfer topics and interventions.

3.1.1. Output indicators

Output indicators were developed in line with the requirements of the EU monitoring and evaluation framework (EC, 2017; Regulation 2021/2115). The process involved categorizing the indicators of the 2014–2022 (Regulation 834/2014) and 2023–2027 CAP program period (Regulation 2021/2115) into five thematic categories: public funding for knowledge transfer activities, agricultural advisors, dissemination of knowledge through various information sources, number of advice and training for farmers, and knowledge exchange and innovation initiatives, such as the European Innovation Partnership (EIP) projects (Table 1).

Table 1. The structure of the evaluation framework for the knowledge transfer system promoting agri-environment practices within the EU's Common Agricultural Policy.

Level	Category	Indicator No.	Indicator
Output	Public funding sources	I_1a	Public expenditure for knowledge transfer (KT)
		I_1b	Share in the RDP budget
		I_2a	Public expenditure for KT for agri-environmental topics
		I_2b	Share of the RDP budget for KT
	Agricultural advisors	I_3	Number of advisors
		I_4	Advisor-to-farmer ratio
		I_5	Number of training courses for advisors
	Information sources	I_6	Number of publications and media articles
		I_7	Number of website visitors
		I_8	Social media reach (number of followers)
		I_9	Number of organized information events
	Advice and training for farmers	I_10	Number of participants in training
		I_11	Number of training courses for farmers
		I_12	Number of individual advice
		I_13	Number of demonstration activities
		I_14	Number of participatory training
	KT and innovation initiatives	I_15	Number of supported projects
Result	Normative and behavioural result indicators	I_16	Attitudes towards training participation
		I_17	Subjective norms regarding training participation
		I_18	PBC regarding training participation
		I_19	Intention for further participation in training
	Satisfaction with KT activity	I_20	Overall satisfaction with training
		I_21	Satisfaction with the organization of training
		I_22	Satisfaction with presenter of the training
		I_23	Satisfaction with the training content
Impact	Objective knowledge	I_24	Total
		I_24a	Knowledge about agri-environmental measures
		I_24b	Knowledge about nature and farmland species
		I_24c	Knowledge about agri-environmental practices
	Normative and behavioural impact indicators	I_25	Attitude towards agri-environmental (AE) practices
		I_26	Subjective norms regarding AE practices
		I_27	PBC regarding AE practices
		I_28a-d	Intentions for implementation of various AE practices
		I_29	Intentions for the market valorization of AE efforts
		I_30	Intentions for participation in AE measures
		I_31	Intentions for acquisition of additional AE knowledge

3.1.2. Result and impact indicators

When designing the survey instruments, we drew upon the Theory of Planned Behaviour (TPB) (Ajzen, 1991) to gauge result and impact indicators, aiming to capture the effects of knowledge transfer initiatives, such as shifts in normative and behavioural aspects (EC, 2022; ECA, 2015). The TPB provides a valuable lens for predicting an individual's intention to engage in a specific behaviour by considering three behavioural and social factors: the individual's positive or negative assessment of engaging in a specific behaviour (attitudes toward the behaviour), perceived social pressure to engage in the behaviour (subjective norms), and perceived ease or difficulty to perform the behaviour (perceived behavioural control, PBC) (Fishbein and Ajzen, 2010). To comprehensively evaluate knowledge transfer activities, we expanded the TPB constructs with additional theoretical elements where required (Ajzen, 1991) (Table 1).

The survey instrument for measuring result indicators (hereinafter: ‘results survey’) is provided in the Appendix, Table A1. It is designed to be conducted after knowledge transfer activities, such as training sessions, demonstration events, individual advice and participatory sessions. A departure from the standard evaluation approach is particularly evident in the part that probes respondents’ attitudes, subjective norms, PBC, and intentions towards participating in such activities. Additionally, the survey assesses respondents’ satisfaction with the activity they attended, including their satisfaction with presenters or moderators, organization and content (Gopal et al., 2021). Respondents rate statements using a 7-point Likert-type scale, ranging from strongly disagree ( = 1) to strongly agree ( = 7). The survey also gathers demographic details (i.e. respondents’ gender, age, education and household income) and farm characteristics (i.e. location, production focus, engagement in agricultural policy measures and the farm's prospects).

The survey instrument for measuring impact indicators (hereinafter: ‘impacts survey’) is presented in the Appendix, Tables A2 and A3. It is designed to be conducted on a general population of farmers, as it aims to assess the broader impacts of knowledge transfer, including its diffusion on non-beneficiaries (Bonan & Pagani, 2018; Kreft et al., 2023). A distinctive feature of this survey is a short test with multiple-choice questions, which measures farmers’ objective knowledge. In the case of the pilot presented in this study, the test was on the following topics: agri-environmental practices and their impact on the environment; agri-environmental measures (AEMs) and their requirements; and nature and farmland species identification (Appendix, Table A2). The second part of the survey includes TPB-based questions regarding implementing practices of interest, in our case, the practices promoting agri-environmental practices, using statements with a 7-point Likert-type scale response. The following part aims at quantifying the impact of knowledge transfer initiatives on farmers’ behaviour, and it includes questions regarding farmers’ intention to implement an array of agri-environmental practices, their intention to valorize agri-environmental efforts on their farm, to acquire knowledge on agri-environmental topics, and their intention to participate in the AEMs. Finally, the concluding section features demographic and farm characteristics queries similar to the results survey.

3.2. Pilot study

3.2.1. Output indicators

The output indicators were piloted by assessing their values for the Slovenian agricultural policy in 2020. For this assessment, we reviewed relevant sources of policy implementation, including annual reports detailing the implementation of the RDP in the 2014–2022 programming period (MKGP, 2021) and reports from the JSKS (JSKS, 2021).

3.2.2. Result indicators

The pilot study for result indicators was carried out between December 2021 and January 2022 within the context of mandatory annual training for farmers participating in the AECM and OF measures. The training lasted four to six hours and was provided by the Slovenian public advisory service. The content of the training encompassed the conditions and requirements for implementing measures. In addition, the OF training included topics on water and soil conservation, organic beekeeping, seed cultivation, and marketing of organic products, and the AECM training covered landscape features and climate-smart agriculture (KGZS, 2021).

For the 27 online training sessions, all participating farmers were subsequently invited to partake in the online survey. In the case of seven in-person training sessions, printed surveys were distributed by agricultural advisors upon the conclusion of the training. Participants in both settings were appropriately informed about the research's purpose, ensured anonymity, and informed about data storage procedures. Specific ethics approval for this study was not required nor obtained. Of the 8,150 farms enrolled in the AECM and OF measures, all of which were obligated to undergo the training, the survey was answered by 2,873 farmers (35.3%), comprising 2,618 (91%) online and 255 printed surveys. Before conducting statistical analyses, questionnaires with more than 20% missing data were excluded. Additionally, we excluded ‘protest’ questionnaires where all answers were identical or if respondents did not recognize negative statements. The final sample comprised 2,481 surveys (86,4% of the initial respondents’ pool) (Table 2).

Table 2. Socio-economic profile of survey respondents for result and impact indicators.

Variables	Result survey	Impact survey
Number	2481	305
Average farmer's age (years)	53.3 (SD = 12.8)	53.8 (SD = 12.5)
Average farm size (ha)	18.95 (SD = 84.7)	11.2 (SD = 12.8)
Gender – male	72.4%	66.4%
Farmers education
non or primary	11.1%	15.3%
secondary	58.5%	67.7%
higher	30.3%	17.0%
Farm type (family farm)	96.2%	99.0%
Livestock breeding	47.2%	66.7%
Household income from farming
<25%	45.4%	50.7%
>75%	16.4%	16.8%
Enrolled in AECM	83.7%	18.9%
Enrolled in Organic farming	63.9%	7.9%
Farm future
increase or maintain farm size	86.8%	84.3%
reduced or abandoned farm	13.2%	15.7%

Indicator assessment relied on calculating the median of responses within each construct, reflecting the specific indicator. Indicator values thus can range from 1 to 7, with 1 indicating a highly unfavourable state and 7 representing an exceptionally favourable one. Visualization of outcomes was facilitated using boxplots.

3.2.3. Impact indicators

The impact survey targeted a sample population comprising all farmers registered in the Slovenian Ministry of Agriculture's Register of Farm Holdings, making them eligible for CAP support. The pilot surveying was conducted at two regional public agricultural advisory offices during March and April 2022. Participation invitations were sent to all farmers who had submitted an annual application for CAP Pillar I direct payments at these specified units within the designated survey period. Due to the length and complexity of the survey, the surveying process was facilitated by trained interviewers who assisted respondents in navigating the questionnaire, ensuring reliability and coherence in their responses. In addition, the interviewers ensured that respondents were well informed about the research's objectives and anonymity guarantees and obtained voluntary consent to participate. A total of 305 farmers completed the survey (Table 2).

Similarly to the result indicators, the statistical analysis involved calculating medians within each survey construct, corresponding to each indicator. Later, these distributions were visually represented with boxplots. The assessment of the indicator on knowledge (I_24) was calculated as an average score of correct answers of all respondents; thus, the indicator can range from 0 to 10, with 0 indicating a highly unfavourable state (meaning all respondents would answer on average all questions wrongly) and 10 representing an exceptionally favourable one (meaning all respondents would answer on average all questions correctly). Descriptive and quantitative analyses were performed using Excel (Microsoft, version 10) and R software (R Core Team, version 4.0.2).

3.3. Interviews with AKIS actors

Interviews were conducted with the 13 representatives of the AKIS actors to assess the suitability and feasibility of the proposed evaluation framework. The interviewees represented government bodies, agricultural advisory services, research institutions, and evaluation firms.

The semi-structured interview began with questions regarding the role of knowledge transfer in agriculture and its current evaluation approach. Next, the interviewer ensured a comprehensive presentation of the proposed evaluation framework, followed by an assessment discussion based on the following criteria (Fitzpatrick et al., 2011; Lipicer et al., 2009): (1) relevance (alignment of acquired information with the policy context and needs); (2) usefulness of the results in practice; (3) suitability (alignment with the established legal foundations); and (4) feasibility considering the required resources and efforts.

The interviews were conducted in the spring of 2023, ranging from 50 to 110 minutes. Each recording was transcribed, followed by multiple readings and a coding process (Saldana, 2015). During this process, relevant segments were assigned a specific code. Text segments sharing the same code were grouped and differentiated from those under different codes. Next was the phase of text organization, combining semantically related text or codes. The coding process facilitated a reduction in data volume and enabled the connection of fragmented meanings of the research topic into semantically and contextually coherent wholes. The reconstruction of the obtained data into new, complete units enabled a deeper understanding of the data. The interviews were analysed using ATLAS.ti (Cleverbridge, version 8).

4. Results and discussion

4.1. Pilot study results

4.1.1. Output indicators

The developed evaluation framework aims to assess the effectiveness of knowledge transfer measures on three indicator levels: outputs, results and impacts. In total, the framework includes 31 main indicators (Table 1). On the output level, the proposal suggests annual monitoring with indicator structure remaining in the scope of guidelines of 2014–2022 (Regulation 834/2014) and the 2023–2027 CAP programming period (Regulation, 2021/2115). The results of the pilot study are presented in Appendix, Table A4.

The budget for knowledge transfer interventions during the 2014—2020 period was relatively low, consisting of only 0.82% of the Slovenian RDP funding, which is below the European average (3.63%) (Beck et al., 2021) and the target value for Slovenia (3.90%) (ENRD, 2016). A significant portion of these funds (66.40%) was spent on addressing agri-environmental topics, which reflects the importance of improving farmers’ understanding of the agri-environmental objectives. However, the limited overall budget for knowledge transfer probably limits its potential reach and effectiveness (Erjavec et al., 2018).

There is a noticeable lack of advisors and their training in the field of agri-environment. The average advisor-to-farmer ratio is one advisor per 170 farms. In comparison, other European countries’ public advisory systems exhibit a lower ratio, averaging just above 100 farmers per advisor, while private advisory organizations have a ratio of around 50 (Knierim et al., 2017). This indicates a potential strain on advisors’ capacity to provide adequate support, particularly personalized one-on-one support. Furthermore, there is a limited amount of training for advisors. For example, in 2020, only five training sessions were organized on agri-environmental topics; even then, these topics were indirectly covered. Advisors’ limited scope of expertise may present one of the important barriers (Ingram et al., 2022), as advisors may focus on administrative aspects rather than providing comprehensive support to farmers (Stupak et al., 2019).

Examining output indicators also highlights shortcomings regarding the methods employed to deliver knowledge to farmers. The training sessions were mainly in the form of relatively standard and generic lectures and, at the time of the analysis, did not include demonstration activities and other innovative knowledge transfer methods based on participatory and group approaches. This indicates a potential gap as these methods are well-known for enabling the utilization of novel and local knowledge, which is particularly valuable when addressing agri-environmental issues (Knook & Turner, 2020). The provision of individual advice represented an important approach to supporting the farmers. However, as indicated above, the question of the long-term quality of advisory service arises considering the limited capacity and lack of training of the agricultural advisors on several emerging topics in agriculture.

4.1.2. Result indicators

The evaluation of result indicators is proposed to be implemented after the knowledge transfer activities, such as training sessions, field demonstrations, individual advice or EIP projects. Evaluation should be carried out annually to monitor and assess participants’ satisfaction with the implementation of the measures and their immediate effects. To capture immediate effects effectively, the survey should be conducted promptly after the activity (Colquhoun et al., 2016), ensuring optimal participation while attendees are still available for feedback. Additionally, a follow-up assessment after a few months later can be considered to observe any changes in participants’ attitudes and intentions.

In our pilot study, we conducted a post-training assessment of the training for the farmers enrolled in the AECM and OF measures. The results further confirmed the inadequacies in the Slovenian knowledge transfer system in the agri-environmental field, which were indicated by the output indicators. While the participants expressed overall satisfaction with the training (rating 6 out of 7), the quality of the presenter (7) and the training organization (7), their satisfaction with the content of the training was moderate (4) (Figure 2). Farmers noted that the training was too standardized and offered limited diversity in topics, which were repeated annually. They also highlighted the absence of field visits and innovative knowledge transfer tools and methods. This indicates a need for more tailored and site-specific advice (Oyinbo et al., 2019) and farmers’ preference for using field demonstrations and other group-based approaches over lectures delivered to a larger audience (Šumrada et al., 2022).

Figure 2. Boxplots of the result indicators measuring farmers’ normative and behavioural aspects regarding participation in agri-environmental training and their satisfaction with them (n = 2,467).

The indicators design based on the TPB revealed a positive attitude (7) and subjective norms (6) towards acquiring agri-environmental knowledge. Farmers also exhibited a relatively high ability to attend such trainings (6) and expressed a strong intention for further participation (7). These findings are consistent with prior studies (e.g. Thomas et al., 2020) and suggest that farmers engaged in the AEMs are receptive to acquiring agri-environmental knowledge.

4.1.3. Impact indicators

Impact indicators aim to assess knowledge transfer interventions’ long-term and broader effects. The proposed evaluation methodology suggests surveying a representative sample of the general population of farmers at regular intervals, such as every three years.

Farmers in our pilot study demonstrated a medium level of agri-environmental knowledge (rating 5 out of 10). Among the topics assessed, farmers are well aware of the agri-environmental practices and their environmental impact (7). On the other hand, their knowledge of the AEMs and their requirements was relatively poor (4). This finding may partially explain their low intention to participate in the AEMs, as previous research has shown that well-informed farmers are more likely to engage in such measures (Brown et al., 2020). Farmers’ knowledge about biodiversity and species identification was slightly higher (5). However, the relatively low results of the knowledge assessment indicate the importance of providing farmers with more information and training opportunities to enhance their understanding of local biodiversity, as this can positively affect the adoption of agri-environmental practices (Czajkowski et al., 2021).

Farmers generally displayed a positive attitude (rating 6 out of 7) and positive subjective norms (6) towards the environment and the implementation of agri-environmental practices (Figure 3). Previous research shows that positive pro-environmental attitudes are typically associated with a higher likelihood of adopting agri-environmental practices (Tama et al., 2021) and enrolling in the AEMs (Brown et al., 2020). However, in our pilot study, these positive attitudes did not translate into an intention to implement such behaviours. The intention to participate in the AEMs was relatively low (rating 2 out of 7). Similarly, there was a relatively low intention to conduct voluntary practices related to biodiversity conservation on arable land (2) and grasslands (4). However, farmers demonstrated a moderate willingness to participate in the preservation of the landscape features (5) and a high intention to implement direct conservation actions (7), such as pesticide application practices that are friendly to pollinators and provide habitats for beneficial organisms (e.g. instalment of dry-stone walls, ponds, hotels for insects and bird nesting boxes).

Figure 3. Boxplots of impact indicators measuring farmers’ knowledge, attitude, and behavioural intention in the agri-environment field in 2022 (n = 305).

The relatively low interest in enrolling in the AEMs and implementing certain agri-environmental practices suggests that other behavioural factors may affect farmers’ decisions (Brown et al., 2020). Some reasons were indicated during the assessment of perceived behavioural control (PBC), which refers to individuals’ perceptions of their ability to engage in a behaviour. In the pilot study, farmers identified several constraints, including time and financial considerations and a lack of knowledge or technical skills. Furthermore, a significant constraint was the misalignment of agri-environmental practices with their existing farming practices and production goals. This aligns with previous research, indicating that farmers often prioritize production-related factors (Włodarczyk-Marciniak et al., 2020) and perceive environmental measures primarily as a means of achieving production and economic goals (Brown et al., 2020; Novak et al., 2022). Therefore, despite farmer's positive attitudes towards nature conservation, these attitudes alone are insufficient to motivate their adoption of agri-environmental practices on a large scale (Wąs et al., 2021).

Finally, the results of our pilot study showed that farmers had a moderate intention to acquire agri-environmental knowledge (rating 5 out of 7) (Figure 3). This intention was lower compared to a similar one in the result survey regarding further participation in knowledge transfer activities, which could indicate a disparity in the motivation for knowledge acquisition between farmers participating in the result survey and those in the impact survey. However, this can be partly expected as the participants in the result study had all already been enrolled in the AEMs, whereas the sample in the impact survey intended to represent the general farming population. Farmers also displayed a relatively high level of intention (rating 6) to valorize the agri-environmental practices on their farms, e.g. by using their efforts to improve the market visibility of their farms or to achieve higher product prices. These highlight farmers’ potential economic and market-related motivations in embracing sustainable practices and their recognition of the added value they may bring.

The results of the evaluation pilot study shed light on the complexity of addressing farmers’ educational needs in the agri-environmental field in Slovenia. Farmers’ knowledge of environmentally sustainable practices, benefits, and supporting policy instruments can differ. Therefore, it is important to develop tailored and targeted interventions that consider specific knowledge gaps observed among different groups of farmers, regions, farming systems and contexts (Thomas et al., 2020). This calls for using a diverse range of knowledge transfer approaches and methods. In particular, incorporating participatory and demonstration activities seems promising (Alif et al., 2024; OECD, 2015) since these approaches can foster group learning and encourage the integration of local knowledge (Knook & Turner, 2020). In addition, the pilot shows the importance of providing training not only to farmers but also to advisors to enhance their capacity to deliver relevant and appropriate advice (Ingram et al., 2022; OECD, 2015). It is thus important to recognize the significance of knowledge transfer activities within the agricultural policy and allocate sufficient budgetary resources to support them (Erjavec et al., 2018).

4.2. Assessment of the evaluation framework

4.2.1. Relevance, usefulness and suitability

The interviews with Slovenian representatives of AKIS showed significant demand for a more comprehensive and high-quality evaluation framework for assessing knowledge transfer interventions in agriculture. They have underscored the inadequacy of the current evaluation practices to support evidence-based policy-making, which is required by EU regulation for the CAP strategic plans (Regulation, 2021/2115). Specifically, the existing indicators are loosely defined and primarily focus on the output levels (EC, 2022). As a result, respondents recognized that the planning of the knowledge transfer activities heavily relies on the subjective beliefs of the policymakers due to the lack of objective resources for providing insight into the actual effects (Weiss, 1999). This issue was not only identified in the context of knowledge transfer but also in other areas of the CAP (Erjavec & Rac, 2023). However, respondents have emphasized that the knowledge transfer domain seems particularly marginalized.

The interviewees agreed that the proposed evaluation framework was relevant and that its results reflect the current situation relatively accurately, highlighting key shortcomings and enabling a better understanding of the policy impacts:

If I had been given these indicators beforehand to assess them according to my own judgment, I would have probably assessed them very similarly to the results. […] From this perspective, I can say that these indicators seem reliable to me. If I, who have been working with these people in this field for so long, have this feeling, and the indicators confirm it, then it means that the tool is useful and that you have hit the mark. [Interview 6, knowledge transfer expert]

The interviewees acknowledged the practical value of the proposed indicators for informing future policy planning and evidence-based decision-making. They emphasized that indicators are instrumental for the effective management of the knowledge transfer system, as well as for the design of diverse approaches and methods. From the pilot results, they identified long-term needs, such as building advisory services capacity and allocating adequate budgetary resources to support knowledge transfer initiatives. Moreover, interviewees recognized the adaptable nature of the framework and the survey instruments, signifying their potential to assess knowledge transfer measures across various agricultural fields, such as digitalization and farm management.

The interview results also highlighted certain limitations in the proposed framework and provided potential suggestions for improvement. Additional indicators were proposed by the interviewees across all three indicator levels. At the output level, the interviewees recommended adding the number of specialized advisors for different knowledge transfer fields of interest, the ratio of advisors to farms based on the farm size and structure, and the regional distribution of the implemented knowledge transfer activities. At the results level, they suggested offering farmers the option to express their interests in the potential training topics. Finally, for the impact level, the indicators assessing farmers’ actual behaviour should be added, e.g. by monitoring farmers’ enrolment in various groups of the AEMs. In addition, they would add an assessment of several context-related indicators, e.g. the farm bird index and conservation status of habitat types.

When considering the suitability of the evaluation framework, adherence to legal foundations was considered (Lipicer et al., 2009), especially the requirements of the relevant EU regulations and guidelines (e.g. EU, 2017; Regulation, 2021/2115). The interviewees agreed that the proposal demonstrates adequacy and flexibility with these. Moreover, they recognized that it surpasses the reporting needs of the European Commission, which, in their opinion, are less exact and, as already pointed out, guidelines lack adequate indicators of the result and impact level. However, the adaptive nature of these seems to ensure the possibility of integrating data from the proposed framework into the reporting needs.

4.2.2. Feasibility

Despite the indicated need for a more comprehensive assessment of the impact of knowledge transfer intervention, the interviewees identified several factors that could hinder the feasibility of the proposed framework. Firstly, they raised the issue of the effective involvement of the farmers in data collection, which is a known challenge when using questionnaires as an evaluation tool (Coon et al., 2020). Some interviewees questioned the credibility of survey data, as farmers may tend to provide pleasing and positive responses. This concern is especially relevant when surveys are conducted by government institutions or advisory services (Mase et al., 2015) because respondents may worry about the anonymity of their responses. Interviewees also noted that farmers often face a heavy survey burden, which can discourage their participation or could result in superficial and quick answers due to time constraints or survey fatigue (Coon et al., 2020).

To address the data collection challenges, the interviewees put forward several suggestions. Clear communication with farmers is crucial because it might ensure they understand the purpose of the survey and recognize the importance of their participation. This is particularly important for environmental topics in rural contexts, as they are often assumed to be less relevant (Brook et al., 2003; Coon et al., 2020). When designing questionnaires, it is important to consider a suitable structure that farmers can easily comprehend (Quinn, 2010). Suggestions included providing farmers with valuable content, including feedback on the survey results or other non-monetary compensations, such as additional individual advice. Finally, it is essential for evaluators or survey administrators to address farmers’ potential concerns about anonymity by clearly communicating the data management procedures in line with the established principles of ethical research. Potential biases could be further reduced by employing external research institutions or evaluation agencies to administer data collection.

Data availability was recognized to be another hindering factor (Castaño et al., 2019). During our data collection for the pilot assessment of output indicators, we encountered a lack of consistent and systematic reporting, relying on information from previous monitoring and evaluation sources. When examining reports from different time frames, we found inconsistent indicator names and unclear and variable data presentation. These challenges were also confirmed through expert interviews. Furthermore, a similar situation was observed when evaluating the effects of CAP cross-compliance, where inconsistencies were apparent between ex-ante and ex-post evaluations due to the involvement of multiple actors with different criteria, methods, and interpretations during the evaluation process (Bozzini & Hunt, 2015). Therefore, it is essential to ensure consistent reporting following transparent evaluation criteria to enhance the credibility and effectiveness of future evaluations.

Weakly defined intervention logic can also significantly impact evaluations (Donaldson et al., 2019). The interviewees highlighted that in the Slovenian knowledge transfer system, the issue lies in the diverse interests of stakeholders and the limited availability of data and evaluations, which results in unclear or unspecified needs. Consequently, interventions targeted numerous and overly broad objectives (ECA, 2017) with weak connections to the intended outcomes, which is a significant hurdle when striving for a comprehensive evaluation design (Lovec et al., 2020).

Limited human resources can present another significant constraint (Cagliero et al., 2021; El Benni et al., 2023). In the context of the CAP, national managing authorities, usually ministries, are responsible for policy design and implementation. However, the implementation is entrusted to paying agencies, while private consulting groups or research institutions are typically responsible for the evaluations (Castaño et al., 2019). Based on the interviews, limited human resources manifest among those bodies on various levels and could hinder the implementation of a proposed evaluation framework. Firstly, only a few individuals are often engaged in the evaluation process (Erjavec & Rac, 2023). For example, in Slovenia, only one person is partially dedicated to planning evaluations at the authorized agriculture ministry. Moreover, there is a limited number of institutions involved in conducting evaluations. The issue is further emphasized by the limited coordination between those actors, which can also negatively impact the quality of evaluations (Castaño et al., 2019):

When we started conducting [name of the evaluation], it became evident that there was a lack of understanding of our role as evaluators. It felt like we were hitting a wall right from the beginning. We realised that people at the ministry had previously undergone an audit, so they treated us like auditors and provided minimal responses without putting in any effort to contribute to the success of the evaluation. […] Most of the time, they perceived evaluations as a mere obligation to meet the requirements of the European Commission. They were also somewhat defensive if the results did not meet their expectations. The atmosphere became very hostile, as if everything we proposed was wrong. [Interview 13, evaluator]

As indicated by the above quote, the next challenge can be a lack of knowledge and understanding of the evaluation process among the actors involved, which is particularly evident among the officials (Erjavec & Rac, 2023). They often perceive evaluation as an additional administrative burden and are sceptical towards the results. Similarly, evaluators often lack the necessary competencies and expertise in the specific field being evaluated, which hampers their ability to identify crucial mechanisms of change and relevant contextual factors (Van Belle et al., 2010). Considering the already limited workforce, comprehensive evaluations can thus logically represent an overload for the officials and other actors involved (ECA, 2017).

According to the interviewees, the proposed evaluation methodology is ambitious compared to current approach (Regulation, 2021/2115) and it enables the acquisition of data and insights crucial for real-time and evidence-based policy planning. This is essential for addressing current issues of formalistic and poorly designed intervention logic (Erjavec & Rac, 2023), particularly considering the European Commissions’ priority for data-driven and targeted strategic planning of CAP (ECA, 2017). However, interviewees expressed concerns about the practicality and systematic implementation of the methodology if applied extensively across multiple policy domains. They considered the methodology feasible for in-depth thematic evaluations of the specific policy domains.

This is the cost of evidence-based policy. The funding and resources are not a problem if we do it only in the field of knowledge transfer, but this is just one of the 35–40 topics. If all areas have to be evaluated similarly, the costs become a serious issue: financially, in terms of human resources, and in other aspects. But if you don't do it, then it's not evidence-based. Otherwise, you'll be throwing millions of euros out of the window and not knowing what you've achieved. [interview 8, researcher]

The interviewees’ reflections raise questions about the system's readiness to adopt an evaluation burden that exceeds the legally envisaged requirements. This aligns with findings from other studies (e.g. Castaño et al., 2019; Mayne, 2009), which suggest that factors hindering the development of such evaluations may stem from a limited evaluation culture. This refers to the evaluation maturity or, in other words, the level of a supportive environment for promoting evaluations as an essential tool for informing the policy (Dahler-Larsen & Boodhoo, 2019; Mayne, 2009). Based on the interviews, the overall evaluation culture in Slovenian agricultural policy, particularly in the knowledge transfer field, is relatively weak. The evaluation process is often viewed as a mere formality to justify the allocation of public funds rather than a tool for knowledge generation and learning:

In Slovenia, these evaluations are often carried out solely because the EU demands and funds them. […] The [ministry] focuses on trying to transpose the EU's requirements precisely into Slovenian legislation. The main logic remains to obtain as much available funding as possible. However, they do not engage in the more challenging task of ensuring that the European policy framework can genuinely benefit Slovenia and develop its own path. [interview 10, researcher]

A similar situation is mirrored in at least some other Member States (Cagliero et al., 2021; Petit, 2019). Therefore, further research is needed at the Member State level to better understand the evaluation culture, systemic support, and the roles of the policymakers and stakeholders to ensure a proper transition towards evidence-based CAP. As suggested in the interviews, the factors listed above can significantly influence the successful implementation of the evaluations and utilization of their results. Hence, it is crucial to tackle these factors to ensure a well-informed and effective evaluation system.

5. Conclusions

Our study aimed to construct a comprehensive evaluation framework to assess the effectiveness of knowledge transfer and innovation interventions in agriculture by using a theory-based approach. Specifically, our goal was to fill the gaps in the CAP monitoring and evaluation framework by creating indicators that could capture the broader impacts of interventions, including shifts in normative and behavioural aspects resulting from knowledge transfer initiatives. This objective is particularly significant considering the current priorities of the CAP, which emphasize the importance of better policy targeting and results orientation (Beck et al., 2021; ECA, 2017).

Our extended evaluation framework adhered to a three-level hierarchy of indicators to ensure alignment with the established common framework for monitoring and evaluating CAP's Rural Development Programmes (Regulation, 2021/2115). At the output indicator level, our study contributed to the existing framework by systematically categorizing indicators from the previous and current monitoring and evaluation systems. Moving to the result and impact levels of the indicators, we extended the existing system by introducing new indicators and appropriate survey instruments. These were built upon the Theory of Planned Behaviour (Ajzen, 1991) and supplemented with additional theoretical components to assess knowledge transfer activities. While the evaluation framework was explicitly adapted for agri-environmental themes, the skeleton of the evaluation framework is designed to be versatile and applicable across diverse knowledge transfer activities and interventions.

The application of the TPB as a part of the evaluation framework constitutes a novel approach in its utilization, especially in the domain of knowledge transfer in agriculture (Hall et al., 2019; Sok et al., 2021). Our study has demonstrated its applicability as a valuable tool within this realm, as it has allowed us to gain insightful perspectives on the effectiveness of knowledge transfer interventions, particularly in assessing the anticipated changes in the farmers’ values and their behaviour. Furthermore, by engaging farmers, i.e. beneficiaries of the interventions, into the evaluation process and the ensuing policy cycle represents an additional strength of the study (ECA, 2015). Active engagement of the beneficiaries may reframe the policy as less exogenous for farmers and can grant them a sense of ownership, thus empowering them to contribute to policy formulation. This is vital for the long-term strengthening of sustainability and effectiveness of agricultural policies (Luján Soto et al., 2021). Moreover, using this evaluation methodology also enables the expedited dissemination of such information. This is crucial because delayed and ad-hoc evaluations often hinder acquiring high-quality evaluation results and their timely integration into the policy planning process (Bozzini & Hunt, 2015; ECA, 2017).

While the proposed evaluation framework offers several benefits, its practical implementation can encounter distinctive challenges. A primary concern revolves around the necessity for unbiased, continuous, and prolonged data collection – a task susceptible to numerous challenges. Insights gained from interviews with the AKIS actors indicated that these might include ineffective recruitment of farmers, survey fatigue and a lack of sufficient financial and human resources for ensuring data collection over multiple CAP programming periods. Furthermore, the need to establish the prevalent evaluation culture among the stakeholders and the political commitment to evidence-based policy-making was emphasized, thus elevating evaluation from a mere resource justification tool to a means of knowledge generation, which can effectively uncover policy impacts and thus inform decision-making.

This study contributes to research on education and knowledge transfer policy interventions where longitudinal studies are needed to understand the sustained impacts and their potential to accelerate behavioural change over time. Furthermore, cross-national comparative analyses could provide insights into the effectiveness of these interventions within diverse cultural and policy contexts. Lastly, integrating qualitative methods could deepen our understanding of the underlying factors that shape farmers’ attitudes and behaviour (Prager et al., 2017). These avenues of research can enrich our comprehension of the intricate dynamics surrounding knowledge transfer and facilitate evidence-based policies and practices that bolster agricultural sustainability and innovation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This study was supported by the Slovenian Research Agency (ARIS) under [grant number P4-0022 (B)]; and the Ministry of Agriculture, Forestry and Food under [grant number CRP V4-2020]. The authors would like to thank the Slovenian Research Agency (ARIS) for co-funding this publication and the Slovenian public advisory service (Javna služba kmetijskega svetovanja) for their help in data collection.