Developing electronic tools for continuing professional development for Finnish physicians: a usability study in a primary health care setting

Abstract

In Finland, Continuing Professional Development (CPD) is voluntary with an ethical and legal obligation for physicians. As medical knowledge is expanding and health care delivery becoming more complex, there is a need to promote physicians’ continuing professional development throughout their careers.

The Taitoni-platform is a personal web tool designed to manage and support physicians’ CPD through both formal and informal learning activities. The tool is accompanied by and linked to an interactive national Continuing Medical Education (CME) calendar and contains data of individual qualifications and workplace history. Platform use and usability are being tested at Helsinki Health Centre. The users included both general practitioners and hospital specialists. In the future, Taitoni may provide data for a possible national revalidation process.

Keywords::

• competence
• continuing medical education
• telephony

Introduction

Physicians have been trusted to be committed to ethical practice and continuing professional development (CPD) since Hippocrates.¹ The development of treatments, ever improving technologies and more complex service systems with economic constraints set high demands on physicians’ performance and expertise.² Physicians’ expertise consists of multiple competencies that are described as an individual's ability to acquire skills in a given area. In medicine these include clinical skills, scientific knowledge and moral judgement.^3,4 Medical skills are accompanied by a vast number of other competencies such as economic reasoning, communication and computing skills. In the European Union (EU), there is regulation of mutual recognition of professional qualifications in health care.⁵ Seven of ten countries in the EU have some obligatory element of CPD, mainly calculation of attendance in continuing medical education (CME) in hours per year.⁶ Society puts its trust in physicians, but equally the physicians need to demonstrate their trustworthiness. So far in Finland, CPD has been voluntary, but the physicians have an ethical and legal commitment to maintain their skills and as in many other countries, the societal climate is becoming more restrictive.⁷ Currently the Finnish Medical Association and the Finnish Medical Society, Duodecim–Swedish Medical Society have set national targets for supporting physicians’ CPD by collaborating in building a comprehensive assessment tool for all Finnish physicians.

The challenge in assessment is the constant strain between practicality and comprehensiveness. Practicality, which arises from time and management constraints, prefers a broader framework of competencies with less strict subcompetencies whereas comprehensiveness is followed by relatively low cost-effectiveness.⁸ However, so far comprehensive assessment of effectiveness in the workplace has been relatively low.⁹ Research from other professional fields indicates that core competencies could include three to four items¹⁰ and our previous work supports this.¹¹ However, we did choose the seven CanMEDS roles for core competencies since they are broadly used in the medical community.¹²

This article describes the process of developing and piloting the web-based CPD-assessment tool in Finland conducted by Pro Medico. Figures 1 to 4 show screenshots of the Taitoni-tool interface. Physicians can record, assess and represent their learning aims and CPD activities outwith their work environment. Moreover, the tool will propose CME events based on individual preferences and needs of the users. The tool has a curriculum vitae (CV) function with linkage to the national physician licensing system through the Finnish Medical Association. In the future, the tool could be linked with competence management systems within the health care organisations and later it can be used to facilitate revalidation in Finland, if necessary. There is no previous tool in use for this purpose in Finland.

Figure 1. Front page.

The main questions include the following:

• Are Finnish physicians willing to use the CPD tool?

• How is the tool used?

• What are the experienced advantages and disadvantages for physicians in using this tool?

Methods

CPD tool

The development of the CPD tool was based on four frameworks: 1) the model for describing competencies in medical environments was adapted from Canada, the CanMEDS.¹³ 2) The structure for learning goals and learning activity categories was derived from self-directed and goal-orientated adult learning¹⁴ and from the nature of learning at the workplace.¹⁵ 3) Intention to act assessment was adapted from behavioural change theories³ and 4) the usability of the actual application was tested with multiple methods focusing on different aspects of usability.¹⁶ CanMEDS vocabulary was translated and meanings of the words were assessed in a separate project.⁹ Tool content and functionality were tested with non-participating physicians in each phase before launching them to the pilot population. The name of the tool is Taitoni, which means ‘my skills’ in Finnish (Figure 1). There are four main domains in the tool: learning goals (Figure 2), learning events (Figure 3), individual reports (Figure 4) and CME calendar; at a later phase there is the possibility of producing a CV.

Figure 2. Learning aims.

Figure 3. List of learning events.

Figure 4. Reports of learning events.

Learning was recorded with predefined formative fields with an option for additive informal notes. The types of learning events were presented as follows: 1) Educational event either at or away from the workplace, 2) group work, 3) e-Learning, 4) self-directed learning, 5) process development and 6) feedback and audits. In all learning activities, physicians were asked to assess their learning session by answering the following: a) What had they learned with formatted sentences that covered acquiring new knowledge or skills, assessing current behaviour and intentions to future actions? b) Which CanMEDS roles were covered by the learning event? 3) How applicable were the learned items to their work with Likert-scale from 1 (weak) to 10 (good)? 4) How their learning met their individual learning goals? The answers to these are described in more detail later.

Goal setting supports task-orientated learning. Individuals are expected to record their individual learning goals, which can be anything from a single learning skill (e.g. stitching) to lifelong tasks (e.g. improving communication). However, in our tutorial we propose concrete and achievable goals which can vary interchangeably: at the beginning, on-going or completed. Physicians are asked to choose CanMEDS competence areas and keywords for each goal. Later, when recording learning events, these individual goals will appear on the event recording site and on the reporting site as well. This enables physicians to see whether learning efforts were linked with any previously set goals.

Pro Medico runs the national CME calendar. Physicians in the pilot are able to seek training from the open electronic CME calendar within the Taitoni tool and then directly import these details into their individual portfolios. Registration for events is not included, but later in development it may contain the attendance certificate imported from the current system run by the Finnish Medical Society, Duodecim. The CME calendar includes national and regional training programme opportunities, and also the Helsinki Health Centre in-house training, all of which are open to every physician working there. This feature was added due to feedback in the middle of the pilot phase. Finnish Medical Subheadings (FinMesh) are used as keywords, accompanied by predictive text input technology for keywords used in linkage to the CME calendar to personalise CME proposals.

The technology of the tool is being developed incrementally and each new feature is tested with actual users. The model for functions of the tool was adapted from the Royal College of Physicians and Surgeons of Canada, MAINPORT,¹⁵ which has been used for CPD planning, documentation and management for specialists in Canada since 2008. During the pilot stage, the participants have a CPD learning event-recording tool and they will have extensions for the electronic CME calendar and for producing a CV.

Setting and participants

The study was conducted at the Helsinki Health Centre, which provides primary health care services for about 600,000 Helsinki citizens and, in addition, specialist medical care services in the fields of psychiatry, internal medicine, geriatrics, neurology and psychiatry. Other specialised health care services are purchased from the Hospital District of Helsinki and Uusimaa. The Health Centre hosts five departments that are responsible for clinical operations: primary health care, oral health care, home nursing, psychiatry and the City Hospital. It has over 90 service facilities and employs about 650 physicians. Pro Medico provided tools, materials and project management. There were representatives from the Helsinki Health centre in the working group during the pilot.

Attendance at the pilot was voluntary without reimbursement and all participants could remain users after the pilot. All physicians working either at the primary health care or at the hospital were invited to join the pilot. Initial invitation to attend was presented to all Primary Care physicians at their annual training event. Chief physicians from primary health care and from the hospital service were equally given more information. There were about 350 physicians eligible at the health stations and about 300 physicians eligible at the hospitals. Recruitment was done with electronic letters from the Director of the Health Centre and Director of Pro Medico with one reminder. The Chief physicians at the primary health care and at the hospitals encouraged the physicians to participate.

A total of 80 volunteers were enrolled: 53 from the health stations from 17 locations and 27 from the City Hospital. Of the participants, 70 (88%) were women and their mean age was 41.8 years (SD = 10.1 years) (Table 1). The pilot started in mid-October 2011 and ran for six months (until mid-April 2012).

Table 1. Participants of the pilot in Helsinki Health Centre for the Taitoni web tool.

		Age		Gender
	n	Mean	SD	n female	% female
Workplace
Hospital	27	43.4	9.4	23	85
Health centre	50	40.8	10.3	45	90
Other	3	44.3	15.4	2	67
Status
Specialist	38	47.5	7.1	34	90
Non-specialist	42	36.7	9.7	36	86
Total	80	41.8	10.1	70	88

Physicians in the pilot received a letter explaining the project at the beginning of the pilot. They were encouraged to ask questions by either phone or e-mail from a support person in Pro Medico. Health centre units and hospitals were visited for one-hour tutorials on request, and eleven tutorials (65% of locations) were arranged in their workplaces. The tutorials included a presentation on the aim of the tool, content and usage.

Data collected with Taitoni

Taitoni provided unidentifiable formative data on learning events, learning descriptors, applicability of events for individuals work, learning goals and CanMEDS roles for learning events and goals.

Usability testing

The usability of Taitoni was tested using different user-centred design methods. Feedback about use of the tool was requested from the users with a feedback box in Taitoni. In addition, experiences and comments were collected during tutorials at health stations and hospitals. An electronic questionnaire including the 10-item System Usability Scale³ was used to measure the usability. The questionnaire was administered as an electronic questionnaire after one month of usage in November 2011 and after six months of usage in May 2012. One reminder was sent to non-respondents. The response rate in November was 70% and that in May was 54%. A separate feedback session was arranged for all the pilot physicians after the pilot period with a presentation of the data collected from the pilot group.

Results

There were 80 physicians who enrolled voluntarily for the study. Of them 64 visited the Taitoni tool site during the six-month period and 45 inserted learning events. The total number of recorded learning events was 354. The frequency of site visits was stable excluding low frequencies at Christmas; average visits per week were 53. Of those who inserted learning events, the mean number of events was 7.9 per user (SD = 6.9, range 1–29). The details of different learning events are presented in Table 2. Of educational events (n = 296), one third were held at the workplace (n = 94, 32%) and two thirds were held elsewhere (n = 125, 42%), but for 26% (n = 77) of the events, the location was not stated. Usefulness of the learning event was evaluated as a single item: ‘Usefulness for my work’ with a Likert-scale from 1 (weak) to 10 (good). Altogether 349 events (99%) were evaluated according to usefulness. The mean was 7.3 (SD 2.5).

Table 2. Educational events inserted in Taitoni during six-month pilot phase.

Type of event	n	%
Educational event	296	84%
Group work	18	5%
e-Learning	0	0%
Learning project	28	8%
Process development	8	2%
Feedback and evaluation	4	1%
Total	354	100%

Altogether 120 (mean 2.9) learning goals were listed. At least one learning goal was inserted by 42 users (53%,). Both learning goals and learning events required the selection of CanMEDS roles, and 315 (80%) of learning events and 97 (81%) of learning goals fulfilled this criterion. The details of CanMEDS roles are presented in Table 3.

Table 3. CanMEDS roles related to learning events and learning goals inserted in Taitoni.

	Learning events		Learning goals
CanMEDS role	n	%	n	%
Medical expert	248	32%	137	25%
Communicator	93	12%	67	12%
Collaborator	101	13%	70	13%
Manager	116	15%	77	14%
Health Advocate	63	8%	49	9%
Scholar	67	9%	79	14%
Professional	94	12%	73	13%
		100%		100%

Note. Several CanMEDS roles could be listed to relate to a single learning event or goal.

The 10-item System Usability Scale was sent to the users after one month. By November, 80% of the physicians had visited the Taitoni site and the usability index was 54, when the aim was to have points over 60.¹⁶ By May, in the follow-up questionnaire, the usability index was 53.

Feedback from the users was mainly positive. The formative structure of fields required for learning events and learning aims was both criticised and appreciated. The physicians reported difficulties in setting learning goals as they found that their personal goals were either too broad or too narrow to report. Technical problems came up and were solved as and when identified. In particular older browser versions were problematic. Initiatives raised from pilot users were adopted during the pilot, such as the extension of free notes and the development of an in-house CME calendar.

Discussion

Finnish medical societies want to ensure that physicians can sustain their competencies and that support for CPD/CME will be available. This development project has demonstrated that 26% of clinicians are willing to engage in a non-compulsory reporting system for CPD at workplace level but only 13% actually used the tool. This result is comparable with the well-known theory of diffusion of innovations by Rogers, where about one-tenth of the population are early adopters of technical innovations.¹⁷ Key features for implementing new tools are listed by Rogers: the tool needs to provide relative advantage and be simple, accessible, effective and compatible with everyday routines.¹⁷ In the pilot study, the tool was accessible and simple, but in order to reach larger physician populations, it needs to generate practical benefits for physicians and also to be compatible with clinical datasets e.g. decision support system feedback. Alternatively the use of the tool may be mandatory. Time investment for CPD has to be seen as worthwhile.

Among the physicians using the CPD tool, the average number of inserted events for six months was 7.9 and the average number of CME events away from the worksite in six months was 2.7 (estimating 5.4 days per year), which is close to primary care physicians’ self-reported CME days per year (6.6 days).¹⁸ It was interesting to see that CanMEDS competence areas were distributed rather evenly and there was an association of competency distribution between goals and events.

Usability test results were below the recommended level of 60 points, which was slightly disappointing. Major difficulties appeared in relation to FinMeSH keywords. In the future FinMeSH keywords will link physicians’ individual tasks to the CME calendar, so that the calendar can customise the CME training offered to meet individual needs. The lesson to be learned, again according to Rogers, was that an innovation needs to be assimilated into a physician's routine. Also in the future, FinMeSH will enable more sophisticated and less distractive functions; thus increasing usability.

The CPD tool created a dataset from which data could be condensed from individual data to physician population figures and averages. Later it might be possible to produce benchmark data to allow individuals to compare their learning events or goal setting with their colleagues, in different specialities. Prior to this pilot, a website was produced for physicians based on an annual physician questionnaire to allow them to compare their learning needs and support for learning, and this was found useful. In this pilot, we produced a report with figures of the total group and sub-groups (e.g. hospital vs. primary care), which was presented to the pilot group and the chief physicians at Helsinki Health Centre. In discussions with the chief physicians, we agreed that organisational competence development would profit from many-sided data on CME activities. However, more detailed data would be needed for managing competences. In the future, electronic CPD tools should provide this type of data not only to physicians, but also to other health care personnel. Learning is an investment for quality and deserves to be recognised.

Conclusion

This study provided information and experience on the process of developing and implementing a new electronic CPD tool at the work environment for physicians in Helsinki, Finland. The uptake and usage of this innovation involving electronic web-based technology was encouraging. However, further work is necessary to identify real professional benefits.

Declaration of interest

Transparency

This paper was produced in the development project of Pro Medico. Pro Medico is an independent association funded by the Finnish Medical Association, The Finnish Medical Society Duodecim and The Finnish Swedish speaking Medical Society, FLS. Helsinki Health Centre physicians contributed to the pilot as a part of their everyday work.

Author Disclosure

KP, TL, PS and AH have disclosed that they have no relevant financial relationships.

Peer Reviewers Disclosure

Peer Reviewers 1 and 2 have disclosed that they have no relevant financial relationships.

Acknowledgments

Physicians of the Helsinki City Health Centre participating and chief physicians, Antti Iivanainen, Ari Aimolahti, Leea Muhonen, from Medical Society Duodecim Peter Eriksson, Ilkka Kunnamo, Juha-Pekka Turunen, Finnish Medical Association Teemu Linna and Data Prisma Ltd Jukka-Pekka Kurki.