What matters while assessing quality of social services? Stakeholders’ perspective in Czechia

ABSTRACT

Quality of public services including social services is an issue frequently discussed by policy makers, service providers, and by those who represent service users. The aim of this study was to explore how stakeholders perceive quality of social services, with a particular focus on (1) what did participants rate as important; (2) what is the relative importance of each domain and how does that differ across stakeholder groups and (3) does importance vary by participant characteristics. A specially designed questionnaire was completed by 217 service providers, by 249 public administration representatives and 205 service users of residential care and in-home support. The subjective quality of life of service users was rated as the most important indicator of service quality by all three stakeholder groups. Particularly important were items that related to the nature of the relationships and interactions between staff and service users. There were some differences between stakeholder groups and also by respondent characteristics – public administration respondents, older service users and providers of residential care were more likely to rate health care as more important than other respondents. Implications for how quality is measured are discussed.

ABSTRAKT

Kvalita veřejných služeb, včetně sociálních služeb je tématem často diskutovaným mezi tvůrci sociálních politik, poskytovateli sociálních služeb a těmi, kteří zastupují uživatele služeb. Cílem této studie bylo zjistit, jak respondenti – zástupci veřejné správy, poskytovatelé sociálních služeb a uživatelé vnímají kvalitu sociálních služeb, se zvláštním zaměřením na to: (1) co hodnotí respondenti jako důležité; (2) jaká je relativní vnímaná důležitost každé domény a jak se tato vnímaná důležitost liší mezi kategoriemi respondentů a (3) jak se tato důležitost liší v závislosti na charakteristikách respondentů. Speciálně navržený dotazník vyplnilo 217 poskytovatelů služeb, 249 zástupců veřejné správy a 205 uživatelů sociálních služeb. Subjektivní kvalita života uživatelů služeb byla všemi třemi kategoriemi respondentů hodnocena jako nejdůležitější ukazatel kvality služeb. Obzvláště důležité pak byly položky, které se týkaly povahy vztahů a interakcí mezi zaměstnanci a uživateli služeb. Existovaly určité rozdíly mezi kategoriemi respondentů a také podle jejich charakteristik. Respondenti veřejné správy, starší uživatelé služeb a poskytovatelé rezidenční péče hodnotili zdravotní péči mnohem důležitěji než ostatní respondenti. V článku jsou diskutovány také některé přístupy k hodnocení kvality.

KEYWORDS:

• Quality
• assessment
• social services
• social inclusion
• quality of life

KLÍČOVÁ SLOVA:

• kvalita
• hodnocení
• sociální služby
• sociální začleňování
• kvalita života

Introduction

Social care services endeavour to maximise the quality life of those who are more or less dependent on support provided by another person. The aim of social services is not to cure the impairment but to compensate the person for the impact of their impairments. How to assess ‘the impact’, processes and outcomes of social care services has been discussed by researchers and policy makers for more than twenty years, often linked to reforms in social policies, strategic planning programmes or their implementation.

Although the aims and nature of Government social care reforms in different countries have differed, there have been some common patterns, such as a trend towards a purchaser/provider split and diversification of the provider market to give service users more choice and control over their support (Šiška & Beadle Brown, 2020).

Secondly, the economic crisis followed by retrenchments in public funding has raised questions for policy makers on how to assess outcomes of interventions aiming to generate what Qureshi and Nicholas (2001) call ‘change outcomes’ and often more challenging ‘maintenance outcomes’. Difficulty in measuring social care outcomes is also inevitably related to different perspectives and judgments of stakeholders on quality and how it is defined as well as to external factors such as cultural, social and political forces and how services have historically been organised. In countries from the so-called former Eastern bloc such external structures changed as part of fundamental political and economic transformation which happened in the nineties. However, the agenda of community-based services as opposed to institutional care emerged only relatively recently in Czechia. Similarly, the issue of assessing quality of social care services has been challenged by stakeholders a decade after it came into force. The newly adopted Act on Social Services 2006 (2006 Act) came into force with high and relatively unified expectations shared by stakeholders such as service providers, policy makers, public administration same as those represented service users such as disabled peoples’ organisations. The new legal provision was seen as a driver for change towards empowering service users and improving quality of social services in general (MoLSA, 2005). In the rest of this introduction, we will briefly outline developments in social services in Czechia considering political transformation in the nineties and during the first decade of the new millennium. We will then present what is already known about research on defining and measuring service quality before going on to present the findings from the current study.

The 2006 Act was drawn up in an attempt to compensate for the limited legislative basis for social services, which had been set up in the 1980s and did not reflect the human rights agenda such as the right to self-determination and the right to community participation. The 2006 Act states that the general aim of services is to maintain the highest possible quality and dignity in the lives of service users. Social services which fall into the social care category comprise counselling, personal assistance, sheltered living support, and larger institutional settings for the elderly, and for persons with disabilities. The largest group of beneficiaries are older people, then people with disabilities, followed by families with children and those living on the fringes of society for various reasons. The providers include municipalities and regions, non-governmental organisations, churches and private organisations. The MoLSA holds responsibility for quality assurance.

The 2006 Act enables service users to be directly involved in purchasing their service from a variety of providers including independent for profit and not-for-profit providers. In addition, service users should be involved in the evaluation of the service delivery and protected from violation of their rights. These assumptions had been seen as drivers of change for cultivating quality of care and for deinstitutionalisation. However, there remains a gap between the rhetoric of the 2006 Act and the reality of the lives of people with disabilities. Real choice about how and where people receive their service remains limited. People with disabilities and their families are often forced to opt for large residential provision, not because they see particular merits in this option but because alternatives are not available (Šiška & Beadle-Brown, 2011). Despite significant improvement, for example, in living conditions of residential services, the real impact of the quality of care inspection system as the key catalyst for change has been questioned (Šiška & Čáslava; in press; Kocman & Paleček, 2013).

One issue is in the conceptualisation of quality of care. This concept is seen as complex and made up of different elements. Donabedian (1966, 1980) suggested that the key indicator of service quality should be the outcomes experienced by those supported. Although the conceptualisation of outcomes and quality of life is a key area of research especially in the field of disability (e.g. Schalock et al., 2002), in policy and practice, conceptualisation and agreement on what constitutes good outcomes and quality of life is still a subject of debate or lack of clarity. The concept of Quality of life (QoL) also appears in other areas such as in economics, medicine and the social sciences and is measured in a variety of ways, including both subjective (e.f. happiness, satisfaction, feeling well and safe) and objective measures (such as employment status, accessibility, income, meaningful occupation etc). However, in order to deliver and understand outcomes such as quality of life Donabedian suggested that it is essential to also think about processes (i.e. the practices used by those providing support) and the structures (e.g. policies and procedures, physical environment, number of staff, resources, skills and attitudes of workers, training etc.) present in the system. Although there is very little research on how care quality standards and quality assurance processes in social services have been developed, most of those for which there is information available have worked on the basis of examining structures, processes and outcomes, although with varying emphasis on each element (Malley & Fernández, 2010).

Such an approach was also applied to the care quality standards in Czechia, introduced in the 2006 Act, which drew on the underlying principle of quality of life as a human right. However, within three years, criticisms were already being raised about the inspection process and in particular that they were not serving as a protection for service users, especially the most vulnerable. Service providers and the Ombudsman (2019) have argued that assessing the quality of social care demonstrates conceptual and analytical challenges and the system is regarded as falling short with regards to improving the quality of social services. Rather, the inspection systems have been found to strengthen poor practices (Kocman & Paleček, 2013). After more than a decade, service providers, in particular, remain dissatisfied with the quality assurance system accompanied by a heavy administrative burden and a lack of coherence in the way regulators interpret the standards.

This has raised a core question in Czechia – who is eligible to frame quality of care and to decide on the respective domains to be assessed. Some inspiring ideas can be found in health care, for example, Brown (2007) highlights the general need to first reach a consensus across the health sector on what quality means to the key stakeholders. The key stakeholders include service providers, the regulators who define service quality, and patients who use the services. According to McGlynn (1997), patients, care providers and investors define quality differently and may differ in their views of how it should be assessed: ‘To some degree, quality is in the eye of the beholder’ (p. 1). Stichler and Weiss (2001) noted the different meanings that quality has for patients and professionals. Professionals tend to define quality within professional standards of care, patients describe the interpersonal aspects of care and the ability of staff to respond to their needs.

Donabedian (1980, 1988) argued that the concept of quality must contain criteria that are acceptable and complete. Pittam et al. (2015) identified what was considered by service providers, government officials, service users and their families as important in assessing the quality of service. In this study, stakeholders rated safety and person-centred care as the most important domains of quality. In addition, the authors underlined the importance of knowing the opinions of all stakeholders for a deeper understanding of quality, although this creates some difficulties related to whose views carry more weight in the case of disagreement.

In conclusion, the domains which are used to conceptualise the quality of care should be seen as important and accepted by all stakeholders. In addition, we argue that the development of new systems of care quality assessment should commence with a systematic identification of similarities and differences in the perception of quality of care by the stakeholders.

The aim of this study was to understand how core stakeholders in Czechia perceive quality and which domains the stakeholders consider as most important for assessing the quality of social services.

The three research questions which guided the measure development and the analysis were:

• What items did each stakeholder group perceive as most important?

• What was the relative importance of the domains by stakeholder group?

• Did perceptions of importance vary by personal characteristics such as age and gender or by service type?

Methods

This study was a cross-sectional questionnaire study exploring the views of three categories of stakeholders – service users, service providers and public administrators working in the area of social care. In this study, service users were (1) those with disabilities and the elderly using residential care settings (housing, healthcare and support combined), and (2) those service users who lived in their own homes and receive in-home support.

Participants

Service providers. From the total number of registered services in April 2019 (N = 2610) providing long-term care, 1000 providers were randomly selected and contacted with a request to complete a survey.

Public administration. The social services departments of all 14 regional authorities in the country and all municipalities with responsibility for social affairs (N = 204) were contacted with the survey.

Service users. The aim was to recruit approximately 300 service users across all 14 regions of Czechia and from the main types of social services (i.e. residential social care settings for the elderly and for persons with disabilities) in approximately the same proportion as found in the total sample of registered services. For example, 20% of people using social care services are living in residential services for the elderly and so the aim was to have approximately 20% of our sample living in residential services for the elderly. Across the whole sample, the aim was to have approximately equal numbers of older adults and people with disabilities. No other criteria were applied.

Services users were recruited through service providers who were contacted initially through research team networks and then snowball sampling. Service providers then approached a sample of service users who they thought might be able to complete the questionnaire with support from trained assistants from the research team and asked them whether they would be willing to participate.

Measures

The questionnaire consisted of 7 domains related to service quality. Each domain contained several items, which were formulated as statements describing different aspects of quality. Respondents were asked to assess the significance of each item for the quality of a service on a five-point scale: with 1 = not at all important, 2 = of little importance, 3 = I’m not sure, 4 = important and 5 = very important.

There were four versions of the questionnaire: – one for service providers, one for public administration representatives and two for service users. The questionnaires for service providers and public administration respondents differed only on a small number of items. The questionnaires for service users included an easy to read version to support the inclusion of those with intellectual disabilities or older people with cognitive disabilities or communication difficulties. The latter consisted of just five domains – the Context and Management domains were not relevant to service users.

The different versions of the questionnaire were piloted with a sample of 10 respondents. Six were service providers or Public administration representatives and 4 were service users.

Validity and reliability

Content validity

Domains and items were identified drawing on the outputs of a thematic analysis of the content of indicators or quality criteria from relevant domestic and international models of quality assessment. It was assumed that the content of these models had been validated during preparation phase and therefore represented external criteria with an appropriate content validity.

The first source reviewed was The Czech national Quality Standards of Social Services. The quality standards cover 15 areas and serve as the legal framework for regulating social services in Czechia. Other sources were two models developed by the National Association of Social Service Providers – Quality Standards for Residential Facilities and the Quality Mark. The European model (SPC, 2010) has highlighted the impact of the wider context in which services are provided. The UK-based study for Quality Watch (Pittam et al., 2015) was also considered in the development of the questionnaire.

Seven core domains were identified for the questionnaire as a result of this review. These were the core domains which were present in the most analysed models. Four of these areas are also present in the national quality standards. However, three of these areas were absent from the national quality standards – health care, subjective quality of life of service users, and the broader context of the environment in which service is provided. These three areas were included in the questionnaire based on the recommendations of a number of critical reviews by relevant representatives of regional governments and services providers in Czechia.

Subsequently, individual items from the above quality assessment models and tools used in different countries were used to identify a list of items for each of the seven domains. This process was conducted by the project team and potential domains and items were discussed until consensus was reached. Statements for the questionnaire were then formulated for each item. Table 1 presents the final domains and list of items.

Table 1. Domains and items with Cronbach Alpha.

	Cronbach alpha
Domain/item	Public admin.	Service provider	Service users
Domain 1 Social care	0.88	0.93	0.82
1.1. Assess users’ individual needs and planning 1.2. Accredited care programmes provided by trained staff 1.3. Regular leisure, educational, cultural, spiritual activities for service users 1.4. Support to participate in activities outside 1.5. Systematic social work, advocating for interests of service users 1.6. Targeted sensory, physical, cognitive stimulation and activation 1.7. Assistance in the use of public services and provisions 1.8. Support social contacts with family and friends 1.9. Availability of assistive technologies and compensatory aids 1.10. Prompt and effective adjustment of care when conditions of service users change 1.11. Effective and ethically appropriate solutions to the problems associated with dementia
Domain 2 Subjective quality of life of service users	0.93	0.89	0.72
2.1. The service user experiences the atmosphere of the service as positive 2.2. The service user perceives the behaviour of the staff as friendly 2.3. The service user experiences supportive and patient behaviour of staff 2.4. The service user experiences a sense of self-control over daily activities 2.5. The service user experiences respectful behaviour of the personal, who is familiar with his history 2.6. The service user experiences the feeling that his privacy is respected 2.7. The service user experiences a sense of security in the service 2.8. The service user experiences a sense of trust 2.9. The service user experiences a feeling of emotional support from the staff
Domain 3 Health Care	0.91	0.88	0.71
3.1 The health of service user is is carefully monitored and evaluated 3.2 The service is able to provide palliative care 3.3 The service has enough qualified nurses for day-long care 3.4 Medical care available when needed 3.5 Physiotherapy is provided when needed 3.6 Meals are controlled by a nutrition expert
Domain 4 Management	0.92	0.93
4.1 The organisation has an effective risk management system 4.2 Quick recognition and resolution of critical and emergency events 4.3 Effective personnel management 4.4 An effective quality management system 4.5 Good organisational culture, teamwork
Domain 5 Quality of the environment	0.82	0.85	0.42
5.1. Suitable and well-equipped areas for activities 5.2. Pleasant environment in the area 5.3. Comfortable accommodation offers privacy 5.4. Quality laundry and catering services
Domain 6 Context	0.87	0.86
6.1 The service is reasonably economically saturated 6.2 Transparent and predictable rules of public regulation 6.3 The public administration requires only the necessary information 6.4 The service is stable and has good prospects 6.5 The rules of regulation by the public administration allow the service professional independence
Domain 7 Ethics	0.95	0.92	0.82
7.1 Respect for service users’ rights 7.2 Finding solutions and making decisions together with the client 7.3 Service provider plans in cooperation with service user 7.4 Effective resolution of situations threatened by violence or abuse 7.5 Open and understandable information and communication with service user 7.6 Regular survey of the satisfaction of service users and their families 7.7 Standardised procedure for solving ethical problems 7.8 Complaints are handled fairly and are used to identify and resolve problems 7.9 The service provider is part of the local community and participates in its activities 7.10 The service provider works with family and friends
Overall Cronbach Alpha (7 Domains) Public administration and service provider questionnaires	0.97	0.97
Overall Cronbach Alpha (5 domains) Service User questionnaire			0.91

Reliability

Internal consistency of the questionnaire was evaluated using Cronbach’s alpha, which is a measure of internal consistency of research instruments. Cronbach’s alpha was determined for the instrument as a whole and its domains on the basis of the polychoric correlations of the individual items. The higher the correlation between items on the scale, the higher the Cronbach’s alpha statistic. As can be seen from Table 1, internal consistency was good with overall scale statistics over 0.9 for all three stakeholder groups. Only one individual domain had a lower Cronbach alpha (0.42) and that was domain D5 Quality of the environment for service users.

Construct validity

The reported high consistency of the instrument and its domains made it possible to evaluate the construct validity. We carried out exploratory factor analysis based on polychoric correlations. The analysis was performed using the principal axis factoring method with the varimax rotation. The number of factors was not restricted for the initial analysis and the Kaiser rule recommending retention of factors with eigenvalues greater than 1 was implemented. We found that seven factors, explaining in total between 58 and 64% of the variance for the individual groups of stakeholders, fulfilled this condition. Thus, the rotation of these factors was carried out. We computed the factor loadings of the individual items and found the data strongly supported the 7-domain model based on the aforementioned content analysis with the highest factor loading corresponding in almost all cases to the factor predicted by the model. Subsequently, confirmatory factor analysis was used to verify the 7-factor structure. It was found that this solution fitted well to the data and was superior to the competing models for the all groups of stakeholders studied. The construct validity of the questionnaire was thus confirmed, and the evaluation of the results obtained by 7 domains was justified. The further analysis of the psychometric properties of the instrument and its development will be given in a separate paper.

The study procedure

Service providers and public administration representatives were contacted by MoLSA with the information about the project and an invitation to participate. The link to the relevant survey was contained in the invitation. The questionnaire was made available online through Google Forms for an eight-week period during 2019.

Due to the nature of the needs of the service users recruited through providers – most had mild intellectual disabilities or were older adults with some cognitive or communication difficulties – the service users’ questionnaire was administered face-to-face by six trained research assistants. Research assistants visited the service users where they lived and supported them to complete the questionnaire, providing, for example, help reading the questions or recording the responses as needed by each individual.

The research assistants were trained to facilitate the questionnaire completion prior to the start of the study – how to address service providers and service users, how to facilitate the questions, and how to record and process the answers. They were made aware of the ethical procedures related to the study.

Data analysis

The returned data were analysed using MS Excel with inbuilt accessory XLSTAT 2019. Both descriptive and inferential analysis was conducted. In each category of stakeholders, the significance of the deviations of individual items from the average was tested using a single-sample t-test. The significance of differences between quality domains was tested for each stakeholder group using Friedman related measures analysis of variance with Nemenyi post hoc tests. The statistical significance of differences between stakeholder groups was also tested using Kruskall–Wallis non-parametric ANOVA and Mann–Whitney U tests.

Ethical approval

Principles of good ethical practice spelled out in the National Ethical Framework for Research 2005 were adhered to during the research study. All participants were informed in an easy-to-understand way that data gathered would be used in reports and papers in anonymous form so that they would not be identified. They were also informed that their participation was voluntary – they did not have to participate. No identifying information was recorded on either the on-line or the hard copy questionnaires.

Results

Response rates and respondent characteristics

Table 2 shows the composition of the final sample. As can be seen there were 249 responses from public administration representatives. Responses were received from all 14 regional authorities and from all 204 municipalities invited but, for some municipalities, more than one survey was returned. A quarter of the public administration surveys were completed by the heads of departments for social affairs, who were responsible for the implementation of social policy in their territory. The majority were women and mostly between 40 and 59 years of age.

Table 2. Response rates and respondent characteristics.

	Public administration	Service providers	Service users
Total number of questionnaires completed by group	249^a	217	205
% of the total number invited to participate who returned the questionnaire	100%	22%	67%
% of total number of services registered (for these two groups and types of services) who returned the questionnaire^b	–	9%	–
Number of responses (and response rate of those invited) relating to residential services^c	–	120 (24.5%)	–
Number of responses (and response rate on those invited) relating to in home/supported living	–	97 (19%)	–
Number living in services for the elderly (residential and in-home support)	–	–	103
Number living in services for people with disabilities (residential and in home support)	–	–	102
Respondent characteristics
% Female	88%	87%	60%
Age group (% in each group)
20–29	6%	6.5%	3%
30–39	20%	21%	8%
40–49	36.5%	35%	17%
50–59	33%	30%	10%
60–69	4%	7%	14%
70 or older	0.5%	0.5%	47%

^a N.B for some municipalities and regions more than one survey was returned.

^b 217 service providers represented 9% of all service providers for older adults and people with disabilities.

^c 120 out of the 217 providers who completed the questionnaire provided residential services and 97 provided in home/supported living settings. These two figures represent 24.5% and 19% of those who were invited to participate respectively.

For service providers, the return rate was 22%, which is similar to that often achieved in online surveys. The 217 service providers completing the survey represented 9% of all residential and supported living/in home support services for older adults and for people with disabilities in Czechia. Most of the respondents were quality managers, directors and social workers. As for public administration, the majority were female and between 40 and 59 years of age.

Table 2 also shows some of the basic characteristics of the service user respondents. Sixty percent were female, and the largest proportion was over 70. In addition, most service users were living in residential homes (89%), with only 11% living in their own home with support.

Distribution of scores

It is important to note that overall scores were generally very positive, with a high ratio of ratings of 4 (significant) and 5 (very significant) across all groups, meaning that even values below the average for the group represented a predominantly positive preference. Although the trend towards high ratings was common to all groups, there were significant differences in the distribution of data. Both Service providers and Public Administration responses were characterised by little variation around a relatively high mean. In contrast, for the Service user ratings, there was more variability around a slightly lower mean, with more lower scores and fewer high scores. Table 3 illustrates this pattern.

Table 3. Domain scores and comparisons along with the items on each domain which were rated as ‘most important’ – i.e. significantly higher (at p < 0.01) than the participant group average for that domain.

Domain/items rated as most important	Mean score (SD); median (min–max)			Statistical comparisons (Kruskall–Wallis/Mann–Whitney)
	Public administration N = 249	Service providers N = 217	Service users N = 205
D1 Social Care Overall mean score	4.33 (0.41); 4.36 (1.46–5.00)	4.40 (0.41); 4.45 (2.55–5.00)	3.89 (0.65); 3.90 (1.27–5.00)	KW X^2 = 94.49 p<0.001
D1.1 Assess individual needs	4.65 (0.6); 5.00 (1.00–5.00)	4.72 (0.48); 5.00 (3.00–5.00)	Not sig.
D1.8 Support social relationships	4.63 (0.59); 5.00 (2.00–5.00)	4.76 (0.46); 5.00 (2.00–5.00)	Not sig.
D1.10 Prompt response to service user situation	4.66 (0.52); 5.00 (1.00–5.00)	4.65 (0.56); 5.00 (2.00–5.00)	4.40 (0.84); 5.00 (1.00–5.00)
D1.11 Effective and ethical response to dementia	4.47 (0.68); 5.00 (2.00–5.00)	4.60 (0.52); 5.00 (2.00–5.00)	4.42 (0.76); 5.00 (1.00–5.00)
D2 Subjective Quality of life of service users	4.47 (0.48); 4.56 (1.67–5.00)	4.65(0.34); 4.78 (3.56–5.00)	4.31 (0.41); 4.40 (2.56–5.00)	KW X^2 = 75.43 p < 0.001
D2.1 Service users experience positive atmosphere	Not sig.	4.64 (0.54); 5.00 (2.00–5.00)	4.63 (0.55); 5.00 (2.00–5.00)
D2.2 Service users perceive staff as friendly	4.49 (0.61); 5.00 (2.00–5.00)	4.67 (0.51); 5.00 (3.00–5.00)	4.60 (0.57); 5.00 (2.00–5.00)
D2.3 Service users experience staff as supportive and patient	4.48 (0.62); 5.00 (2.00–5.00)	4.63 (0.52); 5.00 (3.00–5.00)	Not sig.
D2.5 Service users experience staff as respectful and staff know their history	4.48 (0.63); 5.00 (2.00–5.00)	4.67 (0.48); 5.00 (3.00–5.00)	4.25 (0.78); 5.00 (1.00–5.00)
D2.6 Service users feel privacy is respected	4.61 (0.62); 5.00 (1.00–5.00)	4.79 (0.41); 5.00 (4.00–5.00)	4.34 (0.88); 5.00 (1.00–5.00)
D2.7 Service users feel safe	4.56 (0.65); 5.00 (2.00–5.00)	4.71 (0.48); 5.00 (3.00–5.00)	4.60 (0.61); 5.00 (2.00–5.00)
D2.8 Service users feel they can trust those who support them	4.56 (0.60); 5.00 (2.00–5.00)	4.71 (0.46); 5.00 (3.00–5.00)	4.59 (0.59); 5.00 (2.00–5.00)
D2.9 Service users feel emotionally supported	4.50 (0.70); 5.00 (1.00–5.00)	4.69 (0.52); 5.00 (2.00–5.00)	Not sig.
D3 Health care	4.34 (0.57); 4.56 (1.33–5.00)	4.19 (0.74); 4.27 (1.00–5.00)	3.64 (0.77); 3.83 (1.33–5.00)	KW X^2 = 112.03 p < 0.001
D3.1 Health of service users carefully assessed and monitored	Not sig.	Not sig.	4.33 (0.91) 5.00 (1.00–5.00)
D3.2 Service can provide palliative care	4.67 (0.58); 5.00 (1.00–5.00)	Not sig.	4.42 (0.79); 5.00 (1.00–5.00)
D3.3 Service has enough qualified nurses for day-to-day care	4.50 (0.69); 5.00 (1.00–5.00)	Not sig.	Not sig.
D5 Quality of the Environment	4.28(0.46); 4.25 (1.50–5.00)	4.43 (0.54); 4.50 (2.25–5.00)	3.97 (0.68); 4.00(1.00–5.00)	KW X^2 = 59.92 p < 0.001
D5.3 Comfortable accommodation offers privacy	4.59 (0.57); 5.00 (1.00–5.00)	Not sig.	Not sig.
D7 Ethics	4.31 (0.52); 4.30 (1.20–5.00)	4.54 (0.39); 4.60 (3.30–5.00)	3.96 (0.56); 4 (2.64–5.00)	KW X^2 = 125.24 p < 0.001
D7.1 Rights respected	Not sig.	4.69 (0.47); 5.00 (3.00–5.00)	4.32 (0.79); 5.00 (2.00–5.00)
D7.2 Finding solutions and making decisions with service users	Not sig.	4.66 (0.49); 5.00 (3.00–5.00)	Not sig.
D7.3 Service provider plans in partnership with service users t	4.47 (0.66); 5.00 (1.00–5.00)	4.67 (0.48); 5.00 (3.00–5.00)	4.22(0.93); 5.00 (1.00–5.00)
D7.4 Effective resolution of violent or abusive situation	4.50 (0.70); 5.00 (1.00–5.00)	4.68 (0.52); 5.00 (3.00–5.00)	4.38 (0.82); 5.00 (1.00–5.00)
D7.5 Open and understandable information available	4.46 (0.62); 5.00 (1.00–5.00)	4.69 (0.49); 5.00 (3.00–5.00)	Not sig.
D7.10 The service provider works with family and friends	Not sig.	4.63 (0.60); 4.00 (2.00–5.00)	Not sig.
Domains only in Public Administration and Service provider questionnaires
D4 Management	4.21 (0.60); 4.27 (1.67–5.00)	4.48 (0.49) 4.50(2.00–5.00)		MW z = 5.32 p < 0.001
D4.1 Clear working procedures and staff competencies	Not sig.	4.72 (0.50); 5.00 (2.00–5.00)
D6 Context	4.25 (0.56); 4.29 (1.40–5.00)	4.40 (0.59) 4.60 (2.20–5.00)		MW z = 3.17 p = 0.002
D6.1 Service is reasonably resourced	4.47 (0.69); 5.00 (1.00–5.00)	4.65 (0.63); 5.00 (1.00–5.00)
Statistical comparison within group across domains (Friedman X^2 and p value)	X^2 = 97.26 p < 0.001 (7 domains)	X^2 = 85.83 p < 0.001 (7 domains)	X^2 = 66.75 p < 0.001 (5 domains)

Which individual items did participants rate as more important?

Given the overall high scores and the fact that even those scores that were below the mean for each stakeholder group still represented relatively high importance scores, the analysis reported below focuses on those items that appeared to be rated as more important. This was defined as an item having a higher importance rating than the domain average for that stakeholder group with statistical significance at p < 0.01. This level of significance was used in order to reduce the likelihood of type 2 errors due to the number of comparisons conducted. Table 3 illustrates the items within each domain which were rated as more important by each stakeholder group. The section below first presents the domains completed by everyone, followed by the two domains only completed by public administration and service providers.

Domain 1: social care

Within the domain of social care, four items were significantly more important for public administration respondents and service providers: Assessing individual needs; supporting social relationships of people; providing a prompt response to deal with whatever situation the service user was currently experiencing; and providing an effective and ethical response to dementia. The latter two were also rating as more important by service users.

Domain 2: subjective quality of life of service users

As illustrated in Table 3, all but one individual item (2.4 The service user experiences a sense of self-control over daily activities) were rated as more important for at least two out of the three stakeholder groups. On two items, 2.7 Service users feel safe and 2.8 Service users feel they can trust those who support them, all three groups rated these as statistically more important with mean scores above 4.5 and median scores of 5. Figure 1 illustrates the hierarchy of items rated as important for each stakeholder group for Domain 2. As can be seen there are similarities between public administration and service provider ratings and a somewhat different pattern for service users. Item 2.1 (service users experience positive atmosphere) was the highest rated item of all the questions on the service user questionnaires.

Figure 1. Domain 2 items rated significantly higher than mean domain score by each stakeholder group and presented in the order of importance as denoted by the mean score, arranged from highest to lowest.

Domain 3: health care

None of the individual items on this domain was rated as more important for service providers. However, as can be seen in Table 3, item 3.2 (Service can provide palliative care) was rated as significantly important by both public administrator respondents and service users.

Domain 5: quality of the environment

Again, only one item, this time by public administration respondents, was rated as significantly more important – 5.3 Comfortable accommodation offers privacy.

Domain 7: ethics

On Domain 7, six individual items were rated as significantly more important for at least one stakeholder group. As illustrated in Table 3, two items were significantly more important for all three stakeholder groups – these were item 7.3 (Service provider plans in cooperation with services user) and item 7.4 (Effective resolution of violence or an abusive situation). Service providers and public administration representatives agreed on the importance of having open and understandable information (Item 7.5), while service providers and service users agreed on the greater importance of respect for service user rights (Item 7.1).

Domain 4: management (service provider and public administration only)

Only one item on this domain was rated as significantly more important and only by service providers and that was item 4.1 (There are clear working procedures and staff competencies).

Domain 6: context (service provider and public administration only)

Item 6.1 (Service is reasonably resourced) was the only item in this domain rated as significantly important by both public administration and service providers (see Table 3).

Figure 2 illustrates the order of the items rated as more important by each stakeholder group.

Figure 2. Domain 7 items rated significantly higher than mean domain score by each stakeholder group and presented in the order of importance as denoted by the mean score, arranged from highest to lowest.

Relative importance of domains by stakeholder groups

How did stakeholder groups differ in their ratings of each domain?

Table 3 presents the mean and median scores for each domain and for each stakeholder groups along with the results of the Kruskall-wallis or Mann–Whitney U analysis for each domain across stakeholder groups. Dunn post-host hoc tests (significant at p < 0.01) indicated that on all domains service provider and public administration ratings were significantly higher than service user ratings. On all domains apart from D3 Health Care, service provider ratings were also significantly higher than public administration ratings.

Which domains, if any were rated as more important for each stakeholder group?

As can be seen from Table 3, the Friedman analysis found a significant difference between domains for each stakeholder group. Nemenyi post hoc tests were used to develop a hierarchy for each stakeholder group as summarised in Figure 3.

Figure 3. Hierarchy of the perceived importance of the domains for the individual groups of stakeholders. Domains were the different stakeholders agreed on the relative importance are placed in the intersection of the corresponding sets.

Service providers

D2 Subjective Quality of Life of Service users was rated significantly higher than domains D1, D3, D5 and D6 (all at p < 0.001). On the other hand, there was no significant difference (p > 0.05) between D2 and domains D4 and D7. D1, D5 and D6 were significantly higher than D3 Health Care (p < 0.001). This suggests the following hierarchy in terms of relative importance to service providers:

1. D2, D4, D7

2. D1, D5, D6

3. D3

Public administration

D2 Service user subjective Quality of life was rated significantly higher by public administration respondents (p < 0.001) than all other domains. Although only significant at p < 0.05, D3 Health Care was rated significantly higher (at p < 0.05) than D5 Quality of the Environment, D4, management and D6 Context but not D1 Social Care, and D7 Ethics. This suggests the following hierarchy:

1. D2

2. D3, D1, D7,

3. D5, D6, D4

Service users

As for service providers and public administration, D2 (service user subjective quality of life), was by far the highest rated domain for service users and was significantly higher than all other domains (p < 0.001). D3 (Health care) was significantly lower than all other domains (p < 0.003). There were no significant differences between D1, D5 and D7 giving a resulting hierarchy of:

1. D2

2. D1, D5, D7

3. D3

Do ratings of importance vary by participant characteristics or service type?

Table 4 presents the Spearman’s Rank Order correlation coefficients and biserial correlation coefficients for age and gender.

Table 4. Relationships between age, gender, type of service provided and perceived importance of individual domains.

	Domain	Spearman‘s correlation coefficient for age and perceived importance, biserial correlation coefficient for gender and perceived importance
		Service providers			Public administration		Service users
		Service type	Age	Gender	Age	Gender	Age	Gender
1	Social care	0.127	−0.066	0.213 (p < 0.001)	0.052	0.080	0.027	0.042
2	Subjective quality of life of of service user	−0.023	−0.109	0.239 (p < 0.001)	0.066	0.039	0.045	−0.053
3	Health care	0.251 (p  < 0.001)	−0.008	0.066	−0.051	0.162 (p = 0.011)	0.169 (p = 0.016)	0.030
4	Management	0.123	0.057	0.171 (p = 0.012)	0.042	0.136 (p = 0.032)	N/A	N/A
5	Quality of the environment	0.287 (p  < 0.001)	−0.067	0.169 (p = 0.013)	−0.037	0.094	0.125	−0.085
6	Context	−0.133	−0.047	0.220 (p < 0.001)	0.075	0.009	N/A	N/A
7	Ethics	0.055	−0.064	0.156 (p = 0.022)	−0.029	0.113	−0.027	−0.107

Note. Items in bold are significant at p < 0.01. Exact p values also indicated where p < 0.05.

As can be seen from the Table 4, coefficients were generally low and very few were significant at p < 0.01. The only significant associations were for gender and importance ratings by service providers – where female respondents tended to rate D1 Social care, D2 Service user subjective quality of life and D6 Context significantly higher than male respondents.

Not enough service users lived at home with service support coming into them to look at relationships between current service type and importance ratings for service users. However, biserial correlations between type of provided service (dichotomic variable: residential vs. in-home support) and importance rated by service provider highlighted that those that provided residential services (n = 120) rated D3 health care and D5 Quality of the environment higher (p < 0.001) than providers of in-home support (n = 97).

Discussion and summary of findings

The findings suggest that the constructed set of quality indicators was valid, reliable and representative and the mutual correlations between them were factors that justify their division into seven thematic domains. There was some variation in perceived importance between the three groups of stakeholders. However, the highest rated domain by all stakeholders was the subjective quality of life of service users, which included indicators such as expressing feelings of trust and security, perceptions of a positive atmosphere in the service and experiencing staff as friendly. Service providers also prioritised Rights (D7) and Effective management (D4). Service users rated elements related more to their direct experiences higher – so Rights and Quality of the Environment were next most important after subjective quality of life.

Health care (D3) differentiated stakeholders to some extent – with Health care being rated as more important by representatives from public administration than by service providers and service users. However, older service users rated health care as more important than younger service users and those providing residential care rated health care and quality of the environment as more important than those supporting people in their own homes – this is likely to be a result of those in residential services being more severely disabled, older or more frail than those living in their own homes.

There are some limitations that need to be considered. Firstly, as noted earlier, ratings by service providers and public administration representatives were higher on all domains than ratings by service users. As such this makes it hard to compare the absolute importance between these two groups and those by service users. Secondly, the very high ratings by both service providers and public administration respondents may also, at least in part, be accounted for by social desirability – they may be responding as they think they should respond. However, the likelihood of social desirability was reduced by the survey being anonymous. Services users were supported to complete the questionnaire which may have introduced some influence on their responses. However, support was provided by trained fieldworkers rather than by their own staff, which would have helped ensure they felt they could respond openly.

Finally, the sample was skewed towards those providing or living in residential care – only a very small number of people living in their own home with support and providers of such support were included. There was some indication that there were differences between the views of those who provided residential care and those who were supporting people in their own home, although this was confounded by the fact that those who lived in residential care were generally older and had greater support needs – so concluding whether the difference was due to the type of setting or the nature of the people supported was not possible. However, the differences were only in two domains and did not impact on the domains rated as most important, i.e. the subjective quality of life of service users and human rights.

Implications for quality monitoring and future research

Despite the limitations, the study has highlighted the variability but also the points of consistency in how different stakeholders perceived service quality. The findings are likely to initiate discussion and potentially consensus on how service quality can be conceptualised and how it should be measured – a consensus that would bypass the current bias ruled by the legal system which prioritises the view of the state regulator.

The importance of the subjective quality of life of services users was a key point of agreement. As noted in the introduction, the quality standards introduced in the 2006 Act had come under substantial criticism in particular from service providers. Although quality of life of service users is reflected to some extent in the quality standards, overall, the standards and the inspection methods focus on assessing processes rather than outcomes. The processes themselves are assessed in terms of the presence of documentation as an evidence of meeting requirements defined in the quality standards such as documentation on person-centred plans, contracts between service providers and service users on service provision etc. rather than the quality of support or the nature of the relationships between staff and those in receipt of services (Kocman & Paleček, 2012).

Although still highly rated overall, the relative importance of the quality of healthcare varied between stakeholders and, also by age of service user and by type of service provided. Services users who were older rated healthcare as more important than service users who were younger. Service providers who provided residential care homes rated health care as more important than those providing support into people’s homes. However, these may be related as most of the service providers of residential care homes were providing for older adults, rather than younger adults with disabilities. For these service providers, healthcare is a significant part of what they do and therefore it is perhaps not surprising that they rate healthcare as more important than other domains. Finally, municipality representatives also rated health care as more important than other stakeholders.

This may partly reflect the current dissatisfaction in the Czechia over the fragmentation of the system and the fact that health and social care are seen as very separate and not always working together (MoLSA, 2020). In the context of the UN Convention on the Rights of Persons with Disabilities (UN CRPD), people should be accessing housing, health care, social care, transport, education etc. from their community rather than as part of a ‘total institution’ (Goffman, 1962). However, meeting people’s needs require attention to be paid to all aspects of their lives and co-ordination between sectors such as health, social care, housing, education, so that a holistic support package can be established. This is, however, not easy to establish in practice and is an international issue. For example, in England the concept of joint working between health and social care in particular and ‘Pooled budgets’ were seen as key for the modernising of social services (Great Britain, 1998). However, in 2014, the Commission on the Future of Health and Social Care in England called again for an end to the fragmentation between health and social care and recommended ‘a single, ring-fenced budget for health and social care, with a single commissioner’. The Kings Fund (Humphries & Wenzel, 2015) put forward recommendations for integrated commissioning which they had hoped would to ensure that there was integrated commissioning in all parts England by 2020. More broadly, Šiška et al. (2015) found that lack of co-ordination and organisation across levels of government and between agencies was a key barrier in the implementation of community living and the UN CRPD Article 19 specifically.

The finding that stakeholders’ views focus primarily on quality of life and the quality of interactions between those providing support and those who are receiving services is not unique to Czechia. For example, in Ireland consultation of service users and families around quality in mental health services found that ‘respectful, empathetic relationships’, ‘an empowering approach to service delivery’ and a ‘quality environment respecting the dignity of the individual and the family’ are key (Mental Health Commission, 2005).

Similarly, Czechia is not unique in the fact that quality assessment focuses primarily on processes and procedures, which only sometimes been found to correspond with other assessments of outcomes for service users (Beadle-Brown et al., 2008; Netten et al., 2012). However, in some countries, quality assessment processes have changed recently, and better agreement has been found between inspectors’ ratings and other measures of outcomes and service quality (Towers et al., 2019). In Ireland, as in the UK, inspection processes now include the use of observations of practice, conversations with individuals, and the involvement of experts by experience.

The importance of the subjective quality of life of services users was a key point of agreement. This finding however, poses important questions for how this would/should be assessed/measured. To help people complete this questionnaire for the study it was necessary to provide support to almost all service users. While some people with intellectual disabilities and older people who have dementia or other cognitive difficulties might be able to complete surveys or be interviewed about their quality of life, many cannot. There are also well documented limitations of relying solely on self-report of QoL even for people with milder intellectual disability – for example, the fact that limited experience often brings difficulties with making comparisons or judging how good their current situation is. Alternatively, people can be less likely to say what they really think about a service for fear of losing their place, upsetting staff or getting into trouble (Inclusion Europe, 2019). There are also well-documented issues related to asking others to rate or comment on the quality of life of other people, for example, as a proxy or informant (Bertelli et al., 2017; Lefort & Fraser, 2002).

This has led researchers to argue for the importance of observation Mansell (2011) and the need for inspectors or others looking at quality to visit services and not just to visit to look at the paperwork or the quality of the environment but to look at the quality of the support and interactions between those providing support and those receiving support (Beadle Brown et al., 2020). Items such as those in this survey – i.e. whether staff are friendly, respectful, supportive, positive, patient etc. cannot be ascertained from paperwork – it has to be either reported directly by people themselves or observed. This has substantial implications for measuring quality – any measure that requires someone to rate or make a judgement about quality, requires those who are observing to understand what good (and bad) looks like and what is possible to achieve even for those with the most complex and profound needs and not bound by restrictions of unsuitable environments. Such methods also need observers to have training, not just in the tool to be used but in observer discipline. Finally, consideration to inter-observer reliability needs to be given. Although getting reliability on such ratings can be difficult, it is possible to do so (See Beadle-Brown et al., 2012 as an example).

Acknowledgement

The authors thank Dorota Šišková, Lenka Dlouhá, Marie Maňáková, Michaela Komrsková, Nikol Gruberová, Petr Bláha, Vendula Lipková, Camille Latimier, and Šárka Káňová for their great contribution to the study, namely for assistance with preparation of the research tools and data collection.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the European Social Fund, Operational Programme Employment, Project Development and Support for Quality Systems in Social Services [grant number CZ.03.2.63/0.0/0.0/15_017/0006219].

Notes on contributors

Jan Šiška

Jan Šiška is an associate professor. His research interests include comparative policy and practice in social services, deinstitutionalisation and community-based services particularly for persons with disabilities and mental health problems.

Pavel Čáslava

Pavel Čáslava is a consultant for the Association of Social Service Providers, Czech Republic and chairman of the Ethics Committee. He is interested in systemic issues of social services, regulation and evaluation of quality and ethical issues of social welfare policies and systems.

Jiří Kohout

Jiří Kohout is an associate professor with interests in quantitative research methods in social sciences.

Julie Beadle-Brown

Julie Beadle-Brown is a professor in intellectual disabilities. Her research is focused on active support for persons with intellectual and developmental disabilities, empowerment and community-based services.

Zuzana Truhlářová

Zuzana Truhlářová is an assistant professor. Her research interests consist of social work, and social services with particular focus on long-term social care.

Markéta Kateřina Holečková

Markéta Kateřina Holečková is as assistant professor and expert for the Government administration in social welfare.