A problem-based approach to the advancement of heuristics for socio-technical evaluation

ABSTRACT

With ubiquitous, mobile computing, health care systems, and smart factories, socio-technical phenomena continue to emerge that challenge traditional design and evaluation methods. We perceive such phenomena as the intertwinement of technical artifacts and social practices. Previous work shows that there is no sufficient method to evaluate the quality of this socio-technical intertwinement. Hence, our goal was to develop socio-technical heuristics, in short ST-heuristics, that can be applied by individuals to detect issues. Drawing inspiration from the success of usability heuristics in the field of human–computer interaction, we first applied a literature review to develop an initial set of ST-heuristics derived from six domains comprising groupware/computer-support cooperative work, job design, usability, socio-technical design principles, privacy, and process design. We then conducted two studies to evaluate and improve this set using empirical data from 13 cases from health care, industry, and engineering education fields. In total, we analysed 306 problems. The results substantiate a final set of eight ST-heuristics which allow for evaluating the socio-technical intertwinement in situ. We perceive the contribution of this work as a starting point for evaluators to uncover crucial issues and to improve current practice. We discuss the developed set of ST-heuristics within existing literature.

KEYWORDS:

• Heuristics
• socio-technical systems
• evaluation
• system design
• work systems
• social practice

Acknowledgment

We are grateful to Dr. Rainer Skrotzki, who helped during the grouping and discussions of the items and heuristic assignments. We thank Rolf Molich for his comments on the first version of the heuristics and Felix Thewes for the technical and organisational support of the empirical work with the problem database. Many thanks go to Dr. Jan Nierhoff for data collection and commenting on previous versions.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 The visibility principle says, ‘The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.’ (https://www.nngroup.com/articles/ten-usability-heuristics/).

2 We added one individual to the group of experts for this phase because of her/his background in computer science and STS design, which increased the group to six experts rather than the originally planned group of five.

3 One expert had to be replaced because the original expert was not available during the time period of this phase.

Additional information

Funding

This work was funded by the Ministerium für Kultur und Wissenschaft des Landes Nordrhein-Westfalen, Germany.