Comparative workflow modeling across sites: Results for nursing home prescribing

Abstract

Workflows associated with health care delivery vary between settings, and understanding similarities and dissimilarities can inform context-sensitive practice change. Clinical workflows are complex, dynamic, and context-dependent, and comparing workflow across multiple settings can support tailored implementation of practice-change interventions. We propose a methodology for comparative workflow modeling and evaluate its use through application to antibiotic prescribing in six nursing homes in two states in the United States. After collecting multi-site workflow data and developing a cross-site workflow model from nursing home field visits, we constructed a comparison matrix of workflow task occurrences and variations and used it to perform cross-site comparisons across all six NHs.

Keywords:

• workflow modeling
• comparison
• context
• adaptation
• tailoring
• human factors
• implementation science
• mixed methods

1. Introduction

1.1. Background

Healthcare outcomes are shaped by care processes, which are themselves shaped by the setting in which they take place (Carayon, 2006). Therefore, contextual differences between settings matter for efforts to improve healthcare systems by designing and implementing changes to healthcare processes across settings (Holden et al., 2013; Pfadenhauer et al., 2017). Adoption and effectiveness of process changes can be enhanced by contextually adapting interventions (Escoffery et al., 2018; Wiltsey Stirman et al., 2019) and tailoring the strategies used to implement them (Perry et al., 2019). Strategies that are effective in one context may be ineffective in another (Waltz et al., 2019). This reinforces the need to maximize the fit between intended changes and the practice setting context (Chambers et al., 2013).

Care processes are helpfully understood as workflows, defined as “the flow of people, equipment, information and tasks, in different places, at different levels, at different timescales continuously and discontinuously, that are used or required to support the goals of the work domain” (Carayon, 2012). Health care workflows are complex and dynamic, and studying them can be challenging since tasks are rarely standardized and often distributed among multiple roles in the organization (Holman et al., 2016; Ozkaynak et al., 2013). Workflows have been studied in an increasing number of health care settings including operating rooms, emergency departments, primary care, and daily-living settings, (Harders et al., 2006; Ozkaynak et al., 2015; 2016; Wooldridge et al., 2017). The structure of health care workflows has been associated with outcomes, such as hospital length of stay (Ozkaynak et al., 2015). The ubiquity of healthcare workflows has provided an opportunity for healthcare systems engineering researchers and practitioners to enter a variety of health care settings to redesign workflows for jointly optimized patient and professional outcomes. Workflow modeling has helped identify targets for workflow changes in problems such as health information technology integration, patient safety, and care coordination (Asan et al., 2015; Cain & Haque, 2008; Garg et al., 2005; Rickles et al., 2018; Steckowych & Smith, 2018). Results of published workflow studies and improvement efforts have tended to characterize workflows within specific sites rather than comparing workflows directly and systematically across multiple sites.

Because of the need to maximize the fit between changes and context, comparing workflow across multiple sites is an important problem for healthcare researchers and improvement teams. It can identify workflow features that need to be modified [citation blinded for peer review – Journal of Clinical Rheumatology] and can inform tailored efforts to integrate new interventions into existing workflows [citation blinded for peer review – Arthritis Care & Research]. Few workflow studies have systematically compared workflows across settings; in emerging studies, workflows have been compared across small numbers of sites, but concrete procedures for systematically performing such comparisons have not yet been specified [citation blinded for peer review – Journal of Clinical Rheumatology]( Hoonakker et al., 2017; Ozkaynak et al., 2015; Unertl et al., 2012).

1.2. Objective

Our objective is to put forth a methodology for comparative workflow modeling using antibiotic prescribing in nursing homes as an exemplar. By describing the steps of the methodology in general and then demonstrating how to use them in a challenging application context, we sought to equip others to adopt the methodology to study or improve other workflows in other settings.

2. Methods

We first describe the steps of the methodology, the functions they serve, and how they can be achieved with different kinds of workflow data. We then describe the case study of antibiotic prescribing in nursing homes and the procedures used to apply the methodology.

2.1. Comparative workflow modeling methodology

The proposed methodology consists of four steps: obtain multi-site workflow data, specify a cross-site model as a basis of comparison, construct a multi-site comparison matrix, and perform cross-site comparisons. They are depicted as a linear sequence in Figure 1 for accessibility, but each step can be iterated as needed, depending on the purpose and the situation.

Figure 1. Methodology for comparative workflow modeling.

1. Obtain multi-site workflow data. Comparing workflows across sites requires data that are identifiable by site and that are relevant to how the workflow of interest is performed or perceived. Data can be obtained from new primary data collection or through secondary use of existing data.

2. Specify a cross-site workflow model. Comparing data from different sites to a common basis of comparison enables a systematic identification of similarities and dissimilarities. To serve as a basis of comparison across sites, a model of the workflow of interest must include a set of features (e.g., tasks) that may possibly be found in the data from each site. Specifying a model includes either developing a new model for a given project or selecting an existing model from previous work.

3. Construct a multi-site comparison matrix. The purpose of the comparison matrix is to characterize the workflow at every site in terms of the same set of features. The multi-site matrix enables the cross-site comparisons in the next step, but it could also be a sufficient output for some purposes.

4. Perform cross-site comparisons. Having characterized the workflow data in terms of the same features enables identifying similarities and dissimilarities across sites in relation to those features.

The rationale for using a common model as a basis of comparison is rooted in Rosen’s modeling relation (Rosen, 1978), which has been used in systems engineering to describe clinician-patient interactions and work constraints (Bisantz et al., 2014; Vicente, 1999). The modeling relation describes how a model serves as a representation of a real-world system by having aspects of the system mapped (i.e. encoded) into features of the model in such a way that those features can again be found in data from the system (i.e. decoded). Figure 1 thus establishes a modeling relation between multiple sites and a single model, with the first arrow depicting encoding and the second arrow depicting decoding. The cross-site workflow model serves as a representation of the workflow of interest across multiple sites. That model then serves as a basis for comparisons between the sites, by finding workflow features from the model in the workflow data from each site.

Having a small set of simple steps clearly delineated will likely increase the number of workflow studies and improvement efforts that systematically compare workflows across multiple sites. The following exemplar demonstrates the application of these steps to compare a complex workflow across six sites in a challenging and less-studied setting. We discuss the scalability and generalizability of the methodology after presenting the case study procedures and results.

2.2. Case study domain

Rising antibiotic resistance in long term care facilities, especially nursing homes (NH), has been attributed to antibiotic overuse (Crnich et al., 2015; Daneman et al., 2015). NH antibiotic stewardship programs attempt to promote appropriate antibiotic prescribing practices, but the results of these efforts have been mixed (Crnich et al., 2015; Feldstein et al., 2017). Prior research suggests that antibiotic stewardship interventions that consider workflows in the NH that influence providers’ decision process can facilitate correct antibiotic use (Katz et al., 2017) and may favorably influence overall care quality and resident safety (Castle et al., 2011; Werner et al., 2013).

In order to implement interventions to improve antibiotic prescribing in NHs, site-specific features of the prescribing workflow need to be understood. Existing NH workflow literature has focused on staffing (Bishop et al., 2009; Burgio et al., 2004), integration with information technology (Or et al., 2014; Vogelsmeier et al., 2008), and coordination with other health care systems (Alexander et al., 2014; Tsai et al., 2016). However, we do not know of any detailed characterization of prescribing workflow, which is necessary to inform intervention design and tailored implementation to improve the workflow and ultimately reduce antibiotic overuse.

An important aspect of NH antibiotic prescribing is that, like a number of other healthcare workflows, it does not occur frequently enough for direct observations to be feasible. Additionally, electronic health records have low adoption across NHs and cannot serve as a source of data about these workflows. We therefore used field-based interviews to collect data about workflows in this case study, coupled with investigator field notes. This case study illustrates the steps of comparative workflow modeling, noting that the methodology is applicable for other kinds of data, techniques, and situations, as described later in the paper.

2.3. Procedures

2.3.1. Step 1. Obtain multi-site workflow data

We assessed the antibiotic prescribing workflow during one-day field visits in six NHs in two United States (U.S.) states (Wisconsin and Pennsylvania), by interviewing purposefully-sampled participants from three key roles in each NH: providers, nursing staff and leadership staff. The interviews lasted up to 60 min each and followed semi-structured interview guides for nursing and leadership staff (Appendix 1) and providers that we developed and pilot-tested in a separate NH not included in the study (Appendix 2). Nursing and leadership staff interviews focused on the work performed in the NH between the time a change in the condition of a resident is suspected to the time communication with a provider occurs and an initial antibiotic-prescribing decision is made. Provider interviews focused on the factors that influence their decisions to prescribe an antibiotic. The output of this step consisted of interview audio-recordings, transcribed verbatim by a professional transcription agency. Transcripts did not contain participant identifiers except for their role (nurse, leader, prescriber). Sites each received a blinded label (i.e. A, B, C) and a table cross-referencing the labels was stored in a separate location.

2.3.2. Step 2. Specify a cross-site workflow model

Each field visit produced field notes from observations of morning clinical staff meetings and afternoon shift hand-offs, and from an end-of-visit debrief among the field assessment team (ER, JF, CC) to summarize key features of antibiotic prescribing at that site. The field assessment team conducted nearly all interviews and observations in pairs. After the six site visits, the field assessment team met weekly over two months to construct a cross-site model of the workflow based on the multi-site field visits. Through multidisciplinary review of field notes from observations and debriefs, we constructed a flowchart and revised it iteratively until we achieved consensus. The output of this step consisted of a set of discrete actions or decisions (i.e. tasks) taking place before an antibiotic prescribing event, their disposition in relation to each other, and their definitions.

2.3.3. Step 3. Construct a comparison matrix

Interview transcripts were imported into MAXQDA 12 software (VERBI GmbH, Berlin) to facilitate analysis. We conducted directed content analysis of the transcripts using a codebook consisting of the tasks in the cross-site model with definitions and examples. We identified occurrences of a task by coding transcript segments describing execution of the task at the site. When transcript segments described a task from the model but with deviations from the generic definition of the task in the codebook we identified this as a variant of the task by using an analytic memo documenting aspects of the definition that were not satisfied. A variant could include a task where the site’s normal workflow was different from the cross-site model, as well as tasks that were executed inconsistently within the site, even if the site’s normal workflow was consistent with the cross-site model. We similarly planned to use analytic memos to document any descriptions of antibiotic-relevant tasks that were not already in the codebook, but none emerged from the coding process in this case study. If any such task had been identified, we would have added it to the codebook as a new task and reiterated transcript review looking for other instances of that task across other transcripts. The output of this step consisted of a table characterizing each site in terms of whether and how (in generic form or variant form or both) each task was present in the site’s interview transcripts.

2.3.4. Step 4. Perform cross-site comparison

The comparison matrix was analyzed to identify notable cross-site patterns of similarities and dissimilarities in task occurrence (mention of a task being performed) and variation (how performance of a task differed from the generic definition). The outputs of this step included adding annotations and additional columns to the table depicting the comparison matrix and documenting the details of the most common variants in an additional table and an appendix.

2.4. Rigor and reproducibility

As with any other modeling or analysis effort, rigor and reproducibility are important in order for a comparison of workflows across sites to be trustworthy. There are numerous strategies for increasing and assessing rigor and reproducibility, and for this case study, we used guidelines relevant to qualitative ergonomics (Valdez et al., 2017). Content analysis underwent two coding cycles. In the first cycle, the primary analyst (MT) blinded transcripts by site label and used tasks from the cross-site model to code segments from the transcripts. The analyst structurally coded a subset of the transcripts to create code definitions and compile illustrative examples. These were iteratively reviewed with the first author (ER) then with the field assessment team (ER, JF, CC). Final definitions and examples were consolidated in a refined codebook. In the second coding cycle, the primary analyst finished structurally coding all transcripts and recorded any variations from the generic workflow tasks as analytic memos. This process facilitated a systematic search for disconfirming evidence, thus maintaining rigor in qualitative analysis.

3. Results

3.1. Output 1: multi-site workflow data

Obtaining multi-site data (Step 1) through one-day field visits at six sites produced field notes from three investigators as well as transcripts from 44 individual or group interviews with 68 participants from three key roles in each NH: 16 interviews with 19 providers, 11 interviews with 25 nursing staff, and 17 interviews with 24 leadership staff. All six sites were nonprofit NHs, all but one were not hospital-based, half were part of multi-facility organizations, and the average daily census across sites ranged from 78 to 162, with a mean of 107.

3.2. Output 2: Cross-site workflow model

Specifying a cross-site workflow model (Step 2) produced Figure 2, which shows the flowchart that we created iteratively through multidisciplinary review of field notes, to serve as a basis for comparison across sites. Task definitions and examples are in Appendix 3.

Figure 2. Cross-site workflow model of antibiotic prescribing in nursing homes.

3.3. Output 3: Multi-site comparison matrix

The construction of the comparison matrix (Step 3) produced Table 1, which characterizes all the sites in terms of the same workflow features (i.e., tasks in this case study). The table shows how each workflow task (rows) is represented in each site (columns A-F). A task could be present in a generic form as defined in the cross-site model or in some variant of that definition. In some sites, the data included instances of the task in its generic form as well as instances of the task in variant form. In some sites, the data included no instances of some tasks, in either form. In addition to this main output, intermediary outputs of Step 3 included transcript segments coded by workflow task, which informed an additional analysis of workflow barriers and strategies to reducing antibiotic overuse in NHs [citation blinded for peer review – Journal of the American Geriatrics Society], and the analysis codebook, refined after inter-rater reliability assessments and consensus deliberations. The refined codebook is available in Appendix 3 for other researchers or improvement teams to use to analyze other datasets of antibiotic prescribing workflows in nursing homes or as a template for other codebooks in other settings.

Table 1. Multi-site comparison matrix of workflow tasks with cross-site comparisons.

Workflow Tasks		Sites						Notable Patterns
		A	B	C	D	E	F	All G	All V	No V
1	Resident condition change is recognized	GV	GV	GV	GV	GV	GV	Yes	Yes
2	Staff conducts initial survey of condition	G	G	G	G	G	G	Yes		Yes
3	Involve nursing staff	G	G	G	GV	G	GV	Yes
4	Assess resident	GV	GV	GV	GV	GV	GV	Yes	Yes
5	Transfer to emergency dept.	G	G			G	G
6	Place resident on 24-hour board	G	G	G	G	G	G	Yes		Yes
7	Initiate temporary care plan	GV	GV	G	GV	G	G	Yes
8	Requires provider input?	G		G	G	G	G
9	Notify provider	GV	GV	V	G	GV	G	Most
10	Prepare for provider communication	GV	GV	GV	GV	GV	GV	Yes	Yes
11	Attempt to reach provider	GV	G	G	G	GV	GV	Yes
12	Directly communicate with provider	GV	GV	GV	GV	GV	GV	Yes	Yes
13	Indirectly communicate with provider	V	GV	GV	G		V
14	Shift hand-off with incoming nurse	G	G	G	GV	GV	G	Yes
15	Repeat tasks by incoming nurse		G		GV		G
16	Decision-making process by provider	G	G	G	G	G	G	Yes		Yes
17	Initial decision to prescribe antibiotics	G	G	G	G	G	G	Yes		Yes

Tasks presence:

G = presence of task based on Generic task definition.

V = presence of task as Variant from generic task definition.

Patterns within sites:

Light gray (GV) = presence of task both in Generic form and in Variation form.

Thick outline (V) = presence of task only in Variant form.

No color (G) = presence of task only in Generic form.

Black = no evidence of task in the data in either form.

Patterns across sites:

No color = task present in all sites in Generic form only – Standard practice.

Light gray = task present in all sites with majority Generic form only – Semi-Standard practice.

Dark gray = task present in all sites in Generic form and Variant form – Ubiquitous practice.

Black = task not present in one site or more.

3.4. Output 4: Cross-site comparisons

Performing cross-site comparisons (Step 4) can yield different kinds of outputs, as appropriate for the data and situation. Given this case study’s focus on qualitative descriptions of tasks, our comparisons yielded notable patterns of task occurrence and variation (additional columns augmenting the matrix in Table 1), and details of common variants found across a majority of sites (Table 2 and Appendix 4). In summary, all 17 tasks from the generic workflow map were present in at least half the sites, and 10 were present in all sites: five staff work-up tasks (Tasks 1, 2, 3, 4, 6, 7) and four nurse-to-provider communication tasks (Tasks 10, 11, 12, 14). In this case study example, no new tasks related to antibiotic prescribing emerged from the data that were not already in the model, although that could have been a possible output of the methodology (each new workflow feature would augment the matrix with one additional row).

Table 2. Common variants across sites.

Common Variations	Tasks	Sites
		A	B	C	D	E	F
Family member or resident recognizes change	1	x	x		x	x	x
Provider or non-nursing staff recognizes change		x					x
Staff overlooks resident condition changes			x	x
Insufficient documentation	4	x	x	x	x	x	x
Incomplete nurse assessment					x	x	x
Only provider assesses						x	x
Insufficient preparation for provider communication	10	x	x	x	x	x	x
Fax labs and orders	12	x			x		x
Difficulty reaching provider for direct interaction			x	x			x
Receptionist or answering service	13	x	x	x			x

Task Names: Task 1 = Resident condition change is recognized; Task 4 = Assess resident; Task 10 = Prepare for provider communication; Task 12 = Directly communicate with provider; Task 13 = Indirectly communicate with provider.

3.4.1. Notable patterns

We identified three notable patterns across sites in this case study, which we named standard practice, semi-standard practice, and ubiquitous practice (Table 1, columns on the right and footnotes). Standard practice tasks (2, 6, 15, 17) were present across all sites in their generic form only. Semi-standard practice tasks (3, 7, 11, 14) were present across all sites in their generic form with some occurrences in variant form. Ubiquitous practice tasks (1, 4, 10, 12) were present across all sites in both generic form and variant form; this would include Task 9 (notify provider) with the exception of site C where the task only occurred in variant form. Task 13 (indirectly communicate with the provider) is also worth noting: in two of the sites (A, F), indirect communication of NH nurses’ clinical assessment to the provider took place through non-clinical intermediaries – clinic receptionists – rather than clinic nurses. This would be an example of variant worth examining when implementing strategies to improve information flow between NH nurses and providers. The most inconsistently occurring tasks (5, 13, 15) were also the ones with the lowest within-role percent task mentions (Table 3), as expected, and may merit attention during tailored implementation efforts. These were generic tasks not present in all sites, and seem related to organizational boundaries (5, 13) and temporal boundaries (15).

Table 3. Percentage of participants mentioning each workflow task.

Workflow Task	N (%)	P (%)	L (%)
1. Recognize resident condition change	89	43	42
2. Conduct initial survey	50	0	33
3. Involve nursing staff	78	0	50
4. Assess resident	89	57	50
5. Transfer to emergency dept.	28	7	8
6. Place resident on 24-hour board	72	0	100
7. Initiate temporary care plan	67	14	25
8. Requires provider input?	44	7	8
9. Notify provider	50	14	0
10. Prepare for provider communication	89	14	33
11. Attempt to reach provider	94	43	8
12. Directly communicate with provider	94	86	67
13. Indirectly communicate with provider	22	0	0
14. Shift hand-off with incoming nurse	50	0	33
15. Repeat tasks by incoming nurse	22	0	0
16. Decision-making process by provider	39	86	8
17. Initial decision to prescribe antibiotics	67	50	50

Colors: White: ≥ 66%. Light gray: 34-65%. Dark gray: ≤33%.

N=nurses, P=providers. L=leaders.

3.4.2. Common variations

We found recurring cross-facility variations in five of the workflow tasks (Table 2). Task variants consisted of mentions of deviations from the generic task definitions in terms of how the tasks were performed. These included instances where the discrepancy between the task as expected and the task as performed was consistent in a given facility as well as instances where the discrepancy was situation-specific. These variants highlight challenges in the execution of staff work-up tasks, which included the recognition of resident condition change, nurse assessment, and communication preparation and coordination (Details in Appendix 4), and may be targets for improvement interventions or implementation tailoring.

3.5. Rigor and reproducibility

To ensure the trustworthiness of the content analysis of qualitative data, we used two types of triangulation: triangulation of data sources and triangulation of analysts, following established standards of rigor in qualitative ergonomics (Valdez et al., 2017). For triangulation of data sources, the percentage of participants within each role (e.g. nursing, provider, leadership) mentioning a task was also calculated as an alternative to member-checking (Table 3). Nurses had the highest overall percent of task mentions, followed by leaders and then providers, as expected given their higher degree of involvement in most of the tasks.

For inter-rater reliability through triangulation of analysts, a secondary analyst (SB) received training to use the codebook and coded one third of the transcripts. Agreement rose from weak (Kappa = 0.43) to strong (Kappa = 0.86) (McHugh, 2012). Both analysts were not engaged in the field visits so as to avoid unconscious bias during coding (Morse, 2015). Iterative consensus building discussions among the analysts (MT, SB) and first author (ER) were extensively documented to resolve instances of disagreement. Inter-rater reliability calculations were performed with MAXQDA 12 after each consensus building discussion.

4. Discussion

Health care settings are heterogeneous worksystems and methods for comparing workflow across multiple settings remain poorly specified. The work in this paper builds upon prior literature to propose a step-by-step methodology for comparing workflows across sites, setting the foundation for applications to other workflows in other settings. The proposed comparative workflow modeling methodology consists of four simple steps that can be iterated as needed (Figure 1): obtaining multi-site workflow data (Step 1), specifying a cross-site workflow model to use as a basis for comparison (Step 2), constructing a multi-site comparison matrix (Step 3), and performing cross-site comparisons (Step 4). We applied the four steps of in a case study of antibiotic prescribing in nursing homes. Step 1: Data about the workflow of interest were collected from six sites in one-day field visits that included interviews with 19 providers, 25 nursing staff, 24 leadership staff. Step 2: To specify a cross-site workflow model, the field assessment team conducted iterative weekly review of field notes over two months, reaching multidisciplinary consensus on a flowchart containing 17 unique workflow tasks (Figure 2), with definitions and examples (Appendix 3). Step 3: Those tasks then served as a basis for comparison to characterize each site’s workflow by using directed content analysis to determine whether each task is present in the site’s data as defined generically, in a variant form, both, or neither. The resulting comparison matrix characterized each site in terms of whether and how each task is present (Table 1). Step 4: Cross-site comparisons based on the matrix included notable patterns across sites and the examination of variations present in a majority of sites (Table 2 and Appendix 4). Through our application to a complex workflow (antibiotic prescribing in this case) in a challenging and less-studied setting (nursing homes in this case), we found that our methodology of workflow comparison was feasible for identifying cross-site similarities and dissimilarities in workflow features (task occurrences and variations in this case).

This methodology leverages workflow modeling to establish a common basis of comparison, enabling a systematic identification of similarities and dissimilarities between workflows at multiple sites, and can be feasibly applied with rigor and reproducibility. The following sections describe case study implications and limitations, as well as methodology implications, strengths, and limitations.

4.1. Case study implications

The NH antibiotic prescribing workflow starts with an assessment of resident change in condition. In a typical NH, tasks 1 through 11 are activities usually performed by nurses. As such, it is not surprising that we found a high percentage of mentions that these tasks occurred among nurse participants compared to mentions among providers and leaders. Direct provider communication (Task 12) by nature involves nurses and providers talking with each other and it is not surprising that the level of nurse-provider agreement about task occurrence was relatively high. Communication across shift change is susceptible to communication errors (Welsh et al., 2010). The way shift-to-shift communication takes place is often specific to the organization (e.g. NH) and the low level of agreement (Tasks 13 to 15) reflects this difference. Tasks 16 and 17 focus on the prescribing decision, with providers being more involved in Task 16 and nurses being more involved in Task 17, as reflected by the agreement levels.

Our case study compared workflows across sites using multi-role interviews, addressing the infeasibility of directly observing infrequent or unpredictable phenomena like antibiotic prescribing. The methodology was feasible to apply with qualitative data while upholding the established standards for rigor in qualitative research, such as triangulation of data sources, audit trails, the search for disconfirming evidence, triangulation between data sources (participant roles), and triangulation between analysts (inter-rater reliability) (Valdez et al., 2017). In particular, triangulation between data sources offered an alternative to member-checking. In this way, we bypassed skepticism about the validity and ethics of member-checking by directly comparing sources against each other (Birt et al., 2016; McConnell-Henry et al., 2011; Morse, 2015). Our methods also facilitated a natural process for seeking out evidence-confirming and evidence-disconfirming passages: since one generic workflow map served as the basis for comparison, we were primed to consider cases that did not fit the expected workflow.

4.2. Case study limitations

The case study example also has some limitations. First, creating the cross-site workflow model using multidisciplinary review and iterative consensus relied in this case study on having a multi-disciplinary field assessment team (human factors, systems engineering, and infectious disease medicine) with experience conducting qualitative research in nursing homes. This may not be readily available in other teams. However, once a cross-site workflow model has been developed, it can be reused as the basis for future comparisons of a given workflow in a given domain (e.g. antibiotic prescribing in nursing homes, in this case). This obviates the need to create new workflow models for each new facility and reduces the time spent on future analyses. Second, we did not find evidence of the presence, variation, or absence of four of the 17 tasks in at least one facility. This is a limitation of having to use interview data instead of direct observations for relatively infrequent or unpredictable phenomena (e.g. prescribing) that may occur rarely or sporadically during planned observations. However, the relatively small extent of this limitation (4 of 17 tasks) speaks to the benefit of including data from different professional roles to triangulate their perspectives. Furthermore, these four tasks can now be a target for special attention when planning tailored implementations of practice change interventions. Third, given that we performed workflow assessments in six sites across two states, we had a limited time at each facility, which may limit the depth and corroboration of the findings in the absence of prolonged or repeated visits. However, our methodology findings did highlight tasks, variants, and sites to target for further investigation. Finally, we did not leverage the richness of our qualitative data in depth, which was beyond the scope of this comparative case study. Additional studies can examine details and nuances through more in-depth analyses that search for themes that emerge from the data themselves, as we have reported elsewhere [citation blinded for peer review – Journal of the American Geriatrics Society], as well as apply our comparative workflow modeling methodology in larger sets of NHs and in other clinical settings.

Our case study application of comparative workflow modeling to tasks elicited from interviews at six sites is relatively simple and does not showcase all of the potential of comparative workflow modeling as a methodology. In the case study, the workflow features that we considered were task occurrence and variation. We only constructed one comparison matrix, ignoring for the purposes of this case study other features of individual tasks such as duration, features of the workflow as a whole such as task sequence, and features the entire worksystem such as interplay between organizational and external factors like government regulations. This is a characteristic of this case study, not a limitation of the methodology. We used the comparison matrix to compare task configurations across sites, without considering other features of these tasks like cognitive workload or time pressure, or higher level features like interactions between technological and organizational factors. However, there is a wide range of ways to apply comparative workflow modeling (e.g. our application with quantitative data [citation blinded for peer review – Journal of Clinical Rheumatology]), as we discuss below in the section on generalizability.

4.3. Methodology implications for understanding workflows

The proposed methodology for comparative workflow modeling can be used by others to leverage the extensive existing studies on health care workflows and workflow modeling, by extending them to facilitate systematic comparisons across scalable numbers of sites. The identification of similarities and dissimilarities in the methodology is made from the perspective of a cross-site workflow model, so care needs to be taken in the specification of the model. The existing literature includes much detail about models of specific workflows and techniques to model workflows. The model can be selected from existing examples of published studies for different workflows in different healthcare settings (Harders et al., 2006; Ozkaynak et al., 2015; 2016; Wooldridge et al., 2017). In some situations, developing a new model is necessary to examine a less studied workflow in a challenging setting. In such a case, there are numerous approaches to developing a workflow model (sometimes called process model) that are reviewed elsewhere (Wooldridge et al., 2017). The proposed methodology has the potential to expand and amplify the impact of these existing models and modeling approaches by enabling their use in systematic comparisons of workflows across a scalable number of sites.

4.4. Methodology implications for improving workflows

Because healthcare workflows are shaped by the setting in which they take place, a step-by-step methodology to compare workflows across sites is a significant contribution for efforts to improve healthcare systems by designing and implementing changes to healthcare processes across settings (Holden et al., 2013; Pfadenhauer et al., 2017). This reinforces the need to maximize the fit between intended changes and the practice setting context (Chambers et al., 2013). The outputs of comparative workflow modeling can inform efforts to maximize fit. Workflow features that are similar across sites can be used to implement new interventions, by integrating them widely across sites. Workflow features that are dissimilar across sites require attention when implementing new interventions, to tailor how the interventions are integrated in different sites. In the case example, standard practice tasks that were present in all sites in generic form only (e.g. staff conducts initial survey of condition) are amenable to integrate new interventions widely (e.g. the Stop and Watch early warning tool for support staff). Ubiquitous practice tasks that were present in all sites in both generic and variation forms (e.g. prepare for provider communication) would require further examination of common variations to inform design and implementation of new interventions (e.g. examining barriers like incomplete assessment, missing documentation, or incomplete documentation as we reported elsewhere [citation blinded for peer review – Journal of the American Geriatrics Society]). Tasks that exhibit more dissimilarity across sites, including absence or presence in variant form only (e.g. indirectly communicate with provider) require additional site-specific examination to address potential site-specific barriers (e.g. communication modalities in sites A and F) or omit an intervention that is not needed (e.g. direct communication is already the norm at site E).

4.5. Methodology scalability through feasibility and reusability

A key challenge in efforts to study or improve workflows is that similarities between different instances of the same workflow are challenging to articulate due to a vast richness in details and nuances (Holman et al., 2016). Other studies present illustrations that model-based articulations of workflow features can help capture some of this richness systematically with a task list [citation blinded for peer review – Journal of Clinical Rheumatology](Wetterneck et al., 2012) or higher level system model (Wooldridge et al., 2017). Yet studies comparing workflows have typically had small numbers of sites and have not offered concrete steps to follow for a systematic comparison that can be expanded to more sites (Hoonakker et al., 2017; Ozkaynak et al., 2015; Unertl et al., 2012). The lack of concrete examples of scalable approaches limits the scope of workflow studies or improvement efforts. The methodology for comparative workflow modeling put forth in this paper supports scalability by leveraging workflow modeling as a basis for comparison.

4.5.1. Feasibility

Model-based comparison increases the feasibility of comparing workflows. The absence of a workflow model serving as a basis of comparison would necessitate comparing each site’s workflow directly to the workflow of another site, and to repeat such pairwise comparisons between all sites. However this would requires k(k-1)/2 pairwise comparisons instead of k comparisons between each site and the common model (k = number of sites). Performing k(k-1)/2 = 15 pairwise comparisons instead of performing k = 6 comparisons between each site’s workflow and the generic workflow would be less efficient, more prone to errors, and not feasible with much more than 6 sites. For example, 10 sites would require 45 pairwise comparisons instead of 10 comparisons to a generic workflow and 15 sites would require as many as 105 pairwise comparisons instead of only 15 comparisons to a generic workflow.

4.5.2. Reusability

Using a cross-site workflow model as a basis for comparison for a number of sites lends that model empirical support as a model of this kind of workflow (here, antibiotic prescribing) in this kind of setting (here, nursing homes). The same model can subsequently be reused to guide data collection and analysis in other sites. In contrast, pairwise comparisons between sites would not yield such a reusability benefit. Organizing, communicating, and interpreting the results of pairwise comparisons would be much more challenging in the absence of a common language (e.g. names and definitions of workflow tasks), especially as the number of sites grows.

4.6. Methodology generalizability to other data types and techniques

The proposed methodology does not require specific kinds of data collection or analysis techniques to obtain multi-site workflow data (Step 1), specify a cross-site workflow model (Step 2), construct a comparison matrix (Step 3), and perform cross-site comparisons (Step 4). By providing a step-by-step framework that can be used for different data, techniques, and situations, the proposed methodology paves the way for an expanded range of healthcare systems engineering research and practice to study and improve workflows across sites.

4.6.1. Generalizability of step 1

To obtain multi-site workflow data, our case study used qualitative data from interview transcripts. Other examples of workflow data can include quantitative data from time-and-motion observations [citation blinded for peer review – Journal of Clinical Rheumatology], electronic health record data (Ozkaynak et al., 2015), or process mining data (Litchfield et al., 2018).

4.6.2. Generalizability of step 2

To specify a cross-site workflow model, our case study used tasks arranged in a flowchart, with corresponding definitions, developed by multidisciplinary review of field notes after data collection. Other examples of workflow models can include a task checklist developed from examining policies and procedures or semi-structured observations [citation blinded for peer review – Journal of Clinical Rheumatology](Wetterneck et al., 2012), a swim-lane process map contextualizing tasks in terms of related system factors (Wooldridge et al., 2017), or a high-level process diagram emphasizing transitions across organizational boundaries rather than individual tasks (Werner et al., 2017).

4.6.3. Generalizability of step 3

To construct a comparison matrix, our case study focused on whether and how tasks are present at each site, summarizing the results of directed content analysis of qualitative data using the tasks of a cross-site model as analysis codes. Other examples of workflow features can include the frequencies and durations of specific tasks as in our quantitative study comparing specialty clinic workflows [citation blinded for peer review – Journal of Clinical Rheumatology], other task-level features like cognitive load, time pressure, or need for attention (Carayon et al., 2006), workflow-level features like sequence, dependencies, or distribution across roles (Werner et al., 2017), or worksystem-level features like technology, organization, and environment (Wooldridge et al., 2017).

4.6.4. Generalizability of step 4

Finally, to perform cross-site comparisons (Step 4), our case study concluded with the identification of notable patterns across and within sites, and the detailed examination of the task variants that are present across a majority of sites. Other examples of analyses across workflows can include testing for statistically significant differences [citation blinded for peer review – Journal of Clinical Rheumatology], clustering techniques (Ozkaynak & Brennan, 2012), or more novel analysis techniques (Wooldridge et al., 2018).

It may be possible to emerges 2 and 3 and perform direct comparisons between pairs of sites if the number of sites is small and not likely to be expanded in subsequent studies. In general, however, these steps make the process more efficient and less prone to errors because the number of comparisons exhibits linear rather than quadratic growth as the number of sites grows.

4.7. Methodology limitations

The proposed methodology has a few limitations and we offer suggestions to mitigate them. First, because the intent is to compare across sites but data collection often occurs sequentially one site at a time, there is a possibility that the order in which the sites are visited may cause the generic model to over- or under-represent information from some of the sites. We addressed this by waiting until all site visits were completed to develop the cross-site workflow model, so that field notes from the site visits could be reviewed and discussed concurrently. Second, while the use of a cross-site model as a basis of comparison is a strength, it may introduce bias if its development is informed by the same site visits that produce the data analyzed to compare workflows across sites. We addressed this by triangulating data sources and data analysts from the site visits. The multidisciplinary field assessment team used the field notes to develop the generic model and subsequently analysts who has not participated in the site visits or model development coded the interview transcripts. One researcher (the first author) served as a link between both sub-teams to ensure continuity. Third, while adequate for the purpose of comparison, the analysis based on a common model might leave out relevant details, especially when using qualitative data. We addressed this by using analytic memos to document task variants (Table 2 and Appendix 4) and any tasks that emerged from the data but were not already in the model (we did not find any in this case study). Additionally, we conducted another study in which we identified workflow barriers and strategies inductively [citation blinded for peer review – Journal of the American Geriatrics Society]. Finally, others may find instances of healthcare workflows that have so much lack of structure and regularity within and across sites that the proposed methodology is challenging to apply. We expect that even in such cases, the process of attempting to apply the methodology to parts of the workflow would yield useful findings for the workflows in questions and possible opportunities to extent or refine the methodology.

5. Conclusion

We proposed a methodology of comparative workflow modeling and evaluated its use through a case study of prescribing workflows in nursing homes. By using a common workflow model for all sites, we were able to systematically compare workflow tasks and detect potential task variants across multiple sites that could be targets for interventions. By describing the steps of the methodology in general and then demonstrating how to use them in a challenging application context, we have equipped others to adopt the methodology to study or improve other workflows in other settings.

Acknowledgements

The authors would like to thank Grace Welham for project management support in the data collection phase and the nursing home staff and providers that supported the field visits and participated in interviews. This project was funded under grant number AHRQ R18HS022465-01A1 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services. The opinions expressed in this document are those of the authors and do not reflect the official position of AHRQ or the U.S. Department of Health and Human Services.

Disclosure statement

All authors declare no relationships or conflicts of interest.