A sensitivity analysis of the Children’s Treatment Network trial: a randomized controlled trial of integrated services versus usual care for children with special health care needs

Abstract

Background

The value of integrated care through comprehensive, coordinated, and family-centered services has been increasingly recognized for improving health outcomes of children with special health care needs (CSHCN). In a randomized controlled trial (RCT), the integrated care provided through the Children’s Treatment Network (CTN) was compared with usual care in improving the psychosocial health of target CSHCN. In this paper, we aimed to estimate the effect of CTN care by conducting multiple analyses to handle noncompliance in the trial.

Methods

The trial recruited target children in Simcoe County and York Region, ON, Canada. Children were randomized to receive CTN or usual care and were followed for 2 years. The CTN group received integrated services through multiple providers to address their specific needs while the usual care group continued to receive care directed by their parents. The outcome was change in psychosocial quality of life at 2 years. We conducted intention-to-treat, as-treated, per-protocol, and instrumental variable analyses to analyze the outcome.

Results

The trial randomized 445 children, with 229 in the intervention group and 216 in the control group. During follow-up, 52% of children in the intervention group did not receive complete CTN care for various reasons. At 2 years, we did not find a significant improvement in psychosocial quality of life among the children receiving CTN care compared with usual care (intention-to-treat mean difference 1.50, 95% confidence interval −1.49 to 4.50; P = 0.32). Other methods of analysis yielded similar results.

Conclusion

Although the effect of CTN care was not significant, there was evidence showing benefits of integrated care for CSHCN. More RCTs are needed to demonstrate the magnitude of such an effect. The CTN study highlights the key challenges in RCTs when assessing interventions involving integrated care, and informs further RCTs including similar evaluations.

Keywords:

• children with special health care needs
• chronically ill
• family-centered care
• randomized controlled trial
• noncompliance

Introduction

Children with special health care needs (CSHCN) are those “who have or are at increased risk for a chronic physical, developmental, behavioral, or emotional condition, and who also require health and related services of a type or amount beyond that required by children generally”.¹ In Canada, about 13%–18% of children are considered to have special needs.² These children usually have poor quality of life³ associated with emotional symptoms,⁴ psychologic problems,⁵ and health risk behaviors.⁶ Two thirds of parents with a disabled child do not get the necessary treatments for their child and more than one third do not know where to look for help.⁷ These parents endure enormous financial and mental burdens due to their child’s illnesses.⁸ Since most CSHCN live at home, parents have to quit their jobs, work fewer hours, turn down promotion, or compromise job performance⁹ to take care of their child with special needs. “Finding services, battling waitlists, travelling long distances, briefing numerous professionals from different sectors on their child’s history, and coordinating services from multiple agencies” are just part of their daily life.¹⁰ These parents commonly describe continuing strain and chronic fatigue concerning their child’s well-being,¹¹ and over 35% of them report high levels of emotional distress.¹² Their families are also affected by high divorce rates.¹³

CSHCN are usually diverse in nature and need an individualized approach to address their specialized and multiple needs.¹⁴ The value of continuous, comprehensive, coordinated, and family-centered care¹⁵ has been widely recognized. Delivery of this new type of care requires a more integrative approach than the traditional one through: collaboration with parents to decide on case-specific goals; coordination of an individual team of service providers; and provision of a single point of access to address families’ multiple needs. As an example, the medical home¹⁶ in the US is a new development in health care delivery for families with CSHCN. It has led to reductions in wait-time and unmet health care needs for target children,¹⁷ increased use of preventive care,¹⁸ decreased risk of comorbidities,¹⁹ and alleviation of family burden.²⁰ Adoption of these strategies in the Canadian system has been limited. There are examples of innovative interventions (for example, the chronic care model in British Columbia and the Family Health Network in Ontario), but these interventions target adult patients.¹⁴ Compared with adults, CSHCN are affected by a larger number of uncommon conditions and demand more tailored services and a more complex model of care.

In 2006, a new model of care for CSHCN and their families, known as the Children’s Treatment Network (CTN), was launched in Simcoe County and York Region in Ontario, Canada. The CTN model of care is funded by the government and is based on collaboration of local service providers from different agencies and organizations. It provides a single point of access and service navigation for children. The CTN model is unique, in that it assigns a service coordinator and an individual team of health service providers for each target family, develops a single plan of care, and uses an electronic record system to share clinical information and assessment of the child. Some services available to families through the CTN include developmentally appropriate therapy, speech therapy, augmentative communication, in-home social support, behavior therapy, and early childhood education (www.ctn-simcoeyork.ca). Before the launch of the CTN, parents in both communities had limited access to the specialized treatments needed for their children and had to travel outside of the region to access those services. Funded by the Ministry of Children and Youth Services and the Ministry of Health and Long-term Care In Ontario, Canada, Thurston et al²¹ conducted a randomized controlled trial to compare the integrated interventional program of care organized through the CTN versus usual care for managing children with multiple and complex health needs in Simcoe County and York Region (trial registration in clinicaltrials.gov NCT01379443). This CTN trial was subject to substantial noncompliance, where about 52% of children in the intervention group only received parts of CTN integrated care. Some analyses of the CTN were published previously.^22,23

The primary goal of the CTN trial was to assess the effect of CTN integrated care on improving the psychosocial function of target children in Simcoe County and York Region. In this study, we compared the intention-to-treat (ITT) method with alternative methods of analyzing the data. We also aimed to assess the sensitivity of ITT analysis for participants’ noncompliance with the CTN intervention.

Materials and methods

We describe the CTN trial here by adopting the Consolidated Standards of Reporting Trials guideline.²⁴ Using a parallel design, each eligible child was randomized to an intervention group or a control group with a 1:1 allocation ratio.

Participants

Participants were recruited from the Simcoe County and York Region in Ontario, Canada. The Simcoe York District Health Council²⁵ projected the total number of children in both regions to be 390,498 in 2007. According to a national survey, 1% of children were estimated to have severe disabilities and receive rehabilitation services.²⁶

Children aged 0–19 years were recruited if they had any of the following conditions: cerebral palsy, brain injury, developmental difficulties, Down syndrome, spina bifida, autism, physical disability, developmental disability, pervasive developmental disorder, or a chronic medical condition. Children who were receiving palliative care, requiring emergency services, or living outside of the region at the time were excluded. Non-English speaking families without an English translator were also excluded.

Potential children were identified through the Community Care Access Centres, school boards, and Early Intervention Services agencies. Eligible families were contacted by mail with an invitation package, which contained a study information letter, and a parental consent and participant contact information form. A second mail-out was done to families who did not respond to the initial mail-out. Only children whose parents had provided written informed consent could participate. Children were enrolled from May to December 2007. A unique patient ID was created for each enrolled child.

Ethics

There were no known harms or safety risks to the children and their families involved in the trial. The research ethics board at Hamilton Health Sciences/McMaster Health Sciences approved the study.

Intervention

Each child in the CTN group was assigned a service navigator who conducted a comprehensive assessment to identify the child’s health conditions. A trained service coordinator then followed up with the family. In working with the parents, an individual team of service providers was formed according to the child’s health and social needs. For example, a team comprising an augmentative communication services consultant, early interventionist, occupational therapist, physiotherapist, speech and language therapist, and service coordinator was assembled for a child with cerebral palsy. This integrative team together with family members developed a single plan of care for the child. The service coordinator organized the delivery of services according to the plan. The team met with the family for ongoing assessment and revision of the plan on a regular basis. All assessments and clinical notes concerning the child were documented in an electronic record system shared by all team members. Figure 1 shows the detailed components of CTN integrated care. Families in the control group continued to manage services for their child in a self-directed manner. This group did not have access to the CTN integrated care; however, they had access to all other service providers (including CTN partners) as requested.

Figure 1 Components of the Children’s Treatment Network versus usual care.

Abbreviations: CTN, Children’s Treatment Network; RCT, randomized controlled trial; REB, research ethics board.

Fidelity to integrated child and family team care was evaluated by: content in the child and family team of providers (extent, scope, and agreement in collaboration); quality of team functioning, measured by the Partnership Functioning Scale;²⁷ and the frequency and duration coverage of the prescribed services as outlined in the single plan of care compared with providers’ entries on electronic record.

Outcome and variables

The outcome was the change in the child’s psychosocial quality of life (QoL) in 24 months. This was reported by parents using the short form Pediatric Quality of Life (SF PedsQL).²⁸ Psychosocial QoL was the sum of the emotional, social, and school functions and was extrapolated to have a score from 0 (the worst QoL) to 100 (the best QoL). Information on each child’s age, sex, and admission diagnosis was collected. The age, marital status, educational level, and family income of the parent who was the most knowledgeable (PMK) about the child were recorded.

Parental distress score

The Kessler scale (K10)²⁹ was used to measure PMK symptoms of depression and anxiety. Ten questions measured these feelings: sad, nervous, restless, hopeless, worthless, everything was an effort, tired for no good reason, so nervous that nothing could calm down, so restless and could not sit still, and depression during the past month. The PMK rated each item on a five-point scale (1, “all of the time” to 5, “none of the time”). Scores range from 10 (no symptom of distress) to 50 (severe distress).

Positive parenting score

The National Longitudinal Survey of Children and Youth parenting scale was used, and five questions adapted from the Parent Practices Scale³⁰ were used to measure positive parenting behaviors (eg, praise, play). The PMK rated each item (eg, “Do something special with your child that he/she enjoys”) on a five-point scale (0, “never” to 4, “many times each day”). Higher scores indicate more positive parenting behaviors.

Social support score

An eight-item shortened version of the Social Provisions Scale³¹ was used to measure the level of social support that the PMK received in guidance, reliable alliance, and attachment. PMK rated each item on a four-point scale (0, “strongly disagree” to 3, “strongly agree”). Higher scores represent a greater level of social support.

Family function score

Thirteen items taken from the National Longitudinal Survey of Children and Youth population survey, based on a subscale of the McMaster Assessment Device,³² were used to measure size was calculated to be 240 for a two-sided various aspects of family functioning in problem-solving, communication, affective responsiveness, affective involvement, and behavior control. The PMK rated each item on a four-point scale (0, “strongly agree” to 3, “strongly disagree”). Scores of negatively oriented items (eg, “We avoid discussing our fears or concerns”) were reversed so that higher scores represent greater family dysfunction.

Sample size

Using a minimum clinically important difference of ten on the target children’s psychosocial QoL³³ and an estimated standard deviation of 28 estimated from our pilot sample, the sample size was calculated to be 240 for a two-sided test with 5% alpha and 80% statistical power. To allow for ten independent variables and an attrition rate of 20%, the required sample size was 425.

Randomization

Using a parallel design, eligible children with parental consent were randomized to the intervention and control groups with a 1:1 allocation ratio. Children were stratified by region (Simcoe/York), Community Care Access Centre, and age (pre-school/school). They were randomized within stratum by using a block size of six. The randomization list was generated by the Health and Social Service Utilization Research Unit (HSSURU) at McMaster University (fhs.mcmaster.ca/slru). The allocation codes were then sequentially linked to the patient IDs for enrolled children.

Blinding

Trained interviewers at the HSSURU, who were blinded to group allocation, collected data using a standardized questionnaire booklet containing questions on demographics and outcome measures. Data collection was performed by telephone at baseline, and at 12 and 24 months. After enrollment, children remained anonymous and were identified by their patient IDs. Only the statistician responsible for randomization had access to the allocation codes. The data analyst was blinded to group allocation; however, the participants were aware of their allocation. All data were stored and maintained at the HSSURU.

Statistical methods

We adopted the ITT principle by analyzing all patients according to how they were randomized, regardless of whether they complied with the assigned treatment or not. The ITT estimate was obtained by fitting a linear regression model with adjustment for baseline variables. The multiple imputation technique was used to impute the missing data by assuming that they were missing at random. Five imputation datasets were produced and combined to produce an estimate of 98.5% relative efficiency, given a fraction of about 30% missing data.³⁴

In the sensitivity analysis of the impact of noncompliance, we employed three other analytical approaches, ie, as-treated, per-protocol, and instrumental variable analyses. In the as-treated analysis, we compared the treated children (those who received complete CTN integrated care) with the untreated ones (those who received parts or none of the CTN integrated care) to estimate the effect of the intervention. In the per-protocol analysis, we analyzed the children by their randomization, but only ones who complied with the assigned treatment. Children in the control group did not have access to CTN integrated care and were deemed to comply with usual care. Both as-treated and per-protocol analyses compromised original randomization and were prone to selection bias and confounding. We employed the propensity score technique to adjust for those factors. Application of the propensity score in the as-treated and per-protocol analyses could potentially be a novel approach. Because we could not find any study comparing different uses of the propensity score in this context, we included all four commonly used propensity score methods, ie, matching, stratification, weighting, and covariate adjustment. The details of these propensity score methods are given in the Supplementary materials section.

The instrumental variable analysis³⁵ estimated the effect of CTN integrated care through an instrumental variable. The randomization allocation (Z) was associated with the exposure of CTN integrated care (X) but independent of confounders and any factor that explained the outcome (Y). Thus, Z satisfied the conditions to be an instrumental variable, and the treatment effect can be estimated by

${\beta }_{IV}=\frac{{\beta }_{(\text{Z}-\text{Y})}}{{\beta }_{(\text{Z}-\text{X})}}$

where β_(Z-Y) is the association between randomization and the outcome and β_(Z-X) is the association between randomization and the receipt of CTN integrated care. We assumed that the Z-Y association was independent of the Z-X association. Under the assumption, we used the delta method to derive the variance of β_IV for calculating the 95% confidence interval (CI) and P-value. The details are provided in the Supplementary materials section.

We compared the ITT method with other methods of analysis to assess the sensitivity of the results. Figure 2 summarizes the different approaches to analysis. The results are expressed as the mean difference, 95% CI of the mean difference, and corresponding P-value. The level of significance was set at 5%. All analyses were performed using statistical software R version 2.12.1 and Statistical Package for the Social Sciences version 19 software (IBM Corporation, Armonk, NY, USA).

Figure 2 Summary of different analytical approaches.

Abbreviation: PS, propensity score.

Results

Recruitment, baseline data, and participant flow

Between May and December 2007, 2,319 eligible children were identified and their families were contacted by mail. Parents of 465 eligible children returned signed consent forms to indicate their willingness to participate. Later, 20 consented parents did not complete the baseline interview and were excluded from the study. The remaining 445 children were randomized to either the intervention group (n = 229) or the control group (n = 216).

The children were followed until December 2009. During the 2-year follow-up, 53 children in the CTN group did not have a team of service providers assembled or did not have the services available for their specific needs. Another 58 children in the CTN group withdrew from CTN integrated care. Common reasons for withdrawal were: death of the child, movement of the family outside of the region, a parent’s decision to manage the child without assistance, and transfer to another health care center. Overall, 64 (28%) children in the intervention group and 57 (26%) children in the control group were lost to follow-up. Figure 3 shows the flow of participants in the study. All 445 children were analyzed using a multiple imputation technique.

Figure 3 Flow of children in the Children’s Treatment Network trial.

Abbreviation: CTN, Children’s Treatment Network.

Statistical analysis

The children’s baseline characteristics are summarized in Table 1. Categorical and continuous variables are expressed as the frequency (percentage) or mean (standard deviation), respectively. Within the CTN arm, baseline variables were compared between the children who received complete CTN integrated care (treated) and those who received partial or none of the CTN integrated care (untreated). The results of this comparison are reported in Table 2. From the comparison, we did not find any significant differences between treated and untreated children except for parenting style (mean difference 1.4; P = 0.02); however, a mean difference of 1.4 on a score ranging from 0 to 20 did not seem to be a clinically relevant association.

Table 1 Baseline characteristics

Baseline variable	Intervention (n = 229)	Control (n = 216)	P-value
Child’s age, mean (SD)	7.8 (4.3)	8.1 (4.6)	0.60
Child’s sex, frequency (%)
Male	148 (64.6%)	149 (69.0%)	0.33
Female	81 (35.4%)	67 (31.0%)
Child’s admission diagnostic, frequency (%)
Mental and developmental disorders	106 (46.3%)	95 (44.0%)	0.81
Diseases of the nervous system	53 (23.1%)	51 (23.6%)
Congenital abnormalities	39 (17.0%)	34 (15.7%)
Other disease	31 (13.5%)	36 (16.7%)
Child’s psychosocial score, mean (SD)	59.0 (18.6)	59.2 (18.6)	0.85
Parent’s age, mean (SD)	40.5 (7.6)	40.4 (7.7)	0.95
Marital status, frequency (%)
Married including common-law	190 (83.0%)	187 (86.6%)	0.29
Other	39 (17.0%)	29 (13.4%)
Parent’s education, frequency (%)
Secondary	83 (36.4%)	78 (36.1%)	0.95
Post-secondary	145 (63.6%)	138 (63.9%)
Family annual income, frequency (%)
<$30,000	32 (14.0%)	32 (14.9%)	0.96
$30,000–$90,000	121 (53.1%)	114 (53.0%)
>$90,000	75 (32.9%)	69 (32.1%)
Parent’s Kessler distress score, mean (SD)	19.5 (5.8)	20.4 (7.2)	0.16
Positive parenting score, mean (SD)	15.2 (3.1)	15.1 (3.0)	0.82
Social support score, mean (SD)	17.6 (4.7)	17.5 (4.3)	0.74
Family functioning score, mean (SD)	9.1 (6.3)	9.4 (5.9)	0.60

Notes: Continuous and categorical variables are expressed as the mean (standard deviation) or frequency (percentage), respectively. The P-value was calculated based on a t-test for continuous variables and a chi-square test for categorical variables.

Abbreviation: SD, standard deviation.

Table 2 Comparing treated with untreated children in the intervention group

Baseline variable	Treated children (n = 111)	Untreated children (n = 118)	P-value
Child’s age, mean (SD)	7.3 (4.3)	8.3 (4.3)	0.09
Child’s sex, frequency (%)
Male	70 (63.1%)	78 (66.1%)	0.63
Female	41 (36.9%)	40 (33.9%)
Child’s diagnosis on admission, frequency (%)
Mental and developmental disorders	46 (41.4%)	60 (50.8%)	0.13
Diseases of the nervous system	32 (28.8%)	21 (17.8%)
Congenital abnormalities	21 (18.9%)	18 (15.3%)
Other disease	12 (10.8%)	19 (16.1%)
Child’s psychosocial score, mean (SD)	58.1 (19.0)	59.9 (18.4)	0.49
Parent’s age, mean (SD)	40.4 (6.9)	40.6 (8.3)	0.81
Marital status, frequency (%)
Married including common-law	92 (82.9%)	98 (83.1%)	0.97
Other	19 (17.1%)	20 (16.9%)
Parent’s education, frequency (%)
Secondary	40 (36.0%)	43 (36.8%)	0.91
Post-secondary	71 (64.0%)	74 (63.2%)
Family annual income, frequency (%)
<$30,000	19 (17.3%)	13 (11.0%)	0.34
$30,000–$90,000	58 (52.7%)	63 (53.4%)
<$90,000	33 (30.0%)	42 (35.6%)
Parent’s Kessler distress score, mean (SD)	20.2 (5.9)	18.8 (5.6)	0.09
Positive parenting score, mean (SD)	15.7 (3.2)	14.8 (3.0)	0.02
Social support score, mean (SD)	17.7 (4.6)	17.6 (4.9)	0.86
Family functioning score, mean (SD)	9.1 (6.4)	9.1 (6.3)	0.97

Notes: Continuous and categorical variables are expressed as the mean (standard deviation) or frequency (percentage), respectively. The P-value was calculated based on a t-test for continuous variables and a chi-square test for categorical variables.

Abbreviation: SD, standard deviation.

We conducted different analyses to estimate the effect of CTN integrated care. The results did not show a significant improvement in the children’s psychosocial QoL in the CTN group compared with the usual care group (mean difference 1.50, 95% CI–1.49, 4.50; P = 0.32). The results for the estimated treatment effect are reported in Table 3. In general, the conclusion from the ITT analysis was consistent with that drawn from the as-treated, per-protocol, and instrumental variable analyses. We also noticed some systematic patterns in the direction, magnitude, and precision of the estimates. All estimates except for the as-treated estimates favored CTN integrated care. The per-protocol estimates were the closest to 0 which represented no difference between CTN integrated care and usual care. Both ITT and instrumental variable estimates showed a larger effect of CTN integrated care over usual care than the per-protocol estimates. The instrumental variable estimate had the largest 95% CI. For the as-treated and per-protocol analyses which compromised the original randomization, adjusting for confounders and imbalance by matching on the propensity score produced estimates with a wider 95% CI than did the other propensity score methods. Figure 4 shows a comparison of the estimates obtained from different analyses.

Figure 4 Comparing the estimates of treatment effect from different analyses.

Note: Mean difference refers to the difference in the change in psychosocial score in 2 years between groups.
Abbreviations: ITT, intention-to-treat; AT, as-treated; PP, per-protocol; IV, instrumental variable; PS, propensity score.

Table 3 Summary of estimates of treatment effect

	Multiple imputation				Raw data
	MD	95% CI		P-value	MD	95% CI		P-value
ITT analysisa	1.50	−1.49	4.50	0.32	–a	–	–	–
AT analysis
Matched by PS	−2.60	−7.45	2.24	0.29	−1.82	−6.49	2.86	0.45
Stratified by quintiles of PS	−0.89	−4.34	2.57	0.61	−1.17	−5.01	2.68	0.55
Weighted by PS	−0.75	−3.75	2.25	0.62	−0.80	−4.10	2.50	0.63
Adjusted by PS	−1.12	−4.62	2.39	0.53	−1.15	−4.99	2.70	0.56
PP analysis
Matched by PS	0.67	−3.44	4.78	0.75	2.26	−2.60	7.12	0.36
Stratified by quintiles of PS	0.37	−3.33	4.06	0.85	0.47	−3.58	4.52	0.82
Weighted by PS	0.21	−3.24	3.66	0.91	0.70	−3.16	4.56	0.72
Adjusted by PS	0.02	−3.23	3.27	0.99	−0.20	−3.71	3.31	0.91
IV analysis	3.10	−3.08	9.29	0.33	5.10	−0.78	10.97	0.09

Note:

a ITT analysis requires the inclusion of all randomized patients and was only performed when patients’ missing observations were imputed by multiple imputation.

Abbreviations: MD, mean difference (difference in change of psychosocial score over 2 years between groups); CI, confidence interval; ITT, intention-to-treat; AT, as-treated; PP, per-protocol; IV, instrumental variable; PS, propensity score.

Discussion

The ITT analysis did not show a significant improvement in psychosocial QoL among children receiving CTN integrated care than those receiving usual care in Simcoe County and York Region over 2 years. We conducted as-treated, per-protocol, and instrumental variable analyses to assess the sensitivity of this conclusion under substantial noncompliance with the CTN intervention. These alternative analyses also showed no significant difference between groups. Previously, two CTN studies have been published,^22,23 which explored the interactions between multiple factors and system integration on the child’s psychosocial QoL and examined the associations between multiple factors and level of psychiatric distress experienced by parents.

Our findings were limited by a number of factors. First, the child’s psychosocial QoL was reported by their parents. Parents might have limited knowledge concerning their children’s health-related QoL. The parents’ responses reflected their own perception of children’s disease-related experiences, which might not be the same as how the children felt. For example, it was found that children with a congenital below-the-elbow deficiency reported better QoL than that perceived by their parents.³⁶ Second, the early development of the CTN model of care was associated with suboptimal intervention fidelity. Implementation of the CTN was a major undertaking in changing systems and operations management. It took longer than anticipated for the network hosts to hire and train appropriate staff at the local team level, get local team sites operational, and engage all the needed agency partner staff in the teams. The delay in creating the child and family teams was the key challenge in organizing services for some children. About 52% of the children assigned to the CTN group did not receive CTN integrated care as planned during the follow-up period. The limited success in truly engaging and retaining families in the process of care is often the reason for failure to achieve the desired health outcome.³⁷ Third, there was potential intervention contamination in the usual care group. Given that families within the same community might be randomized to a different group, families in the usual care group might learn about the intervention from those assigned to the CTN group. The CTN trial was limited by insufficient resources and funding to employ a large-scale multicenter cluster design, which could protect against intervention contamination. Consequently, contamination of control children may have reduced the estimate of intervention effect and could have led to a type II error in the results.³⁸ Fourth, our findings were limited by a relatively short period of observation after the initial launch of the intervention. For chronically ill children, the effect of integrated care on health outcomes would often accumulate over time. Thus, an interventional gain might be observed over a longer period of follow-up as practitioners became more proficient and team functioning became more collaborative. Fifth, our results were limited by missing data due to loss of patients to follow-up. We assumed that those data were missing at random and employed multiple imputation in the analysis. However, the assumption of data being missing at random is inherently untestable. Finally, the population targeted by the CTN may represent a subset of the defined CSHCN population accessing the services. Thus, our results may not be generalizable to a broader population of CSHCN, some of whom may not be accessing the services. The complex model of care and the heterogeneous population under study are the main challenges in assessing integrated health service interventions.³⁹

Our findings should be interpreted in light of previously published evidence on the CTN intervention.^22,23 For the subgroup of children who received complete CTN integrated care, their QoL was influenced by complex interactions between team integration and parenting factors, and overall, their QoL improved after 2 years.²² In a cross-sectional analysis of the CTN children,²³ we found that symptoms of psychiatric distress experienced by parents were associated with level of social support, family functioning, child behavior, and parenting style. All those factors could be mediated through coordinated and family-centered care. Also, the costs of total health and social services were higher in those parents with more symptoms of psychiatric distress. From an economic perspective, continuous and comprehensive services can save costs because giving people the services they need not only improves their QoL but also reduces use of other duplicative or otherwise fruitless services.⁴⁰ A recent systematic review showed that coordinated and family-centered care was associated with improved outcomes for CSHCN.⁴¹ A study by King et al⁴² showed that services for children with neurodevelopmental disorders were most beneficial when they were delivered in a family-centered fashion and addressed parent-identified issues. The current evidence is primarily from nonrandomized studies. More randomized controlled trials are needed to assess the effect of integrative care that is continuous, comprehensive, coordinated, and family-centered for CSHCN. The CTN intervention is the first of its kind in Canada. It aligns with the advocacy of comprehensive care for children’s mental health,⁴³ family-centered rather than child-centered interventions,⁴⁴ and partnerships with community psychology in the system of care.⁴⁵ Considering that 13%–18% of Canadian children had special health care needs, the CTN appears to be a milestone in the improvement of QoL for CSHCN through integrative care. At the same time, multiple community, family, parent, and child factors around the management of CSHCN make it challenging to evaluate this type of intervention. The ongoing implementation and evaluation of the CTN is an important step forward for Canadian community-based child care programs targeting this vulnerable population. The methodologic limitations in our study inform further trials when evaluating similar interventions.

Conclusion

In this trial, we assessed the effect of CTN integrated care versus usual care on psychosocial QoL in CSHCN from Simcoe County and York Region over 2 years. We did not find a significant improvement in QoL for children in the CTN group compared with those in the usual care group. The value of continuous, comprehensive, coordinated, and family-centered care for CSHCN is increasingly being recognized. At the same time, more research is needed to demonstrate the magnitude of the effect of integrated care on those children and their parents. Given that noncompliance is a common issue in randomized controlled trials, especially pragmatic trials, a better understanding of the performance of different analytical approaches is also crucial. Future simulation studies are needed to provide insights into this question.

Author contributions

CY conceived the sensitivity analysis, proposed statistical methods, performed all analyses, interpreted the results, and drafted and revised the manuscript. GB advised on important intellectual content and revised the manuscript. GB was also commissioned by the CTN to design, carry out, analyze, and report the original trial. LT and JB contributed to the statistical design of the sensitivity analysis and revision of the manuscript. All authors have read and approved the final manuscript.

Acknowledgments

CY is supported in part by funding from the Father Sean O’Sullivan Research Centre studentship award, the Canadian Institute of Health Research Training award in Bridging Scientific Domains for Drug Safety and Effectiveness, and the Canadian Network and Centre for Trials Internationally program.

Supplementary materials

Propensity score methods

The propensity score (PS) is the probability of receiving a treatment conditional on observed characteristics. Balancing on PS mimics randomization and produces comparable groups that are balanced on prognosis and observed confounding. We collected baseline variables that included child’s age, sex, admission diagnosis, parent’s age, marital status, education, distress, parenting style, family income, social support, and family functioning. Using those variables, we calculated the PS for receiving complete Children’s Treatment Network (CTN) integrated care by a logistic regression model. Four PS methods were used to balance between treated and untreated children in the AT and PP analyses, ie, matching, stratification, weighting, and covariate adjustment.

In PS matching, we created matched pairs of treated and untreated children by matching them within 0.2 of the standard deviation of the logit of the PS. We used a 1:1 ratio to match the nearest children without replacement. This matching algorithm produced the least bias.¹ We then used the generalized estimating equation model to analyze the matched data. An exchangeable correlation structure was used.

In PS stratification, children were divided into five equal strata by the quintiles of their PS.² The outcome between treated and untreated children was compared directly in each stratum. The overall treatment effect is given by

${\beta }_{overall}=\frac{{\displaystyle {\sum }_i^s{\beta }_i}}{s},$

where β_i is the estimated treatment effect in stratum i. The variance of β_overall is calculated by

$Var({\beta }_{overall})=\frac{1}{{\displaystyle {\sum }_i^s{w}_i}},$

where w_i is the inverse of the estimated variance of β_i for stratum i.

In the PS-weighted method, we balanced the children by the inverse probability of receiving CTN integrated care. The weight (w_i) for each child can be calculated by

$w_i=\frac{X_i}{e_i}+\frac{(1-X_i)}{1-e_i},$

where X_i is the treatment indicator and e_i is the estimated PS for child i. Confounding between the observed variables and treatment was eliminated in the weighted sample. We then directly compared the outcome between treated and untreated children using a weighted linear regression model.

In the covariate adjustment method, we adjusted the PS as the sole covariate in the regression model for estimating the treatment effect.

Calculating standard error for instrumental variable estimate

Suppose we have two random variables, X and Y. A Taylor series expansion of f(x, y) about the values (x₀, y₀) is given by

$\begin{array}{c}f(x,y)=f(x_0,y_0)+(x-x_0)\frac{\partial f(x,y)}{\partial x}{\left|{}_{(x_0,y_0)}+(y-y_0)\frac{\partial f(x,y)}{\partial y}\right|}_{(x_0,y_0)}+o(\cdot )\\ \approx f(x_0,y_0)+(x-x_0)\times \frac{\partial f(x,y)}{\partial x}{\left|{}_{(x_0,y_0)}+(y-y_0)\frac{\partial f(x,y)}{\partial y}\right|}_{(x_0,y_0)}\end{array}$

where o(⋅) represents the higher order terms which are omitted in this approximation. The instrumental variable (IV) estimator for the exposure X and the outcome Y is given by

${\beta }_{IV}=\frac{{\beta }_{(Z\to Y)}}{{\beta }_{(Z\to X)}}$

where β_(Z→Y) and β_(Z→X) are the parameters of association between Z and Y and between Z and X, respectively. By the Taylor series expansion on β_IV, we have

${\beta }_{IV}\approx \frac{b_{(Z\to Y)}}{b_{(Z\to X)}}+({\beta }_{(Z\to Y)}-b_{(Z\to Y)})\frac{1}{b_{(Z\to X)}}-({\beta }_{(Z\to X)}-b_{(Z\to X)})\frac{b_{(Z\to Y)}}{b_{(Z\to X)}{}^2}$

where b_(Z→Y) and b_(Z→X) are two values at which β_IV is differentiable. The variance of the IV estimator can then be approximated as

$Var({\beta }_{IV})\approx \frac{1}{b_{(Z\to X)}{}^2}Var({\beta }_{(Z\to Y)})+\frac{b_{(Z\to Y)}{}^2}{b_{(Z\to X)}{}^4}Var({\beta }_{(Z\to X)})-\frac{2b_{(Z\to Y)}}{b_{(Z\to X)}{}^3}Cov({\beta }_{(Z\to X)},{\beta }_{(Z\to Y)}).$

Under the assumption that β_(Z→χ) is independent of β_(Z→Y), the variance of the IV estimator is then

$Var({\beta }_{IV})\approx \frac{1}{b_{(Z\to X)}{}^2}Var({\beta }_{(Z\to Y)})+\frac{b_{(Z\to Y)}{}^2}{b_{(Z\to X)}{}^4}Var({\beta }_{(Z\to X)}).$

We substitute b_(Z→Y) and b_(Z→X) by the estimates of ß_(Z→Y) and ß_(Z→X), respectively, and approximate the $\sqrt{Var({\beta }_{(Z\to Y)})}$ and $\sqrt{Var({\beta }_{(Z\to X)})}$ by the associated standard error. Thus, we can obtain an approximate variance of the IV estimate. In our analysis, we used the least squares estimate of ß_(Z→Y) and associated standard error obtained from the linear regression model. For the exposure X (a binary indicator of whether or not a patient received complete CTN integrated care), the association between X and the IV can be calculated by

${\beta }_{(Z\to X)}=P(X=1|Z=1)-P(X=1|Z=0)$

where P(X = 1|Z = 1) represents the proportion of treated patients in the CTN group; and P(X = 1|Z = 0) is always zero because the children in the usual care group are deemed to be untreated. The variance of β_(Z→X) is

$Var({\beta }_{(Z\to X)})=nP(X=1|Z=1)(1-P(X=1|Z=1))$

where n is the number of children in the CTN group.

References

• AustinPCOptimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studiesPharm Stat201110215016120925139
   PubMed Web of Science ®Google Scholar
• RosenbaumPRRubinDBReducing bias in observational studies using subclassification on the propensity scoreJ Am Stat Assoc198479387516524
   Web of Science ®Google Scholar

Disclosure

The authors report no conflicts of interest in this work.