The test-retest reliability of non-navigated transcranial magnetic stimulation (TMS) measures of corticospinal pathway excitability early after stroke

Abstract

Purpose

Motor evoked potential (MEP) characteristics are potential biomarkers of whether rehabilitation interventions drive motor recovery after stroke. The test-retest reliability of Transcranial Magnetic Stimulation (TMS) measurements in sub-acute stroke remains unclear. This study aims to determine the test-retest reliability of upper limb MEP measures elicited by non-neuronavigated transcranial magnetic stimulation in sub-acute-stroke.

Methods

In two identical data collection sessions, 1–3 days apart, TMS measures assessed: motor threshold (MT), amplitude, latency (MEP-L), silent period (SP), recruitment curve slope in the biceps brachii (BB), extensor carpi radialis (ECR), and abductor pollicis brevis (APB) muscles of paretic and non-paretic upper limbs. Test-retest reliability was calculated using the intra-class correlation coefficient (ICC) and 95% confidence intervals (CI). Acceptable reliability was set at a lower 95% CI of 0.70 or above. The limits of agreement (LOA) and smallest detectable change (SDC) were calculated.

Results

30 participants with sub-acute stroke were included (av 36 days post stroke) reliability was variable between poor to good for the different MEP characteristics. The SDC values differed across muscles and MEP characteristics in both paretic and less paretic limbs.

Conclusions

The present findings indicate there is limited evidence for acceptable test-retest reliability of non-navigated TMS outcomes when using the appropriate 95% CI for ICC, SDC and LOA values.

Clinical Trial Registration

Current Controlled Trials: ISCRT 19090862, http://www.controlled-trials.com

IMPLICATIONS FOR REHABILITATION

• This study identified that Non-navigated Transcranial Magnetic Stimulation (TMS) demonstrates low reliability of TMS measures in upper limb with variation between muscles and measures in sub-acute stroke

• When using non-navigated TMS to explore corticospinal pathway excitability the individual target muscle and TMS measure should be taken into consideration

• Non-navigated TMS may be more useful in exploring group differences rather than individual differences in corticospinal pathway excitability

• Non-navigated TMS could provide a means of measuring recovery in clinical practice and could inform the development of more effective interventions but this needs further development before it can be used as a clinical recovery biomarker

Keywords:

• Test-retest reliability
• transcranial magnetic stimulation (TMS)
• smallest detectable change
• stroke
• upper limb

Introduction

Transcranial Magnetic stimulation (TMS) is a non-invasive method of brain stimulation [1,2] that has been used extensively in healthy participants to probe the Central Nervous System, investigate brain plasticity and to better understand the underlying neurophysiology of motor control [3–5]. This technique has increasingly been applied to clinical populations, in particular stroke for diagnostic or prognostic purposes [6,7] and to evaluate underlying neurological change in clinical trials [8,9].

Rehabilitation of motor function tailored to individual stroke survivors (precision stroke rehabilitation) is promised by the incorporation of biomarkers of recovery, into clinical trials and clinical practice [10]. Biomarkers are expected to advance the science and practice of precision stroke rehabilitation by distinguishing subgroups of stroke survivors and ascertaining whether specific rehabilitation interventions drive recovery. An international consensus recommends measuring the functional integrity of the corticospinal tract (CST), using TMS to identify the presence or otherwise of a motor-evoked potential (MEP) [11]. If TMS is going to be incorporated into clinical practice and neurorehabilitation, there must be robust investigation of the psychometric properties of the technique; in particular, the test-retest reliability of TMS outcome measures [12,13].

A systematic review of the reliability of TMS measures in healthy adults and other work [13,14] has highlighted methodological issues in the interpretation of TMS reliability, including the lack of consideration for the 95% confidence intervals (95% CI) for intraclass correlation coefficient (ICC) values when interpreting the findings, small sample sizes and lack of consideration of measurement error.

The majority of test-retest studies have concentrated on healthy participants [15,16], there has been less investigation of test-retest reliability of TMS measures in chronic stroke individuals [13,17–20]. This lack of evidence is even more pronounced for subacute stroke [13,21].

When TMS is used with neuro-navigation, some TMS measures (such as motor threshold, MEP latency and amplitude) in some distal upper limb muscles have been found to possess good to excellent test-retest reliability in neurologically intact adults [13,16], people within six months of stroke; and people greater than six months after stroke [13]. Consequently, neuro-navigated TMS may be a valuable tool to advance the science of precision stroke rehabilitation. However currently fully equipped TMS laboratories including full neuronavigation systems, are practically exclusively found in Universities or research centres, clinical access to this neuronavigation may be limited. This has a direct impact on this tool’s ability to be widely used, if all the equipment including advanced neuronavigation is required for every TMS session to check appropriate stimulation parameters. Consequently, if TMS is used widely in clinical neurorehabilitation, TMS used without neuro-navigation also needs to demonstrate good to excellent test-retest reliability.

The results of the test-retest reliability using non-navigated TMS in stroke survivors six months after stroke, are variable and range from poor to good in the upper and lower limbs. The MEP amplitude demonstrated the most variable findings, ICC values ranging from ICC = 0.205 in the vastus lateralis [22] to an ICC = 0.98 (lower level of the 90% confidence interval 0.94) in the (first dorsal interosseous) FDI muscle [18]. However, participants in these studies were not clinically representative, particularly because people with severe paresis were excluded. In addition, these findings may not be transferable to the sub-acute phase after stroke. This is important because most people receive the majority of rehabilitation during the first few weeks after stroke when neural plasticity is more rapid and therefore considered the optimal time for targeted interventions [23,24]. During this time, it is essential to know whether a change in any TMS-derived measure is attributable to neural factors or measurement ‘error’ [25]. In sub-acute stroke participants it has been demonstrated for a range of TMS measures including MEPs, total conduction time in APB muscle [21] and FDI [13] that there was more variability resulting in lower ICC, higher measurement error than in healthy groups and chronic stroke groups. There is also a lack of examination of the test-retest reliability of TMS measures in a range of muscles, with the majority of studies targetting the hand muscles [13,18,20,21]. This missing information is crucial, given that TMS values can vary between M1 representations of different muscles along a proximal-distal gradient [26] and do not target the clinically key forearm muscles.

To address this, in the current study the muscles of investigation were the biceps brachii, extensor carpi radialis, and abductor pollicis brevis of the paretic and less paretic limbs. These muscles were selected because they are essential for the completion of activities of daily living such as dressing and grooming and are commonly affected following stroke. To ensure that the most commonly used TMS measures were included in this study a range of measures were assessed including motor threshold (MT), amplitude, latency (MEP-L), silent period (SP), and recruitment curve slope.

This study aims to determine the test-retest reliability of MEP measures, elicited using non-neuro-navigated TMS in people between two and 60 d of stroke onset.

Materials and methods

Design and ethics

A prospective correlational study embedded within a randomised clinical trial (RCT) [27].

Ethical approval was obtained for this study and the study was conducted in accordance with the Declaration of Helsinki. Participant testing was conducted at the University Exercise Laboratory and a UK University Hospital Stroke Rehabilitation Centre, participants were tested at the same location on both occasions. Identical hardware, software, equipment, and consumables were used in both locations.

Test-retest reliability was assessed over two identical TMS sessions separated by one to three days. The short inter-session time was due to the neural recovery and cortical reorganisation being at its most rapid in the first three months since ictus [23] and similar to a previous sub-acute reliability study [13].

Participants

Thirty adults in the sub-acute phase (early) after stroke were recruited from an ongoing RCT but had not started the intervention period and met the inclusion criteria: aged 18+ years, after anterior circulation ischemic or hemorrhagic stroke within the previous 2–60 d; scoring >11/33 on Motricity Index pinch subsection with more paretic upper limb; unable to complete Nine-Hole Peg Test in 50 s or less with paretic upper limb (to determine participants had enough residual upper limb motor control/movement to complete the RCT intervention); no obvious spatial neglect and no contraindications to TMS [27]. As the study was investigating the test-retest reliability of TMS measures, only participants who exhibited an MEP (in at least one of the target muscles in either hemisphere) at the baseline time point of the trial were invited to participate in this study. A full medical history, details of stroke, and clinical scores were obtained for all stroke participants (Table 1).

Table 1. Participant characteristics.

Age	73.3 ± 10.6years (range: 46–91)
Gender	Male n = 16 Female n = 14
Stroke type^a	Ischemic = 26 Haemorrhagic = 4
Stroke side^b	Right n = 23 Left n = 7
Side of brain lesion^c	Right n = 9 Left n = 21
Time since stroke (days)^d	36 ± 16 (range: 11–70)
ARAT total score Baseline^e	29 ± 17.4 (range: 0–53)
WMF total score baseline^f	42 ± 16.28 (range: 4–66)
Grip force	7.08 ± 6.35 kg (range 0–20)
Pinch Force	3.18 ± 2.73 kg (range 0–7.5)
Medication	N = 29 including medication for Hypertension (n = 8), Cholesterol (n = 9), Blood thinner (n = 14), Atrial Fibrillation (n = 5), Diabetes (n = 4), Osteoarthritis (n = 4); SSRI (n = 2), Thyroid hormone treatment (n = 1)

^a Type of stroke as determined by MRI.

^b Stroke side refers to more paretic side of the body, more stroke affected side.

^c Side of brain lesion as determined by MRI.

^d Time since stroke in days at randomisation.

^e The Action Research Arm Test (ARAT) total score at Baseline (max score 57-Scores of less than 10 points, between 10 and 56 points, and 57 points correlate with poor, moderate, and good recovery respectively.

^f The Wolf Motor Function Test (WMF) total score baseline (max score 75 Lower scores are indicative of lower functioning levels).

Procedure

Procedures were identical in sessions one and two. The experimenter had extensive experience in the use of TMS. The experimenter was the same for both sessions. Participants were seated comfortably in an office chair with their forearms relaxed on pillow placed on their lap. Arms were consistently positioned across sessions.

Surface EMG electrodes were placed in parallel along the muscle fibers of the bilateral biceps brachii (BB) and extensor carpi radialis longus (ECR) (ConMed Cleartrace ECG surface electrodes 20 mm), cup electrodes (Nicolette) placed over the abductor pollicis brevis (APB), ground electrode at the olecranon process. The EMG signal was sampled at 1000 Hz, amplified 1000x, band-pass filtered at 15–450 Hz, and saved for offline analysis. This process was standardised across all sessions.

Single pulse monophasic TMS was delivered using a Magstim 200² (Magstim Company Ltd) stimulator with a figure-of-8 coil (90 mm in diameter) [28]. The coil was placed so that the axis of intersection between the two loops was orientated at approximately 45° to the sagittal plane, to induce a posterior to anterior current flow across the motor strip in the primary motor cortex (M1). Once the hotspot for each muscle was established, the coil position was marked on the head with a permanent pen and used throughout the experiment. The hot spot was determined for each muscle individually at each session [29]. Resting motor threshold (rMT), was established as the lowest stimulation intensity at which MEPs with peak-to-peak amplitude of approximately 50 µV were evoked in at least 5 out of 10 MEPs, at that hotspot location. Active Motor threshold (AMT) was established when half of the successive TMS pulses (in at least 5 out 10 trials while the participants maintained a constant background contraction) produced an MEP > 200 µv [30]. There is evidence that after stroke there is variability in force production when maintaining a specific percentage of the maximal voluntary contraction in both upper and lower limbs [31,32]. Furthermore, there is greater variation with lower percentages of the MVC, which within this experiment would be between 10-20% MVC [32]. This evidence along with the participant burden and fatigue of completing multiple upper limb assessments prior to TMS data collection participants were asked to maintain a slight muscle contraction during TMS data collection with the muscle of interest during active trials and this was monitored by palpation, vision, and visual EMG by the Experimenter (a qualified physiotherapist).

An active recruitment curve was obtained from 100% AMT to 130% AMT increasing in 10% increments (bilateral limbs). Five TMS pulses were delivered at each intensity. This process was repeated for each muscle (BB, ECR, APB) in both upper limbs. When collecting active recruitment curves the non-paretic BB hotspot (which corresponds to stimulation of non-stroke lesioned hemisphere) was always probed first, followed by non-paretic ECR, non-paretic APB, paretic (stimulation of lesioned hemisphere) BB, paretic ECR and paretic APB. The order of the TMS measures was purposefully fixed so that any order effect, if present, would have a consistent influence on measurement variability. All participants were asked to attend to their upper limb during data collection, instructed to relax all other muscles and the level of arousal and fatigue was monitored by an experimental team [33].

Data analysis

Active Recruitment curve was collected for each muscle in both hemispheres. From these the following measurements were extracted MEP amplitude, silent period (measured at 130% active motor threshold), MEP latency (at 120% active motor threshold), and recruitment curve slope.

Following completion of data collection, all MEPs were visually assessed (to determine if there was any electrical noise or a lack of an MEP and trials without an MEP or with electrical noise were not analysed). Then using a custom-made script, values were extracted for: peak-to-peak MEP amplitude; visual analysis of the MEP was used to determine MEP latency [18], and the Cortical Silent Period [34]. The cortical silent period was assessed at 130% AMT as there is evidence that the cortical silent period has the smallest coefficient of variation at this threshold (vs 110% and 120% AMT [30]. The raw MEP amplitudes were fitted using both a Boltzmann sigmoid function [29,35] and a linear function [36] to estimate the slope of the recruitment curve (RC). Data were fitted to both sigmoidal and linear function as there has been issues fitting data, especially from more proximal upper limb muscles [35,37].

The data were tested for normality and found to exhibit a normal distribution. The test-retest reliability of TMS measures was assessed using the intra-class correlation coefficient (ICC) model 2,1 with associated 95% confidence intervals [95%CI] and the limits of agreement (LOA). The ICC model was used to determine agreement between the two sessions [12] and interpreted with the associated 95% CI such that an ICC > 0.70 is acceptable reliability, determined by the lower 95% CI. The LOA provided further estimates of measurement agreement and evaluated if there was a biased pattern of error (systematic or random).

The SDC provides a value for the minimum change that needs to be observed, in order to be confident that the observed change is real and not, potentially, a product of measurement error in the instrument [38]. The smallest detectable change (SDC) was calculated using the formula: $1.96 x \sqrt{2} x \mathit{\text{SEM}}$ [21].

Results

Sixty-seven stroke survivor participants in one of the three trial centres were screened for inclusion. Of these, 43 people (63%) were suitable to have TMS, 36 (84%) of those provided informed consent of whom 30 (83%) were included in data analysis. The 6 people who were consented but not included in the analysis had only completed one of the two required sessions. Participant characteristics are detailed in Table 1. Thirty participants were included with a mean age of 73.3 (range 46–91) years and a mean of 36 (SD 16 d, range 11–70) days after stroke. Participants were included irrespective of mild, moderate or severe upper limb dysfunction as demonstrated by a mean Action Research Arm Test score of 29 of a possible total of 57 (range 0–53) and a mean Wolf Motor Function Test score of 42 of a maximum possible total of 75 (range 4–66) [27]. These scores demonstrate that testing was conducted on a clinically representative sample.

No adverse TMS events occurred. There were 10% of trials across the participants not included in the analysis due to either a MEP not being present, or electrical noise inhibiting detection and analysis of the MEP.

Test-retest reliability of MEP characteristics

The test-retest reliability of TMS measures (including AMT, silent period, MEP latency and MEP amplitude) in individuals with stroke was variable within and between MEP characteristics and muscles (Tables 2–4).

Table 2. Test-retest reliability and smallest detectable difference (SDC) of: motor threshold (% stimulator output), motor evoked potential latency (MEP-L, ms) and silent period (ms).

	Non-Paretic UL	Paretic UL	Non-Paretic UL		Paretic UL
	ICC [95% CI]	ICC [95% CI]	Mean difference (95% LOA)	SDC	Mean difference (95% LOA)	SDC
RMT
BB	0.47 [0.14,0.71]^a	0.67 [0.38,0.84]^b	−0.40 (−19.94; 19.87)	22.20	−3.67 (−27.27;19.93)	25.76
ECR	0.81 [0.64,0.90]^a	0.73 [0.49,0.80]^c	−0.29 (−14.49; 13.92)	12.68	−6.04 (−31.77;19.69)	18.72
APB	0.66 [0.39,0.83]^d	0.77 [0.54,0.89]^e	−1.60 (−11.81;8.61)	21.01	−5.17 (−25.37;15.04)	12.63
AMT
BB	0.65 [0.38,0.82]^a	0.69 [0.45,0.84]^d	−0.20 (−13.87;12.76)	12.30	−2.18 (−18.90;14.55)	14.66
ECR	0.76 [0.56,0.88]^a	0.84 [0.68,0.93]^d	−0.29 (−8.37;7.04)	7.64	0.09 (−16.34;10.88)	13.85
APB	0.64 [0.37,0.80]^d	0.81 [0.62,0.91]^f	−0.26 (−12.33;9.59)	10.81	−1.96 (−19.86;15.95)	17.52
MEP-L
BB	0.71 [0.46,0.85]^g	0.65 [0.32,0.84]^g	0.20 (−3.18;3.59)	3.31	0.27(−3.84;4.38)	14.51
ECR	0.35 [0.00,0.64]^g	0.30 [0.00,0.64]^g	−0.34 (−4.13;3.45)	7.15	0.23 (−9.79;10.24)	8.72
APB	0.67 [0.40,0.83]^c	0.77 [0.49,0.91]^b	0.55 (−2.99;4.09)	3.66	−0.05 (−2.67;2.56)	2.63
SP
BB	0.53 [0.20,0.75]^f	0.45 [0.04,0.72]^b	−7.90 (−88.15;72.34)	85.50	−3.43 (−74.35;67.49)	72.79
ECR	0.19 [0.00,0.51]^f	0.75 [0.50,0.89]^b	−13.89 (−107.43;79.65)	97.22	−10.66 (−68.06;46.75)	56.53
APB	0.66 [0.38,0.83]^g	0.84 [0.62,0.94]^i	−11.67 (−78.38;55.65)	71.86	−0.68 (−48.72;47.35)	51.56

^a n = 30; ^bn = 24; ^cn = 27; ^dn = 29; ^en = 25; ^fn = 26; ^gn = 28; ^hn = 22; ⁱn = 18.

ICC: intraclass correlation coefficient; CI: confidence interval; LOA: limits of agreement; UL: upper limb; RMT: resting motor threshold; BB: biceps brachii; ECR: extensor carpi radialis; APB: abductor pollicis brevis; AMT: active motor threshold; MEP-L: motor evoked potential latency.

Table 3. Test-re-test reliability and smallest detectable difference (SDC) of motor evoked potential (MEP) amplitude (µv) at different stimulation intensities.

	Non-Paretic UL	Paretic UL	Non-Paretic UL		Paretic UL
	ICC [95% CI]	ICC [95% CI]	Mean difference (95% LOA)	SDC	Mean difference (95% LOA)	SDC
BB
100% AMT	0.46 [0.14, 0.70]^a	0.35 [0.00, 0.63]^b	0.17 (−0.60;0.94)	0.77	0.03 (−0.71;0.77)	2.94
110% AMT	0.57 [0.28, 0.77]^a	0.24 [0.00, 0.55]^b	0.20 (−0.91;1.30)	0.89	0.30 (−2.03;2.63)	1.95
120% AMT	0.55 [0.25 0.76]^a	0.23 [0.00, 0.54]^b	0.17 (−1.50;1.85)	1.19	0.26 (−1.63;2.16)	1.70
130% AMT	0.51 [0.20, 0.73]^a	0.11 [0.00, 0.47]^c	0.24 (−1.85;2.33)	1.57	0.55 (−2.73;3.82)	2.75
ECR
100% AMT	0.46 [0.14, 0.70]^a	0.39 [0.02, 0.67]^c	−0.32 (−2.79;2.15)	2.42	−0.07 (−1.19;1.05)	1.23
110% AMT	0.43 [0.10, 0.67]^a	0.34 [0.00, 0.63]^d	−0.57 (−3.25;2.11)	2.62	−0.18 (−1.87;1.50)	1.49
120% AMT	0.35 [0.02, 0.62]^a	0.31 [0.00, 0.62]^c	−0.59 (−3.65;2.47)	2.99	−0.16 (−1.82;1.49)	1.54
130% AMT	0.41 [0.08, 0.66]^a	0.33 [0.00, 0.65]^e	−0.54 (−3.37;2.28)	2.72	−0.12 (−1.80;1.56)	1.59
APB
100% AMT	0.77 [0.56, 0.88]^b	0.34 [0.00, 0.65]^f	0.05 (−1.71;1.61)	1.55	−0.21 (−3.42;2.99)	2.90
110% AMT	0.69 [0.44, 0.84]^b	0.74 [0.48, 0.87]^f	−0.29 (−2.47;1.89)	1.96	−0.04 (−2.67;2.53)	1.94
120% AMT	0.67 [0.40,0.83]^b	0.88 [0.75, 0.95]^g	−0.04 (−2.85;2.76)	2.46	−0.06 (−2.08;1.96)	1.88
130% AMT	0.66 [0.38, 0.82]^b	0.77 [0.49, 0.90]^h	−0.06 (−3.06;2.94)	2.65	0.08 (−2.99;3.15)	3.00

^a n = 30; ^bn = 29; ^cn = 25; ^dn = 27; ^en = 24; ^fn = 26; ^gn = 23; ^hn = 21.

ICC: intraclass correlation coefficient; CI: confidence interval; LOA: limits of agreement; UL: upper limb; BB: biceps brachii; AMT: active motor threshold; ECR: extensor carpi radialis; APB: abductor pollicis brevis.

Table 4. Test-retest reliability of the recruitment curve slope (RCS) for all participants.

Non-paretic UL active	Paretic UL active
RCS-sigmoidal
ICC (95% CI)
BB	0.21	[0.00, 0.77)	0.16	[0.00, 0.78)
ECR	0.22	[0.00, 0.63)	0.78	[0.00, 0.98)
APB	0.12	[0.00, 0.68)	0.00	[0.00, 0.91)
M-diff (95% LOA)
BB	0.13	(0.00,0.65)	0.00	[0.00,0.23)
ECR	0.00	(0.00,0.35)	0.21	(0.00,0.54)
APB	0.26	(0.00,0.57)	0.31	(0.00,0.60)
RCS-linear
ICC (95% CI)
BB	0.48	[0.00; 0.76]	0.03	[0.14, 0.20]
ECR	0.01	[0.22, 0.20]	0.02	[0.04, 0.08]
APB	0.04	[0.34, 0.25]	0.04	[0.55, 0.48]
M-diff (95% CICI)LOA)
BB	0.01	(−0.07;0.08)	0.02	(−0.07;0.10)
ECR	−0.01	(−0.08;0.07)	0.00	(−0.04;0.04)
APB	0.01	(−0.08;0.11)	0.02	(−0.07;0.12)

UL: upper limb; ICC: intra-class correlation coefficient; 95% CI = 95% confidence intervals; BB: biceps brachii; ECR: extensor carpi radialis; APB: Abductor Pollicis Longus; M-diff: mean difference; LOA: limits of agreement.

For RMT, AMT, SP and MEP latency the majority of estimated ICC values were good or moderate, the exceptions were non-paretic RMT BB, ECR (both limbs) latency and ECR silent period which had poor ICC values (Table 2). MEP amplitude test-retest reliability in RC stimulation intensities across all muscles had estimated ICC values of poor or moderate with the exceptions of good ICC values in APB 100% AMT (both paretic and non-paretic limbs) and APB 120/130% AMT paretic limb (Table 3). The RC slope generally had poor test-retest reliability across all muscles in both arms regardless of whether the fit was sigmoidal or linear (Table 4). However, across all MEP measures (AMT, RMT, latency, SP, MEP amplitude, RC slope) the ICC confidence intervals were wide, with the lower end falling below acceptable test-retest reliability (Tables 2–4).

The LOA analysis demonstrates small mean differences between session 1 and session 2, suggesting small group differences between the two sessions. However, the upper and lower 95% LOA were wide, suggesting variation in individuals in turn indicating measurement error (Tables 2–4). For example, the mean difference (LOA) for: MEP latency was 0.23 (−9.79, 10.24) ms and AMT ECR was 0.09 (−16.34, 10.88) percentage of stimulator output (Table 2). There were differences in the mean difference and LOA between paretic and non-paretic limbs, generally the non-paretic limb demonstrated smaller mean differences and narrower 95% LOA (exceptions SP ECR and APB, and latency APB). For example, the motor threshold in non-paretic ECR mean difference and 95% LOA were −0.29 (−14.49; 13.92) whereas the paretic ECR demonstrated −6.04 (−31.77; 19.69) measured as a percentage of stimulator output.

Smallest detectable change (SDC) in MEP characteristics

Example group SDC’s for motor threshold ranged from 7.64% of stimulator output for AMT of non-paretic ECR to 25.76% for RMT of paretic BB, (Supplementary Table 1 and 2) indicating that a change of greater than 7.64 or 25.76% of stimulator output would indicate a change in measurement above that of overall measurement error in the respective muscles. Generally, the non-paretic limb demonstrated smaller SDCs than the paretic limb in 13 of the MEP characteristics investigated. The few exceptions to this were the SP for all muscles and the RMT and MEP-L in APB.

Discussion

This present study characterised the test-retest reliability of frequently used TMS measures in sub-acute stroke participants in a range of upper limb muscles across two separate sessions. To the best of our knowledge, this study is the first to investigate a variety of upper limb muscles across the distal-proximal gradient, including the clinically meaningful forearm (ECR) and Biceps muscles, in a sub-acute stroke population. Using the different measures of reliability and agreement, the authors conclude that TMS has low reliability between sessions at an individual level early after stroke.

If TMS is to be used in clinical practice and research to ascertain whether interventions promote recovery or compensation, then the SDC must be considered. The smallest detectable change (SDCs) of eight MEP characteristics in three upper limb muscles bilaterally (48 values) is included. The SDC results for a range of MEP characteristics demonstrate high values, with the paretic limb generally having higher values than the non-paretic limb and variation across the three target muscles. The smallest values were for MEP latency and the largest for the silent period. The sub-acute stroke participants had substantially higher values, for example, in AMT, 10.8% compared to 4% for healthy participants [17]. This demonstrates the impact that post-stroke neurological changes can have on TMS measures. It, therefore, warrants more work to identify these stroke-related changes to ensure measurement error is not larger than any potential observed change. This is especially pertinent if TMS measures are to be used as a biomarker for recovery [11].

Limits of agreement (LOA) is used to determine measurement error which is otherwise known as agreement [13]. If there is agreement between measures, the measurement error is small, and the more reliable and less variable the measure is. The LOA were wide across all TMS measures and muscles, indicating poor absolute reliability, high levels of measurement error or a lack of agreement between individuals. Measurement error is subject to change depending on muscle, population and TMS measure in focus. Previous work has suggested that more proximal muscles such as Biceps may have higher measurement error because it receives fewer cortico-motor neuronal projections than distal muscles [39].

ICC was used to measure the degree to which stable individuals in a sample maintain their position relative to each other [40]. This study found that most TMS measures were not reliable due to the lower end of 95% CI of the ICC being below 0.7. Despite many of the estimated ICC values being within moderate or good reliability, they must be interpreted in conjunction with 95% CI values. To date, reliability results have been compared across studies including in different muscles and different samples. This has fundamental limitations due to the effect age has on TMS measures [37] and that TMS measures do not respond uniformly across all muscle representations [35]. Comparing ICC values is also statistically problematic, as they are dependent on sample variation [37]. However, much of the previous literature concluded that the ICC values were reliable but did not provide a 95% CI value; they may not have concluded acceptable reliability if adherence to the requirement of a lower 95% CI of at least 0.70 was used.

Previous research has demonstrated that distal muscles have a greater response to brain stimulation compared to proximal muscles [26]; the greater response (of distal muscles) to stimulation may be a contributing factor to their reliability. This may be in part due to the neurophysiological control mechanisms of upper limb muscles. Distal muscles such as FDI may have more reliable responses to TMS due to the higher density of corticospinal neurons projecting to the muscle and the larger motor map accessible to TMS in comparison to a proximal muscle such as Biceps [41]. The more proximal muscles also appear to have weaker cortical inhibitory control than distal muscles [26,35]. There has been relatively little research investigating the reliability of TMS measures from more proximal muscles such as BB, with the majority in healthy volunteers [39,41,42]. Even in healthy participants, it has been demonstrated that reliability is lower than in the hand muscles [41,42] and dependent on the TMS measure. The present study has a lower ICC for RMT (0.67) than Sankarasubramanian and colleagues (ICC 0.74)[39]. However, the latter did not provide 95% CI values. Therefore, although the authors concluded the ICC suggests that RMT intensity in BB was reliable, we concluded using the lower end of the 95% CI RMT was not. Kamen and colleagues [43] displayed higher ICC values in RMT than the current study (0.95/0.99) as did [42] (0.882) but these were both in healthy individuals and it is established that reliability tends to be lower in stroke patients regardless of the muscle [13]. Given the small sample of studies with BB as a target muscle for TMS reliability, it is suggested that further investigation is needed [13,39,42].

As outlined in the introduction, very few studies have investigated reliability in a sub-acute stroke population [13,20]. The ICC [95% CIs] of the TMS measures from sub-acute stroke population, in the study reported here were lower than those found for neuro-navigated TMS in sub-acute stroke survivors. Neuro-navigated TMS improves spatial accuracy of coil placement on the hotspot and aims to decrease variation in MEP amplitude [44–46]. This study purposely aimed to investigate the reliability of TMS measures whilst not using neuronavigation. This was to examine the transferability of this tool to a clinical setting, where neuronavigation is less widely used in comparison to neurophysiological lab settings. The authors consider this an important development in translational clinical neuroscience as many (but not all) of the stroke clinical trials involving TMS have used neuronavigation. However, if this technique is to be applied in a clinical setting the test-retest reliability without neuro-navigation needs to be considered.

Studies using neuronavigated TMS have produced some measures that demonstrate high ICC or low measurement error in healthy and stroke participants. However, some MEP characteristics using neuro-navigated TMS are not reliable, namely: recruitment curve slope, short-interval intracortical inhibition, and intracortical facilitation [13]. So, it is possible that even with the use of neuro-navigation, some MEP characteristics do not have acceptable reliability in subacute stroke [13,47]. If neuro-navigated TMS does produce reliable measures of MEP characteristics in muscles of interest in a clinically relevant population, its feasibility in clinical practice will require further investigation.

Other factors could influence our results, including a heterogenous sample: this current study included participants with a wide range of functional impairment, including those with very low clinical scores indicating a high level of impairment. Previous work investigating reliability in stroke was conducted with participants with less motor impairment [13]. However, in the current study, the authors included a spectrum of stroke survivors from mild to severe motor impairment to have a sample closer to the clinical population. The target muscle could also be a factor as this study aimed to investigate the clinically functionally relevant muscles, both proximal and distal. The differing muscle response may be explained by the proximal to distal gradient [48] and the functional role of each muscle [49]. There are other potential influences on the reliability of TMS-derived measures of MEP characteristics. Inter-individual variability in response to TMS has been reported due to factors such as gene expression, e.g., brain-derived neurotrophic factor, BDNF [50]. Also, the TMS data collection method may contribute to differences in findings between studies [40]. Standardisation of data collection and processing [33] would facilitate improved comparisons between studies and may contribute to improved measurement reliability.

The key strengths of this study are that the stroke participants were representative of the sub-acute stroke population with no contraindication to TMS early (mean 39 d). MEP characteristics of clinical interest for discerning recovery from compensation were investigated for both proximal and distal upper limb muscles. The test-retest reliability in stroke survivors has been focused on chronic stroke populations (greater than six months after stroke) ranging from 9 to 23 with an average of 19 participants. There are only two studies that focus on sub-acute TMS reliability with 18 and 20 participants. To the best of the author’s knowledge, the achieved sample size of 30 is the largest stroke survivor sample in a TMS reliability study. The sample of stroke participants included a wide range of impairment level, including individuals more severely affected, a population that is often less represented in stroke research.

This study addresses statistical and procedural limitations of previous studies. In particular, the lack of use of the lower 95% CI of an ICC value and not including a means of assessing measurement error (for example SDC) [13,17]. In response to the recent systematic review [35], the critical appraisal tool proposed by IFCN [33] was used to check TMS procedures and improve the quality of reporting. This demonstrated that 90% of the factors were reported, except for the following: ‘‘History of specific repetitive motor activity,” “Prior motor activity of the muscle to be tested”, and “time between MEP trials” [33].

There were some limitations to this study. The main one is that five TMS pulses were delivered at each stimulation intensity, which is below the number advised [51,52]. It remains possible that more pulses would have changed the reliability values. It is also possible that increasing the number of stimulation intensities could influence the reliability findings. It has been suggested that increasing the number of TMS trials can reduce SDC and measurement error [53]; future studies should consider this. However, five stimuli have been used previously (as well as recommended in previous guidelines) and reported to have high reliability [30,54,55]. The use of non-navigated TMS can be seen as a limitation, but this was part of a conscious decision by the research team to reflect the current clinical environment were neuro-navigation is rarely available. Several pragmatic decisions to reduce the burden on stroke participants very early after stroke could be viewed as limitations. These included not conducting an RMT recruitment curve, not having sub-acute stroke participants perform an MVC, and not having stroke participants maintain a specific percentage of MVC contraction during AMT which may have impacted the reliability. While the experimenter provided consistent instructions and used observation to evaluate the muscle contraction during TMS this method could be more objective and standardised in future research and in clinical settings. For example, using for dynamometer for MVC and maintaining a percentage of the MVC as well as using EMG feedback in a similar way to this study but having feedback to the participant to maintain a specific contraction during TMS data collection [56].

Conclusions

In summary, this study found unacceptable test-retest reliability for a variety of non-navigated TMS measures (motor threshold, MEP latency, cortical silent period, recruitment curve slope) in the Biceps, ECR or APB muscles in sub-acute stroke participants. The different conclusions from earlier studies may result from this study addressing many of the limitations of previous studies (including the use of 95% CI). The present findings indicate there is still limited evidence for acceptable reliability of TMS outcomes in sub-acute stroke. The next steps could be to undertake a direct comparison between the reliability of neuro-navigated and non-neuro-navigated TMS focused on the same muscles in the sub-acute stroke population. The results of a definitive comparative study would provide the knowledge required to inform the future use of biomarkers to advance knowledge of the mechanisms of recovery and thereby the science of precision stroke rehabilitation.

Professor Pomeroy is grateful for the support of the National Institute for Health Research (NIHR) Brain Injury MedTech Co-operative based at Cambridge University Hospitals NHS Foundation Trust and University of Cambridge. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.

Acknowledgements

We acknowledge the contribution of the FAST INdiCATE Investigators for contributions to the conduction of the FAST INdiCATE Trial namely: Susan M Hunter, Heidi Johansen-Berg, Nick Ward, Elizabeth Chandler, Christopher John Weir, John Rothwell, Alan M Wing, Michael J Grey, Garry Barton, Nick Malachy Levy, Claire Havis, Roger N Lemon, Jane Burridge, and Amy Drummond. In addition, we are grateful to all participants in the study reported here who gave their time for research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.