A preliminary study of the reliability of soccer skill tests within a modified soccer match simulation protocol

ABSTRACT

Aim

This study examined test-retest reliability of soccer-specific skills within a modified version of the soccer match simulation (SMS) protocol.

Methods

Ten professional youth academy soccer players (18 ± 1 years) from the United Kingdom completed 30 minutes of the modified SMS on two occasions under standardised conditions. During each trial, participants performed 20-m dribbling, short passing (4.2-m), long passing (7.9-m), shooting skills, and 15-m sprints within four blocks of soccer specific activity.

Results

Collapsed normative data (mean (SD)) for trial 1 and trial 2 for dribbling speed was 2.7 (0.2) m/s, for sprint speed 5.9 (0.4) m/s, for short pass speed 11.1 (0.5) km/h, for long pass speed was 12.2 (0.5) km/h, and for shooting speed was 13.3 (0.4) km/h. Mean results from trial 1 and trial 2 were not different for all measures evaluated (P > 0.05). Good to excellent reliability (ICC 0.76-0.99) was observed for long and short passing speed, shooting speed, sprint speed, and long pass accuracy, with CVs typically < 5-10%. Moderate reliability (ICC 0.50-0.75) was observed for dribbling speed. Poor reliability (ICC <0.50) was observed for dribbling accuracy and shooting accuracy.

Conclusions

The reliability of the modified version of the SMS protocol is promising for most of the skills assessed, with the exception of dribbling and shooting accuracy in this group of professional youth soccer players. The modified protocol is easy to implement within professional clubs without specialist equipment, but due to the limited sample size the reliability requires further confirmation in a larger sample.

KEYWORDS:

• Football
• reproducibility
• accuracy
• speed

Introduction

Soccer simulation protocols aim to replicate movement patterns and physiological demands of match-play (Drust et al. 2000; Nicholas et al. 2000; Thatcher and Batterham 2004; Russell et al. 2011). Free running intermittent exercise simulation protocols are designed to simulate the activity pattern characteristics of soccer, however, several factors, such as the omission of game-specific skills (Russell et al. 2011), and the use of a non-grass surface might reduce the ecological validity of these protocols (Russell et al. 2011). As such, modified versions of protocols have been implemented to investigate soccer-specific skills (Ali et al. 2008; Rostgaard et al. 2008; Foskett et al. 2009).

Russell et al. (2011) developed the Soccer Match Simulation (SMS) protocol. The SMS is a modified version of the Loughborough Intermittent Soccer Test (LIST) which was the first intermittent exercise simulation protocol designed to simulate the activity pattern characteristics of soccer. The LIST includes 75-min of intermittent activity followed by a run to exhaustion (Nicholas et al. 2000), however, it does not include game-specific skills. The SMS includes additional movement components, and soccer-specific skills are embedded to enhance the ecological validity of the protocol (Russell et al. 2011). This means the SMS has application to studies investigating interventions on both the physical and skill components of soccer players performance (Russell et al. 2011; Harper et al. 2017). The SMS protocol has been successfully used to evaluate performance, physiological responses, and the efficacy of nutritional interventions (Kingsley et al. 2014; Harper et al. 2017; Rodriguez-Giustiniani et al. 2019). However, it is important to know the reliability of sport-specific tests like the SMS, as day to day variation needs to be understood in order to determine differences between trials in intervention studies. Reliability refers to the reproducibility/precision of values from a test, assay, or other measurement in repeated trials on the same individuals (Hopkins 2010). Russell et al. (2010) assessed the reliability of the skills contained in the SMS and reported moderate, to moderately strong, relative reliability for passing (speed, precision, and success), shooting (accuracy), and dribbling (speed and accuracy). When assessed over 120-min (two 45-min halves plus 2 additional 15-min periods) Harper et al. (2016) reported the physiological and performance responses were reliable but did not include skill outcome measures of passing or shooting.

In our recent study, we utilised a modified version of the SMS protocol to assess soccer skill performance (Rodriguez-Giustiniani et al. 2019). The protocol was modified for ease of use in a professional soccer academy setting, as well as to provide greater insight into potential differences between dominant and non-dominant foot. In order to assess the passing variables, we utilised readily available passing targets (i.e., soccer mannequins) instead of banners with target boxes and an illumination system (Russell et al. 2011). Furthermore, for the first time, the assessment of passing and shooting was discriminated between the dominant and non-dominant foot. This involved increasing the number of passes and shots taken during the match simulation protocol. Discriminating between feet could be of importance as fatigue is associated with a decrement in central control (Welsh et al. 2002) and may be more likely to affect skill performance with the non-dominant foot (Rodriguez-Giustiniani et al. 2019).

Previous studies on skill-based sports like squash (Bottoms et al. 2007) and tennis (McRae and Galloway 2012) have shown positive effects of nutritional interventions on the maintenance of skill performance, particularly on weaker/non-dominant shots such as backhand drive-in squash. Indeed, passing speed was better maintained in the non-dominant foot when evaluating the effects of a carbohydrate-electrolyte beverage on soccer skill performance using a simulated soccer match protocol (Rodriguez-Giustiniani et al. 2019). Taken together, it seems reasonable to distinguish between the dominant and the non-dominant foot when investigating the reliability of soccer-skill assessments. Due to the changes that we have made in the delivery of the SMS, and the skills within, it is relevant to assess the reliability of this modified version of the protocol. We theorised that this modified protocol would demonstrate similar reliability to previous studies using the SMS. Specifically, we aimed to quantify the absolute and relative test–retest reliability and compare the magnitude of these statistics to previous investigations on the SMS.

Methods

Participants

Ten male well-trained professional outfield soccer players (5 midfielders, 3 defenders, 2 strikers) from the United Kingdom, who were accustomed to skill assessments as part of their regular training, were recruited from a local Professional Football Club development squad in order to participate in this investigation. All players had five or more years of playing experience, had been training consistently for 1 year or more, were regularly participating in match-play with their squad, and were free from injury at the time of the recruitment and testing (age: 18 ± 1 years, body mass: 75.0 ± 6.5 kg, stature: 179.2 ± 5.6, body mass index: 23.4 ± 1.0 kg/m²). The experimental procedures were approved by a local Ethics of Research Committee and the study was conducted in accordance with the declaration of Helsinki.

Study DesignPlayers attended one preliminary visit for a familiarisation session before undertaking two main trials in which test re-test reliability was assessed. All visits were expected to be separated by a minimum of 2 days and a maximum of 7 days. From the ten players, three completed their first main trial 2 days after the familiarisation visit, four after 4 days of the familiarisation, two 6 days after the familiarisation, and one player completed the first trial 7 days after the familiarisation. All the participants completed the two main trials 7 days apart. Players followed 48-hr habitual diets (avoiding caffeine and alcohol) and recorded food consumed before the familiarisation visit. The pre-familiarisation diet was replicated for both main trials. Players refrained from strenuous exercise 48-hr before the familiarisation and main trial days. All testing sessions were performed on an indoor artificial grass pitch (length: 37-m; width: 19-m; ceiling height: 6.5-m). Soccer balls were inflated to a pressure of 14 psi before each trial.

Familiarisation and main trials procedures

All trials started in the afternoon to reflect the time at which this group typically engages in soccer match play. At the training ground, researchers provided players with a standardised breakfast (2 eggs, 2 slices of bread, 1 medium-sized banana providing 423 kcal, 46 g carbohydrate, 26 g protein, 14 g fat). A pre-trial standardised meal also was provided 2-h before beginning the main trials, with the meal containing 2 g carbohydrate ⋅ kg⁻¹ of body mass (pasta in a tomato sauce) plus 500 ml of water. Upon attendance for testing (familiarisation and main trials), body mass (SECA Quadra 808, Hamburg, Germany) was assessed immediately after voiding of bladder and bowels.

Familiarisation and main trials commenced after a 15-min standardised warm-up (consisting of running, dynamic stretching, and 20-m sprints) that preceded each trial. During each trial, participants were required to perform soccer dribbling, passing, and shooting skills, and 15-m sprints throughout four blocks of a modified soccer match simulation (SMS) protocol (Russell et al. 2011) lasting a total of 30 minutes (Figure 1(a)). Each block of the SMS protocol consisted of 3 repeated cycles of three 20-m walks, one walk to the side, five 20-m jogs, one 20-m backwards jog, two 20-m strides and an alternating timed 15-m sprint or a 20-m dribble (Figure 1(b)), followed by passing and shooting assessments (Russell et al. 2011).

Figure 1. Protocol schematic (a) and simulated soccer match outline (b).

Simulated soccer match protocol, skills testing and analysis

The SMS includes exercise blocks consisting of 3 repeated cycles of three 20-m walks, one walk to the side, an alternated timed 15-m sprint or a 20 m dribble, a 4-s passive recovery period, five 20-m jogs at a speed corresponding to 40% VO₂max, one 20-m backwards jogs at 40% VO₂max and two 20-m strides at 85% VO₂max followed by passing and shooting assessments. So as to assess the reliability of our modified version of the protocol, the participants completed four blocks of the abovementioned cycles. In order to assess dribbling speed and accuracy, players dribbled a ball between 7 cones (cones 2–7 were placed 3-m away from the preceding cone, and cones 1 and 7 were 1 m away from each end of the course; Figure 2(a)). Participants were required to dribble the ball as quickly and accurately as possible from one end to the other over the 20-m total distance. Participants dribbled towards a video camera that was placed directly in line with the cones. For the sprint assessment, players ran as fast as possible through timing gates (Brower®, USA) placed 15-m apart, with a 1-m run-in. At the end of each block of activity, players performed a bout of passing where they directed alternate passes towards target zones placed to the left and right at distances of 4.2-m (short pass) and 7.9-m (long pass).

Figure 2. Schematic representation of the (a) dribbling assessment and the (b) passing and shooting assessments.

Soccer mannequins (Diamond Football®, Senior Pro Free Kick) with their bases were used as passing targets. The base (0.5-m wide) was the central zone of the target, with cones at a distance of 0.5-m of each side as the lateral zones of the target. A pass to the centre area was worth 10 points and the two lateral areas were worth 5 points. Passes that missed the target areas were scored as 0. Passing bouts consisted of 8 passes (2 with the dominant and 2 with the non-dominant foot to the short pass target, and 2 with the dominant and 2 with the non-dominant foot to the long pass target; Figure 2(b)). Then, a shooting skill assessment was performed, for this, participants were instructed to kick the ball as firmly and accurately as possible to a shooting target. Shooting target zones were at a distance of 15-m in the four corners of the goal. These target areas have been identified as optimal ball placement to beat a goalkeeper when shooting (Ali et al. 2007). Each shooting target was divided into two areas, a centre area (75 cm x 60 cm) and an extended area (100 cm x 90 cm) with the centre area worth 10 points and the extended area worth 5 points. Shots that missed the areas on the target were scored as 0. The bouts of shooting consisted of 8 shots (4 with the dominant foot and 4 with the non-dominant foot; Figure 2(b)). To enhance ecological validity, no prior touches were allowed to control the ball before a pass or a shot (Dooan et al. 2001).

Video footage of the skill tests was captured using GoPro cameras (GoPro®, Hero 5), one was placed 1-metre apart from the last cone of the dribbling course and the other 1-metre behind the passing and shooting zone. Manual digitisation (Kinovea® version 0.8.15; Kinovea Org., France) yielded dribbling speed, dribbling precision, passing accuracy, and passing speed, as well as shooting accuracy and speed. Passing and shooting speed was calculated from the time interval between ball contact with the foot and subsequent ball contact with the target area.

Inter and intra rater reliability

In order to determine inter-rater and intra-rater reliability, we selected data from three participants across the two trials for each one of the variables evaluated through digitisation. For the assessment of inter-rater reliability two experienced video analysts within a professional football club analysed the data which was compared with the data gathered by the investigators of this study. To assess intra-rater reliability we compared repeated measures from one of the authors of the study and two video analysts for each one of the variables assessed.

Statistical analysis

Statistical analysis was carried out using SPSS software (Version 16.0; SPSS Inc., USA) and a custom-made spreadsheet (Hopkins 2015). Systematic bias was evaluated as the mean change between trial 2 and trial 1, determined using paired t-tests and statistical significance set at P < 0.05. Soccer skills relative and absolute reliability was assessed using the intra-class correlation coefficient (ICC) and the coefficient of variation (CV), respectively. The ICC was derived from the methods of a two-way mixed effects model for consistency in a single measure (i.e., ICC3,1). Interpretation of the ICC values was based on guidelines provided by Koo and Li (Koo and Li 2016): ICC’s less than 0.50 were indicative of poor reliability, values between 0.50 and 0.75 indicated moderate reliability, values between 0.76 and 0.90 indicated good reliability, and values greater than 0.90 indicated excellent reliability. The coefficient of variation was calculated from the standard error of measurement. All the aforementioned statistics were presented with 95% confidence intervals (CI).

Results

Descriptive and reliability statistics along with mean differences (95% CI) for each of the repeated skill assessments are presented in Table 1 and 2. Mean results from trial 1 and trial 2 were not significantly different for all measures evaluated (P > 0.05). Reliability statistics for the assessed skills are also presented in Table 1 and 2. Long passing speed with both feet, with the dominant foot, and with the non-dominant foot were the most reliable outcomes that were identified since the ICC showed excellent reliability.

Table 1. Descriptive and reliability statistics obtained from two trials assessing soccer-skills performance within a modified SMS protocol.

Skill	Trial 1	Trial 2	Mean change (95% CI) Raw values %		t-test (P-value)		ICC (95% CI)	CV (%) (95% CI)
Dribbling Speed (m/s)	2.7 ± 0.2	2.7 ± 0.2	0.0 (−0.1, 0.1)	−0.1 (−5.3, 4.9)		0.99	0.68 (0.14, 0.91)	7.6 (7.5, 7.7)
Dribbling Precision (cm)	1.2 ± 0.4	1.1 ± 0.3	−0.1 (−0.4, 0.3)	0.1 (−0.4, 0.3)		0.61	0.15 (−0.49, 0.69)	23.1 (22.9, 23.3)
Sprint Speed (m/s)	5.9 ± 0.4	5.9 ± 0.4	0.0 (−0.1, 0.2)	0.0 (−0.2, 0.1)		0.69	0.87 (0.58, 0.97)	6.9 (6.7, 7.2)
Short Pass Accuracy Both Feet (points)	56 ± 10	54 ± 6	−2 (−6, −2)	2.2 (−1.9, 6.1)		0.25	0.85 (0.43, 0.95)	13.9 (8.9, 18.9)
Long Pass Accuracy Both Feet (points)	44 ± 7	43 ± 5	−1 (−3, 2)	0.1 (−0.2, 0.3)		0.65	0.76 (0.22, 0.92)	15.5 (11.8, 19.2)
Short Pass Speed Both Feet (km/h)	11.1 ± 0.4	11.0 ± 0.5	−0.1 (−0.3, 0.2)	0.6 (2.3, −3.4)		0.58	0.84 (0.44, 0.96)	3.4 (3.2, 3.7)
Long Pass Speed Both Feet (km/h)	12.1 ± 0.6	12.2 ± 0.4	−0.1 (−0.1, 0.2)	0.1 (−0.2, 0.1)		0.33	0.95 (0.81, 0.99)	4.3 (3.9, 4.6)
Shooting Accuracy Both Feet (points)	69 ± 6	71 ± 5	2 (−2, 6)	−1.9 (−0.1, 0.2)		0.33	0.40 (−0.27, 0.81)	6.3 (2.9, 9.7)
Shooting Speed Both Feet (km/h)	13.6 ± 0.3	13.6 ± 0.4	0.1 (−0.2, 0.1)	−0.1 (−0.2, 0.1)		0.27	0.89 (0.62, 0.97)	2.5 (2.3, 2.7)

Data presented as mean ± standard deviation, CI: confidence interval. P-value determined from test re-test data using paired sample t-test for all measurements’ outcomes (n = 10).

SMS: soccer match simulation, CI: confidence interval, ICC: intra-class correlation coefficient, CV: Coefficient of variation.

Table 2. Descriptive and reliability statistics obtained from two trials assessing soccer passing and shooting performance with the dominant and non-dominant foot within a modified SMS protocol.

Skill	Trial 1	Trial 2	Mean change (95% CI) Raw values %			t-test (P-value)	ICC (95% CI)	CV (%) (95% CI)
Short Pass Accuracy Dominant Foot (points)	31 ± 5	30 ± 3	−1.0 (−2.0, 4.0)	1.0 (−2.0, 3.9)		0.47	0.53 (−0.10, 0.86)	11.7 (9.2, 14.2)
Short Pass Accuracy Non-Dominant Foot (points)	25 ± 5	23 ± 5	−2.2 (−2.5, 6.9)	2.2 (−2.6, 6.7)		0.32	0.22 (−0.34, 0.66)	15.5 (12.4, 18.6)
Long Pass Accuracy Dominant Foot (points)	24 ± 6	24 ± 3	−0.1 (−1.2, 1.4)	0.1 (−1.2, 1.4)		0.86	0.92 (0.72, 0.98)	18.6 (15.8, 21.4)
Long Pass Accuracy Non-Dominant Foot (points)	20 ± 4	19 ± 3	−0.7 (−1.6, 3.0)	0.7 (−1.6, 3.0)		0.51	0.56 (0.05, 0.84)	15.3 (13.1, 17.5)
Short Pass Speed Dominant Foot (km/h)	11.6 ± 0.6	11.4 ± 0.8	−0.2 (−0.2, 0.5)	0.2 (−0.2, 0.5)		0.36	0.76 (0.39, 0.92)	5.6 (5.2, 6.0)
Short Pass Speed Non-Dominant Foot (km/h)	10.7 ± 0.3	10.7 ± 0.3	0.0 (−0.2, 0.14)	0.0 (−0.2, 0.1)		0.63	0.78 (0.33, 0.94)	2.9 (2.2, 3.1)
Long Pass Speed Dominant Foot (km/h)	10.7 ± 0.3	10.8 ± 0.5	0.1 (−0.3, −0.4)	0.2 (−0.3, 0.0)		0.09	0.95 (0.81, 0.99)	3.7 (3.5, 4.0)
Long Pass Speed Non-Dominant Foot (km/h)	11.4 ± 0.5	11.3 ± 0.5	−0.1 (0.0, 0.2)	0.1 (0.2, 0.2)		0.02	0.96 (0.85, 0.99)	3.7 (3.4, 4.0)
Shooting Accuracy Dominant Foot (points)	39 ± 5	38 ± 4	−1.0 (−2.1, 4.1)	1.1 (−1.9, 4.0)		0.43	0.43 (−0.23, 0.82)	8.5 (5.7, 11.3)
Shooting Accuracy Non-Dominant Foot (points)	29 ± 7	32 ± 6	3.5 (−6.7, −0.3)	−3.6 (−7.0, −0.2)		0.04	0.75 (0.26, 0.93)	18.3 (14.3, 22.3)
Shooting Speed Dominant Foot (km/h)	14.1 ± 0.3	14.0 ± 0.3	−0.1 (−0.2, 0.4)	0.1 (−0.2, 0.4)		0.49	0.56 (−0.07, 0.87)	2.4 (2.2, 2.6)
Shooting Speed Non-Dominant Foot (km/h)	13.0 ± 0.5	13.1 ± 0.5	0.1 (−0.1, 0.2)	0.0 (−0.1, 0.2)		0.65	0.87 (0.55, 0.97)	4.1 (3.8, 4.4)

Data presented as mean ± standard deviation, CI: confidence interval. P-value determined from test re-test data using paired sample t-test for all measurements’ outcomes (n = 10).

SMS: soccer match simulation, CI: confidence interval, ICC: intra-class correlation coefficient, CV: Coefficient of variation.

Short pass speed with both feet and shooting speed with both feet showed good reliability along with a CV <5%, whereas sprint speed also presented good reliability with a CV <10%. Good reliability and a CV in the 10–20% range was reported for both short and long pass accuracy with both feet. Dribbling speed was moderately reliable with a CV <10% whereas shooting accuracy and dribbling precision both showed poor reliability. When assessing passing and shooting performance with the dominant and the non-dominant feet separately (Table 2), we observed that long pass accuracy with the dominant foot showed excellent reliability. We reported good reliability with a CV <5% for short pass and shooting speed with the non-dominant foot, whereas short pass speed with the dominant foot had good reliability and a CV <10%. Shooting speed with the dominant foot, short pass accuracy with the dominant foot, long pass accuracy with the non-dominant foot, and shooting accuracy with the non-dominant foot were all moderately reliable while short pass accuracy with the non-dominant foot and shooting accuracy with the dominant foot showed poor reliability.

Inter-rater and intra-rater error reliability was good to excellent for almost all variables studied, except for long pass with non-dominant foot which only achieved moderate intra-rater reliability in analyst 2 for trial 1 (Table 3). The coefficient of variation was below 10%for most of the variables assessed except in four cases (dribbling precision, intra-rater reliability, analyst 2; shooting accuracy with both feet, intra-rater reliability, researcher; shooting accuracy non-dominant foot, intra-rater reliability, analyst 1; shooting speed dominant foot, analyst 1) in which the CV values were between 10–20%. We did not observe any of the CV values to be > 20%.

Table 3. Inter-rater and intra-rater reliability.

	ICC (95% CI)
	Inter-rater reliability		Intra-rater reliability: researcher		Intra-rater reliability: analyst 1		Intra-rater reliability: analyst 2
Skill	Trial 1	Trial 2	Trial 1	Trial 2	Trial 1	Trial 2	Trial 1	Trial 2
Dribbling Speed (m/s)	0.99 (0.96, 1.00)	0.98 (0.84, 0.99)	0.99 (0.54, 1.00)	0.99 (0.89, 1.00)	0.99 (0.98, 1.00)	0.99 (0.98, 1.00)	0.97 (0.21, 0.99)	0.89 (0.72, 0.92)
Dribbling Precision (cm)	0.99 (0.97, 1.00)	0.98 (0.82, 1.00)	0.98 (0.40, 1.00)	0.98 (0.33, 0.99)	0.99 (0.97, 1.00)	0.99 (0.96, 1.00)	1.00	0.94 (0.38, 0.99)
Short Pass Accuracy Both Feet (points)	0.99 (0.96, 1.00)	0.99 (0.90, 1.00)	1.00	0.98 (0.56, 1.00)	0.99 (0.80, 1.00)	0.98 (0.65, 1.00)	0.83 (0.21, 0.98)	0.91 (0.44, 0.99)
Short Pass Accuracy Dominant Foot (points)	0.95 (0.63, 0.99)	0.96 (0.05, 0.99)	1.00	0.99 (−0.01, 0.99)	0.95 (0.30, 0.99)	0.87 (0.31, 0.99)	0.97 (0.22, 0.99)	0.96 (0.41, 0.99)
Short Pass Accuracy Non-Dominant Foot (points)	0.99 (0.96, 1.00)	0.98 (0.83, 0.99)	0.98 (0.62, 0.99)	0.98 (−0.46, 0.99)	0.99 (0.91, 1.00)	0.97 (0.34, 0.99)	0.99 (0.43, 0.99)	0.78 (0.13, 0.90)
Long Pass Accuracy Both Feet (points)	0.99 (0.95, 1.00)	0.97 (0.68, 0.99)	1.00	0.99 (0.93, 1.00)	0.99 (0.79, 1.00)	0.97 (0.03, 0.99)	0.85 (0.23, 0.97)	0.99 (0.82, 1.00)
Long Pass Accuracy Dominant Foot (points)	0.99 (0.86, 0.99)	0.99 (0.92, 1.00)	0.98 (0.77, 1.00)	0.96 (0.41, 0.99)	0.99 (0.81, 1.00)	0.96 (0.20, 0.99)	0.98 (0.89, 0.99)	0.97 (0.74, 1.00)
Long Pass Accuracy Non-Dominant Foot (points)	0.91 (0.14, 0.99)	0.99 (0.36, 1.00)	1.00	1.00	0.98 (0.59, 1.00)	0.98 (0.50, 0.99)	0.91 (0.30, 0.99)	0.95 (0.30, 0.99)
Short Pass Speed Both Feet (km/h)	0.97 (0.36, 0.99)	0.98 (0.83, 0.99)	1.00	1.00	0.99 (0.61, 1.00)	0.96 (0.10, 0.99)	0.99 (0.91, 1.00)	0.99 (0.91, 1.00)
Short Pass Speed Dominant Foot (km/h)	0.99 (0.94, 1.00)	0.98 (0.85, 0.99)	1.00	1.00	0.98 (0.03, 0.99)	0.96 (0.03, 0.99)	0.95 (0.53, 1.00)	0.99 (0.79, 1.00)
Short Pass Speed Non-Dominant Foot (km/h)	0.98 (0.84, 1.00)	0.96 (0.66, 0.99)	1.00	1.00	0.91 (0.22, 0.99)	0.93 (0.37, 0.99)	0.99 (0.81, 1.00)	0.99 (0.81, 1.00)
Long Pass Speed Both Feet (km/h)	0.98 (0.85, 0.99)	0.91 (0.18, 0.99)	0.96 (0.31, 0.99)	0.94 (0.41, 0.99)	0.98 (0.47, 0.99)	0.86 (0.61, 0.99)	0.98 (0.59, 1.00)	0.85 (0.32, 0.97)
Long Pass Speed Dominant Foot (km/h)	0.99 (0.96, 1.00)	0.97 (0.69, 0.99)	0.94 (0.94, 0.99)	0.97 (0.48, 0.99)	0.98 (0.58, 0.99)	0.98 (0.14, 1.00)	1.00	0.89 (0.74, 0.93)
Long Pass Speed Non-Dominant Foot (km/h)	0.97 (0.80, 0.99)	0.95 (0.66, 0.99)	0.96 (0.67, 0.99)	0.88 (0.43, 0.99)	0.93 (0.07, 0.99)	0.92 (0.67, 0.99)	0.73 (0.38, 0.84)	0.97 (0.73, 0.99)
Shooting Accuracy Both Feet (points)	0.95 (0.61, 0.99)	0.97 (0.75, 0.99)	0.96 (0.25, 0.99)	0.97 (0.56, 0.99)	0.93 (0.11, 0.99)	0.87 (0.42, 0.99)	0.91 (0.77, 0.99)	0.98 (0.33, 0.99)
Shooting Accuracy Dominant Foot (points)	0.94 (0.59, 0.99)	0.91 (0.59, 0.99)	1.00	1.00	0.95 (0.12, 0.99)	0.93 (0.10, 0.99)	0.94 (0.59, 0.99)	0.98 (−0.03, 1.00)
Shooting Accuracy Non-Dominant Foot (points)	0.99 (0.90, 1.00)	0.95 (0.61, 0.99)	0.91 (0.58, 0.99)	0.90 (0.67, 0.99)	0.88 (0.30, 0.99)	0.92 (0.48, 0.99)	0.89 (0.81, 0.98)	1.00
Shooting Speed Both Feet (km/h)	0.98 (0.88, 1.00)	0.98 (0.89, 0.99)	0.96 (0.56, 0.99)	0.99 (0.74, 1.00)	0.95 (0.31, 0.99)	0.96 (0.01, 0.99)	0.93 (0.78, 1.00)	0.98 (0.46, 0.99)
Shooting Speed Dominant Foot (km/h)	0.99 (0.88, 1.00)	0.96 (0.64, 0.99)	0.97 (0.73, 0.99)	0.96 (0.79, 0.99)	0.84 (0.12, 0.99)	0.97 (0.46, 0.99)	0.95 (0.64, 0.99)	0.99 (0.93, 1.00)
Shooting Speed Non-Dominant Foot (km/h)	0.97 (0.56, 0.99)	0.98 (0.85, 0.99)	0.96 (0.26, 0.99)	0.90 (0.61, 0.99)	0.97 (0.48, 0.99)	0.98 (0.43, 0.99)	0.98 (0.85, 0.99)	0.96 (0.41, 0.99)

Discussion

The aim of this study was to examine the preliminary reliability of soccer skill tests within a modified version of the SMS protocol (Russell et al. 2011). In particular, we have observed that when assessing long passing speed (both feet, dominant foot, and non-dominant foot), sprint speed, short pass accuracy with both feet, long pass accuracy (both feet and with the non-dominant foot) short pass speed (both feet, with the dominant foot, and with the non-dominant foot), and shooting speed (both with and with the non-dominant foot) this version of the SMS protocol demonstrated good to excellent reliability. However, the modified protocol revealed poor reliability for dribbling precision, short pass accuracy (with the non-dominant foot) and shooting accuracy. To our knowledge this study is the first protocol of this kind that assesses the reliability of soccer specific skills performed with the dominant and non-dominant foot in professional youth soccer players, on an artificial grass surface.

Reliability of dribbling speed and precision

The number of successful dribbling tasks has been identified as a key contributor to match success (Zago et al. 2016). The present study found moderate reliability for dribbling speed but poor reliability for dribbling precision. This finding is in contrast to good reliability for both dribbling speed and dribbling precision reported previously (Russell et al. 2010). The reason for the difference may, in part, be due to how the skills were assessed. Specifically, integrating the dribbling skill assessment within the SMS in the present study may have reduced the reliability, in comparison to when the dribbling assessment was performed in isolation (Russell et al. 2010). Under circumstances where the player performs the dribbling test between efforts, they are likely to take time to focus exclusively on the required skill.

Harper et al. (2016) investigated the reliability of physiological and performance responses to the SMS across 120 minutes of soccer-specific exercise. In order to do this, these authors used an extended version of the SMS in which they included two additional 15-min periods of intermittent exercise and skill testing, on top of the two 45-min halves of the original SMS protocol. All performance variables assessed were expressed as an average per 15-min of exercise. Harper et al. (2016) demonstrated moderate reliability for dribbling speed in both the 0–15 min (r = 0.71) and the 16–30 min (r = 0.52) time-points within the SMS. Hence dribbling speed over a 20-m seems to be a moderately reliable soccer skill to assess when embedded within the SMS protocol. On the contrary, dribbling precision seems to be poorly reliable within the SMS protocol. Harper et al. (2016) demonstrated moderate reliability during the first 15-min of activity only (r = 0.64). However, correlation values (using Pearson’s correlation) thereafter (30–120 mins) corresponded with poor reliability. As both mental and physical fatigue influence skill performance (Reilly and Holmes 1983; Ramsbottom et al. 1988; Rampinini et al. 2009; Smith et al. 2015), Harper et al. (2016) attributed the respective lower reliability values to the fact that skill performance was measured in a fatigued state (Foskett et al. 2009). Thus, with respect to previous observations our data would suggest that dribbling precision over 20-m when assessed within an SMS protocol has poor reliability even when not fatigued, particularly beyond the initial 15 minutes of activity (Harper et al. 2016).

Reliability of passing speed and accuracy

To retain possession of the ball, accurate passing is a frequent and essential skill throughout soccer match-play (Hughes and Franks 2004; Rampinini et al. 2009). Longer passing sequences and a greater number of successful passes are associated with more goals scored (Hughes and Franks 2004). Russell et al. (2010) reported moderate reliability (ICC = 0.51) when assessing passing accuracy. As previously highlighted by Russell et al. (2010), it is possible to make comparisons between studies that report reliability in different ways. Using the intraclass correlation coefficient, our data revealed good reliability for passing accuracy using both feet, for both short and long passes, however, the CV revealed that there was only a moderate rating on variability. Regarding passing speed, we observed good reliability for the short pass and excellent reliability for the long pass with both feet. Russell et al. (2010) observed similar reliability for passing speed with an ICC of 0.76. Ali et al. (2007) aimed to assess the reliability of the Loughborough Soccer Passing Test (LSPT) in elite and non-elite soccer players. While the outcomes from the LSPT (time taken, time penalties for incorrect actions and total time) are not aligned to the outcome measures of the SMS passing assessment (i.e., accuracy, speed), the reliability of these outcomes are comparable with those shown by Russell et al. (2010). When comparing reliability for passing accuracy using the ICC, in the current study to the total time for the LSPT (as global marker of precision) we demonstrated greater reliability. Thus, passing performance (accuracy, speed) with both feet appears to be a reliable skill to assess within a SMS. We suggest the improvement observed in the present study may be due to players being able to use their routine soccer footwear (boots), instead of trainers (indoor surface). In addition, replacing the passing markers with mannequins may have also helped the players passing performance by providing a more realistic target.

Skill performance declines with fatigue (Mohr et al. 2005), and it appears that the non-dominant side may be more susceptible to this decline (Bottoms et al. 2007; McRae and Galloway 2012; Rodriguez-Giustiniani et al. 2019). Therefore, it is important to discriminate between dominant and non-dominant foot when assessing passing performances. We previously reported that long passing speed was better maintained with carbohydrate ingestion versus placebo during the latter stages of a SMS protocol in the non-dominant foot only (Rodriguez-Giustiniani et al. 2019). To our knowledge, that was the first study to differentiate between dominant and non-dominant foot when measuring soccer passing performance. The present data reveal that reliability of passing accuracy for the long pass is good for the dominant foot, and moderate for the non-dominant foot. However, this level of reliability was not evident for the short pass, which was moderate for the dominant foot and poor for the non-dominant foot.

By reducing the passing distance between the player and target, a reasonable assumption would have been improved, or at least similar, reliability to that of the long pass. We observed that reliability for passing speed was excellent (ICC: > 0.90) for both the dominant and the non-dominant foot for the long pass, whereas it was classified as good for both the dominant and the non-dominant foot for the short pass. Thus, since short pass targets were positioned at a 30° angle from the ball contact point whereas long pass targets were located at a 60°angle from the ball contact point, we speculate that the more lateral positioning of the short pass targets may have negatively influenced the reliability. The effect of increasing the passing distance further, to >8 m (typical of goal keepers and common for all outfield players), on reliability of passing remains to be established. Nevertheless, dominant, and non-dominant foot passing skills (accuracy, speed) within the SMS seem to be reliable assessments within this group of players, on an artificial grass surface when football mannequin targets are used.

Reliability of shooting speed and accuracy

The aim of soccer is to score more goals than the opposition, thus, shooting is a crucial skill (Stone and Oliver 2009). When assessing the reliability of soccer-skills, Russell et al. (2010) revealed poor reliability (ICC = 0.38) when assessing shooting accuracy. Our data also show poor reliability for shooting accuracy when considering data for both feet combined. As previously stated, the skill assessments in Russell’s study (Russell et al. 2010) were evaluated in isolation and not integrated within the blocks of running per se as in the present study. As comparisons are possible where reliability is reported in dimensionless units, both the present data and that presented by Russell et al. (2010) appear to be more reliable than the Loughborough Soccer Shooting Test (Ali et al. 2007). Ali et al. (2007) evaluated the reliability of the Loughborough Soccer Shooting Test (LSST) in elite and non-elite soccer players, they reported poor reliability (ICC = 0.26) for shooting success and precision in both groups. Moreover, when discriminating between the dominant and the non-dominant foot the present data also demonstrate poor or moderate reliability for shooting accuracy with the dominant and the non-dominant foot, respectively.

There may be several reasons for poor shooting reliability, such as the increased distance versus the long pass (15-m v 8-m). The increased shooting speed generated when performing a shot (Table 2) may also add variation to the skill, as well as technique performed in the execution of shooting (front of foot/laces) versus passing (instep) per se. Our results are consistent with data presented by both Russell et al. (2010) and Ali et al. (2007) who reported poor shooting reliability, over distances of 15 m and 16.5 m respectively. It should be noted that only two of the 10 participants in the present study were classified as strikers. Players adopt playing positions due to their suitability and skill. Therefore, position-specific participants may be required to increase the reliability and assessment of specific skills such as shooting.

Intra- and inter-rater reliabilityWe observed good to excellent intra-rater reliability for almost all of the variables when assessed by the study researcher and two video analysts within a professional football team. This indicates that the analysts’ assessments were on the whole consistent on repeated analysis. When multiple raters assessed the studied variables, excellent inter-rater reliability was observed. This excellent inter-rater reliability indicates that different raters can consistently assess the experimental trials. Therefore, it is recommended that experienced video analysts/researchers are in charge of the digitisation when assessing skill outcomes using this modified version of the SMS protocol. For large-scale projects, the ability to use multiple experienced researchers/analysts offers important methodological considerations on data assessment and input

Limitations

It has been stated that proficient skill performance is affected by cognitive factors such as decision-making and game intelligence (Williams and Reilly 2000). In the current study, we did not use a randomised lighting system for target identification when assessing passing and shooting as used in the original version of the protocol (Russell et al. 2011). This modification was made to increase ease of implementing the protocol in professional club settings. Therefore, it is unknown how inclusion of decision-making and visual searching, would have influenced the skill reliability. We acknowledge that including such perceptual demands would increase the ecological validity. However, we believe that the modified version of the SMS used in the present study could be more practically applicable in professional club settings as the equipment used (mannequins) are readily available.

Due to the fact that more than 40 participants are recommended for reliability studies (Atkinson and Nevill 1998; Hopkins 2010) we recognise that our results are only preliminary and further reliability assessment of this version of the SMS is warranted. However, we believe that our results are applicable to professional youth football players. Russell et al. (2010) also reported construct validity when they assessed the reliability of the skills contained within the SMS and observed that professional football players performed better than recreational players. There are no data regarding this aspect for our modified version of the protocol, but it would be of interest to test for construct validity in a further larger scale study.

In conclusion, this modified version of the SMS protocol showed encouraging reliability, especially for dribbling speed, sprint speed, short and long pass speed, shooting speed, and long pass accuracy. How skill reliability may change as the academy players transition to senior teams, and the reliability of other soccer-specific skills such as heading, and ball control remain to be established. This testing protocol has potential application for research settings out of the laboratory when investigating strategies that aim to improve skill performance in professional soccer players.

Disclosure statement

Ian Rollo (IR) is an employee of the Gatorade Sports Science Institute, a division of PepsiCo, Inc. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of PepsiCo Inc.

Additional information

Funding

No funding is associated with this study.