The relationship between trained ratings and untrained listeners’ judgments of global coherence in extended monologues

ABSTRACT

Background: Global coherence rating scales have been used by a number of researchers to examine spoken discourse in populations with and without acquired neurogenic communication disorders. The 4-point global coherence (GC) scale in the current study has demonstrated reliability and convergent validity. However, we have not yet established how a global coherence rating corresponds to functional communication.

Aims: The current study explores the clinical meaningfulness of the 4-point GC rating scale. Survey questions and ratings were developed to examine discourse quality and functional coherence as perceived by untrained listeners. Ratings by untrained listeners were compared to trained discourse ratings using the established 4-point GC scale.

Methods and Procedures: Twelve discourse samples, scored by a trained rater, were selected for the current study from a previously collected set of discourse transcripts. Transcripts were extended monologues in response to one of four possible open-ended questions that were re-recorded by the trained rater to remove any distracting features of the original recordings, such as articulatory errors. Twenty-four untrained listeners rated the discourse samples using a short questionnaire that included questions for each sample about: topic maintenance, inclusion of unnecessary information, and the listener level of interest and attention. Questions for untrained listeners were based on operational definitions of global coherence and discourse quality respectively. These untrained ratings were compared to trained ratings of global coherence. Outcome measures were compared using non-parametric tests and a Spearman Rank Order test was also used to examine relationships among variables.

Main Contribution: Untrained listeners’ ratings for topic maintenance and inclusion of unnecessary information were significantly different between trained low, medium, and high global coherence ratings. Topic maintenance and inclusion of unnecessary information were also both significantly correlated with global coherence. Untrained listeners’ ratings of their level of interest and attention for a sample were significantly different between trained medium-high and low-high global coherence ratings. Interest and attention ratings were also significantly correlated with ratings of global coherence.

Conclusions: Untrained raters did differentiate between levels of global coherence using ratings of topic maintenance, inclusion of unnecessary information, and their level of attention and interest. Global coherence was also significantly correlated with the untrained ratings. These findings provide preliminary external validity for the global coherence scale and support its clinical utility.

KEYWORDS:

• Global coherence
• discourse
• discourse analysis
• external validity
• untrained raters

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplemental material

Supplemental data for this article can be accessed here.

Additional information

Funding

This work was supported by the National Institute on Aging [R21AG033284-01A2] at the National Institutes of Health, as well as the National Parkinson Foundation Center of Excellence, and the UF-INFORM database.