Short Text Coherence Hypothesis

Abstract

In this paper, we experimentally study the degree to which the length of a short text affects its comprehensiveness and readability, within quantitative linguistics. The quantitative linguistics focus mainly in analysis of large text collections and one of the major scientific theories in use is the Menzerath-Altmann law. In this paper we attempt to define the quantitative analysis framework for short texts consisting approximately of one or two sentences, due to the fact that they are considered very important in many scientific fields. To achieve the aim of this paper, a coherence statistical testing process of three variables was created for short texts. The implementation of that was possible through experimental and statistical evaluation. Upon completion of the above-mentioned evaluation, the statistical results showed that short text coherence, comprehensiveness and readability are fully achieved in short texts consisting of 14 words, when three predetermined variables are associated and vice versa. To prove the above hypothesis the theory of Vector Space Model and Kendall’s Coefficient of Concordance were used. The assessment of statistical results concluded that the above hypothesis can be fully met for a number of cases with a probability p > 99%. Moreover, in the experiment were used short texts in English language but it was proven that language can be considered irrelevant. To corroborate this, a smaller scale experiment with short texts in the German language was conducted and hypothesis was confirmed that the proposed model of this paper can be applied in all short texts regardless of their linguistic origin.