Interactive natural language acquisition in a multi-modal recurrent neural architecture

Figures & data

Figure 1. The overall MTRNN architecture with exemplary three horizontally parallel layers: input-output (IO), context-fast (Cf), and context-slow (Cs), with increasing timescale τ, where the Cs layer includes some context-controlling (Csc) units. While the IO layer processes dynamic patterns over time, the Csc units at first time step (t=0) contain the context of the sequence, where a certain concept can trigger the generation of the sequence.

Figure 2. The MTRNN with context abstraction architecture providing exemplary three horizontally parallel layers: context-slow (Cs), context-fast (Cf), and input-output (IO), with increasing timescale τ, where the Cs layer includes some context-controlling (Csc) units. While the IO layer processes dynamic patterns over time, the Csc units abstract the context of the sequence at last time step (t=T). The crucial difference to the MTRNN with context bias is an inversion of the direction of procession and an accumulation of abstract context instead of production from a given abstract context.

Figure 3. Effect of the self-organisation forcing mechanism on the development of distinct concept patterns for different sequences of contrary cosine waves: training effort (a) and mean $d_{\mathsf{a}\mathsf{v}\mathsf{g}}$ and $d_{\mathsf{r}\mathsf{e}\mathsf{l}}$ with standard error bars over varied ψ (b), each over 100 runs; representative developed Csc patterns (c–f) for different sequences for selected parameter settings of no, small, “good”, and large self-organisation forcing, respectively.

Figure 4. Architecture of the multi-modal MTRNN model, consisting of an MTRNN with context bias for auditory, two MTRNNs with context abstraction for somatosensory as well as visual information processing, and CAs for representing and processing the concepts. A sequence of phonemes (utterance) is produced over time, based on sequences of embodied multi-modal perception.

Figure 5. Scenario and manipulation action recording for multi-modal language learning scenario.

Figure 6. Schematic process of utterance encoding. The input is a symbolic sentence, while the output is the neural activity over $\left|I_{\mathsf{I}\mathsf{O}}\right|$ neurons times $T_{\mathrm{a}}$ time steps.

Figure 7. Schematic process of visual perception and encoding. The input is a single frame taken by the NAO camera, while the output is the neural activity over N neurons, with N being the sum over $\mathrm{shape}+\mathrm{colour}+\mathrm{positionfeatures}$.

Figure 8. Representations in the multi-modal language acquisition scenario.

Table 1. Standard meta and training parameter settings for evaluation.

Parameter	Description	Value	Parameter	Description	Value
$\left|I_{\mathsf{a}\mathsf{,}\mathsf{I}\mathsf{O}}\right|$	Auditory IO neurons	44	${\tau }_{\mathsf{a}\mathsf{,}\mathsf{I}\mathsf{O}}$	Timescale auditory IO	2
$\left|I_{\mathsf{a}\mathsf{,}\mathsf{C}\mathsf{f}}\right|$	Auditory Cf neurons	80	${\tau }_{\mathsf{a}\mathsf{,}\mathsf{C}\mathsf{f}}$	Timescale auditory Cf	5
$\left|I_{\mathsf{a}\mathsf{,}\mathsf{C}\mathsf{s}}\right|$	Auditory Cs neurons	23	${\tau }_{\mathsf{a}\mathsf{,}\mathsf{C}\mathsf{s}}$	Timescale auditory Cs	70
$\left|I_{\mathsf{s}\mathsf{,}\mathsf{I}\mathsf{O}}\right|$	Somatos. IO neurons	5	${\tau }_{\mathsf{s}\mathsf{,}\mathsf{I}\mathsf{O}}$	Timescale somatos. IO	2
$\left|I_{\mathsf{s}\mathsf{,}\mathsf{C}\mathsf{f}}\right|$	Somatos. Cf neurons	40	${\tau }_{\mathsf{s}\mathsf{,}\mathsf{C}\mathsf{f}}$	Timescale somatos. Cf	5
$\left|I_{\mathsf{s}\mathsf{,}\mathsf{C}\mathsf{s}}\right|$	Somatos. Cs neurons	23	${\tau }_{\mathsf{s}\mathsf{,}\mathsf{C}\mathsf{s}}$	Timescale somatos. Cs	50
$\left|I_{\mathsf{v}\mathsf{,}\mathsf{I}\mathsf{O}}\right|$	Visual IO neurons	19	${\tau }_{\mathsf{v}\mathsf{,}\mathsf{I}\mathsf{O}}$	Timescale visual IO	2
$\left|I_{\mathsf{v}\mathsf{,}\mathsf{C}\mathsf{f}}\right|$	Visual Cf neurons	40	${\tau }_{\mathsf{v}\mathsf{,}\mathsf{C}\mathsf{f}}$	Timescale visual Cf	5
$\left|I_{\mathsf{v}\mathsf{,}\mathsf{C}\mathsf{s}}\right|$	Visual Cs neurons	23	${\tau }_{\mathsf{v}\mathsf{,}\mathsf{C}\mathsf{s}}$	Timescale visual Cs	16
$\left|I_{\mathsf{a}\mathsf{,}\mathsf{C}\mathsf{s}\mathsf{c}}\right|$	Auditory Csc units	12	α	Teacher forcing	0.1
$\left|I_{\mathsf{s}\mathsf{,}\mathsf{C}\mathsf{s}\mathsf{c}}\right|$	Somatos. Csc units	12	${\eta }_{\mathsf{m}\mathsf{a}\mathsf{x}}$	Maximal learning rate	1.0
$\left|I_{\mathsf{v}\mathsf{,}\mathsf{C}\mathsf{s}\mathsf{c}}\right|$	Visual Csc units	12	${\eta }_{\mathsf{m}\mathsf{i}\mathsf{n}}$	Minimal learning rate	$1.0\times {10}^{-6}$
${\mathbf{\text{C}}}_{\mathsf{a}\mathsf{,}\mathsf{0}}^0$	Initial Csc values range	$\pm 0.01$	${\xi }_{+}$	Increasing factor	1.01
${\mathbf{\text{C}}}_{\mathsf{s}\mathsf{,}\mathsf{T}}^0$	Init. final Csc r. somatos.	$\pm 1.00$	${\xi }_{-}$	Decreasing factor	0.96
${\mathbf{\text{C}}}_{\mathsf{v}\mathsf{,}\mathsf{T}}^0$	Init. final Csc r. visual	$\pm 1.00$	${\eta }^0$, ${\beta }^0$, ${\zeta }^0$	Initial learning rates	0.05
${\mathbf{\text{W}}}^0$	Initial weights range	$\pm 0.025$	${\psi }_{\mathsf{v}}$	Self-organisation f. visual	$5.0\times {10}^{-5}$

Figure 9. Effect of the self-organisation forcing mechanism on the development concept patterns in the MultiMTRNNs model: mean mixed $F_1$ -score (a) and mixed edit distance (b) – “mixed” measures indicate a combination of training and test results with equal weight, mean of average and relative pattern distances (c), and intra- and intra-cluster distances (d) with interval of the standard error, each over 100 runs and over varied ${\psi }_{\mathsf{v}}$, respectively; representative developed Csc patterns (e–g) reduced from $\left|I_{\mathsf{C}\mathsf{s}\mathsf{c}}\right|$ to two dimensions (PC1 and PC2) for selected parameter settings of no, “good”, and large self-organisation forcing, respectively. Different words for shapes and colours are shown with different coloured markers (black depicts “position” utterance).

Figure 10. Activity in the Csc units after the model has been activated by proprioception and visual perception for the final internal states (somatosensory and visual) and the initial internal states (auditory), reduced from $\left|I_{\mathsf{C}\mathsf{s}\mathsf{c}}\right|$ to two dimensions (PC1 and PC2) and normalised, each. Visualisation a, c, e are shown for an representative example for low and b, d, f for high generalisation.