Tail-recursive Distributed Representations and Simple Recurrent Networks

Abstract

Representation poses important challenges to connectionism. The ability to compose representations structurally is critical in achieving the capability considered necessary for cognition. We are investigating distributed patterns that represent structure as part of a larger effort to develop a natural language processor. Recursive auto-associative memory (RAAM) representations show unusual promise as a general vehicle for representing classical symbolic structures in a way that supports compositionality. However, RAAMs are limited to representations for fixed-valence structures and can often be difficult to train. We provide a technique for mapping any ordered collection (forest) of hierarchical structures (trees) into a set of training patterns which can be used effectively in training a simple recurrent network (SRN) to develop RAAM-style distributed representations. The advantages in our technique are three-fold: first, the fixed-valence restriction on structures represented by patterns trained with RAAMs is removed; second, the representations resulting from training correspond to ordered forests of labeled trees, thereby extending what can be represented in this fashion; third, training can be accomplished with an auto-associative SRN, making training a much more straightforward process and one which optimally utilizes the n-dimensional space of patterns.

KEYWORDS:

• Recursive distributed representation
• recursive auto-associative memory
• compositionality
• finite-state machine
• language

Additional information

Notes on contributors

STAN C. KWASNY

E-mail: [email protected] and [email protected]

BARRY L. KALMAN

E-mail: [email protected] and [email protected]