A high storage capacity neural network content-addressable memory

Abstract

A new neural network design is developed and studied for error-correcting content-addressable memory (CAM). Relative to the size of the network, the design enables the storage and error-correcting retrieval of many more random patterns than has previously been possible. This increase in storage capacity is achieved by using hidden units to assist in the process of storing and retrieving patterns over the visible units. Key components of the design are the mean-field-theory learning algorithm adapted for CAM (MFT-CAM), an error-correcting retrieval algorithm for networks with hidden units, and an architecture that is not fully connected. Random pattern storage capacity appears to scale linearly with the number of hidden units, with a slope in the range 13–18, the largest networks investigated stored 4–6 bits per connection. As the ratio of the number of hidden to visible units increases, the algorithm must contend with a random-weight. Identity-map effect that creates spurious stable visible states and limits the error-correcting basins of attraction. In networks without hidden units, the error-correcting basins are found to be somewhat larger than those of the symmetric Perceptron algorithm, which has previously been shown to be a special limit of the MFT-CAM algorithm.