FROM IMAGE SEQUENCES TO NATURAL LANGUAGE: A First Step toward Automatic Perception and Description of Motions

Abstract

We present our work concerning the connection of a vision system to a natural language system. That is, automatic processing transforms the original sequence of TV images into natural language descriptions concerning moving objects. It is the first time that this transformation has been achieved entirely by computer. A vision system that has been developed in Karlsruhe is briefly introduced. By analyzing displacement vector fields, trajectories of object candidates are recognized. The natural language system CITYTOUR is presented. The verbalization of spatial relations between static and moving objects can be studied with this system. The present state of the connection is described, and the data resulting from the vision system are partially used and verbalized by CITYTOUR.