AVIS: a connectionist-based framework for integrated auditory and visual information processing

This paper presents a general framework that facilitates the exploration of a single information-processing system in which auditory and visual information are integrated. The framework allows for learning, adaptation, knowledge discovery, and decision making. An application of the framework is a person-identification task in which face and voice recognition are combined in one system. Experiments are performed using visual and auditory dynamic features which are synchronously extracted from visual and auditory information flows, The experimental results support the hypothesis that the recognition rate is considerably enhanced by combining visual and auditory dynamic features.