Colour association to sound : a perceptual experiment using a CIELab haptic response interface and the Jyväskylä film music set

Abstract

While some cross-modal associations might have psychobiological basis, other patterns of association might be acquired or cultural (cf. [5], [11], [4]). Stimuli perceived through different sensory organs via parallel brain pathways may be associated at a higher level if they both happen to have the same effect on emotional state, mood, or affective state ([14]). If the perceived input under-specifies an event, more complex cognitive processing mechanisms kick in ([7]). This process is not primarily ecological and might be mediated by emotion ([12]). Research on crossmodal matching has provided evidence that many non-arbitrary and universal correspondences exist. Audio-visual correspondences may be based on amodal correspondences, for example, the loudness of sound and the luminosity of light ([14]). [13] showed that most cultures display word clusters near ‘red’, ‘green’, ‘yellow’, and ‘blue’ (in addition to ‘white’ and ‘black’), and argued that “focal colours” really are universal. Bresin ([2]) derived 24 colours from a scheme of selecting approximately equal distances in colour parameters in HSL (Hue, Saturation, Lightness) “space”. This produced a set where the colour patches are arguably more evenly distributed, from a perceptual point of view, than those in the two studies mentioned above. Bresin found correlations between colour parameters and the affective intent in music excerpts, i.e. listeners matched colours to music excerpt played with a certain ‘feeling’. As in [1] but more general, colour brightness was associated with positive emotion and darkness with negative emotion. Palmer ([12]) investigated colour association to classical music excerpt where tempo and tonal mode were manipulated. The authors found that colours of high saturation and brightness, and colours more towards yellow (‘warmth’) were selected for music stimuli in fast tempo, and that conversely, de-saturated (‘grayer’), ‘darker’, and blue colours were selected for music of slow tempo in minor mode. Furthermore, they claimed strong support for emotion as a mediating mechanism for the cross-modal associations. A review of the research provoked the idea that colour association to sound might be context-dependent. When associating colour to music, natural soundscapes, and ‘soundscape compositions’, do people use different strategies? Which musical features influence colour association? Can emotion mediate between musical features and colour association? We designed an experiment to investigate a) correlations between visual colours defined by linear parameters and music stimuli with previously validated affect; b) correlations between the colour parameters and computational acoustic and musical features; and c) the multiple regressions onto colour parameters of affective ratings (emotions), psychoacoustic descriptors, and musical features.