Does the use of multi-sensory devices in the arts create a better experience for the viewer?

Sarah Rachel Kemp
Manchester Metropolitan University, Fine Art - Printmaking

“Synaesthetes may be more likely to participate in creative activities (Rich, Bradshaw & Mattingley 2005), and some studies have suggested a correlation between synaesthesia and creativity (Domino 1989; Dailey, Martindale & Borkum 1997).”[i]

After reading an extract on sense experience from the text Phenomenology of Perception by Maurice Merleau Ponty, I decided to find out more about the condition of synaesthesia. I discovered the quote above and became intrigued by the idea that creativity could be linked to a specific genetically predisposed condition in the brain, as I had always believed that while some people are just naturally gifted when it comes to creativity, that to think in a creative way could be learned and developed through practice. If the experience of perceiving more than one sense at the same time is simply a natural part of the way we experience the world, then maybe multi-sensory art can be better appreciated by more people than an art piece that involves the use of only one of our senses to experience it? I decided to research further into the condition of synaesthesia in order to help me determine if the use of multi-sensory techniques and devices in the arts creates a better experience for the viewer.

What is Synaesthesia?

Synaesthesia is probably best described as being a neurological condition in which two or more senses are joined. Common forms include grapheme-colour synaesthesia where letters or numbers are perceived as coloured, ordinal linguistic personification where numbers, days of the week and months of the year evoke a certain personality, and spatial-sequence or number form synaesthesia where numbers, days of the week and months of the year have a precise location in space. Studies suggest that the condition may be as common as 1 in 23 people and that it may be passed on genetically. Synaesthetic experiences have also been reported by individuals who have been under the influence of psychedelic drugs, who have suffered a stroke, or as a result of blindness or deafness i.e. sense deprivation.[ii] Despite synaesthesia affecting each individual differently, certain characteristics have been identified in synaesthetes in general, such as left to right confusion and problems with maths and writing, but often there appear to be benefits of the condition including a heightened ability to remember things such as phone numbers and good spelling abilities. It has also been found that a high number of synaesthetes are creative people and in some way involved in the arts. Some people with synaesthesia have used their experiences as a synaesthete to aid their creative process and many non-synaesthetes have attempted to create art that captures what it is like to experience synaesthesia, in an attempt to heighten the experiences of their audience or viewer. This makes me wonder whether synaesthete artists are more naturally creative than most people because of having synaesthesia, or whether having synaesthesia causes the individual to turn to art as a means of expressing their heightened sense of perception? In this busy multi-sensory environment that we live in, should art be a soothing remedy to our senses or should it reflect the nature of our technology and cultural tendency to bombard our minds with sensory information? I will initially research into synaesthesia in the arts in order to discover about perception and its affect on creativity in general. By looking at research into synaesthesia, and then examining the use of multi-sensory devices in the arts today, I hope to gain a better understanding of these issues and determine whether the experience of the creative product is intensified by the use of multi-sensory techniques, or whether the use of synaesthesia in the creative process actually leaves the viewer with a feeling of sensory overload.

Creativity

When looking at the links between synaesthesia and creativity it must first be necessary to define what I mean by the term creativity. I have never subscribed to the ‘messenger of God’ theory of creativity[iii] which suggests that finished works are planted into the minds of creative people in their entirety, and that the creative person is merely a vessel through which the creation must pass in order to come into being. Instead I would prefer to use the analogy of the creative person as being like a dog with a bone, who gnaws away at it for hours until finally satisfied that it has gained all it can from it. This promotes the theory that the creative person will grasp at an idea and refuse to let go of it, mulling over problems and playing about with all the possible ways of solving them or interpreting the original idea. In other words creativity could be seen as synonymous with persistence and lateral thinking. By thinking of creativity in this way it would generally follow that creative thinking is simply a device which can be learned by anyone, in order to help a person to problem solve or express an idea. In the case of artists it may be that in order to express an idea or something that is significant to them emotionally, it is natural for them to turn to a creative way of thinking, as this may enable them to express their ideas consciously in order that they can then analyse them and ultimately understand them. This theory fits in well with my own personal way of working as an art practitioner, as I like to take my idea or subject matter and play about visually with different ways of expressing it, before I find my method of working or create a final work that I am satisfied with. The above theory also promotes the idea of the importance of direct experience in creativity and creative thinking. The creative person would work through all the possible ways of expressing their ideas or solving a problem, and come up with the most efficient or effective option, tapping into the limbic system in the brain which controls aspects such as emotion, memory, attention and consciousness. Richard E. Cytowic states in his book The Man Who Tasted Shapes that the limbic systems “ ability to determine valance and salience (value and relevance) yields a more flexible and intelligent creature, one whose behaviour is unpredictable and creative.”[iv] When considering Synaesthesia, this idea seems of importance due to the fact that Synaesthetes are very in tune with their limbic system. Creative people who tap into their limbic system and give their work an emotional charge seem to produce more powerful works that resonate more effectively with the viewer. This is not always the case, but as emotion makes our brain function more effectively it also provides us with intuition as to what works well aesthetically. Perhaps then, this means that synaesthetes are more adept at perceiving aesthetically? It could be seen that synaesthetes have a more direct experience than non-synaesthetes when it comes to their perceptions, due to the fact that more than one sense can be evoked and felt at the same time, producing not a better sense of perception but just a richer and more varied one. In respect of the arts, if synaesthetes use their abilities of an alternative means of perception as the focus of their work or the driving force behind it, does it therefore mean that their works are richer and more varied perceptionally than the work of non-synaesthetes, or do the difficulties of trying to convey their sensory fusion to the viewer mean that the work is not a true portrayal of what they perceive?

Synaesthesia in the Arts

Synaesthesia in art refers to a number of artistic experiments which combine or synthesise different areas within the arts; music and art being just one example of this. It has manifested itself in different genres including visual music, abstract film, computer animation, symbolist poetry and literature, and multimedia art. The term synaesthesia when applied to the arts should be used with caution however, as there is a need to differentiate between genuine congenital synaesthesia which is measured and researched scientifically and the use of artificial synaesthesia in order to attempt to demonstrate or evoke cross-sensory experiences in the arts. As scientific methods have only been developed in the last two decades to diagnose genuine synaesthetic experiences it is virtually impossible to assess synaesthesia in artists before this time, making it necessary to rely on interpretations of autobiographical and biographical sources.[v] The term ‘synaesthesia’ in art can be applied to either art created by synaesthetes or art created to portray the synaesthetic experience. It is generally used as an attempt to understand the relationship between the experiences of congenital synaesthetes, the experiences of non-synaesthetes, and the appreciation of this art by both people with synaesthesia and people without it. Art created by a synaesthete may also evoke experiences similar to synaesthesia in a non-synaesthete viewer, but we can not really assume that all ‘synaesthetic’ art will accurately reflect what it genuinely feels like to experience synaesthesia for a congenital synaesthete, which is why this type of art is often referred to as being artificially synaesthetic. Historically, synaesthetic art has consisted of a number of variations such as colour organs, musical painting and visual music. All of these genres of art have been intended to evoke an experience of cross-sensory fusion in their viewers, however the creators were not necessarily synaesthetes or even aware of what synaesthesia is. There are many modern synaesthete artists who have described in detail the way in which they use synaesthesia to aid them in the creation of their artworks, including such artists as Elizabeth Stuart Jones, Jane Mackay, Carol Steen and Marcia Smilack. Their work helps to demonstrate the complexities of the interplay between what they experience on a personal level and their resultant artistic creations.

Synaesthete Artists
-Carol Steen

Carol Steen, who I have previously mentioned, is an example of a practicing synaesthete artist who uses her synaesthetic experiences as the basis of her works as well as to help develop and create them. Steen has several forms of synaesthesia, seeing numbers and letters in colour, seeing colours when hearing certain sounds, and seeing colours and shapes elicited by certain forms of touch (such as acupuncture). Initially she was interested in sculpture as an art form as it didn’t involve colour at all, which came as a relief from the sometimes overwhelming visuals she would experience in her mind. Steen doesn’t describe having synaesthesia as being at a disadvantage to non-synaesthetes however, more that she had been given the opportunity to either involve herself with the colours she saw or to ignore them completely. In fact the only disadvantages appear to be that there was a lack of information on the condition when she was growing up, leading to a sense of alienation from most other people around her. Steen didn’t talk about her condition with anyone until she was around the age of twenty. She discovered that her father saw things in the same way as her when she was sat around the dinner table with her family and declared that “the number five is yellow”[vi], to which her father replied that it was in fact yellow ochre. The colours that Steen experiences have remained consistent throughout her life and as an artist she is never short on inspiration, creating her paintings by tapping into her unusual abilities as a synaesthete. Several years ago she attempted to paint the images she experienced in her mind after undergoing acupuncture (see Fig. 1 in appendices p.26) and described it as “like watching an amazing movie”[vii]. The problem with working in this way was that it was impossible for Steen to remember everything that she saw during the experience, so she has since adapted her working methods to painting music that she plays to herself in her studio. This allows her to replay the music so she can recreate what she sees rather than relying on her memories of the experience. Generally the images she sees are of explosions of colour or brilliantly coloured shapes which move in different directions. This produces an abstract image for the viewer when translated into the medium of paint, however I would say that it is not an accurate reproduction of the experience as Steen feels it, as she seems to view the colours as a translucent flowing sensation which is shown when she describes the colours as being like “sunlight shining through a gemstone”[viii], whereas when you look at her works they are first and foremost static objects on canvas and the colours are in pigment form, with the medium of oil paint giving them a sense of opacity. This makes me wonder whether it is an exercise in futility for synaesthete artists to attempt to portray what they experience, as they may never be able to translate what they feel in an abstract almost ethereal way into a solid medium. Perhaps however, it is simply the choice of medium that Steen uses in her works that limits the effectiveness of portraying her experience to the viewer?
-David Hockney David Hockney is probably the most well known modern practicing artist who uses his synaesthesia to inform his works but not to be the focus of them. This can be seen most evidently in his work with designing theatre sets for such productions as Jarry’s Ubu Roi at the Royal Court Theatre in London in 1966, Stravinsky’s Rake's Progress in 1974 and Mozart’s Magic Flute in 1978 for the Glyndebourne Festival, a triple bill of works by Satie, Poulenc and Ravel as well as a triple bill of Stravinsky pieces for the Metropolitan Opera in New York, Wagner's Tristan und Isolde in Los Angeles, Puccini's Turandot in San Francisco and Strauss's Die Frau ohne Schatten at Covent Garden. When listening to the music for The Rake’s Progress before beginning designing, Hockney felt that “the music seemed to [him] very linear and spiky” [ix] which is most likely what led him to use crosshatching to stylise the costumes and sets (see fig. 2 in appendices p.27). He took the crosshatching effect directly from Hogarth’s engravings (see fig. 3 appendices p.28) and the sets and costumes consisted of black, red, blue or green lines over a white ground, which were the colours of 18th century printer’s ink. Hockney later said that he “thought the design worked; it was wonderful with the music; it seemed vibrant and lively to [him] and the audience appeared to think the same”[x]. It seems like a lot of the time Hockney does not consciously use his synaesthesia, but it is simply there in the background, which is shown when he stated that he “actually felt those wiggly lines,”[xi] and that the “colour too was getting bolder; [he] simply felt it again”[xii] when he was creating the painting Nichols Canyon (see fig. 4 in appendices p.29). This may suggest that the most effective way of producing work that elicits a synaesthetic-like experience is to just use your natural feelings about what works together sensually, and tap into your innate sense of metaphor to achieve an aesthetically and sensually pleasing work. Perhaps also, where Steen’s work loses its effectiveness at conveying the feelings of having synaesthesia to the viewer is that it can only be experienced by one sense at a time; the sense of sight. Hockney’s opera sets may be more effective in demonstrating how the music affects him as the audience can experience the sound and visuals simultaneously, providing them with a richer sensory experience and maybe forcing them to engage in the opera in a similar way to someone with synaesthesia; a multi-sensory way. In comparison to Steen’s work, Hockney’s multi-sensory opera sets seem to be more effective at conveying synaesthetic effects, but does this necessarily mean that it is better to use multi-sensory devices in all his works or should the use of multi-sensory devices only be used in appropriate situations? The nature of opera is to combine visuals and sounds to create a narrative or spectacle, thereby using the senses of sight and hearing together in a synaesthetic way. However it is not necessary to hear music whilst viewing a painting for example. The power of a lot of Hockneys works (and most other painters, printmakers or sculptors works) seems to come from the fact that the narrative is told within the subject matter and also from the qualities of the way it is constructed. Hockney’s A Bigger Wave, 1989 (see fig. 5 in appendices p.30), for example is a powerful painting because of its scale, the dark stormy colours used, and the way the paint has been applied to the surface of the canvases in order to convey the force behind the wave and the impending threat of the wave crashing onto the beach. It is simply painted, showing bold brush strokes and needs nothing else; no sound effects, no background noise, no dramatic lighting, to convey the power of the sea to the viewer. You don’t need multi-sensory devices to feel what it would be like to experience the wave directly in person. The use of multi-sensory devices would be redundant and perhaps even detract from the power of the medium used. There have however, been many attempts throughout the arts to set colours to sounds and music to pictures, in a successful way.

Coloured Music and Multi-Sensory Contrivances

There have been many examples in the arts of the use of synaesthetic ideas by people who are not themselves synaesthetic, in order to create an effect of sensory fusion in their works. A common fusion is that of music and colour; termed coloured music. The origins of coloured music stem from the Renaissance period and in particular from the theories of Athanasius Kircher (1602-1680), a Jesuit music theorist and mathematician. He stated that every musical sound has an objective link to a certain colour[xiii], and from the eighteenth century onward a variety of keyboard-like instruments were created that simultaneously projected coloured lights and produced musical sounds by pressing their keys. The idea of a link between colour and sound was also typical of late Romanticism, seen for example in the works of Schönberg, who was not synaesthetic himself but experimented with the effects of coloured music in the short opera Die Glückliche Hand, where the score demands shifting colours to be shown with the music in order to mirror the emotions of the characters portrayed in it. Multi-sensory concerts of music and light known as ‘son et lumière’ were popular in the nineteenth century, and made use of the previously mentioned colour organs and keyboards as well as occasionally including odours to add to the sensory experience. Links between music and colour can also be seen in the Impressionist period of music’s history, and in fact art and music paralleled each other in their development at this time. Painters such as Monet, Renoir, Manet and Degas left behind the techniques of the Classical artists and favoured a lighter way of painting, often trying to capture the movement of colour and light. Similarly such composers as Debussy, Ravel, Delius, Satie and Faure began to create delicate musical textures by stepping away from the confines of the Classical and Romantic periods in music; taking themes from art such as sunlight through the leaves, clouds, the reflections of light dancing on water, and expressing them musically. The Impressionist composers favoured the orchestra as a medium because of the varied timbres (the equivalent of musical shades or colours) that could be achieved, and almost all of the elements of music such as melody, colour, harmony, rhythm and form were rearranged to form new tonal palettes. The piano was popular as an instrument because the damper pedal could be used to create the effect of harmonies that vibrate and hang in the air, and low registered reed instruments could be counter-pointed with the violins’ high registers, against which the percussion and harp would add a sense of lightness to the piece. Combining all this with the fragmentation and overlapping of phrases which created a flowing effect by altering the meter and providing an off beat syncopation for rhythm, the Impressionist composers were actually painting with music.[xiv] Their music is incredibly emotive because of the way it uses art-like textural devices to paint a visual image, creating a sensory fusion in your mind unlike that of many other musical periods. The listener really does feel the music. Painters have been moved by music in the same way that there have been composers who have been inspired by art and those who have created deliberately contrived colour and sound compositions. Georgia O’Keeffe was one such painter who created a series of pictures inspired by music; Music-Pink and Blue II (see Fig. 6 in appendices p.31) being one such example of her works. She was influenced by Kandinsky’s ‘On the Spiritual in Art’ and despite not being synaesthetic herself she still seems to create a sense of the nöetic understanding she has of the subject matter in her works. In a letter from 1930 she shows her understanding of how colours and expression can affect the viewer and perhaps convey a sense of the experience of the subject matter, not just what is there visually. “ I know I cannot paint a flower. I cannot paint the sun on the desert on a bright summer morning but maybe in terms of paint colour I can convey to you my experience of the flower or the experience that makes the flower of significance to me at that particular time.”[xv]

The above quote is significant in terms of it hinting at the links between the richness and diversity of perception as felt by synaesthetes, and the complexities and nöetic quality to the sense of aesthetics felt by those with creative and artistic vision. It seems important when considering if multi-sensory techniques and devices create confusion for the viewer of an art work or whether it adds to the experience, because it suggests that synaesthesia may not just be an experience felt by a privileged few, but a nöetic capability to which everyone has access; making sensory fusion a natural and pleasing form of perception for everyone, and multi-sensory art works effective as a mode of experience for the viewer. It is easy to see how art can inspire music and music can inspire art, and the results can often be pleasing to the viewer, however it may be difficult to take an art work and a piece of music and marry the two together effectively. It should be a lot easier however, and more effective to produce the sound and the image together as if they are one entity, thus making the effect and experience a lot better for the viewer or audience. This has been achieved in recent years in the rise of hypermedia, the internet, and by a group of people known as VJ’s.

The VJ Culture: Synaesthesia Today

VJing is in essence very similar to DJing except that the VJs (or visual jockeys) mix video clips instead of sounds. The term was first used in the New York club Peppermint Lounge in the 1970s, and although it was initially an underground movement found itself introduced to the mainstream audience through outlets such as MTV which played music videos on a widely accessible TV channel. The modern VJ today sees himself as being rather like a film maker or alchemist of the moving image, mixing live visuals in front of real people in various environments, whether in a club, concert venue, theatre, art gallery or at a summer music festival, the VJ creates audio-visual marriages to engage the senses of the audience. The existence of the VJ culture comes from an amalgamation of developments on the social, artistic, cultural and technological fronts, but more immediately it emerged with the rise in popularity of electro music at the end of the 1970s, breaking through at the beginning of the 1980s with house music. The nature of house parties meant that the stage act that you would traditionally expect to see at a venue with live music was absent, so the integration of multiple screens in order to project visuals was used throughout the space as a natural replacement for the lost act or DJ. This places the emphasis of the house party on the total spectacle of the event and seems important when considering multi-sensory devices, as it suggests a need for some sort of synaesthetic experience through the stimulation of vision as well as hearing, that can be focused on by the audience whilst listening to the music; in other words a need for a multi-sensory experience. The presence of the multi-sensory aspect is also evident when looking at how the house parties (and indeed many VJ shows) are formed. Aside from the total experience, the individual components that go together to form the experience are created in a synchronised way. The DJ spins records whilst the VJ triggers loops and the lighting is controlled by a different technician. All this is done with hardly any pre-arranging and comes together in an improvised but synchronised way; it is a multi-media approach that is synaesthetic both in its construction and in its end product.[xvi] The use of VJing techniques can be seen most evidently in recent years in the concert visuals that accompany live music shows. One such example of a popular show was that of Andy Warhol’s Exploding.Plastic.Inevitable. that toured as a road show in 1966 and included a mixture of music, theatre, film and performance to create a total spectacle for the audience. In the shows, Warhol simultaneously projected two or more of his films or slides onto the Velvet Undergound as they performed their music. It also included a light show, dancers and interviewers who interacted with the audience by asking confronting questions. This style of performance art seems to refer back to the Cabaret Voltaire and the Dada movement in its avante-garde approach to dealing with audience interaction and creating a spectacle of multi-sensory experiences. It must have seemed unusual and exciting to the audience who may not have experienced anything like it before. Similarly, another example of the effective use of multi-sensory devices in the form of VJing in the concert environment is that of the popular rock band Pink Floyd’s rock opera The Wall. This is infamous in its use of spectacular visuals and staging that engaged the audiences’ senses in a synaesthetic way. The band used theatrically oversized scenery in the form of a wall, onto which they integrated audio-visuals by projecting animations and film images, thus creating a synaesthetic experience for the audience. Both Pink Floyd and Warhol used their film images and videos within the audio performance to great effect, because although the music of the Velvet Underground and Floyd stand well on their own without the multi-media aspect of combining audio with visuals, the nature of a live show demands a more rich experience sensually for the audience or they might as well simply play their records, CDs or audio files to themselves at home. The essence of a live concert is in the entire spectacle and the atmosphere created by the stimulation of the audiences’ visual, tactile, olfactory, and auditory senses simultaneously, promoting the use of multi-sensory devices in this environment.[xvii] The VJ Robin Rimbaud also known as Scanner stated that “VJs have shown us how it’s possible to watch sound, charting the movement and flow of the music through enhanced vision.”[xviii], and that “the finest VJs can translate and transform, transport and transpire the boundaries of what we see and hear.”[xix] After looking at the works of some of the many VJs around today this statement seems to ring true, as when the VJ gets it right the experience can become sensually extremely interesting. The piece entitled Timbre[xx] which was created out of a collaboration between the VJ groups of Coldcut and Hexstatic won many awards, being adopted as a Greenpeace anthem, ‘what you see is what you hear’ because of its imagery of de-forestation and attitude towards green ideals. The piece by hfr-lab entitled Cityscan 4.11[xxi] is also effective as the cityscapes that can be seen visually, scroll up the screen in time with the music becoming pattern-like in their use of subtle layering and movement. The audio and visuals work together in harmony and you experience them as one entity rather than separate media components. It seems obvious that the use of multi-sensory devices can be extremely effective in creating something that it visually and sensually interesting, and may in fact be beneficial to the audience or viewer by creating a spectacle and an atmosphere that adds to an experience. However is it the appreciation of the stimulation of senses in a synaesthetic way that affects the audience or viewer, or is it this spectacle and sense of a shared experience with fellow members of the audience, and the sense of community and belonging that this may fill them with, that makes the experience better and richer? Maybe this question can be answered by looking at the use of hypermedia and the relationship between language and synaesthesia.

Hypermedia

Jay David Bolter defines hypermedia as being the means of linking textual and audio-visual files. In his text entitled Writing Space: The computer, hyper text, and the history of writing, he states that: “A hypermedia display is still a text, a weaving together of elements treated symbolically. Hypermedia simply extends the principles of electronic writing into the domain of sound and image. The computer’s control of structure promises to create a synaesthesia in which anything that can be seen or heard may contribute to the texture of the text.” [xxii]

Certainly the internet and hypermedia used in such places as interactive museum collections, travellers’ kiosks, alternative presentations of literary texts, taped performances, and also learning systems, do employ the use of multi-media devices that stimulate different senses in the viewer, but is it synaesthetic in the way Bolter describes? I believe that in itself it falls short of creating synaesthesia for the viewer. The condition of synaesthesia is essentially non-linear in respect of the fact that the senses involved are triggered simultaneously and not as a result of a leakage of one sense into the other. For example, if a synaesthete hears a piece of music they may also see a coloured shape that enters their field of vision. They do not hear the sound THEN see the shape, they hear the sound AND see the shape at the same time with no pause in between. The auditory sense is not triggering the visionary sense. This is where Bolter’s idea of synaesthesia is wrong as hypermedia such as the internet is linear in the way that it uses links to navigate through it to allow the viewer to access different information in different forms of media. The viewer may use the internet in a synaesthetic way by reading a text whilst simultaneously listening to an audio clip, however the way the internet is structured means that the media elements are separate and must be accessed in a sequential rather than simultaneous way. This is the difference between multi-media texts and hypermedia as opposed to other multi-media devices. It shows that multi-media can mean multi-sensory, but that multi-sensory devices are not always necessarily synaesthetic. The internet for example has different elements in its textuality that stimulate different senses however it is down to the viewer or user to decide how they want to experience that information, whether as separate media or combined in a synaesthetic way. When considering if the use of multi-sensory devices provides a better experience for the viewer it may mean that providing a choice as to how to view the media (either in a synaesthetic way or a linear way) allows the viewer or user to create as rich and complex an experience as they like. The difference between multi-media texts (or hypermedia) and other multi-media products such as television, documentaries, computer games, and virtual reality simulations is that with hypermedia the images or sounds are incorporated into the structure of the thing as a whole (as part of the textuality of the piece). With other multi-media products the text is treated as just another image and turned into a graphic. Does this mean that the use of written language (either numerate or literate) stifles the effect of synaesthesia? It is suggested in James C. Morrison’s essay entitled Hypermedia and Synesthesia that computers bring our culture closer to that of preliterate man.[xxiii] This theory however falls apart in the same way as Bolter’s because the emphasis on text and the linear way of processing information on the computer has no relationship with the preliterate sense of orality. This raises some interesting issues regarding the links between synaesthesia, creativity and language. Is it in fact the simultaneous direct or shared experience brought about by a tradition of an oral culture that creates a richly pleasurable multi-sensory experience, or could there be some kind of synaesthetic roots in language itself that suggests a basic sense of a collective multi-sensory sensibility within a culture? Either way, it suggests that multi-sensory experience is central to the way we express ourselves as well as the way in which we experience the environment around us.

Synaesthesia, Language and Metaphor

Non-literate societies did not really use numbers and similarly the computer substitutes the use of numbers for ‘yes’ and ‘no’ options. The ‘primitive’ tribes in Australia and Africa and the Inuits who do not use numbers in a series, also have a binary system that consists of independent numbers for one and two and composite numbers up to six, which is where the idea that computers take us back to the culture of pre-literate man could come from. Humans speak in what is known as natural language rather than formal languages, which is lacking in total precision and logic which is partly why machines and computers have failed to replicate artificial intelligence that effectively mimics human behaviour. Thought and consciousness is rooted in speech not in texts, which is an idea explored in Morrison’s essay when he quotes Tobias Dantzig who says that thought, speech and texts all “have their meanings through reference of the visible symbol to the world of sound.”[xxiv] Put another way, the words that a reader might see on a page are symbols of speech and language that prompt them to form either vocalised sounds, or imagined ones within their consciousness. Even when participating in silent reading, the consciousness can cause motions of the vocal organs that suggest an interplay between the oral and the visual senses. Without knowing it reading can cause us to behave in a naturally synaesthetic way. The evidence that perhaps there is a collective multi-sensory sensibility within a culture can be seen in the integration of expressives into language, which is demonstrated in Austroasiatic languages. Expressives are words that describe emotions, movements, sensations, noises, colours, light patterns, shapes and aesthetic feelings. Synaesthesia can often be noted in these words, and the use of wordplay means that structurally there are unlimited variations that can be created because the speaker can alter and use language to suit them. Thus a synaesthetic kind of expressiveness is built into the language structure. There is in fact a general cultural agreement between the way we individually perceive things such as a square looking like a square in order to identify things in our environment and aid communication between individuals. We also have a shared sense of metaphor in certain things such as a cultural belief that lemons taste sharp. In our everyday language we continually make connections imaginatively between our senses such as ‘I see what you are saying’ and ‘that lemon tastes sharp’ but we mean it in a non-literal, metaphoric way. You could be forgiven for assuming that the western use of metaphor within language is evidence of a cultural recollection of synaesthesia that may have been lost from the consciousness of non-synaesthetes. Metaphor is the means of combining two unrelated things by seeing the similar in the dissimilar. It is easy to see why people may think that synaesthesia is simply an extended form of metaphor, because in order to express what they experience a synaesthete can only use the limited language at their disposal to give a verbal interpretation of the sensory experience by way of an analogy. This can appear to be the same as metaphor. It points to a need within our culture for expressing an experience that is multi-sensory in nature and that language is insufficient in doing this, perhaps even hindering the expression. It brings me back to the idea at the start of this essay that creativity is an outlet for the synaesthetic experience, and that it is most likely inherent to everyone just not always on a conscious level. Metaphoric devices are the kind of artificial synaesthesia that can be seen in symbolist poetry and other forms of literature that I have previously mentioned. In his book The Man Who Tasted Shapes, Richard Cytowic at one point suggests that without the ability to form cross-modal associations which are present in the condition of synaesthesia, then language would never have evolved as the association of two sensations provides humans with the ability to assign names to objects and create abstracts such as language, however he later realises that these associations in non-synaesthetes are the result of experience and reasoning that comes from interaction with our environment becoming like an instinctive process. This is why synaesthesia is different to metaphor as synaesthesia is an involuntary sensation which is not governed by a learned experience. In terms of multi-sensory devices written language shows a link between the oral and the visual senses. Without knowing it reading can cause us to behave in a naturally synaesthetic way which must show the use of multi-sensory, cross modal devices in our minds to be intrinsic to our comprehension of symbols and imagery. The use of metaphoric devices within our natural language points to a need within our culture for expressing how we experience certain aspects of our environment that language is insufficient in doing. It suggests that creativity may be an outlet for the synaesthetic experience not just with synaesthetes but with non-synaesthetes as well.

Intuitive Recognition

Synaesthesia is often seen as a desirable thing for a creative person or someone involved in the arts to have. This is probably because it represents a richer, more varied, or heightened sense of perception which is concerned with direct experience as I have previously mentioned. A creative person might therefore see having this condition as having an aesthetic advantage over other people when creating their works. Richard Cytowic believes that “ satisfying art is a product of deep knowledge and understanding within the artist”[xxv] and also goes on to say that “it is true that art is informed by the intellect and with acquired technique. But the function of the artist is to penetrate the visible world to illuminate the mystery behind it….If successful, the artist’s expression resonates within the inner life of the reader, viewer, or listener who experiences what I have called an intuitive recognition.”[xxvi] This would appear to be the case when looking at the Impressionist painters and musicians who knew all the techniques and conventions of their predecessors, but who chose to break with these and create works that reflected how the subject matter truly felt to them rather than what they knew it should appear like from an intellectual way of thinking. In doing so they created works that are amongst the most famous and popular pieces in history. When you look at these pieces you feel a sense of ‘this is great’ or ‘this is it’, which gives a sort of aesthetic validation or recognition of something that you can’t put into words because it comes from inside as an emotional and subjective reaction. Cytowic calls this an “intuitive recognition”[xxvii]. Our popular culture today seems to celebrate this subjectivity because ‘intuitive recognition’ relates to feelings and emotions rather than intellectualisation, and promotes the idea of the importance of the individual and individual experience. This is possibly why creative people such as artists, musicians, writers and film makers are so popular as they confirm to the individual that it is acceptable for us to have a desire for inner expression and even go so far as to suggest that it is a vital part of having a balanced psyche. It is little wonder then that synaesthetic techniques and ideas have been used so often in the arts. In conclusion, it could indeed be seen that synaesthetes have a more direct experience than non-synaesthetes when it comes to their perceptions, due to the fact that more than one sense can be evoked and felt at the same time, producing more interaction with the environment and a richer and more varied sense of perception. However, when translating these sensations from their own experience using an alternative mode of expression it is impossible for a synaesthetic artist to make the audience feel the sensations in the same way that a synaesthete would. It appears that this can mainly be attributed to the fact that individually we all experience things differently anyway due to the fact a sense experience is highly subjective and individual, but it can also be affected by the limitations of the media used to express the synaesthesia. This can be seen when looking at Steen’s paintings in comparison to Hockney’s theatre sets. The audience may need to experience a stimulation of several of their senses if they are non-synaesthetes in order to provide them with a rich sensory experience and maybe force them to engage in the experience in a multi-sensory way which is similar to that of a person with synaesthesia. This theory is also supported by the popularity of events such as house parties and live concerts where the emphasis is on the spectacle and the entire sensory experience where multi-sensory devices can be extremely effective in creating something that is visually and sensually interesting in terms of the atmosphere that the audience feel. It doesn’t seem to matter whether the synaesthetic experience of the multi-sensory devices used by VJs in these environments are what the audience appreciates, or whether it is the sense of a shared experience that is created by the large crowds, because the emphasis of these events is firmly rooted in direct experience and a need to be involved in the self expression and creativity of those on stage. These events mean that multi-sensory experiences can be felt by anyone. The fact that a kind of artificial synaesthesia can be seen throughout history in multi-sensory contrivances and that it is rooted in our everyday lives through the linguistic devices we use, shows that multi-sensory devices and thinking in a synaesthetic way can become part of our natural means of expression as well as helping us to comprehend symbols in our surrounding environment.
In short multiple senses are the norm, and the use of multi-sensory devices in the arts should not cause a sensory overload when used sympathetically in an appropriate context, and although it may not necessarily always provide a better sense of perception for the viewer, it may add another dimension to an experience and undoubtedly alter the way it is perceived. After all, we experience the world in different ways as an individual but we use all of our available senses with which to interact with our surrounding environment.

-----------------------
[i]Synaesthesia, http://en.wikipedia.org/wiki/Synaesthesia (Accessed 7th May 2007)

[ii]Synaesthesia, http://en.wikipedia.org/wiki/Synaesthesia (Accessed 7th May 2007) Initial information and statistics gained from viewing website.

[iii] Cytowic, Richard E. The Man Who Tasted Shapes, First MIT Press edition, 1998, p.89.
(First published in 1993 by G.P.Putnam’s Sons, New York)
For further information on the messenger of God theory.

[iv] Ibid., p.189.

[v]Synaesthesia, http://en.wikipedia.org/wiki/Synaesthesia (Accessed 7th May 2007)
Information on Synaesthesia in the arts.

[vi] The Health Report, Robin Hughes, http://www.abc.net.au/rn/talks/8.30/helthrpt/hstories/hr080796.htm (Accessed 7th May 2007)
This is a quote from a Radio National Transcript of an episode of The Health Report on Synaesthesia in which Robin Hughes interviews both Dr Richard Cytowic- a neurologist and synaesthesia expert, and the artist Carol Steen who has the condition.

[vii] ‘Synesthesia: When sounds have colors and words have taste’, Richard Schapiro, http://jscms.jrn.columbia.edu/cns/2005-04-19/schapiro-synesthesia/ (Accessed 7th May 2007)

[viii] Ibid,.

[ix] Hockney, D. That’s The Way I See It, Thames & Hudson Ltd, 2002, p.21.
(First published in the United Kingdom in 1993 by Thames & Hudson Ltd.
First paperback edition 1999)

[x] Ibid,. p.28.

[xi] Hockney, Op.cit., p.67.

[xii] Hockney, Op.cit., p.68.

[xiii]Synaesthesia, http://en.wikipedia.org/wiki/Synaesthesia (Accessed 7 May 2007)
For more information on coloured music.

[xiv] The Fine Art of Impressionism: Monet, Debussy and Multiple Intelligences, Mariesse Oualline Samuels, http://www.uh.edu/hti/cu/2005/1/12.pdf (Accessed 19 February 2006)
For information on the links between impressionist art and music.

[xv] Cytowic, Op.cit., p.116.

[xvi] Faulkner, Michael (Editor), Audio-visual art and vj culture, Laurence King Publishing Ltd, 2006, pp.10-22.
For information on the history of VJing and concert visuals.

[xvii] Ibid,. pp.37-41
For information on concert visuals.

[xviii] Rimbaud, Robin, in Faulkner, Michael (Editor), Audio-visual art and vj culture, Laurence King Publishing Ltd, 2006, p.8.

[xix] Ibid,. p.8.

[xx] Coldcut/Hexstatic, Timbre, Audio by Coldcut/Hexstatic, © 1997 Ninjatune, Track 21 of the VJ Book DVD that was developed and produced by D-FUSE (creative director Michael Faulkner) and accompanies Faulkner, Michael (Editor), Audio-visual art and vj culture, Laurence King Publishing Ltd, 2006. For further information go to www.DFUSE.COM/VJBOOK

[xxi] hfr-lab, cityscan 4.11, Audio by RCQ, © 2005 hfr-lab,Track 16 of the VJ Book DVD that was developed and produced by D-FUSE (creative director Michael Faulkner) and accompanies Faulkner, Michael (Editor), Audio-visual art and vj culture, Laurence King Publishing Ltd, 2006. For further information go to www.DFUSE.COM/VJBOOK

[xxii] Bolter, J.D. (1991). Writing Space: The computer, Hypertext,and the history of writing, Hillsdale, NJ: Erlbaum, p.27, in Hypermedia and Synaesthesia, Morrison, James C. http://www.media-ecology.org/publications/MEA_proceedings/v1/hypermedia_and_synesthesia.html
(Accessed 21 Nov 2007)

[xxiii] Hypermedia and Synaesthesia, Morrison, James C. http://www.media-ecology.org/publications/MEA_proceedings/v1/hypermedia_and_synesthesia.html
(Accessed 21 Nov 2007)
For more information.

[xxiv] Dantzig, Tobias. Number:The Language of Science. P.57, in Hypermedia and Synaesthesia, Morrison, James C. http://www.media-ecology.org/publications/MEA_proceedings/v1/hypermedia_and_synesthesia.html
(Accessed 21 Nov 2007)

[xxv] Cytowic, Op.cit., p.217.

[xxvi] Cytowic, Op.cit., p.217.

[xxvii] Cytowic, Op.cit., p.217.

Appendices

Fig. 1

Vision, by Carol J. Steen. The image was evoked synaesthetically by acupuncture needles. The image is abstract and mysterious but does not really show the translucent ethereal quality which Steen experiences in her synaesthesia.

[pic]

Steen, Carol. Vision,, 1996, oil on paper, 15” x 12 ¾”, © Carol Steen. Photo © Jean Vong, in Duffy, Patricia Lynne. Blue Cats and Chartreuse Kittens, Henry Holt and Company LLC, First Edition, 2001, p87.

Fig. 2

Set design for the auction scene from The Rake’s Progress (detail), recreated for the theatre exhibition ‘Hockney Paints the Stage’, at the Walker Art Centre, Minneapolis, 1983. Showing the crosshatching evident in Hogarth’s engravings and Hockney’s reference to the way the music felt linear and spiky to him.

[pic]

Set design for the auction scene from The Rake’s Progress (detail), recreated for the theatre exhibition ‘Hockney Paints the Stage’, at the Walker Art Centre, Minneapolis, 1983 © David Hockney in Hockney, D. That’s The Way I See It, Thames & Hudson Ltd, 2002, p.23.
(First published in the United Kingdom in 1993 by Thames & Hudson Ltd.
First paperback edition 1999)

Fig. 3

William Hogarth, frontispiece to Dr Brook Taylor’s Methods of Perspective, published by Joshua Kirby, 1754, showing the style of crosshatching used by Hogarth which Hockney refers to in his sets and costumes for The Rake’s Progress.

[pic]

William Hogarth, frontispiece to Dr Brook Taylor’s Methods of Perspective, published by Joshua Kirby, 1754, in Hockney, D. That’s The Way I See It, Thames & Hudson Ltd, 2002, p.29.
(First published in the United Kingdom in 1993 by Thames & Hudson Ltd.
First paperback edition 1999)

Fig. 4

Nichols Canyon, 1980, by David Hockney. Showing the influence on his paintings of the way he felt the wiggly lines of the landscape he saw.

[pic]

Hockney, D. Nichols Canyon, 1980, acrylic on canvas, 84” x 60”, ©David Hockney, Private Collection, in Hockney, D. That’s The Way I See It, Thames & Hudson Ltd, 2002, p.67.
(First published in the United Kingdom in 1993 by Thames & Hudson Ltd.
First paperback edition 1999)

Fig. 5

A Bigger Wave, 1989, by David Hockney. The dynamic power of the brushstrokes conveys the atmosphere created by the stormy wave crashing onto the beach. There is no need for the stimulation of other senses by using multi-sensory devices as it may detract from the power and simplicity of the medium used.

[pic]

Hockney, D. A Bigger Wave, 1989, oil on four canvases, overall 60 ¼” x 72 ¼”, © David Hockney, Private Collection, in Hockney, D. That’s The Way I See It, Thames & Hudson Ltd, 2002, p.196.
(First published in the United Kingdom in 1993 by Thames & Hudson Ltd.
First paperback edition 1999)

Fig. 6

Music-Pink and Blue II, 1919 , by Georgia O’Keefe. She attempted to paint her experience of what she saw, showing a sense of the nöetic understanding she has of the subject matter in her works. Does this mean that we can all comprehend synaesthetically if on a nöetic level?

[pic]

O’Keefe, Georgia. Music-Pink and Blue II, 1919, Whitney Museum of American Art, New York.
Image taken from: Humanities Web, http://www.humanitiesweb.org/human.php?s=g&p=c&a=p&ID=950 (Accessed 10 Feb 2008)

Bibliography

Cytowic, Richard E. The Man Who Tasted Shapes, First MIT Press edition, 1998,
(First published in 1993 by G.P.Putnam’s Sons, New York)

Duffy, Patricia Lynne. Blue Cats and Chartreuse Kittens, Henry Holt and Company LLC, First Edition, 2001.

Faulkner, Michael (Editor). Audio-visual art and vj culture, Laurence King Publishing Ltd, 2006.

Hockney, D. That’s The Way I See It, Thames & Hudson Ltd, 2002,
(First published in the United Kingdom in 1993 by Thames & Hudson Ltd.
First paperback edition 1999)

Ponty, Maurice Merleau. Phenomenology of Perception, Routledge, 1995, translated by Colin Smith.

Discography

D-Fuse, VJ: Audio-visual art and VJ culture, [VJ Book DVD], developed and produced by D-Fuse (creative director Michael Faulkner) and accompanies Faulkner, Michael (Editor), Audio-visual art and VJ culture, Laurence King Publishing Ltd, 2006. For further information go to www.DFUSE.COM/VJBOOK

Websites

Synaesthesia, http://en.wikipedia.org/wiki/Synaesthesia (Accessed 7 May 2007)

The Health Report, Robin Hughes, http://www.abc.net.au/rn/talks/8.30/helthrpt/hstories/hr080796.htm (Accessed 7 May 2007)

Synesthesia: When sounds have colors and words have taste, Richard Schapiro, http://jscms.jrn.columbia.edu/cns/2005-04-19/schapiro-synesthesia/ (Accessed 7 May 2007)

Hypermedia and Synaesthesia, Morrison, James C. http://www.media-ecology.org/publications/MEA_proceedings/v1/hypermedia_and_synesthesia.html
(Accessed 21 Nov 2007)

Synaesthesia, Robyn Williams, http://www.abc.net.au/rn/scienceshow/stories/2006/1764423.htm (Accessed 7 May 2007)

The Fine Art of Impressionism: Monet, Debussy and Multiple Intelligences, Mariesse Oualline Samuels, http://www.uh.edu/hti/cu/2005/1/12.pdf (Accessed 19 February 2006)

Humanities Web, http://www.humanitiesweb.org/human.php?s=g&p=c&a=p&ID=950 (Accessed 10 Feb 2008)