Multimodal literacy refers to meaning-making that occurs through the reading,
viewing, understanding, responding to and producing and inter- acting with multimedia and digital texts. It may include oral and gestural modes of talking, listening and dramatising as well as writing, designing and producing such texts. The processing of modes, such as image, words, sound and movement within texts can occur simultaneously and is often cohesive and synchronous. Sometimes specific modes may dominate. For example, when processing screen- based texts the visual mode may dominate whereas the mode of sound may be dominant in podcasts. Two elements that impact on how we teach literacy today in our classrooms:
The impact of technology on reading and writing
Social practices that shape the way in which we communicate and make meaning
DISCUSS in your group
A crucial issue for literacy educators is whether those basic aspects of reading and writing on screen entail different processes from the reading, writing and producing of print- based texts. The theory of multimodality (Kress & van Leeuwen, 2001; Kress, 2003) has been the basis for the contention that the simultaneous processing of different modes of text, image, sound and gesture in visual, media or digital texts is a different function from the linear, sequential reading of print-based texts. As students produce multimodal texts they need to consider and understand features of design such as layout, composition, use of text and image or graphics including aspects such as colour, size, medium, angles and the way these would suit a specific audience. With the increased use and popularity of social networking, literacy practices have adapted to these changed forms of communication. Such rapid development of digital technologies has changed the nature of literacy and will continue to do so. The reality is that, even at this time of writing, newer communication technologies are emerging. More than ever teachers need to prepare their students for the new literacy practices and discourses that have become embedded in online social interaction. More than ever students need to be able to identify the authenticity and ideology of texts and messages, and to critically evaluate the purpose and audience that specific texts are designed for. With the sophistication possible with designing texts, students need not only to be able to use and manipulate new technologies but to be able to consider the best way to use these for their purpose and audience. A text may be defined as multimodal when it combines two or more semiotic systems. There are five semiotic systems in total: Linguistic: comprising aspects such as vocabulary, generic structure and the grammar of oral and written language Visual: comprising aspects such as colour, vectors and viewpoint in still and moving images Audio: comprising aspects such as volume, pitch and rhythm of music and sound effects Gestural: comprising aspects such as movement, speed and stillness in facial expression and body language Spatial: comprising aspects such as proximity, direction, position of layout and organisation of objects in space.
a picture book, in which the textual and
visual elements are arranged on individual pages that contribute to an overall set of bound pages a webpage, in which elements such as sound effects, oral language, written language, music and still or moving images are combined a live ballet performance, in which gesture, music, and space are the main elements. Multimodal texts can be delivered via different media or technologies. They may be live, paper, or digital electronic. A key concept to understand when working with multimodal texts is that the whole is greater than the sum of the parts. This means that we need to make sure we dont just read a text like a picture book, talk about one or two literary features, discuss which picture we like the best then move on to the next activity. Multimodal texts make meaning because all the elements work together to create a whole text. This is the case for picture books, using only image and text, or for video and multimedia, where image, gesture, movement, words and sound all work together to create the final piece.