Home > Issue20 > Cambra, Leal, & Silvestre article

The interpretation and visual attention of hearing impaired children when watching a subtitled cartoon¹

Cristina Cambra, Aurora Leal and Núria Silvestre, Universitat Autònoma de Barcelona

ABSTRACT

The article analyses how hearing impaired children interpret a subtitled cartoon, taking into account their visual attention to both sources of information (the images and the subtitles), and the auditory information from the oral language. The sample was made up of eleven children, aged seven to eleven years old (second to fifth grade of elementary school), who attend ordinary mainstream schools in Barcelona in the oral mode. A narrative cartoon with subtitles in Catalan was used in the study. The eye movement of the subjects was also analysed using an eye-tracker as the subjects watched the audiovisual story. After seeing it, they were asked to orally retell the cartoon story. The results reveal the importance of having time to view the images to interpret the story well. The eye-tracker shows that the hearing impaired children who participated in the study maintained attention on the frames with subtitles, with the exception of the participants in second grade with lower reading skills who did not pay attention to the subtitles towards the end of the cartoon when the conflict of the story was resolved.

KEYWORDS

Hearing impairment, subtitles, eye-tracker, cartoon programme, accessibility, television.

Subtitling is an important tool for hearing impaired people to understand television programmes. Despite this, the use of subtitles does not always ensure comprehension of the message of a programme, as shown in the studies conducted by Cambra et al. (2008, 2009) which explore the contribution of subtitles to the comprehension of a brief televised story presented first without subtitles and later with subtitles to two groups of hearing impaired participants of differing ages (one group from 12 to 19 and another from 6 to 7 years old). The authors conclude that the low level of comprehension coupled with the participants’ reading speed compared to the speed at which the subtitles appear, show that hearing impaired students do not always have enough time to look at the image and read the subtitles. Therefore, in some cases they either totally or partially ignore the subtitles, while in other cases they may miss the information provided by the image. In a study conducted with subjects aged 28 to 54, Jensema et al. (2000b) analysed how subtitles influence the way the subjects watch television, and concluded that reading subtitles dominates eye movements, while watching the action tends to be secondary. Specifically, with a sample of 23 hearing impaired subjects aged 14 to 61, Jensema et al. (2000a) demonstrated that 84% of the time is spent reading the subtitles, while only 14% of the time is spent looking at the images. The study by d’Ydewalle et al. (1991) with hearing subjects who were asked to watch a subtitled programme showed that they also spent a considerable amount of time on the subtitled area of the screen, even if they had the oral version available in their native language. The authors confirm the findings of Pezdek and Hartmann (1983) and Lorch et al. (1979) who highlight the predominance of the visual information and its processing, as opposed to the processing of the sound track in the understanding of TV programmes.

When the viewers are children with a hearing impairment who are just learning how to read and write, the activity of watching television with subtitles is even more complicated, for several reasons. This is true firstly because since their reading skills are weak (Marschark et al. 2012), they become tired of following the subtitles and therefore stop looking at them (Cambra et al. 2009), and secondly because when watching television with subtitles they receive several stimuli simultaneously, not only visual (images and subtitles) but also auditory (oral language and environmental sounds) which further complicates the activity. In a different study, d’Ydewalle and Rensbergen (1989) showed that reading behaviour of hearing children in Grade 2, when watching a foreign television programme with subtitles in their own language, is hardly systematic because the routine of reading subtitles on television has not yet been fully developed. In addition, Koolstra et al. (1999) registered eye movements while children in Grades 2, 4 and 6 were watching an episode of a foreign-language action series subtitled in their own language and indicated that children in Grade 2 were discouraged by the short presentation time of the subtitles to such an extent that they put little effort in trying to follow them. Conversely, children in Grades 4 and 6 showed a pattern very similar to that of adults. As with the adults, the reading time of fourth and sixth graders appeared to increase with the amount of characters in the subtitle and, in addition, they spent proportionally more reading time on two-line compared with one-line subtitles.

Several studies have been conducted to determine whether visual attention is altered by deafness. According to Bavelier and Neville (2002), Bavelier et al. (2006) and Corina and Singleton (2009), auditory loss triggers changes in the attention process and particularly in how information from multiple senses is integrated. In the case of individuals who are born profoundly deaf, there is a redistribution of attention resources towards the peripheral visual field. Put another way, while the attention of hearing people is in the centre of the visual field, people with a hearing impairment pay more attention to the peripheral visual field, which might be interpreted as a reflection of how they use their attention resources (Dye et al. 2007; Dye et al. 2008).

1. Objectives

The study has the two following objectives:

• To analyse the interpretation of a subtitled cartoon story: which elements the subjects ignore and which ones they include in their oral explanations.
• To explore the visual attention of hearing impaired children when watching subtitled cartoons by using the eye-tracker.

2. Method

2.1. Participants

The sample was made up of eleven prelingually deaf children with profound hearing impairment (seven girls and four boys) aged seven to eleven years (Mean (M) = 8.72; Standard deviation (SD) = 1.42) who were in the second to fifth grade. Thanks to the auditory device that they wore (cochlear implant or hearing aids: see table 1) they achieved a hearing enhancement of between 13 and 37 decibels (M = 21.36; SD = 6.87). The hearing devices had been worn for at least four years.

The participants in the study were deaf children of parents without hearing impairment. They attended mainstream schools in Barcelona with hearing classmates and their schooling was based on oral input. At school, they received individualised attention from a speech therapist. None of the subjects used sign language to communicate, nor did they have any associated disability.

The hearing impaired students in the study were in the second, third and fifth grade of elementary school in a classroom with hearing classmates (there are no hearing impaired students in fourth grade in any of the three schools in the sample).

Based on a brief information-gathering interview on the television habits of hearing impaired children, with a view to ascertaining the frequency with which they watch subtitled television, it was observed that most of the hearing impaired children in the sample did not watch television programmes with subtitles (five did not watch subtitled programmes at all and three only occasionally). In addition, two of the children who occasionally watched subtitled television programmes did not read the subtitles because they said they were too quick; only one of the children, a fifth grader, read the subtitles whenever he watched television.

2.2 Materials

2.2.1 Reading competence

Two tests were used to assess the participants’ reading competence: a reading comprehension test and a reading speed test. Further to this, a subtitled cartoon was used to explore their interpretation of the content by examining the information they retained after watching the cartoon and analysing their visual attention.

Reading comprehension and reading speed were assessed based on a standardised test from Spain (Canals 1989). Both tests have five levels of difficulty, each of which correspond to an elementary school grade. The reading comprehension test contains exercises which require the participants to put sentences into the correct order, carry out orders, or answer questions after reading a short text. Each participant was administered the test that corresponded to their year at school, and if they did not pass it they were administered the test from the previous year until they managed to pass the test and their reading comprehension level was determined.

The reading speed test consists of reading an excerpt from a text at the level of each participant’s year at school for one minute and counting the number of words read (wpm). It is important to note that according to the reading speed test, average students in the second grade are expected to read at 62 wpm, third graders at 85 wpm and fifth graders at 111 wpm (see Table 1).

Table 1. Characteristics of the participants.

2.2.2. Audiovisual

The script of the cartoon used in the study follows a narrative structure where there is an introduction to the characters, communication of their intentions regarding what they want to achieve, a conflict, the resolution of the conflict, a consequence and an end. The total length of the cartoon, with subtitles in Catalan, is 105 seconds and it contains 52 subtitled words. This cartoon was chosen because the speed of the subtitles was suitable for children from the second grade onwards.

The title of the cartoon film is The Pirate Treasure. The characters, pirates sailing in a boat (introduction), want to find a treasure chest which is full of toys (intention). As they sail along, they are warned that they are about to hit a rock (conflict). In order to avoid it, they have to turn the ship (resolution of the conflict), but a misunderstanding of one pirate’s orders to another makes the ship hit the rock (consequence). Luckily, everything ends up being a dream (end).

In the cartoon, some information may be deduced solely from the images, such as the introduction, the conflict and the consequence. In contrast, other content, such as the intention, the resolution of the conflict and the end of the story, needs the use of subtitles to allow clear understanding.

The information that can be deduced based on the pictures is the introduction, the conflict and the consequence of the story. The information that has to be gleaned by reading the subtitles is the characters’ intentions, the resolution of the conflict and the end.

2.3. Procedure

The information was gathered by the same researcher at the participants’ schools on a one-to-one basis.

While watching the audiovisual story of the subtitled cartoon, the participants’ eye movement was analysed using an eye-tracker, after first having ensured the calibration of the eye with the camera.

After seeing the cartoon story, the students were asked to orally retell what had happened in the story. Their answers were videotaped and transcribed, to be analysed based on the aforementioned narrative sequences. As in the study we analyse which narrative sequences they included in their answers, we did not consider their linguistic expression as an important factor in our assessment.

3. Results

The results are grouped into two sections according to the goals of the study. The first section represents the results of the subjects' interpretation of the content of the cartoon, and the second section represents the results of the analysis of their visual attention when watching the video.

Before presenting the results, it is important to recall that when hearing impaired children watch cartoons on television, they cannot lip read. That is, the meaning of the content of the cartoons has to be gleaned by reading the subtitles, listening and looking at the images.

3.1. Interpretation of the cartoon The Pirate Treasure

Based on the subjects’ retelling of the story The Pirate Treasure, we explored whether their oral explanations included any of the narrative sequences that appear in the cartoon: the introduction, the intention (which contains two pieces of information), the conflict, the resolution of the conflict, the consequence and the end.

Generally speaking, we can state that the participants in the sample understood the story told in the video The Pirate Treasure, although the oral information, which was transmitted visually (subtitles) and orally (language) at the same time, posed the most difficulties.

As seen in Figure 1, the sequences that could be deduced solely from the images (the introduction, the conflict and the consequence) were included by all the participants when retelling the story. However, the interpretation of the sequences that required supplementary reading of the subtitles because the images and/or the listening cues themselves were not explicit enough was more varied, although satisfactory on the whole. Specifically the second graders, who demonstrated the lowest levels of reading comprehension and speed, were the group of participants that encountered the most difficulties. The most difficult sequences to interpret were primarily the information contained in the second part concerning the characters’ intentions and the resolution of the conflict. The intentions require two pieces of information to be retained: (a) that the pirates were searching for a treasure chest; and (b) that the treasure chest was full of toys to share with their friends. As can be seen, the first part of the intention, which is actually the most important part, was retained more than the second. The resolution of the conflict also posed difficulties since it is a sequence in which the subjects were asked to grasp a series of misunderstandings among the characters, and therefore it was difficult to interpret.
 

Figure 1. Interpretation of the cartoon sequences based on the images or subtitles.

3.2. Visual attention when watching the subtitled cartoon The Pirate Treasure

In addition to the participants’ interpretation of the content of the cartoon, we also explored how they looked at the screen and which visual elements captured their attention the most.

The eye-tracker, which recorded the children’s eye movements as they were watching the cartoon, enabled us to see where their gaze was directed. The results indicate that the participants in the sample kept progressive attention on the frames with subtitles, except for four of the five subjects in the second grade (children 2, 3, 4 and 5). That is, while the majority of the hearing impaired students paid more attention to the frames with subtitles as the story progressed, the second graders mentioned above did not pay any attention to the subtitles from frame 3500 on, which is the sequence in which the conflict is resolved (see Figure 2).

Figure 2. Visual attention to the frames with subtitles among second graders.

 

In order to illustrate these results, we show an example of the data gathered with the eye-tracker (see Figure 3) which reveals how, in the ending sequence in the story The Pirate Treasure, there were subjects (each represented by a different-coloured circle) who did not read the subtitles and instead directed their attention to the image.

Figure 3. Visual attention of the participants in the sample measured with the eye-tracker in the final sequence of the story.

Thanks to the eye-tracker, we have seen that the hearing impaired participants pay attention to the lip movements of the characters in the cartoon, despite the fact they cannot lipread. This is possibly due to the fact that even though they cannot lipread syllables, they still read mouth movements that suggest when the characters are actually speaking. This took place in both the sequences with subtitles (M = 174.09; SD = 66.57) and in those without subtitles (M = 124.45; SD = 59.84). However, the children with a higher reading speed and a higher level of reading comprehension (children 6, 7, 9 and 11) show a clear tendency to watch the characters’ lips less than the other participants (see Figure 4).

 

Figure 4. Visual attention of the subjects on the lips in a image from the video with subtitles, as measured with the eye-tracker. Each participant represented by a different coloured circle.

4. Discussion

Although the results of this study cannot be generalized owing to the low number of participants, such results allow us to reflect on the effectiveness of subtitling for hearing impaired children.

The results reveal that hearing impaired students obtain a more complete understanding of the story when there are sequences without subtitles, thus allowing them time to view the images. It is true to say that the data shows that all the sequences that can be deduced from the images were interpreted correctly, and those that required the subtitles to be read, even though the interpretations were more uneven, were also understood by the majority of subjects. The participants in the sample, with one exception, had an acceptable reading speed to be able to read the subtitles of the cartoon in the study. However, we noticed that the second graders tended to stop directing their visual attention to the subtitles in the second half of the cartoon, thus missing the linguistic information that could not be deduced from the images.

These results lead us to wonder despite the fact that they had a high enough reading speed to read them. First, we must consider that at this age the images exert a powerful attraction and are an easy-to-use resource for gleaning the meaning of the storyline when students have not yet mastered reading and lack a sound reading comprehension level. Likewise, it should be borne in mind that the participants in the sample were not used to watching the television with subtitles, which meant that it was an unfamiliar and complex activity for them, compounded by the new visual stimulus onscreen combined simultaneously with images and oral language.

Based on the results of this study, we suggest two kinds of implications: some for subtitlers and some for families with hearing impaired children.

Subtitlers should consider the specific characteristics of hearing impaired television viewers, who are the prime audience of subtitled cartoons. If the goal is for the subtitling service to be useful for hearing impaired children, it should be borne in mind that watching television programmes with subtitles requires viewers to pay attention to several visual and auditory stimuli at the same time, and that this is a specifically difficult task for hearing impaired children who are still acquiring reading mastery. As a result, subtitles should be distributed throughout the cartoon story only when the oral information cannot be understood from the images alone, thus making it less likely for the viewers to stop reading the subtitles due to tiredness.

Likewise, parents of hearing impaired children should be aware of the importance of getting their children used to always watching television with subtitles, and getting them in the habit of reading the subtitles to determine their relevancy for understanding the content of the programmes from a very young age. Stewart and Clarke suggest several intriguing ideas for beginning to use subtitles in an educational way as the parents watch television with their hearing impaired children. They include stopping the video at any time to talk about the contents and the words and phrases written in the subtitles, explaining the meaning of the information conveyed in the subtitles, and taking advantage of the content of the programme to speak about and discuss a topic, thus encouraging language development (2003: 120–136).

Further research should be done with a more extensive sample to obtain more specific recommendations for the improvement of subtitling of cartoon programmes for hearing impaired children.

Bibliography

• Bavelier, Daphne, Dye, Matthew and Peter Hauser (2006). “Do deaf individuals see better?” Trends in Cognitive Science 10, 512–518.
• Bavalier, Daphne and Helen Neville (2002). “Cross-modal plasticity: where and how?” Nat. Review Neuroscience 3, 443–452.
• Cambra, Cristina, Silvestre, Núria and Aurora Leal (2008). “Función de la subtitulación y la interpretación de la imagen en la comprensión de los mensajes televisivos: la comprensión de una serie por parte de los adolescentes sordos.”Cultura y Educación 20(1), 81–93.
• — (2009). “Comprehension of television messages by deaf students at various stages of education.” American Annals of the Deaf 153(5), 425–434.
• Canals, Ramon (1989). Proves Psicopedagògiques d’Aprenentatges Instrumentals. Barcelona: Onda.
• Corina, David and Jenny Singleton (2009). “Developmental social cognitive neuroscience: Insights from deafness.” Child Development 80(4), 952–967.
• Dye, Matthew, Baril, Dara and Daphne Bavelier (2007). “Which aspects of visual attention are changed by deafness? The case of the attentional network test.” Neuropsychologia 45, 1801–1811.
• Dye, Matthew, Hauser, Peter and Daphne Bavelier (2008). “Visual attention in deaf children and adults.” Marc Marschark and Peter C. Hauser (ed.) (2008). Deaf cognition. Foundations and outcomes. New York: Oxford University Press, 250–263.
• d’Ydewalle, Géry and Johan Van Rensbergen (1989). “Developmental studies of text-picture interactions in the perception of animated cartoons with text.” Heinz Mandel and Joel R. Levin (eds) (1989). Knowledge acquisition from text and pictures. Amsterdam: North-Holland, 233–248.
• d’Ydewalle, Géry, Praet, Caoline, Verfaillie, Karl and Johan Rensergen (1991). “Watching subtiled television. Automatic reading behaviour.” Communication Research 18(5), 650–666.
• Jensema, Carl, Danturthi, Ramalinga and Robert Burch (2000a). “Time spent viewing captions on television programs.” American Annals of the Deaf 145(5), 464–468.
• Jensema, Carl, Sharkawy, Sameh, Danturthi, Ramalinga, Burch, Robert and David Hsu (2000b). “Eye movement patterns of captioned television viewers.” American Annals of the Deaf 145(5), 275–285.
• Koolstra, Cees, Van der Voort, Tom, and Géry d’Ydewalle (1999). “Lengthening the presentation time of subtitles on television: effects on children’s reading time and recognition.” Communications 24(4), 407–422.
• Lorch, Elizabeth, Anderson, Daniel and Stephen Levin (1979). “The relationship of visual attention to children’s comprehension of television.” Child Development 50, 722–727.
• Marschark, Marc, Bull, Rebecca, Sapere, Patricia, Nordmann, Emily, Skene, Wendy, Lukomski, Jennifer and Sarah Lumsden (2012). ”Do you see what I see? School perspectives of deaf children, hearing children and their parents.” European Journal of Special Needs Education 27(4), 483–497 http://dx.doi.org/10.1080/08856257.2012.719106 (consulted 03.09.2012).
• Pezdek, Kathy and Erik Hartmann (1983). “Children’s television viewing: attention and comprehension of auditory versus visual information.” Child Development 54, 1015–1023.
• Stewart, David A. and Bryan R. Clarke (2003). Literacy and your deaf child. Washington: Gallaudet University Press.  

Biography

Cristina Cambra is a Lecturer of the Educational Department of Psychology at the Universitat Autònoma de Barcelona. She is a member of a research group on deafness and language acquisition problems (GISTAL). Her current research deals with the comprehension of subtitled television programmes by deaf people and she has published some articles in this area. She can be reached at: cristina.cambra@uab.cat.

Aurora Leal is a Lecturer of the Educational Department of Psychology at the Universitat Autònoma de Barcelona. She is a member of a research group on deafness and language acquisition problems (GISTAL). She can be reached at: aurora.leal@uab.cat.

Núria Silvestre is a Professor of the Educational Department of Psychology at the Universitat Autònoma de Barcelona. She is the director of a research group on deafness and language acquisition problems (GISTAL). She can be reached at: nuria.silvestre@uab.cat.

Notes

Note 1:
Thisstudy has been conducted thanks to the grant awarded to the authors by the Consell Audiovisual de Catalunya (CAC).
Return to this point in the text