Home > Issue33 > Liao article

The impact of monolingual and bilingual subtitles on visual attention, cognitive load, and comprehension

Sixin Liao, Macquarie University
Jan-Louis Kruger, Macquarie University
Stephen Doherty, University of New South Wales

ABSTRACT

With the proliferation and global dissemination of audiovisual products, subtitles have been widely used as a cost-effective tool to minimise language barriers for audiences of diverse cultural and linguistic backgrounds. However, the effectiveness of subtitles is still a topic of much debate and subject to various conditions, such as the context of use, the subtitle type, and the relationship between the language of the soundtrack and that of the subtitles. Drawing on an analysis of eye movements and a self-reported questionnaire, this study compares the impact of bilingual subtitles to that of monolingual subtitles in terms of viewers’ visual attention distribution, cognitive load, and overall comprehension of video content. Twenty Chinese (L1) native speakers watched four videos with English (L2) audio, each in a different condition: with Chinese subtitles (interlingual/L1 subtitles), with English subtitles (intralingual/L2 subtitles), with both Chinese and English subtitles (bilingual subtitles), and without subtitles. Our results indicate that viewers’ visual attention allocation to L1 subtitles was more stable than to L2 subtitles and less sensitive to the increased visual competition in the bilingual condition, which, we argue, can be attributed to the language dominance of their native language. Bilingual subtitles as a combination of intralingual and interlingual subtitles did not appear to induce more cognitive load or produce more cognitive gain than monolingual subtitles. Compared with the no subtitles condition, however, we found bilingual subtitles to be more beneficial as they provided linguistic support to make the video easier to comprehend and facilitate the learning process.

Keywords

Bilingual subtitles, visual attention distribution, cognitive load, eye tracking, subtitle mode.

1. Introduction

With the proliferation and global dissemination of audiovisual materials, subtitling is now widely used as a cost-effective tool to minimise language barriers when watching videos that involve foreign languages (Doherty 2016; Kruger and Doherty 2016). Bilingual subtitles, a unique subtitle mode that presents subtitles in two different languages simultaneously, are gaining popularity around the world, especially in Mainland China (Li 2016). This is partly attributed to the belief that bilingual subtitles could deliver the benefits of both intralingual and interlingual subtitles, with intralingual subtitles providing the written forms of spoken words that can facilitate vocabulary learning and interlingual subtitles providing the meaning (translation) of words that can enhance viewers’ comprehension and absorption of the content (García 2017).

There exists however an inherent risk that subtitles, as a written form of spoken dialogue, generate redundant information that may overburden the visual processing channel and deplete people’s limited cognitive resources that could have been used to process other essential information (Zheng et al. 2016). When watching subtitled videos, viewers have to cope with a rich combination of multimodal and multiple-source information: visual images (visual-nonverbal), spoken dialogue (audio-verbal), subtitles (visual-verbal) and background sounds (audio-nonverbal) (Gottlieb 1998). This could place high demands on viewers’ attentional and cognitive resources because processing too much information simultaneously has been shown to exceed the capacity of working memory and result in cognitive overload (Kalyuga et al. 1999).

While recent years have seen a growing interest in exploring the impact of subtitles on cognitive load in order to ascertain how subtitles affect comprehension and learning outcomes (Kruger, Hefer and Matthew 2013, 2014; Kruger, Matthew and Hefer 2013; Kruger and Doherty 2016; Szarkowska and Gerber-Morón 2018), empirical and theoretical progress in this field is still limited as previous studies have been centred on monolingual subtitles. Compared with monolingual subtitles, watching videos with bilingual subtitles could be more cognitively demanding due to the concurrent presence of subtitles in two different languages, which, if the audience understands both languages, is likely to generate more redundancy and impose additional cognitive load on working memory. However, due to the scant research in this field, little is currently known about the actual visual and cognitive processing of bilingual subtitles.

In an attempt to shed some light on the issue, the current study sets out to address the question of what, if any, the impact of bilingual subtitles is on attention distribution and cognitive load as well as comprehension. This is done using a mixed-methods approach of eye tracking (online measure), self-reported measures of cognitive load (offline), and free recall test (offline)¹. Drawing on Cognitive Load Theory (CLT) (Sweller 2010; Sweller et al. 2011), we begin with an introduction to the redundancy effect, pinpointing how subtitles, as verbal redundancy, could impact on cognitive load and on the effectiveness of comprehension. Subsequently, we present the results from the above methods and discuss them in relation to the research question of this study. We then conclude with a discussion of the limitations encountered and suggestions for future research and best practice in the usage of bilingual subtitling.

2. Research background

2.1 Cognitive Load Theory and the redundancy effect

The management of cognitive resources is of critical importance to the design of multimedia, especially in the context of instructional design, as learning occurs only when the total load imposed by the learning task is within the limited cognitive capacity of the learner (Paas et al. 2004). Since bilingual subtitles have become commonplace in multimedia learning (García 2017; Lazareva and Loerts 2017), it is essential to establish a body of empirical evidence to determine the impact of bilingual subtitles, as additional and mostly redundant information, on cognitive load and learning.

The conceptualisation of cognitive load has been well-established in CLT (Sweller et al. 2011), one of the most influential theoretical frameworks accounting for cognitive processing during learning (Martin 2014). As this study approaches bilingual subtitles for the purpose of enhancing learning, we argue that CLT is a most appropriate theoretical framework for the current study since it has a well-established empirical basis in educational psychology, instructional design, and educational technology.

Three components of cognitive load have been identified in CLT literature, namely intrinsic cognitive load, extraneous cognitive load and germane cognitive load. Intrinsic cognitive load is created by dealing with the inherent complexity of the task (Van Merriënboer and Sweller 2005; Sweller 2010), while extraneous cognitive load is generated by dealing with instructional features that do not contribute to learning. Germane cognitive load, on the other hand, is created when learners are engaged in processing essential information that contributes to learning (Sweller et al. 1998; Sweller 2010).

As the human brain has finite cognitive resources in working memory, the more intrinsic and extraneous cognitive loads are produced, the fewer cognitive resources remain for the learner to perform and learn from the task at hand, with the result that diminished, if any, learning takes place. Given that intrinsic cognitive load cannot be manipulated by instructional design (Leppink et al. 2013), a vital principle in learning instruction is therefore to reduce extraneous cognitive load so as to free up more cognitive resources, for instance, by optimising the instructional design of the stimuli.

Learners have also been shown to be more likely to experience high extraneous cognitive load when they process redundant information that is unnecessary for learning (Kalyuga and Sweller 2005). More specifically, numerous empirical studies on cognitive load effects have found that presenting the same information in different forms (e.g. presenting verbal information in both written and audio forms) would hinder learning and cause the redundancy effect (Mayer et al. 2001; Diao and Sweller 2007). The redundancy effect is very relevant to subtitling in that subtitles transfer auditory information into a written form and thus could produce verbal redundancy, which is likely to induce extraneous cognitive load. However, subtitles in different linguistic formats generate different degrees of redundancy and could exert a differential impact on cognitive load and, as a consequence, on task performance and learning outcomes within educational settings.

2.2 Subtitles in different linguistic formats

Based on formal linguistic parameters, subtitles can be categorised into three types, namely intralingual subtitles, interlingual subtitles, and bilingual subtitles (Díaz Cintas and Remael 2007). Intralingual subtitles (or same-language subtitles or bimodal subtitles), which are presented in the same language as the spoken dialogue, are primarily used by deaf and hard-of-hearing viewers, but also in language learning and other educational contexts (Doherty 2016; Kruger and Doherty 2016; Doherty and Kruger 2018). Interlingual subtitles (or standard subtitles or L1 subtitles) refer to subtitles that are displayed in a language different from that of the dialogue, normally in the viewers’ native language (Raine 2012). Different from intralingual and interlingual subtitles, which consist of only one language, bilingual subtitles (also known as dual/double subtitles) present subtitles simultaneously in two different languages. This category is mostly used in multilingual countries or regions where two or more languages are spoken, such as Finland, Belgium, Israel, Singapore, Malaysia and Hong Kong (Gottlieb 2004; Kuo 2014; Corrizzato 2015). In Mainland China, for example, bilingual subtitles are gaining currency as China’s dominant TV broadcaster is stepping up its effort to present television programs with subtitles in both English and Chinese in order to attract a wider audience². The increasing usage of bilingual subtitles in online videos is attributed to the efforts of amateur subtitlers who translate foreign language videos online on a voluntary basis (Zhang 2013; Hsiao 2014).

While a number of recent empirical studies have found that intralingual or interlingual subtitles do not cause cognitive overload (Kruger, Hefer and Matthew 2013; Kruger et al. 2014; Perego et al. 2010), these findings may not apply to bilingual subtitles, which increase the amount of text presented on the screen as well as the level of redundancy, particularly for bilingual viewers, who have access to both languages in the subtitles and the language in the auditory track. As the processing of more sources of information and the existence of redundancy have been found to cause higher cognitive load, we suspect that bilingual subtitles could cause cognitive overload. Thus, our research question aims to identify the effects, if any, of bilingual subtitles on viewers’ distribution of visual attention, cognitive load, and comprehension of audiovisual stimuli.

3. Methodology

3.1 Participants

Twenty Chinese native speakers (L1) who used English as their second language (L2) were recruited as participants. As they all met the language requirements of postgraduate programmes at Macquarie University, they were considered to possess similar English language proficiency³. The average age of participants was 25.7 (ranging from 22 to 38, SD = 4.33), with a breakdown of 14 females and 6 males. Ethics approval was granted by the lead author’s institution (Macquarie University, Faculty of Human Sciences Research Ethics Sub-Committee reference number 5201700464).

3.2 Material

3.2.1 Stimulus

Four 5-minute video clips from the BBC documentary series Planet Earth (Fotherhill2006) were used as stimuli in the eye tracking experiment. The videos were selected because they were comparable in terms of the density and complexity of pictorial content, the level of correlation between visual information (image) and verbal information (narration), as well as the speech rate. To ensure that all video clips were comparable in terms of the difficulty of verbal information, we performed a readability test for the transcription of each video clip using Coh-Metrix, a computational tool that evaluates linguistic characteristics at multiple levels of language and discourse (Graesser et al. 2014). In order to answer the research question, we created four experimental conditions for each video clip:

1. English narration with Chinese subtitles (CS);
2. English narration with English subtitles (ES);
3. English narration with both Chinese and English subtitles (BS);
4. English narration without subtitles (NS).

This is a within-subject study with each participant seeing all 4 videos, each in a different condition: CS, ES, BS, and NS. The lead author produced all subtitles using Aegisub subtitling freeware (www.aegisub.org),  and these were checked for quality by a professional subtitler proficient in both languages. The display time of subtitles in the CS, ES and BS groups were the same in order to minimise the impact of other variables when investigating the effects that different subtitles have on attention distribution and cognitive load.

The bilingual subtitles were presented on two lines to avoid excessive obstruction of the image, in accordance with standard conventions (Díaz Cintas and Remael 2007; Kuo 2014), with one line in the target language (Chinese) and the other one in the same language as the original speech (English). To keep the upper line shorter in order to limit the obstruction of other visual information, Chinese subtitles were displayed above English subtitles because they occupy less space than English subtitles due to the different writing systems. Subtitles were displayed in the standard position, i.e. at the bottom centre of the screen (Díaz Cintas and Remael 2007), as illustrated in Figure 1:

Figure 1. Screenshots of four video conditions

In terms of presentation rate, we adhered to standard conventions (ibid.) by presenting English subtitles at a rate of 10 to 14 characters per second (CPS) which produced a near verbatim transcript of the spoken text. The spotting preserved the semantic units and each English subtitle contained no more than 55 characters⁴. The Chinese subtitles were a literal translation in which the lexical, semantic, and syntactic information of the original English version were reproduced. Each Chinese subtitle contained no more than 20 Chinese characters in accordance with standard conventions (Kuo 2014).

3.2.2 Data collection instruments

3.2.2.1 Biographical questionnaire

As a pre-task measure, a short questionnaire was used to obtain biographical information about the participants, including age, academic major, and English language proficiency (IELTS scores: M = 6.74, SD = 0.59).

3.2.2.2 Cognitive load questionnaire

As all videos are comparable in terms of image and verbal complexity, there should be no difference in the inherent complexity of videos and therefore no difference in intrinsic cognitive load across the various conditions. However, we expected to find a significant difference in extraneous load between the conditions for different linguistic formats of subtitles representing different instructional features, which are likely to exert differential impact on extraneous cognitive load.

In order to measure the three types of cognitive load, as described in 2.1, we administered a post-task self-report cognitive load questionnaire developed by Leppink et al. (2014). This instrument was selected because it has been validated and is the first one to differentiate between different types of load. As this study was based on a context of film comprehension, which was different from the problem-solving context in which the study by Leppink et al. (2014) was situated, we adjusted the contextual information in the instrument to reflect the content of our stimuli.

The cognitive load questionnaire contains twelve items with a 0–10 rating scale. Intrinsic cognitive load (IL) was measured with three items that were related to the complexity of the video (“The information covered in this video was very complex”) and one item concerning the effort invested to cope with the complexity (“I invested a very high mental effort in the complexity of this video”). Extraneous cognitive load (EL) was evaluated with three items that were related to the presentation design (“The presentation of information in this video was very unclear”) and one item concerning the effort invested to deal with the presentation design (“I invested a very high mental effort in unclear and ineffective presentation of information in this video”). Germane cognitive load (GL) was evaluated with three items referring to the contribution of the video to information acquisition (“This video really enhanced my understanding of the information that was presented”) and one item related to the effort invested in information acquisition (“I invested a very high mental effort during this video in enhancing my knowledge and understanding”).

We performed an analysis of internal consistency using Cronbach’s alpha coefficients as an indication of reliability (see Table 1). Cronbach’s alpha coefficients of IL and EL across four conditions were high, revealing a high level of reliability of the items used to measure these two types of cognitive load⁵. However, Cronbach’s alpha coefficients were low for GL in three conditions. In order to increase the internal consistency of the GL component, we removed the last item (item 12), which was related to the mental effort in information acquisition⁶. The cognitive load that was evaluated by the remaining items of GL was referred to as GL*.

Table 1. Cronbach’s Alpha Coefficients for intrinsic cognitive load (IL), extraneous cognitive load (EL) and germane cognitive load (GL*) in four conditions

3.2.2.3 Free recall test

We constructed a post-task recall test that asked participants to write down on computer as much information as they could remember from the video they had just watched. There was no time limit imposed on the recall test as time limits have been shown to result in decay (Chen and Cowan 2005).

3.3 Apparatus

We presented all stimuli using an SMI RED eye tracker with a sampling rate of 250 Hz. The screen resolution of the eye tracker’s monitor was 1920 × 1080 pixels and the stimulus covered the entire 23-inch screen. We used SMI iViewX and Experiment Centre 3.0 to design, record, and post-process the eye movement data.

3.4 Procedure

We conducted all data collection sessions individually, due to the need to record eye movements one participant at a time. After an introduction to the study, each participant signed a participation consent form in line with institutional ethics requirements. In order to be able to randomise the texts and the conditions and ensure that no participant would see the same text more than once or be exposed to any condition more than once, the video clips, their treatments, and the order in which viewers watched videos were counterbalanced using a Latin Square design. We therefore randomly assigned participants to one of four groups, each group seeing 4 videos in 4 conditions (Table 2):

Table 2 . Experiment design

We then seated the participants on a stable chair 700 mm from the stimulus screen in a sufficiently illuminated room and invited them to complete the biographical questionnaire on the computer. A thirteen-point calibration was performed on each participant prior to the commencement of the first video. After each video, the participant completed the cognitive load questionnaire and recall test before watching the next video. This sequence was repeated until all four videos and the respective cognitive load questionnaire and recall test were completed.

3.5 Data processing

3.5.1 Eye movement data

We discarded eye tracking data with a tracking ratio of lower than 85% due to the unreliability of the data (Orero et al. 2018), as well as data recorded with technical problems during video playing. This yielded valid eye movement data from seventeen participants in the NS, CS, and ES conditions and sixteen in the BS condition. We manually marked the subtitles and the whole screen in the ES, CS and BS conditions as different areas of interest (AOIs). In the BS condition, we marked subtitles in the two different languages as two separate AOIs. The AOI of Chinese subtitles in BS were marked as “CS_B” and the English subtitles as “ES_B”, as shown in Figure 2 and Figure 3:

Figure 2. Screenshot of AOIs for bilingual subtitles

Figure 3. Screenshot of AOI of the screen

Based on the eye-mind hypothesis (Just and Carpenter 1980) that there is a strong link between what the eye fixates and what the mind attends to, this study used dwell time percentage of visible time (DT%) as a measure of visual attention allocation, and mean fixation duration (MFD) as an indirect indication of extraneous cognitive load (Zu et al. 2017). Dwell time was calculated as the sum of all fixations and saccades within an AOI (SensoMotoric Instruments 2011), starting with the first fixation in an AOI. Visible time was the display time of an AOI. For instance, if a subtitle AOI was displayed for 6 seconds, the visible time was 6000 ms, and if a viewer looked at the subtitle for 3 seconds, the dwell time percentage of visible time was 3000 ms, which is 50%. As this measure is a percentage, it is proportional. We calculated visual attention to the rest of the screen (DT% on the visual image) by dividing the dwell time on the screen (with the dwell time of subtitles subtracted) by the visible time of the video and multiplying that by 100 for a percentage. The event detection parameter for a fixation was a minimum duration of 80 ms within an area of 100 pixels.

3.5.2 Free recall test

We analysed each response to the recall test as a set of idea units (Riley and Lee 1996), in which each idea unit contained one major idea. We awarded one point for each idea unit that corresponded to the image, subtitles, or spoken dialogue of the stimulus. We did not penalise grammatical mistakes and minor misspellings. 0.5 point was awarded if specific names were not given correctly in the recall test. One participant’s recall test data in the ES and one in the BS condition were discarded because of technical problems in video playing. In the final analysis, we included 17 recall tests in the NS and CS conditions and 16 in the ES and BS conditions. We tasked two independent researchers to score the recall test separately after first scoring a sample test, discussing discrepancies and reaching an acceptable agreement on the scoring criteria. We then used the average of the two researchers’ scores. The scale ranged from 1 to 24.5.   

3.5.3 Cognitive load questionnaire

Data of self-reported cognitive load from two participants were discarded due to technical problems in video playing. This resulted in 18 sets of valid data on cognitive load in total.  

3.5.4 Statistical analyses

Statistical analyses were carried out using IBM SPSS Statistics, version 22. Video condition (CS, ES, BS and NS) was treated as an independent variable. Dependent variables included dwell time percentage of visible time in subtitles (DT% in subtitles), dwell time percentage of visible time on the visual image (DT% in the visual image), mean fixation duration in subtitles (MFD), self-reported cognitive load and comprehension performance (scores of free recall test). A significance level of 0.05 was adopted for all statistical analyses.

Since eye-tracking data often violate the normal distribution requirement of inferential statistical tests like the ANOVA or t-tests, data that were not normally distributed were subject to non-parametric tests.

4. Results

4.1 Allocation of visual attention

4.1.1 DT% in subtitles

We conducted a one-way repeated measures ANOVA and found significant differences in the DT% in subtitles between three different subtitling conditions (CS, ES and BS), F (2, 30) = 3.944, p = .030, partial η2 = .208. A post hoc analysis with a Bonferroni adjustment for multiple comparisons (Table 3 and Figure 4) revealed that there was significance in the difference between BS and CS (p = .034).

A paired samples t-test found a significant difference between ES and ES_B (t (15) = 2.815, p = 0.013), but this was not the case between CS and CS_B (t (15) = 0.772, p = 0.452). In other words, participants appeared to spend much less time looking at L2 subtitles in the bilingual condition than in the monolingual condition, whereas they spent a similar amount of time reading L1 subtitles in both conditions.

Note. CS = Chinese monolingual subtitles. CS_B = Chinese subtitles in the bilingual condition. ES = English monolingual subtitles. ES_B = English subtitles in the bilingual condition.BS = bilingual subtitles.
Table 3. Descriptive statistics of DT% in subtitles in monolingual and bilingual conditions

Figure 4. DT% in subtitles in monolingual and bilingual conditions

We did not find a significant difference in the DT% between CS_B and ES_B (t (15) = 0.539, p = 0.598), which indicates that participants spent similar amount of time on the two different subtitles when watching videos with bilingual subtitles. However, on closer inspection of the individual data, nearly half of participants had higher DT% in English subtitles while half of them had higher DT% in Chinese subtitles (Figure 5), which suggests that viewers chose subtitles in one language as a dominant source of visual-verbal information.

Figure 5. Comparison of DT% in L1 and L2 subtitles in the bilingual condition. Negative values = higher DT% in ES. Positive values = higher DT% in CS

4.1.2 DT% in the visual image

We conducted a one-way repeated measures ANOVA on DT% in the visual image and identified a significant difference between conditions, F (3, 45) = 8.382, p < .0005, partial η2 = .358. We used a post hoc analysis with a Bonferroni adjustment for multiple comparisons which revealed that there was a significant difference between NS and BS (p = .028), NS and CS (p = .026), and NS and ES (p = .017). However, no significant differences were found between subtitling conditions: BS and CS (p = 1.000), BS and ES (p = 1.000), ES and CS (p = .547) as shown in Table 4 and Figure 6:

Table 4. Descriptive statistics of DT% in the visual image in four video conditions

Figure 6. DT% in the visual image across four video conditions

4.1.3 Mean fixation duration in the subtitled area

We carried out a one-way repeated measures ANOVA which showed that mean fixation duration (MFD) was not significantly different between conditions, F (3, 45) = 1.289, p = .290, as illustrated in Table 5 and Figure 7:

Table 5. Descriptive statistics on mean fixation duration (ms) in subtitles in monolingual and bilingual conditions

Figure 7 . Mean fixation duration in subtitles in different subtitled conditions. Three outliers were kept in the analysis as an inspection of their values did not reveal them to be extreme.

4.2 Self-reported Cognitive Load

4.2.1 Intrinsic cognitive load (IL)

We conducted a one-way repeated measures ANOVA on IL, which showed that there were significant differences between the four experimental conditions, F (3, 51) = 5.321, p = .003, partial η2 = .238. Our post hoc analysis with a Bonferroni adjustment for multiple comparisons (Table 6 and Figure 8) revealed that there was a significant difference between NS and ES (p = .035), NS and BS (p = .039). IL was highest in NS and lowest in BS.

4.2.2 Extraneous cognitive load (EL)

We then carried out a one-way repeated measures ANOVA on EL and found significant differences in EL between different conditions, F (3, 51) = 5.103, p = .004, partial η2 = .231. Post hoc analysis with a Bonferroni adjustment for multiple comparisons revealed that there were significant differences between NS and CS (p = .040), as well as NS and BS (p = .011). Similar to the trend in IL, EL was highest in NS and lowest in BS (Table 6 and Figure 8).

4.2.3 Germane cognitive load (GL*)

We also conducted a one-way repeated measures ANOVA on GL*, which showed that there were significant differences between different conditions, F (2.198, 37.373) = 8.424, p = .001, partial η2 = .331. Our post hoc analysis with a Bonferroni adjustment for multiple comparisons showed that there was a significant difference between NS and CS (p = .020), NS and ES (p = .017), and NS and BS (p = .009). GL* in the CS condition was the highest and lowest in the NS condition (Table 6 and Figure 8).

4.2.4 Mental effort in information acquisition (ME)

As discussed in section 3.2.2, we performed a separate analysis of the mental effort item of the GL* component of the cognitive load instrument. We therefore performed a Friedman test on the data for the item which showed that mental effort in the subtitled area was significantly different in different conditions, χ2 (3) = 9.245, p = .026, W = .171. We performed pairwise comparisons with Bonferroni correction for multiple comparisons. There was a marginally significant difference between NS and CS (p = .049).

Note. N = 18.
Table 6. Means (SD) of cognitive load and mental effort in different conditions

Figure 8. Self-reported value of cognitive load and mental effort in four video conditions. Outliers were kept in the analysis as an inspection of their values did not reveal them to be extreme.

4.3 Free recall of stimuli

We performed a one-way repeated measures ANOVA, which showed no significant difference in the recall scores between different conditions, F (3, 42) = 1.447, p = .243, as shown in Table 7 and Figure 9:

Table 7. Means of recall test scores in different conditions

Figure 9. Scores of the free recall test in different video conditions

5. Discussion

5.1 The impact of subtitle mode on attention allocation

We aimed to identify the impact, if any, that bilingual subtitles have on the distribution of visual attention, cognitive load, and comprehension on a sample of Chinese native speakers who used English as their second language. As processing too much information and the existence of redundancy have been found to cause higher extraneous cognitive load (DeLeeuw and Mayer 2008), we suspected that bilingual subtitles, which increase the amount of text presented on the screen as well as the level of redundancy, are likely to cause higher extraneous cognitive load and exert a negative impact on comprehension performance.

Our results from eye movement analyses revealed a significant effect of subtitle mode on the distribution of visual attention on subtitles, namely participants spent more time looking at bilingual subtitles than at Chinese subtitles. It was thought that this might be attributed to bilingual subtitles containing two lines of subtitles, whereas there were only one-line subtitles in the monolingual condition. However, this assumption did not hold true as there was no significant difference between the bilingual subtitle and English subtitle conditions in this respect. As the language of the soundtrack remained unchanged (always in L2), we believe that it is the addition of subtitles in a non-native language that results in more attention to the subtitled area. This also provides support to previous arguments that “the number of lines do[es] not play as big a role in the processing of subtitles as previously thought” (Kruger and Steyn 2014: 105). However, it is worth noting that adding subtitles in a non-native language may cause a different interaction between the language of the subtitles and the language of the soundtrack, which could consequently impact on the attention allocated to subtitles, as the viewer may automatically try to read along with the narration, in what we could call the karaoke effect. This assumption is being investigated in our other studies.

Furthermore, we found that participants’ distribution of visual attention to L1/L2 subtitles was not equally sensitive to the visual competition introduced by the presence of the subtitles in another language. The presence of both L1 and L2 in the bilingual subtitles did not appear to significantly alter the visual attention to L1 subtitles as they received the same amount of attention as in the monolingual condition, but it did result in a significant reduction of attention to L2 subtitles. We therefore suggest that participants’ visual attention to L2 subtitles is more sensitive to the increased visual competition in the bilingual condition. There are several possible explanations for that. One is that participants’ reliance on L1 subtitles is more stable than that on L2 subtitles due to the language dominance of their native language (Heredia and Altarriba 2001; Cieślicka et al. 2014). Arguably, in the face of time constraints, participants were inclined to acquire information in their native language, which is easier to process. It is also possible that in the bilingual condition, with two lines of subtitles presented in different languages, subtitles presented on the first line (i.e. L1 subtitles) can grasp viewers’ attention more easily and viewers may feel less motivated to read L2 subtitles once they have gained sufficient information from L1 subtitles. To sanction this assumption, more empirical research is needed to investigate the impact that subtitle positioning in bilingual subtitles has on the distribution of visual attention. Another possibility is that L2 subtitles render more redundancy than L1 subtitles when L2 audio information is available and therefore are less attended to by participants. This would suggest that viewers of subtitles may have the ability to filter more redundant information even though they are unable to completely avoid redundant information in order to save cognitive resources for higher order processing and deeper elaboration of the messages (Reese 1984; Liu et al. 2011).

When we presented both L1 and L2 subtitles in the bilingual condition, our participants did not allocate an equal amount of visual attention to two different subtitles nor did they completely ignore subtitles in one language due to their redundancy. Instead, they chose one language as a main source of visual-verbal information with the second language providing supporting information. Our results indicate that viewers are able to adjust their viewing pattern and choose the less cognitively demanding way to understand the video because paying equal attention to two sets of subtitles would mean that viewers have to shift back and forth between two subtitles, which could consume extra cognitive resources and hinder information acquisition. This reasoning can be explained by the early-selection theories of attention proposed by Broadbent (1958) and Treisman (1968), which posit that stimuli will be filtered at an early stage in order not to overload the limited processing capacity of the human cognitive system. Moreover, the fact that viewers spent time reading subtitles in both languages in spite of their redundancy provides evidence for the automatic subtitle reading behaviour hypothesis as originally proposed by d’Ydewalle et al. (1991).

The finding that a viewer spent an approximately similar amount of time on image regardless of the subtitling mode implies that their reliance on image appears to be more stable than their reliance on subtitles. We propose that this is possibly because there is less redundancy between nonverbal information (image) and verbal information (narration/subtitles) than between visual-verbal information (subtitles) and audio-verbal information (narration), and therefore viewers would rather spend more time on the less redundant information, i.e. image, in order to maximise information acquisition. This again corroborates the view that viewers are able to filter out information that has a higher degree of redundancy.

5.2 The impact of subtitle mode on cognitive load

5.2.1 Self-reported measures

As we found significant differences in three types of cognitive load between the NS and BS conditions, with BS reporting significantly lower scores in IL and EL and higher scores in GL*, we propose that adding bilingual subtitles that contain both L1 and L2 subtitles makes the video easier to understand and allows for more available cognitive resources than not providing viewers with any written text as linguistic support. This finding also supports the growing body of evidence that processing subtitles is cognitively effective and does not cause cognitive overload if optimised spatio-temporally (Kruger, Hefer and Matthew 2013; Lång 2016; Perego et al. 2010).

In contrast to Diao and Sweller’s (2007) findings, we did not find a significant increase in extraneous cognitive load in the presence of redundancy between audio and visual-verbal information in any of the subtitled conditions. It is worth noting, however, that their study compared text only to text with audio, whereas the present study does not have a text only condition due to our focus on video-based stimuli.

As we found no significant differences in cognitive load or mental effort between the bilingual and monolingual conditions, we report insufficient evidence for the argument that bilingual subtitles provide viewers with a cognitive processing advantage by combining the benefits of intralingual and interlingual subtitles. Bilingual subtitles did not induce more cognitive load on viewers as a result of containing more redundant information than monolingual subtitles.

5.2.2 Eye tracking measures

As no significant difference in mean fixation duration was found between L1 monolingual and L2 monolingual subtitles, or between L1 and L2 subtitles in the bilingual condition, we contend that processing L1 and L2 subtitles is equally cognitively demanding regardless of whether the two subtitles are presented separately or simultaneously. We argue that this is because viewers chose one language of subtitles as a major channel for visual-verbal information. As such, they engage in processing the textual information in a similar fashion to monolingual, as they are proficient in both languages.

It is also noteworthy to point out that as our participants spent significantly less time looking at L2 subtitles in the bilingual condition than in the monolingual condition (see results of DT% in subtitles, Table 3 and Figure 4), no difference was observed for the mean fixation duration in L2 subtitles between the two conditions. In other words, the reduction of time participants spent looking at subtitles did not affect the depth of processing of the subtitles (Kruger et al. 2018). This further points to the necessity of making a distinction between the allocation of overt visual attention to and actual reading of subtitles (Kruger and Steyn 2014).

5.3 The impact of subtitle mode on content comprehension

The free recall scores did not differ significantly across the four different conditions, which implies that our participants could remember and comprehend the video equally well regardless of the presence and linguistic formats of subtitles, although the lowest comprehension rate in the NS condition could suggest that subtitles benefit comprehension, which is consistent with a number of studies (Chung 1999; Markham et al. 2001; Hayati and Mohmedi 2011; Wang 2014; Hosogoshi 2016). However, the lack of significance in our results does not support previous studies.

Although bilingual subtitles do not seem to produce more cognitive benefits per se, we argue that the lack of significant difference in both subjective measures (self-reported questionnaire) and recall measures between the bilingual and monolingual conditions at least dispels the concern that bilingual subtitles generate more redundancy and, as a result, more cognitive overload and diminished comprehension.

5.4 Audiovisual redundancy in subtitling research

In contrast to some previous subtitling studies that used videos with an unknown language either in the subtitles or in the soundtrack (Perego et al. 2010; Bisson et al. 2014), our study provided viewers with access to both visual-verbal (subtitles in L1, L2 or L1 and L2) and audio-verbal (narration in L2) channels, which means that participants were exposed to either two or three sources of redundant verbal information at a time. This study presents an attempt to extend research on the redundancy effect from a L1 context to a L2 context, providing some interesting insights into the influence of redundancy on visual processing and cognitive load when watching subtitled videos. First, the absence of significant difference in comprehension performance between the no subtitle condition and subtitled conditions suggests that the presence of subtitles as visual-verbal redundancy does not give viewers a significant advantage in video comprehension. However, our eye movement data revealed that participants spent more than 20% of the time reading subtitles in monolingual conditions (21.55% in the L1 monolingual condition and 32.15% in the L2 monolingual condition) and 33.62% in the bilingual condition. While the two different subtitles in the bilingual condition were redundant to each other, viewers still spent time reading both subtitles. It appears that it is the presence rather than the usefulness of visual-verbal redundant information that plays a bigger role in attracting visual attention. This view is in line with previous studies which found that subtitle reading was an automatic behaviour (d’Ydewalle et al. 1991; Bisson et al. 2014). The fact that the time spent reading the subtitles is significantly shorter than the average time reported by other studies like Perego et al.’s (2010), who reported an average dwell time of 67%, could also be an indication of the fact that the subtitles (both monolingual and bilingual) performed a supplementary function rather than a primary function as you would expect in the case of monolingual viewers watching film with a foreign (rather than second) language soundtrack.  

If viewers cannot avoid redundant information, how they allocate their attentional and cognitive resources among multiple information sources would be of great importance for comprehension, due to the limited processing capacity of working memory. Consistent with the findings of the study by Liu et al. (2011), we found that participants had the ability to filter out information with a higher degree of redundancy using selective attention according to their dynamic needs. Therefore, a question that is worth further investigation is whether or not viewers’ selective attention strategy is a function of their prior knowledge, motivation and learning practice. Research on multisensory processing and integration would provide much insights in this regard (Koelewijn et al. 2010; Talsma et al. 2010; Van der Burg et al. 2010; Van der Burg et al. 2011; Morís Fernández et al. 2015; Quak et al. 2015). This also points to the need for interdisciplinary collaboration in audiovisual research and highlights the potential benefits that subtitling research could gain from other disciplines such as cognitive psychology.

Interestingly, findings of the current study do not support previous claims that processing redundant information causes higher extraneous cognitive load. In contrast, the BS condition, which presumably features more redundancy, reported lower intrinsic and extraneous cognitive load than NS which contains the least amount of redundant information. There could be two reasons for that. First, the redundancy effect is originally based on native language contexts whereas the current study is based on a second language context. Presenting redundant information in viewers’ native language may have a different impact on cognitive load than presenting redundant information in viewers’ second language. This study could be replicated by including video conditions that contain L1 spoken dialogue with L1 and L2 monolingual subtitles to examine if there exists any difference in subtitle processing. Second, the videos used in the current study are less image intensive than the animation used in other studies that explored the redundancy effect. As a result, viewers in the current study may have had more available cognitive resources for the processing of redundant verbal information.

6. Conclusion

This article presents an empirical study as part of the growing body of research that explores the impact of subtitles on cognitive processing and video comprehension. In particular, it contributes to our understanding of the impact of bilingual subtitles on attention allocation and cognitive load, which has not been investigated before.

It was found that bilingual subtitles, as a combination of intralingual and interlingual subtitles, affected viewers’ attention distribution to subtitles in a way different from intralingual and interlingual subtitles. Results showed that viewers’ visual attention to L1 subtitles was more stable than to L2 subtitles and was less sensitive to the increased visual competition in the bilingual condition. This study also dismisses the concern that bilingual subtitles result in cognitive overload and impede comprehension as a result of increased redundancy.

Furthermore, this study enriches our understanding of the redundancy effect by exploring the processing of redundant information in a foreign language context. Findings of the current study also indicate that the presence of redundant information does not necessarily result in an increase in cognitive load and a reduction in learning, as suggested in previous research. The effects of redundant information on comprehension are, to some extent, dependent on viewers’ ability to evaluate the momentary value of different layers of redundancy, and actively select and integrate different sources of redundancy based on their individual and dynamic needs to achieve their learning goal.

However, there are a number of limitations that should be taken into account when replicating the current study in further research. First, although the sample size is in line with most other eye tracking studies in audiovisual translation, a larger sample size could produce more conclusive findings. Second, as viewers were asked to complete the free recall test in their second language, their English writing skills may have interfered with their memory and comprehension. A meta-analysis study conducted by Montero Pérez et al. (2013) also found that the test type used to measure the effectiveness of subtitles had a significant impact on the usefulness of subtitles for listening comprehension. The effectiveness of subtitles on comprehension may be reduced when productive tests (e.g. recall protocol) interfere with other language skills, for example, writing skills. Therefore, a combination of both receptive (e.g. multiple choice questions) and productive (e.g. recalling) tests is therefore advisable for further studies. Future studies could also draw on the Reading Index for Dynamic Text (RIDT) developed by Kruger and Steyn (2014) to provide a more comprehensive picture of viewers’ visual processing of subtitled audiovisual content.

References

• Bisson, Marie Josée, Walter J. B. Van Heuven, Kathy Conklin and Richard J. Tunney (2014). “Processing of native and foreign language subtitles in films: An eye tracking study.” Applied Psycholinguistics 35, 399–418.
• Broadbent, Donald Eric (1958). Perception and Communication. London: Pergamon.
• Carlisle, Joanne F. (1999). “Free recall as a test of reading comprehension for students with learning disabilities.” Learning Disability Quarterly 22(1), 12-22.
• Cieślicka, Anna B., Roberto R. Heredia and Marc Olivares (2014). “It’s all in the eyes: How language dominance, salience, and context affect eye movements during idiomatic language processing.” Mirosław Pawlak and Larissa Aronin (eds). Essential Topics in Applied Linguistics and Multilingualism, Second Language Learning and Teaching. Switzerland: Springer International Publishing, 21-41.
• Cheng, Zhijian and Nelson Cowan (2005). “Chunk limits and length limits in immediate recall: A reconciliation.” Journal of Experimental Psychology: Learning Memory and Cognition 31(6), 1235-1249.
• Chung, Jingmei (1999). “The effects of using video texts supported with advanced organizers and captions on Chinese college students’ listening comprehension: An empirical study.” Foreign Language Annals 32(3), 295-308.
• Corrizzato, Sara (2015). Spike Lee’s Bamboozled: A Contrastive Analysis of Compliments and Insults from English into Italian. Newcastle: Cambridge Scholars Publishing.
• d’Ydewalle, Géry,  Caroline Praet, Karl Verfaillie and Johan Van Rensbergen (1991). “Watching subtitled television: Automatic reading behavior.” Communication Research 18, 650–66.
• DeLeeuw, Krista E. and Richard E. Mayer (2008). “A comparison of three measures of cognitive load: Evidence for separable measures of intrinsic, extraneous, and germane load.” Journal of Educational Psychology 100(1), 223-234. 
• DeVellis, Robert F. (2003). Scale Development: Theory and Applications. 2nd ed. Thousand Oaks, CA: Sage Publications.
• Diao, Yali and John Sweller (2007). “Redundancy in foreign language reading comprehension instruction: Concurrent written and spoken presentations.” Learning and Instruction 17, 78–88.
• Díaz Cintas, Jorge and Aline Remael (2007). Audiovisual Translation: Subtitling. Manchester: St Jerome.
• Doherty, Stephen (2016). “The impact of translation technologies on the process and products of translation.” International Journal of Communication 10(23), 947-969.
• Doherty, Stephen and Jan-Louis Kruger (2018). “The development of eye tracking in empirical research on subtitling and captioning.” Tessa Dwyer, Claire Perkins, Sean Redmond and Jodi Sita (eds). Seeing into Screens. London: Bloomsbury, 46-64.
• Fothergill, Alastair (2006). “Planet Earth.” https://www.bbc.co.uk/programmes/b006mywy/episodes/guide (consulted 13.12.18).
• García, Boni (2017). “Bilingual subtitles for second-language acquisition and application to engineering education as learning pills.” Computer Applications in Engineering Education 1, 1-12.
• Gottlieb, Henrik (1998). “Subtitling.” Mona Baker (ed.). Routledge Encyclopedia of Translation Studies. London: Routledge, 244-248.
• Gottlieb, Henrik (2004). “Subtitles and international anglification.” Nordic Journal of English Studies 3(1), 219–230.
• Graesser, Arthur C., Danielle S. McNamara, Zhiqang Cai, Mark Conley, Haiying Li and James Pennebaker (2014). “Coh-Metrix measures text characteristics at multiple levels of language and discourse.” The Elementary School Journal 115(2), 210-228.
• Hayati, Abdolmajid and Firooz Mohmedi (2011). “The effect of films with and without subtitles on listening comprehension of EFL learners.” British Journal of Educational Technology 42(1), 181–192.
• Heredia, Roberto R. and Jeanette Altarriba (2001). “Bilingual language mixing: Why do bilinguals code-switch?” Current Directions in Psychological Science 10(5), 164-168.
• Hosogoshi, Kyoko (2016). “Effects of captions and subtitles on the listening process: Insights from EFL learners’ listening strategies.” JALTCALL Journal 12(3), 153-173.
• Hsiao, Chi-Hua (2014). “The moralities of intellectual property: Subtitle groups as cultural brokers in China.” The Asia Pacific Journal of Anthropology 15(3), 218–241.
• Ivarsson, Jan and Mary Carroll (1998). Subtitling. Simrishamn: TransEdit HB.
• Just, Marcel Adam and Partricia A. Carpenter (1980). “A theory of reading: From eye fixations to comprehension.” Psychological Review 87(4), 329-355.
• Kalyuga, Slava, Paul Chandler and John Sweller (1999). “Managing split-attention and redundancy in multimedia instruction.” Applied Cognitive Psychology 13, 351-371.
• Kalyuga, Slava and John Sweller (2005). “The redundancy principle in multimedia learning.” Richard E. Mayer (ed.). The Cambridge Handbook of Multimediaandbook of Multimedia Learning. Cambridge: Cambridge University Press, 247–262.
• Kline, Rex B. (2005). Principles and Practice of Structural Equation Modeling. 2nd ed. New York: Guildford.
• Koelewijn, Thomas, Adelbert Bronkhorst and Jan Theeuwes (2010). “Attention and the multiple stages of multisensory integration: A review of audiovisual studies.” Acta Psychologica 134(3), 372-384.
• Kruger, Jan-Louis, Esté Hefer and Gordon Matthew (2013). “Measuring the impact of subtitles on cognitive load.” Proceedings of the 2013 Conference on Eye Tracking South Africa - ETSA ’13, 62–66.
• Kruger, Jan-Louis, Esté Hefer and Gordon Matthew (2014). “Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2.” Journal of Eye Movement Research 7(5), 1–15.
• Kruger, Jan-Louis and Faans Steyn (2014). “Subtitles and eye tracking: Reading and performance.” Reading Research Quarterly 49(1), 105-120.
• Kruger, Jan-Louis, Gordon Matthew and Esté Hefer  (2013). “Measuring the impact of subtitles on cognitive load : Eye tracking and dynamic audiovisual texts.” Proceedings of the 2013 Conference on Eye Tracking South Africa - ETSA ’13, 29–31.
• Kruger, Jan-Louis and Stephen Doherty (2016). “Measuring cognitive load in the presence of educational video: Towards a multimodal methodology.” Australasian Journal of Educational Technology 32(6), 19-31.
• Kruger, Jan-Louis, Stephen Doherty, Wendy Fox and Peter de Lissa (2018). “Multimodal measurement of cognitive load during subtitle processing: Same-language subtitles for froeign-language viewers.” Isabel Lacruz and Riitta Jääskeläinen (eds). Innovation and Expansion in Translation Process Research. Amsterdam: John Benjamins, 267-294.
• Kuo, Szu-Yu (2014). Quality in Subtitling: Theory and Professional Reality. PhD thesis. London: Imperial College London.
• Lång, Juha (2016). “Subtitles vs. narration: The acquisition of information from visual-verbal and audio-verbal channels when watching a television documentary.” Silvia Hansen-Schirra and Sambor Grucza (eds). Eye Tracking and Applied Linguistics. Berlin: Language Science Press, 59-82.
• Lazareva, Elena and Hanneke Loerts (2017). “Double subtitles as an effective tool for vocabulary learning.” Konin Language Studies 5(1), 61-92.
• Leppink, Jimmie, Fred Paas, Gees P. M. Van der Vleuten, Tamara Van Gog and Jeroen J. G. Van Merriënboer (2013). “Development of an instrument for measuring different types of cognitive load.” Behavior Research Methods 45(4), 1058–1072.
• Leppink, Jimmie, Fred Paas, Tamara van Gog, Cees P.M. van der Vleuten and Jeroen J. G. Van Merriënboer (2014). “Effects of pairs of problems and examples on task performance and different types of cognitive load.” Learning and Instruction 30, 32–42.
• Li, Mingyue (2016). An Investigation into the Differential Effects of Subtitles (First Language, Second Language, and Bilingual) on Second Language Vocabulary Acquisition. PhD thesis. Edinburgh: University of Edinburgh.
• Liu, Han-Chin, Meng-Lung Lai and Hsueh-Hua Chuang (2011). “Using eye-tracking technology to investigate the redundant effect of multimedia web pages on viewers’ cognitive processes.” Computers in Human Behavior 27(6), 2410–2417.
• Markham, Paul L., Lizette A. Peter and Teresa J. Mccarthy (2001). “The effects of native language vs. target language captions on foreign language students’ dvd video comprehension.” Foreign Language Annals 34(5), 439–445.
• Martin, Stewart (2014). “Measuring cognitive load and cognition: Metrics for technology-enhanced learning.” Educational Research and Evaluation 20(7–8), 592–621.
• Mayer, Richard E., Julie Heiser and Steve Lonn (2001). “Cognitive constraints on multimedia learning: When presenting more material reults in less understanding.” Journal of Educational Psychology 93(1), 187–198.
• Montero Pérez, Maribel, Wim Van Den Noortgate and Piet Desmet (2013). “Captioned video for L2 listening and vocabulary learning: A meta-analysis.” System 41(3), 720–739.
• Morís Fernández, Luis, Maya Visser, Noelia Ventura-Campos, César Ávila and Salvador Soto-Faraco (2015). “Top-down attention regulates the neural expression of audiovisual integration.” NeuroImage 119, 272–285.
• Orero, Pilar, Stephen Doherty, Jan-Louis Kruger, Anna Matamala, Jan Pedersen, Elisa Perego, Pablo Romero-Fresco, Sara Rovira-Esteva, Olga Soler-Vilageliu and Agnieszka Szarkowska (2018). “Conducting experimental research in audiovisual translation (AVT): A position paper.” The Journal of Specialised Translation 30, 105-126.
• Paas, Fried, Alexander Renkl and John Sweller (2004). “Cognitive load theory: Instructional implications of the interaction between information structures and cognitive architecture.” Instructional Science 32, 1–8.
• Perego, Elisa, Fabio Del Missier, Marco Porta and Mauro Mosconi (2010). “The cognitive effectiveness of subtitle processing.” Media Psychology 13(3), 243–272.
• Quak, Michel, Raquel Elea London and Durk Talsma (2015). “A multisensory perspective of working memory.” Frontiers in Human Neuroscience 9, 1–11.
• Raine, Paul (2012). Incidental Learning of Vocabulary through Subtitled Authentic Videos. MA Thesis. Birmingham: University of Birmingham.
• Reese, Stephen D. (1984). “Visual‐verbal redundancy effects on television news learning.” Journal of Broadcasting 28(1), 79–87.
• Riley, Gail L. and James F. Lee (1996).  “A comparison of recall and summary protocols as measures of second language reading comprehension.” Language Testing 13(2), 173-187.
• SensoMotoric Instruments (2011). BeGaze™ manual: Version 2.4. Teltow, Germany: Author.
• Sweller, John (2010). “Element interactivity and intrinsic, extraneous, and germane cognitive load.” Educ Pshchol Rev 22, 123–138.
• Sweller, John, Paul Ayres and Slava Kalyuga (2011). Cognitive Load Theory. New York: Springer.
• Sweller, John, Jeroen J. G. Van Merrienboer and Fred G. W. C. Paas (1998). “Cognitive architecture and instructional design.” Educational Psychology Review 10(3), 251–296.
• Szarkowska, Agnieszka and Olivia Gerber-Morón (2018). “Viewers can keep up with fast subtitles: Evidence from eye movements.” PLoS ONE 13(6), 1-30.
• Talsma, Durk, Daniel Senkowski, Salvador Soto-Faraco and Marty G. Woldorff (2010). “The multifaceted interplay between attention and multisensory integration.” Trends in Cognitive Sciences 14(9), 400–410.
• Treisman, Annie M. (1968). “Strategies and models of visual attention.” Psychological Review 75(84), 127–190.
• Van der Burg, Erik, Sanne G. Brederoo, Mark R. Nieuwenstein, Jan Theeuwes and Christian N. L. Olivers  (2010). “Audiovisual semantic interference and attention: Evidence from the attentional blink paradigm.” Acta Psychologica 134(2), 198–205.
• Van der Burg, Erik, Durk Talsma, Christian N. L. Olivers, Clayton Hickey and Jan Theeuwe (2011). “Early multisensory interactions affect the competition among multiple visual objects.” NeuroImage 55(3), 1208–1218.
• Van Merriënboer, J. J. G. and John Sweller (2005). “Cognitive load theory and complex learning: Recent developments and future directions.” Educational Psychology Review 17(2), 147-177.
• Wang, Yangting (2014). The Effects of L1/L2 Subtitled American TV Series on Chinese EFL Students’ Listening Comprehension. MA thesis. East Lansing: Michigan State University.
• Zhang, Xiaochun (2013). “Fansubbing in China.” MultiLingual 24(5), 30–37.
• Zheng, Robert, Derek Smith, Marilyn Luptak, Robert D. Hill, Justin Hill and Randall Rupper (2016). “Does visual redundancy inhibit older persons’ information processing in learning?” Educational Gerontology 42(9), 635–645.
• Zu, Tianlong, John Hutson, Lester C. Loschky and N. Sanjay Rebello (2017). “Use of eye-tracking technology to investigate cognitive load theory.” https://arxiv.org/ftp/arxiv/papers/1803/1803.02499.pdf (consulted 20.11.2018).

Biography

Sixin Liao is a PhD student in the Department of Linguistics at Macquarie University in Australia. Her main research interests include interpreting studies and audiovisual translation with a focus on the visual and cognitive processing of subtitled products.

Email: sixin.liao@students.mq.edu.au

 

 

Jan-Louis Kruger is Head of the Department of Linguistics at Macquarie University in Australia. His main research interests are the reception and processing of language in multimodal contexts, including investigations on the impact of audiovisual translation products on cognitive load and psychological immersion combining eye-tracking and subjective measures.

Email: janlouis.kruger@mq.edu.au

 

 

Stephen Doherty is Senior Lecturer in Linguistics, Interpreting, and Translation in the School of Humanities & Languages, Faculty of Arts & Social Sciences, UNSW Sydney, where he also runs the Language Processing Lab. His research is based in language, cognition, and technology. With a focus on psycholinguistics and language technologies, his research investigates human and machine language processing using natural language processing techniques and combinations of online and offline measures (task performance, eye tracking, psychometrics, and electroencephalography).

Email: s.doherty@unsw.edu.au

Disclaimer: Authors are responsible for obtaining permission to use any copyrighted material contained in their article.

Notes

Note 1:
Free recall has been commonly used to assess comprehension in research studies (Carlisle 1999).
Return to this point in the text

Note 2:
“CCTV’s efforts to produce more programs with bilingual subtitles in Chinese and English are bearing fruit.” (title translated by the authors) http://www.cctv.cn/​2015/​11/28/ARTI1448699078542283.shtml (consulted 06.04.2017).
Return to this point in the text

Note 3:
We admit that more meticulous and reliable measures should be used in future research to ensure the homogeneity of samples in terms of language proficiency.
Return to this point in the text

Note 4:
The standard number of characters per line in most guidelines is 37 characters (Díaz Cintas and Remael 2007; Ivarsson and Carroll 1998). However, since only one line was used per language, and due to the wider format of the screen, a line length of 50% longer than the convention was considered to be functional, particularly since the subtitles were created for use on a computer screen with the user at a distance of approximately 70cm from the screen.
Return to this point in the text

Note 5:
Cronbach’s alpha coefficient of 0.7 or above is generally regarded to reflect a sufficient level of internal consistency and reliability (DeVellis 2003; Kline 2005).
Return to this point in the text

Note 6:
Leppink et al. (2014) also reported in their study that adding the last item regarding the mental effort in understanding the video did not increase the internal consistency of the scales used to measure germane cognitive load.
Return to this point in the text

Home | Past Issues | Keyword Search | Editorial Board | Events | About | Contact | Help