Home > Issue38 > Chan et al article

An investigation of subtitles as learning support in university education

Wing Shan Chan, Macquarie University, Sydney
Jan-Louis Kruger, Macquarie University, Sydney; North-West University, Vanderbijlpark
Stephen Doherty, University of New South Wales, Sydney

ABSTRACT

The literature on the benefits of subtitles has yielded contradictory results largely due to inconsistencies in experimental design and the operationalisation of variables such as language proficiency, subtitle language, and subtitle characteristics. As a result, there is insufficient evidence on the cognitive processing and impact of subtitles as learning support for learners in a L2 educational environment. The current study investigates the cognitive processing and resultant impact of subtitles in terms of cognitive load, comprehension scores, and eye movements. Native Chinese speakers (n=70) were recruited to watch five lecture videos with either no subtitles, Chinese L1, or English L2 subtitles. Results showed that reading L1 subtitles improved comprehension, with no significant difference in cognitive load between subtitle groups. By studying learners’ eye movements, we found that L2 subtitles were read differently from L1 subtitles, suggesting reading L2 subtitles is more cognitively demanding. The findings of the current study add further insight to existing research on the effectiveness of subtitles and subtitle language, showing that learners reading L1 subtitles improve performance in an authentic academic context. These findings could have implications for the use of subtitles to support students in English-medium instruction where English is not their first language.

KEYWORDS

Subtitles, cognitive load, educational subtitling, eye tracking, reading, audiovisual translation (AVT), English-medium instruction (EMI).

1. Introduction

Although there has been a growing interest in the use of subtitling in education over the past four decades, the research has often focused mainly on the use of subtitles in language learning and for people with hearing loss (Bird and Williams 2002; Danan 2004; Gerber-Morón and Szarkowska 2018; Zárate 2021). As online learning has become increasingly popular, the use of subtitling in educational videos has also become a growing area of research interest to increase accessibility for a variety of populations including people with hearing loss, foreign language speakers, and also first language speakers. The use of educational subtitling has become essential as a result of increasing globalisation, even more so after the disruption to learning and increase in online delivery of education due to the Covid-19 pandemic. The use of subtitles allows learners from around the world to learn through an online medium in a language of their choice (many students even study overseas through a language that is not their first language (L1)), and in particular English-medium instruction (or EMI). Generally, these students would have achieved a minimum standard of language proficiency in order to study abroad. However, they still face a language barrier that could prevent them from achieving their full potential. This educational phenomenon has created a growing research interest in the most effective ways to support these learners.

The existing body of research has shown the possible benefits of subtitles, mostly in the context of language acquisition and to some extent in content learning. The difference between language learning and content learning is related to the learning goal. In a multimedia learning environment, when learning a second language (L2), the processing of L2 subtitles is directly related to the learning goal, whereas in content learning, the processing of L2 subtitles becomes a tool to assist learners in acquiring knowledge (van der Zee et al. 2017). In addition, the methodologies used in past studies on reading a second language and its comprehension mostly involved static texts, and rarely used dynamic texts like subtitles in videos. The analysis of viewers’ eye movements allows us to discover the different eye movement patterns between reading L1 and L2 subtitles and the potential implication of the underlying processing. The results of this study add to the understanding of the discipline and could have important pedagogical implications in supporting academic success for learners.

2. Research Background

2.1. Contradictory findings on the effect of subtitles

The impact of subtitles is typically believed to be positive and beneficial for comprehension (Danan 1992; Garza 1991; Montero Pérez et al. 2013; Moreno and Mayer 2002), word recognition (Markham 1999), and information retention and recall (Vanderplank 1988) with no evidence of cognitive overload in the context of language acquisition. However, this seems to contradict the well-established findings on a redundancy effect (i.e., that performance is impacted negatively if the same information is presented simultaneously in more than one channel — such as when combining visual presentation with narration on the same topic) in the context of language learning in listening comprehension (Diao et al. 2007), procedural learning (Kalyuga et al. 1999) and concept learning (Mayer et al. 2001) in native language. These studies suggest that simultaneous presentation of written text (as visual input) combined with narration (as auditory input) might induce cognitive overload that impedes performance. It is important to note that these studies were not done in the context of subtitled video. For example, in the study of Kalyuga et al. (1999), written texts were added to visual diagram and audio narrative; and in the study of Mayer et al. (2001), on-screen text was added to a narrated animation in a multimedia learning environment.

In investigating whether a redundancy effect does in fact manifest when students are exposed to subtitled video, some studies, such as Kruger et al. (2013, 2014), and Kruger and Steyn (2013), demonstrated that the redundancy effect does not seem to occur when using subtitled videos in content learning and comprehension in an academic context. Even though these studies did not find an impact of subtitles on comprehension, the results from Kruger and Steyn (2013) indicated a positive correlation between subtitle reading and performance, suggesting that the presence of subtitles is valuable in an academic context.

Since it is difficult to draw any definite conclusion on the effect of subtitles based on these results, the aim of the current study is to investigate the impact of subtitles on performance, when the subtitles are in either the first or second language of students who study in a second language environment. Since the inconclusive results in earlier studies could potentially be attributed to inconsistent practices and standardisation in experimental protocols and frameworks (Orero et al. 2018), this study was designed with an increase in experimental control as will be discussed below.

2.2. Cognitive Load Theory

The theoretical framework for our study is cognitive load theory (CLT) proposed by Sweller (1994, 2003, 2004, 2010, 2011; Sweller et al. 2019), based on Baddeley’s working memory model of memory (see Baddeley and Hitch 1974). This theory suggests that the human cognitive architecture consists of unlimited long-term memory that coordinates human cognitive activities, and a very limited working memory that deals with new information and the transfer of this information to long-term memory. Working memory becomes optimal only when handling knowledge that has been learned previously and stored in long-term memory (Sweller 2003). Since the working memory is so limited, only a certain number of units can be processed at the same time. If too much information is presented at one time, cognitive overload could occur resulting in decreased learning. Cognitive load (CL) can be measured through comprehension, recall, memory retention or recognition. The significant reduction of performance scores could be an indication of the presence of cognitive overload. One of the questions in subtitle reading concerns the vast amount of information presented simultaneously in different channels which could potentially result in overloading the cognitive system and impacting performance.

CLT postulates three types of cognitive load: intrinsic, extraneous and germane load (Sweller 1988). Sweller (2010, 2011) defines them through the concept of element interactivity. Intrinsic load refers to the cognitive load generated by the task complexity and learners’ level of proficiency (Paas et al. 2003). Extraneous load refers to the cognitive load generated by the presentation format of the instruction in terms of level of clarity, which means that bad instructions could increase cognitive load on learners without improving learning (Debue and van de Leemput 2014). Germane load refers to the mental capacity that is available in dealing with the task complexity associated with intrinsic load, and therefore was later considered as part of intrinsic load by Sweller (2010). He proposed that the total cognitive load included only intrinsic and extraneous load because germane load does not contribute an independent source of load to the working memory. This has also changed the cognitive load instrument from measuring three cognitive load components to a two-factor intrinsic/extraneous cognitive load framework developed by Leppink and van den Heuvel (2015), which was adapted in this study. According to Sweller (2004), CLT is aimed at generating instructional design principles to enhance learning in education, and subtitles can be used as an instructional aid. Since adding subtitles to video changes the presentation of information, extraneous load seems to be the most impacted component of cognitive load, and as such it is the main component of cognitive load measured in this study.

2.3. The role of subtitle language in online learning

Past research such as the meta-analysis conducted by Montero Pérez et al. (2013) found that second language learners viewing L2 video subtitles performed better in listening comprehension and vocabulary learning than those viewing no subtitles in the context of language acquisition, however, no L1 subtitles were compared. The study of Markham et al. (2001) showed that second language learners viewing L1 subtitles had improved general comprehension when compared to those viewing L2 and no subtitles in the context of language learning. Previous findings also indicated that learners’ proficiency and subtitle language could be the reason for the differences in processing L1 and L2 subtitles. Winke et al. (2013) found that L2 learners spent more time in reading L2 subtitles if there is bigger distance in linguistic features between L1 and L2 in terms of phonological and logographical differences, such as English and Chinese. Specker (2015) found that the reading patterns and eye movements of native speakers stay consistent in both dynamic and static conditions (see also, Jensema et al. 2000), whereas non-native speakers change their reading patterns. These research findings give us some insight into the possible effects of reading L1 and L2 subtitles in terms of learners’ language proficiency and the linguistics differences in their first and second language.

2.4. Eye movement in reading static text vs. dynamic text

Eye tracking is an invaluable technique in the search for answers relating to the impact of subtitles and its underlying processing (see also, Doherty and Kruger 2018; Szarkowska and Bogucka 2019; Szarkowska and Gerber-Morón 2018) based on the eye-mind hypothesis (Just and Carpenter 1980), namely that there is a close relationship between what the eye fixates and what the mind attends to. Fixation duration — where the eyes remain relatively still — is generally assumed to be the time when information is extracted during reading, so measuring attention is possible at the same time (Holmqvist et al. 2011). Reading research suggests that new information is acquired during fixations, and studies conducted by Sun et al. (1985), Rayner (1998), Rayner et al. (2005a), Rayner et al. (2006), Schotter and Rayner (2012), and generally agree that each fixation typically lasts approximately 200-250ms. Fixations get longer when readers encounter difficult words or challenging content (Rayner et al. 2006), and as such, fixation duration provides some evidence of cognitive load elicited by a particular word or information.

Mean fixation duration (MFD, the average duration of all fixations) and the number of fixations (FC, fixation counts) are the global eye movement measures used in this study. Both fixation durations and number of fixations are indicators for processing challenging information. We also focus on the proportion of subtitles that were skipped (skipped subtitles %) by the participants in each subtitle group as this could be an indicator for the impact of subtitle language if the result is interpreted alongside with the comprehension scores. By comparing these eye movement measures between L1 and L2 subtitle groups, we can observe the ways L1 and L2 subtitles were processed. Based on previous findings, more and longer fixations could be an indication of difficulty in extracting information from the stimuli (Holmqvist et al. 2011; Rayner et al. 2006).

Previous reading studies on the eye movement patterns between reading Chinese and English texts (e.g., Sun et al. 1985; Sun and Feng 1999; Rayner 2004; Rayner et al. 2005b; Feng et al. 2009; Schotter and Rayner 2012) have found that average fixation durations in reading English (270ms) for native English speakers are very similar to those in reading Chinese (260ms) for native Chinese speakers despite the distinctive differences in visual form and writing system (orthography vs logography) of the two languages. Sun and Feng (1999) suggested that fixation duration and reading eye movement patterns are determined by linguistic information rather than the visual form of the text. However, subtitle reading is different from static reading in terms of the reading speed control and attention distribution between various channels of information such as image, audio speech, and possibly other on-screen text other than subtitles (Schotter and Rayner 2012; see also Rayner 1998). Therefore, it is uncertain if eye movement patterns still remain the same when Chinese and English text is read and processed in the form of subtitles.

In this study, we investigated the impact of L1 and L2 subtitles on students studying through the medium of a second language. Our experiment addresses two specific research questions. The first question of this study was to determine whether L1 and L2 subtitles would have an impact on the cognitive load (CL) and performance of Chinese L1 viewers, when compared to unsubtitled educational video. The second question we investigated was whether there are significant differences between the visual processing of L1 and L2 subtitles by these viewers.

The measures used in this study include self-reported cognitive load ratings, a comprehension test and a range of eye tracking measures, with data collected in a laboratory setting.

3. Methodology

3.1. Participants

Seventy Chinese L1 postgraduate students (57 female and 13 male) with a mean age of 30 were recruited at a large, public university in Sydney, Australia. We recruited postgraduate students from Linguistics to participate in the study, assuming they would be skilled readers with high proficiency in English as these students have a language proficiency (IELTS) requirement for university entrance. Ethics was approved (ref: 5201700903) by the ethics committee of the lead author’s institution and individual informed consent was obtained with a small reward being paid to participants for their time spent in the experiment.

3.2. Materials

3.2.1. Stimulus

The stimuli in this study were five 7-minutes excerpts of video lectures on the principles of Micro-Economics (Karunaratne 2012, 2015a, b, c, d). The video format was that of off-screen narration accompanying a paper-hand drawing style video showing only the sheet and the teacher’s hand illustrating the content. The stimuli were chosen as they were part of the curriculum for a diploma program, and the video length were decided to limit the total duration of the experiment. In addition to the subtitles at the bottom of the screen, graphs, formulas and texts emerged on the screen as the video progressed as shown in Figure 1.

Figure 1. Screenshot of the stimuli in English and Chinese subtitles.

The videos were transcribed in English and transcripts were translated into simplified Chinese. The scripts were reviewed by two professional translators for accuracy. Aegisub (2014), a free subtitling software package that allows the creation of professional-standard subtitles, was used to produce all the subtitles. All the video lectures were subtitled in English and Chinese according to established conventions with English subtitles containing no more than 37 characters per line (Díaz-Cintas and Remael 2007; Ivarsson and Carroll 1998) and Chinese subtitles no more than 20 Chinese characters per line (Kuo 2014). The presentation rate of the subtitles was kept to a speed of below 16 characters per second (CPS) for English and below five CPS for Chinese. The average presentation rate was around ten CPS for English, and three CPS for Chinese (see Díaz-Cintas and Remael 2007; Ivarsson and Carroll 1998; Kuo 2014 for guidelines). This is a rather slow presentation rate and was dictated by the slow speech rate of the lecturer, which allowed most subtitles to be verbatim transcripts of the speech. The subtitles were presented in both one-line and two-line formats and the display time of the subtitles for both English subtitles (ES) and Chinese subtitles (CS) were closely synchronised. We created the English and Chinese subtitles using Arial in size 20 and Microsoft YaHet in size 30 respectively for better readability after pilot testing various font sizes.

The five videos were analysed for comparable readability in both English and Chinese to limit the impact of text complexity on performance. We used Flesch-Kincaid Reading Ease to test the readability level of the English transcripts. The test scores range from 0-100, with zero as most difficult and 100 as the easiest, and a score between 60 and 80 should be quite easily comprehended by people with a school level of 7th–9th grade (Flesch 1979). The readability scores of the English transcripts ranged between 50 and 80, indicating the participants should be able to read the English subtitles with ease. We used the Chinese Readability Index Explorer, CRIE 3.0 (Sung et al. 2016) to analyse the Chinese translations for text complexity. Word difficulty can be indicated by the number of strokes in a Chinese character. Characters with 1-10 strokes are easy, and characters with over 21 strokes are difficult. The analysis of the Chinese scripts showed that the difficult word percentages were low (13%–19%) with the low-stroke character percentage ranging between 75% and 90%, indicating that the Chinese scripts are also very easy to read. The overall text analyses are provided in Table 1 and Table 2 for both English and Chinese scripts.

Video	Duration	Number of Sentence	Number of Words	Number of complex words	% Complex word	Average word / sentence	Average syllables / word	Flesch-Kincaid Readability score (%)
1	06:20	39	594	113	19.02	15.23	1.62	54.40
2	07:10	54	647	110	16.87	12.07	1.59	60.30
3	07:10	56	623	73	11.7	11.14	1.43	74.70
4	06:48	59	415	87	20.96	7.03	1.76	50.70
5	05:40	47	408	40	9.78	8.70	1.42	78.20

Table 1. Summary of English subtitles text analysis using Flesch-Kincaid Reading Ease.

Video	Number of sentences	Number of words	Characters	Difficult word	Difficult word %	Average word / sentence	Low-stroke characters (1-10)	Low-stroke characters %
1	40	507	695	94	18.54	12.68	581	83.60
2	38	496	671	85	17.13	13.05	576	85.84
3	56	487	623	65	13.34	7.75	482	77.37
4	55	377	458	49	13.00	6.85	408	89.08
5	29	306	412	45	14.70	10.55	361	87.62

Table 2. Summary of Chinese subtitles text analysis using CRIE 3.0.

3.2.2. Categorisation of redundant information

Since redundancy between the subtitles and other visual information on screen has been shown to have an impact on its processing (Mayer et al. 2001), we have factored redundant information (Redundant) into our model analyses. We manually categorised each subtitle in these five videos into four categories in terms of the presence of redundant information and hand movement of the lecturer guiding visual attention. These four categories are: 1) redundant information with hand movement (RH); 2) redundant information with no hand movement (RN); 3) non-redundant information with hand movement (NRWH); and 4) non-redundant information with no hand movement (NRN). Redundant information, in this case, refers to audio information describing the image (graphs, formulas and written text) on the screen. The distribution (in percentage) of these four categories of redundant information is shown in Figure 2.

Figure 2. Distribution of redundant information among the five videos.

3.2.3. Data collection instruments

A biographical questionnaire was used to collect participants’ background information and language history including age, home language, language used in learning during secondary schooling, and years spent living in an English-speaking country. The analyses showed that none of these factors have an effect on subtitle processing.

We measured only the perceived extraneous load since our study focused on the impact of subtitles as part of the instructional presentation. The 4-item cognitive load rating scale was adapted from the original instrument by Leppink and van den Heuvel (2015). It is a self-evaluated report on an 11-point scale from 0-10 with zero being ‘not at all the case’ and ten being ‘completely the case’. The four items on the instrument are: 1) The explanations and instructions in this video were very unclear; 2) The explanations and instructions in this video were full of unclear language; 3) The explanations and instructions in this video were, in terms of learning, very ineffective; and 4) I invested a very high mental effort in unclear and ineffective explanations and instructions in this video.

We used Cronbach’s alpha coefficients to check for item reliability within the same cognitive load instrument. The analysis in Table 3 shows that the four items have relatively high internal consistency.

Table 3. Reliability test between 4 cognitive load measuring items.

A performance test was conducted based on the comprehension of the content of the video after each video viewing. Due to time limitations, we only tested the participants on five multiple choice questions in each video, resulting in a total of 25 questions. The comprehension questions were sourced from an online sample by Frasca (2007) on this content as well as questions created by the researcher based on the video content.

Eye movement data were collected using SMI RED250 (2011) – an eye tracking device – and its associated software iViewX (2016) and Experiment Centre 3.7 (2010).

3.3. Apparatus

All the stimuli were presented through SMI RED250 (2011). The device has a sampling rate of 250 Hz and a screen resolution of 1920 x 1080 pixels. The stimuli were showed on a 17-inch laptop screen. We programmed and recorded the experiment using iViewX (2016) and Experiment Centre 3.7 (2010); we also used BeGaze™ (2016) to mark the area of interest (AOI) and to process the eye movement data.

3.4. Design and Procedure

3.4.1. Design

Our experiment is a quantitative, between-group design. Participants were randomly assigned to one of the three subtitle groups:

1. English audio with no subtitle (NS, n=18),
2. English audio with English subtitles (ES, n=28),
3. English audio with Chinese subtitles (CS, n=24).

Comprehension scores, self-perceived cognitive load (CL) ratings and eye tracking data on MFD, FC and skipped subtitles % were calculated and compared between these three groups. Correlations between subtitle language, skipped subtitles, redundancy, CL ratings and comprehension scores were analysed.

3.4.2. Procedure

Each participant completed the experiment individually inside a laboratory with sufficient illumination for eye tracking using SMI RED250 (2011). They were briefed about the experimental procedure, then were asked to sign the consent form before the experiment started.

Participants were asked to sit comfortably in front of the eye tracker which was attached to a laptop. Their positions were adjusted within the range of the device, and participants were asked to maintain a similar position throughout the experiment for their eye movement to be recorded successfully. Participants were asked to switch off their phones to avoid distraction, and there was no noise from outside the laboratory after the door was closed as the experiment was conducted in a sound-proof laboratory.

A 13-point calibration (including 4-point validation) was employed to ensure participants’ eyes were calibrated with the device for accurate eye tracking measurement. Full instructions of the experimental procedure were presented on the screen as the experiment proceeded. The first video excerpt started after a successful calibration followed by a biographical questionnaire, and participants were required to complete a 4-item self-report cognitive load rating and then five multiple-choice comprehension questions after each video viewing. There was no time limit for participants to answer the questions. A 30-second break was given after each comprehension test and before the next calibration. After the short break, another calibration started again before the next video, and the process repeated until the experiment finished. The whole experiment lasted approximately one hour.

3.5. Analyses

3.5.1. Comprehension and self-perceived cognitive load measures

We used R (R Core Team 2013) to perform the analyses in this experiment. Since the comprehension and the self-perceived cognitive load data were not normally distributed, we performed a Kruskal-Wallis rank sum test to calculate the difference between three subtitle groups in comprehension scores and self-perceived cognitive load ratings as an alternative to a one-way ANOVA test (Baayen 2008). A Wilcoxon pairwise-comparison test was then performed to find out which comprehension pairwise-comparison was significant (p < .05). No Wilcoxon pairwise-comparison was needed to further analyse the self-perceived cognitive load ratings because the result was not significant.

3.5.2. Eye movement data

Eye movement data with tracking ratio of lower than 85% were removed from analysis to maintain data reliability, resulting in 39% loss of the original data. Areas of interests (AOIs) were created around each individual subtitle. To compare the difference in the processing of L1 and L2 subtitles, we report two eye movement measures: 1) mean fixation duration (MFD), 2) number of fixations (FC); and a measure for skipped subtitles percentage through binomial modelling.

The eye movement data were analysed using Generalised Linear Mixed Model (GLMM) through the function glmer (Brooks et al. 2017) from package lme4 (Bates et al. 2015). GLMM is a mixed effects modelling which accounts for the variation from participants and items (design factors) simultaneously (Brown 2021). It allows researchers to build a fitted model that can factor in both fixed and random effects (Winter 2013). One of the advantages of using a mixed effects modelling over the ANOVA framework is its flexibility as it can manage missing data and unbalanced designs rather than removing the individual responses completely as in ANOVA (Brown 2021). Especially in eye tracking studies, it allows the assessment of the time effect as continuous variable on a categorical variable such as the gaze location or area of interest (AOI). ANOVA, on the other hand, was developed to assess the effects of categorical variables as independent variables (such as subtitle language) on continuous dependent variables such as attention duration (Barr 2008).

Since the dataset contained eye tracking information from two different language systems, the dataset needed to be normalised with characters per second(CPS)and mean word length(MW) centred and scaled before they could be used for analysis. To analyse MFD, MW was treated as an interaction and Redundant as fixed effect; to analyse FC, CPS was treated as an interaction, MW and Redundant as fixed effects. For all eye tracking analyses and the binomial modelling, subtitle language (Group) was treated as a fixed effect, participant and AOI as random effects.

3.5.3. Correlations

Correlations were carried out between subtitle language, skipped subtitles, redundancy, CL ratings and comprehension scores to investigate if there are any significant relationships between these variables.

4. Results

4.1. Comprehension scores

We conducted a Kruskal-Wallis rank sum test as the data was not normally distributed. We used R to calculate the difference between the three subtitle groups in comprehension scores as an alternative to a one-way ANOVA test. The results are shown in Table 4.

Table 4. Kruskal-Wallis rank sum test in three subtitle groups is statistically significant.

A Wilcoxon pairwise-comparison test was then performed, and the results in Table 5 show that the comprehension between CS (M = 3.52, SD = 1.02) and NS (M = 3.13, SD = 1.21) is statistically significant (p < .05, d = 0.35); the comprehension between CS (M = 3.52, SD = 1.02) and ES (M = 3.19, SD = 1.12) is also statistically significant (p < .05, d = 0.31).

Group	N	Mean scores (SD)	pairwise-comparison	Sig.
Chinese_sub	24	3.25 (1.02)	Chinese_sub & English_sub	p=.042 *
English_sub	28	3.19 (1.12)	Chinese_sub & No_sub	p=.042 *
No_sub	18	3.13 (1.21)	English_sub & No_sub	p=.823

Table 5. Wilcoxon pairwise-comparison test for comprehension scores.

The results show that Chinese participants scored significantly higher in comprehension when reading Chinese subtitles than reading English and no subtitles (ES: M = 3.19, SD = 1.12; NS: M = 3.13, SD = 1.21; p < .05); and there is no significant difference in comprehension between reading English and no subtitles. The results seem to suggest that reading L1 subtitles results in an advantage over reading L2 subtitles, with L1 subtitles potentially providing deeper cognitive processing in comprehending the text.

4.2. Self-Perceived Cognitive Load

The results of a Kruskal-Wallis rank sum test for CL show that the difference between Group (NS: M = 10.7, SD = 8.93; CS: M = 10.1, SD = 8.33; ES: M = 9.79, SD = 8.70) is not statistically significant, ηp2(2, N=350) = .502, p > .05). The boxplot in Figure 3 shows no significant difference in cognitive load for Chinese participants reading L1, L2 or no subtitles.

Figure 3. No significant difference in mean CL ratings between subtitle groups.

The non-significant results could be explained by various factors affecting cognitive load for Chinese participants in these three subtitle groups. When Chinese speakers are reading L1 subtitles, their CL increases due to the possibility of them checking the accuracy of the translation, and at the same time dealing with subject matter with Economics-related technical language they do not normally use in their linguistics studies even if it is their first language. Similarly, reading L2 subtitles could trigger a comparison of audio and subtitles; whereas in no subtitles, CL increases could be caused by lack of visual support that is provided by the other two subtitle groups. 

4.3. Mean Fixation Duration

The results in Figure 4 show that the difference in average MFD between English subtitles (M = 275, SD = 4.08) and Chinese subtitles (M = 214, SD = 3.62) group is statistically significant (p < .05, d = 15.82), with MW as centred and scaled value.

Figure 4. Average MFD results show that participants had significantly (p<.05, d=15.82) longer mean fixation durations when reading ES (M=275) than CS (M=214), with MW as centred and scaled value.

The computation outcome using glmer analyses (with MW of the language) in Table 6 shows that Chinese speakers have significantly longer mean fixation duration in reading English subtitles (305ms) than Chinese subtitles (266ms). The results seem to indicate that reading patterns change when Chinese speakers read L2 subtitles compared to L1 subtitles.

Table 6. MFD of Chinese speakers in reading L1 and L2 static and dynamic text.

4.4. Fixation Counts

The results show that the difference in fixations between English subtitles (M = 5.19, SD = 3.66) and Chinese subtitles (M = 3.54, SD = 3.34) is statistically significant (p < .05, d = .47), with CPS and MW as centred and scaled value. The results indicate that Chinese speakers have significantly more fixations when reading English subtitles than Chinese subtitles, which is presented through a boxplot in Figure 5.

Figure 5. Fixations results show that participants have more fixations when reading ES than CS.

The results seem to indicate that it is more challenging for native Chinese speakers to read English L2 subtitles than Chinese L1 subtitles. As evidenced in previous reading research, longer and more fixations are an indication of processing difficulty and could therefore be linked to an increase in cognitive load (Rayner et al. 2006).

4.5. Skipped Subtitles Percentage

A binomial model using glmer analyses was used, and it focused on the odds ratio of the subtitles being processed or not processed based on the language of subtitles (Group) as fixed effects; the participants and each subtitle per video (video/AOI) as random effects. The binomial modelling is based on the concept of probability, and the odds ratio (also known as likelihood) are defined as “the odds is the ratio of the probability that the event of interest occurs to the probability that it does not” (Bland and Altman 2000:1486). In this case, the modelling is to calculate the ratio of the probability of English subtitles being processed to the probability of Chinese subtitles being processed. The results show that the median probability of processed subtitles being processed is approximately 0.9, which indicates that the model has a high accuracy in predicting correctly that the processed subtitles would be processed. The area under our ROC curve is 0.85, indicating that this is a good model fit in this prediction since a higher value of the area under the curve (AUC) that is closer to one, the better the model fits (Robin et al. 2011).

Table 7 shows that 83% of the total subtitles (combining Chinese and English subtitles) were actually processed by the participants, indicating that the model is a good model fit.

Table 7. The actual percentage for processed and non-processed subtitles with 99% quantile.

The computation outcome using glmer analyses in this binomial model shows that the odds for the English subtitles being processed were about 128% higher than the odds of the Chinese subtitles being processed by Chinese speakers (ratio of the odds for the English subtitles being processed was 1:2.28), but the difference is not significant (p > .05). Even though the results are not significant, Chinese speakers in our study have a relatively higher rate of skip reading the subtitles when it was presented in Chinese than in English. This is shown in the line graph in Figure 6 that Chinese subtitles were skipped more by Chinese participants in general.

Figure 6. Skipped subtitles % shows that CS were skipped more by Chinese participants in general.

4.6. Correlations

The correlations in Table 8 show that skipped English subtitles and NRN (non-redundant without hand) redundant content is strongly negatively correlated, r(942) = -0.96, p < .05; and skipped English subtitles and RN (redundant without hand) redundant content is strongly positively correlated, r(942) = 0.98, p < 0.05. The correlations show that, when there is no hand movement as extra visual distraction, with more information being redundant, the more English subtitles were skipped; on the other hand, with more information as non-redundant, the less English subtitles were skipped, in other words the more English subtitles were read.

Table 8. Correlation between redundant type and skipped Chinese and English subtitles for 5 videos.

5. Discussion

Our comprehension results show that Chinese speakers scored significantly higher in comprehension when reading L1 subtitles than reading L2 and no subtitles, and the findings showed no evidence of cognitive overload in information processing when adding subtitles to multimedia learning materials regardless of language. Native Chinese speakers in our study seem to comprehend more effectively in reading L1 subtitles, which could also have facilitated a more efficient processing of the visual information in the videos. These results agreed with the findings of Kruger et al. (2014) where Sesotho students allocated more attention in reading English L2 subtitles but have a higher retention of knowledge when reading Sesotho L1 subtitles. Kruger et al. (2014) suggested that reading L1 subtitles may provide cognitive priming and therefore benefits L1 speakers in deep cognitive processing.

Although we did not find a significant difference in cognitive load between the three subtitle groups, our eye tracking results show that eye movement of native Chinese speakers reading L2 subtitles are significantly different from reading L1 subtitles, with more and longer fixations and fewer skipped subtitles, which suggest that L2 subtitles could be cognitively more taxing to process. Our findings (see Table 6) indicate that native Chinese speakers had longer fixations when reading English L2 subtitles (305ms) than reading Chinese L1 subtitles (266ms), which was read as if it was a written text (where the average fixation from literature is 260ms). The results support the findings of Specker (2015; see also, Jensema et al. 2000) who found that eye movements for native speakers stay consistent when reading text in dynamic and static conditions, whereas non-native speakers change reading pattern when reading subtitles in their second language. The reading patterns of the Chinese speakers in our study seem to imply that when L2 text is presented dynamically as subtitles in a multimedia learning environment, it is more challenging for them to process and therefore they need longer time to fully comprehend the content.

The results from our study also suggest that L1 subtitles may have the advantage of facilitating deep cognitive processing of the learning content that improves comprehension, as suggested by Danan (2004) and Kruger et al. (2014). According to CLT, working memory becomes optimal when handling knowledge that has been learned previously and stored in long-term memory. High language proficiency in first language indicates that relevant linguistics skills have been stored in long-term memory for effective processing during subtitle reading. Therefore, without having to struggle with language issues, learners have more cognitive capacity to process new information when reading L1 subtitles resulting in better understanding.

The results on skipped subtitles percentage with higher skipping rate when reading Chinese subtitles signal that Chinese speakers in this study have a lower reliance on subtitles presented in L1, but when combined with the comprehension scores, it seems that even superficial processing of L1 subtitles result in performance gains.

The current findings indicate that redundant information has limited impact on subtitle reading in terms of mean fixation duration and the number of fixations, regardless of the subtitle language. In addition, the results show that redundancy does not have the negative impact on information processing, in this case subtitles, as has been suggested by past studies (Diao et al. 2007; Diao and Sweller 2007; Kalyuga et al. 1999; Mayer et al. 2001).

However, the correlations show that redundant information correlates positively with skipped L2 subtitles; and non-redundant information correlates negatively with skipped L2 subtitles. The results of the correlations seem to suggest that redundancy may only have an impact on the language with less proficiency in subtitle processing.

The overall findings evidence that native Chinese speakers reading L1 subtitles improved performance in terms of comprehension; and their eye movement patterns show that they read L1 subtitles as if it was a reading task while they spent longer time in reading L2 subtitles. The differences in these reading patterns indicate that there is a change of reading strategy when L2 text is presented as subtitles, as suggested by Specker (2015).

6. Conclusion and Limitations

The results of this study show that reading L1 subtitles is beneficial to learners when they learn through a second language in academic contexts. As online learning is becoming more popular, the presence of subtitles, particularly L1 subtitles, assists learners in comprehending educational content more effectively in a second language multimedia learning environment that leads to improved performance. Previous studies have proven that the presence of subtitles helped maximising learning (Vanderplank 1988) and complementing the process of ambiguous or novel information (Bird and Williams 2002; Danan 2004) in the context of learners reading L2 subtitles for the purpose of second language acquisition. Even though the results indicate that redundant information correlates mainly to L2 subtitles, the current findings show that redundancy has very limited impact on audiovisual text processing. The current study adds further insight to and complements existing research on the effectiveness of subtitles and subtitle language. Our results show that learners reading L1 subtitles in a second language learning environment could improve academic performance.

Given that our results provide initial evidence showing L1 subtitles assist in improving comprehension, L1 subtitles could be utilised with great effect in contexts like Australia, the UK and the US where large number of students study using EMI, whereas L2 subtitles can be presented in assisting language acquisition, which has been proven to be effective in previous studies (Montero Pérez et al. 2013). The results have valuable implications for pedagogy in terms of assisting learners in achieving their highest potential academically without being disadvantaged by possible language barriers.

We have identified a few limitations that can be addressed in future research. The loss of data in this study is quite high. Although a high rate of data attrition is common in eye tracking studies, more accurate eye trackers like the high-speed Eyelink trackers used with a chin rest may result in improved tracking ratio in future studies. Native speakers from different countries such as Vietnam, Thailand or India can be the focus of future investigation. Other topic areas in content learning could be used in future investigation to explore the possible impact of topic area in subtitle processing and learning motivation.

Acknowledgements

We would like to thank the participants who took part in this research and colleagues who helped in the data collection. Thank you also to Dr Peter Humburg who has guided us through the statistical analysis. The lead author was supported by a PhD scholarship from Macquarie University.

References

• Aegisub (2014). https://github.com/Aegisub/Aegisub/releases (consulted 1.6.2017).
• Baayen, Harald (2008). Analyzing Linguistic Data: A Practical Introduction to Statistics using R. Cambridge: Cambridge University Press.
• Baddeley, Alan and Graham Hitch (1974). “Working memory.” Gordon H. Bower (ed.) (1974). The Psychology of Learning and Motivation: Advances in Research and Theory, vol. 8. New York: Academic Press, 47-89.
• Barr, Dale (2008). “Analyzing ‘visual world’ eyetracking data using multilevel logistic regression.” Journal of Memory and Language 59(4), 457-474. DOI: 10.1016/j.jml.2007.09.002.
• Bates, Douglas et al. (2015). “Fitting Linear Mixed-Effects Models Using lme4.” Journal of Statistical Software 67(1), 1-48. DOI: 10.18637/jss.v067.i01.
• BeGaze™ [Computer software] (2016). SensoMotoric Instruments GmbH. Teltow, Germany.
• Bird, Stephen and John Williams (2002). “The effect of bimodal input on implicit and explicit memory: An investigation into the benefits of within-language subtitling.” Applied Psycholinguistics 23(4), 509-533.
• Bland, J. Martin and Douglas G. Altman (2000). "The odds ratio." BMJ 320, 1468. DOI: 10.1136/bmj.320.7247.1468.
• Brooks, Mollie E. et al. (2017). “glmmTMB balances speed and flexibility among packages for zero-inflated generalized linear mixed modeling.” The R Journal 9(2), 378-400.
• Brown, Violet (2021). “An introduction to linear mixed-effects modeling in R.” Advances in Methods and Practices in Psychological Science 4(1), 1-19. DOI: 10.1177/2515245920960.
• Danan, Martine (1992). “Reversed subtitling and dual coding theory: New directions for foreign language instruction.” Language Learning 42(4), 497-527.
• Danan, Martine (2004). “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta 49(1), 67-77. DOI: 10.7202/009021a.
• Debue, Nicolas and Cécile van de Leemput (2014). “What does germane load mean? An empirical contribution to the cognitive load theory.” Frontiers in Psychology 5(1099). DOI: 10.3389/fpsyg.2014.01099.
• Diao, Yali and John Sweller (2007). “Redundancy in foreign language reading comprehension instruction: Concurrent written and spoken presentations.” Learning and Instruction 17(1), 78-88.
• Diao, Yali et al. (2007). “The Effect of Written Text on Comprehension of Spoken English as a Foreign Language.” The American Journal of Psychology 120(2), 237-261.
• Díaz-Cintas, Jorge and Aline Remael (2007). Audiovisual Translation: Subtitling. Manchester: St Jerome.
• Doherty, Stephen and Jan-Louis Kruger (2018). “The development of eye tracking in empirical research on subtitling and captioning.” Tessa Dwyer et al. (eds). Seeing into Screens. London: Bloomsbury, 46-64.
• Experiment Center™ [Computer software] (2010). SensoMotoric Instruments GmbH. Teltow, Germany.
• Feng, Gary et al. (2009). “Orthography and the development of reading processes: An eye-movement study of Chinese and English.” Child Development 80(3), 720-735.
• Flesch, Rudolf (1979). How to write plain English. New York: Harper & Row.
• Frasca, Ralph R. (2007). “Microeconomics.” https://academic.udayton.edu/pmic/Instructors%20manual_copy(1).htm (consulted 10.8.2017).
• Garza, Thomas (1991). “Evaluating the Use of Captioned Video Materials in Advanced Foreign Language Learning.” Foreign Language Annals 24(3), 239-258.
• Gerber-Morón, Olivia and Agnieszka Szarkowska (2018). “Line breaks in subtitling: an eye tracking study on viewer preferences”. Journal of Eye Movement Research 11(3):2. DOI: 10.16910/jemr.11.3.2.
• Holmqvist, Kenneth et al. (2011). Eye Tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press.
• Ivarsson, Jan and Mary Carroll (1998). Subtitling. Simrishamn: TransEdit HB.
• iViewX™ [Computer software] (2016). SensoMotoric Instruments GmbH. Teltow, Germany.
• Jensema, Carl J. et al. (2000). “Eye movement patterns of captioned television viewers.” American Annals of the Deaf 145(3), 275-285. DOI: 10.1353/aad.2012.0093.
• Just, Marcel Adam and Partricia A. Carpenter (1980). “A theory of reading: From eye fixations to comprehension.” Psychological Review 87(4), 329-355.
• Kalyuga, Slava et al. (1999). “Managing split-attention and redundancy in multimedia instruction.” Applied Cognitive Psychology 13(4), 351-371.
• Karunaratne, Prashan (2012). Demand & supply and government actions in markets. https://www.youtube.com/watch?v=Z3Qb94pubAQ (consulted 15.7.2017).
• Karunaratne, Prashan (2015a). Market structure and efficiency. https://www.youtube.com/watch?v=5pxkCxMYUPU (consulted 15.7.2017).
• Karunaratne, Prashan (2015b). Perfect competition and monopolistic competition. https://www.youtube.com/watch?v=3T9GSJeomKE (consulted 15.7.2017).
• Karunaratne, Prashan (2015c). Producer theory and shifting cost curves. https://www.youtube.com/watch?v=C3PDVmPijIY (consulted 15.7.2017).
• Karunaratne, Prashan (2015d). Externalities. https://www.youtube.com/watch?v=GE8XeeaVkJ8 (consulted 15.7.2017).
• Kruger, Jan-Louis et al. (2013). “Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts.” Proceedings of the 2013 Conference on Eye Tracking South Africa (ETSA '13). New York: Association for Computing Machinery, 62-66. https://doi.org/10.1145/2509315.2509331.
• Kruger, Jan-Louis et al. (2014). “Attention distribution and cognitive load in a subtitled academic lecture: L1 vs L2.” Journal of Eye Movement Research 7(5), 1-15.
• Kruger, Jan-Louis, and Faans Steyn (2013). “Subtitles and eye tracking reading and performance.” Reading Research Quarterly 49(1), 105-120.
• Kuo, Szu-Yu (2014). Quality in Subtitling: Theory and Professional Reality. PhD thesis. London: Imperial College London. https://spiral.imperial.ac.uk/handle/10044/1/24171 (consulted 4.1.2018).
• Leppink, Jimmie and Angelique van den Heuvel (2015). “The evolution of cognitive load theory and its application to medical education.” Perspectives on Medical Education 4(3), 119-127. DOI: 10.1007/s40037-015-0192-x.
• Markham, Peter L. (1999). “Captioned videotapes and second-language listening word recognition.” Foreign Language Annals 32(3), 321-328.
• Markham, Paul L. et al.(2001). “The effects of native language vs. target language captions on foreign language students’ DVD video comprehension.” Foreign Language Annals 34(5), 439-445.
• Mayer, Richard E. et al. (2001). “Cognitive constraints on multimedia learning: when presenting more material results in less understanding.” Journal of Educational Psychology 93(1), 187-198.
• Montero Pérez, Maribel, Wim Van Den Noortgate and Piet Desmet (2013). “Captioned video for L2 listening and vocabulary learning: A meta-analysis.” System 41(3), 720-739.
• Moreno, Roxana, and Richard E. Mayer (2002). “Verbal redundancy in multimedia learning: When reading helps listening.” Journal of Educational Psychology 94(1), 156-163. DOI: 10.1037//0022-0663.94.1.156.
• Orero, Pilar et al. (2018). “Conducting experimental research in audiovisual translation (AVT): A position paper.” The Journal of Specialised Translation 30, 105-126.
• Paas, Fred et al. (2003). “Cognitive load measurement as a means to advance cognitive load theory.” Educational Psychologist 38(1), 63-71. DOI: 10.1207/s15326985ep3801_8.
• R Core Team (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/ (consulted 12.10.2019).
• Rayner, Keith (1998). “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 124(3), 372-422.
• Rayner, Keith (2004). “Eye movements, cognitive processes, and reading.” Studies of Psychology and Behavior 2, 482-488.
• Rayner, Keith et al. (2006). “Eye movements as reflections of comprehension processes in reading.” Scientific Studies of Reading 10(3), 241-255.
• Rayner, Keith et al. (2005a). “Eye movements during reading.” Margaret J. Snowling and Charles Hulme (eds) (2005). The Science of Reading: A Handbook. Malden, MA, USA: Blackwell Publishing, 79-97.
• Rayner, Keith et al. (2005b). “The effect of word predictability of the eye movements of Chinese readers.” Psychonomic Bulletin & Review 12(6), 1089-1093.
• Robin, Xavier et al. (2011). “pROC: an Open-Source Package for R and S+ to Analyze and Compare ROC Curves.” BMC Bioinformatics 12(77). DOI: 10.1186/1471-2105-12-77.
• Schotter, Elizabeht and Keith Rayner (2012). “Eye movements in reading: Implication for reading subtitles.” Elisa Perego (ed.) (2012). Eyetracking in audiovisual translation. Rome: Aracne Editrice, 83-104.
• SMI RED250™ [Apparatus] (2011). SensoMotoric Instruments GmbH. Teltow, Germany.
• Specker, Elizabeth (2015). L1/L2 Eye movement reading of closed captioning: A multimodal analysis of multimodal use. PhD thesis. The University of Arizona. http://hdl.handle.net/10150/194820 (consulted 6.11.2017).
• Sun, Fuchuan and Dagan Feng (1999). “Eye movements in reading Chinese and English text.” Jian Wang et al. (eds) (1999). Reading Chinese Script: A Cognitive Analysis. New York: Taylor and Francis, 189-206.
• Sun, Fuchuan et al. (1985). “Comparative patterns of reading eye movement in Chinese and English.” Perception & Psychophysics 37, 502-506. DOI: 10.3758/bf03204913.
• Sung, Yao-Ting et al. (2016). “CRIE: An automated analyzer for Chinese texts.” Behavior Research Methods 48(4), 1238-1251. DOI: 10.3758/s13428-015-0649-1.
• Sweller, John (1988). “Cognitive load during problem solving: Effects on learning.” Cognitive Science 12(2), 257-285. DOI: 10.1016/0364-0213(88)90023-7.
• Sweller, John (1994). “Cognitive load theory, learning difficulty, and instructional design.” Learning and Instruction 4(4), 295-312.
• Sweller, John (2003). “Evolution of human cognitive architecture.” Psychology of Learning and Motivation 43, 215-266. DOI: 10.1016/S0079-7421(03)01015-6.
• Sweller, John (2004). “Instructional design consequences of an analogy between evolution by natural selection and human cognitive architecture.” Instructional Science 32, 9-31.
• Sweller, John (2010). “Element interactivity and intrinsic, extraneous, and germane cognitive load.” Educational Psychology Review 22(2), 123-138. DOI: 10.1007/s10648-010-9128-5.
• Sweller, John (2011). “Cognitive load theory.” Psychology of Learning and Motivation 55, 37-76.
• Sweller, John et al. (2019). “Cognitive architecture and instructional design: 20 years later.” Educational Psychology Review 31, 261-92. DOI: 10.1007/s10648-019-09465-5.
• Szarkowska, Agnieszka and Lidia Bogucka (2019). “Six-second rule revisited: An eye-tracking study on the impact of speech rate and language proficiency on subtitle reading.” Translation Cognition & Behaviour 2(1), 101-124. DOI: 10.1075/tcb.00022.sza.
• Szarkowska, Agnieszka and Olivia Gerber-Morón (2018). “Viewers can keep up with fast subtitles: Evidence from eye movements.” PLoS ONE 13(6), 1-30. DOI: 10.1371/journal.pone.0199331.
• Van der Zee, Tim et al. (2017). “Effects of subtitles, complexity, and language proficiency on learning from online education videos.” Journal of Media Psychology Theories Methods and Applications 29, 18-30.
• Vanderplank, Robert (1988). “The value of teletext sub-titles in language learning.” ELT Journal 42(4), 272-281.
• Winke, Paula et al. (2013). “Factors influencing the use of captions by foreign language learners: an eye-tracking study.” The Modern Language Journal 97(1), 254-275.
• Winter, Bodo (2013). Linear models and linear mixed effects models in R with linguistic applications. arXiv. http://arxiv.org/abs/1308.5499 (consulted 11.1.2019).
• Zárate, Soledad (2021). Subtitling for d/Deaf and Hard of Hearing Audiences. London: UCL Press. DOI: 10.2307/j.ctv14t478b.8.

Data availability statement

The data and the supplementary document that support the findings of this study are openly available in Center for Open Science (OSF) at https://osf.io/x6arp/?view_only=02bf85a075f341f6a675f98f99ac0bf5.

Biographies

Wing Shan Chan completed her PhD in Linguistics at Macquarie University in 2020. Her main research interests include effective instructional designs for learners and pedagogy in university education, especially in supporting non-English-speaking learners learning through English-medium instructions.

Email: wing.chan@mq.edu.au
ORCID: 0000-0002-9104-2470

Jan-Louis Kruger is professor of Linguistics at Macquarie University and extraordinary professor in the UPSET focus area at North-West University in South Africa. His research focuses on the processing of language in multimodal contexts, specifically in audiovisual translation, reading, and interpreting. His main approaches are aligned with cognitive psychology and psycholinguistics. Primarily, his projects focus on investigating cognitive processing when more than one source of information has to be integrated, as in the reception of subtitles or the production of interpreting. He is on the editorial board of the Journal of Audiovisual Translation.

Email: janlouis.kruger@mq.edu.au
ORCID: 0000-0002-4817-5390

Stephen Doherty is Associate Professor of Linguistics, Interpreting and Translation at the University of New South Wales, where he is also lead of the HAL Language Processing Lab. With a focus on the psychology of language and technology, his research investigates human language processing and usage by employing natural language processing techniques and combinations of online and offline methods, mainly eye-tracking and psychometrics.

Email: s.doherty@unsw.edu.au
ORCID: 0000-0003-0887-1049

Home | Past Issues | Keyword Search | Editorial Board | Events | About | Contact | Help