Plastina article

Remaking meaning through intersemiotic translation: the case of online medical journals

Anna Franca Plastina, University of Calabria

ABSTRACT

The intersemiotic translation of written research abstracts is currently practised in online medical journals to facilitate experts’ rapid access to specialised knowledge. Taking a social semiotic approach to translation, this paper considers intersemiotic translation as resemiotisation yielding visual abstracts. The twofold aim is to examine how visual abstracts resemiotise original meanings on the expression, content and context planes, and to seek how intersemiotic translation is rendered to meet the new needs of the specialist readership. A systemic functional-multimodal discourse analysis is conducted on sample abstracts from the British Medical Journal (BMJ) and New England Journal of Medicine (NEJM). Findings show how original experiential meanings are exploited to resemiotise original implicit interpersonal meanings and construe new ones on the expression plane. BMJ content is mainly resemiotised through intersemiotic addition and variation to ease reader comprehension; NEJM content is recreated through intersemiotic exemplification for rapid retention of elaborate information. Register variation, determined by the choices of nonverbal resources, is justified by the online context of visual abstracts. Significant genre shifts through transgeneric translation foreground the new purpose of visual abstracts as a key genre-defining element. Ultimately, the research underlines the importance of reconfiguring the study of translation in a multidimensional semiotic paradigm.

KEYWORDS

Intersemiotic translation, visual abstracts, systemic functional-multimodal discourse analysis, metafunctional meanings, online medical journals.

1. Introduction

The proliferation of multimodal texts in digital media reveals an unprecedented shift towards the visual representation of information which was previously mediated only through the verbal code (Kress 1998). As van Leeuwen (2017: 4) notes, “more recently, the dominance of monomodality has weakened, although it still persists in some practices, for instance in the writing and publishing of academic papers.” Even in this field, however, multimodality seems to be slowly gaining ground in the highly valued genre of medical research abstracts over the past few years. Leading online medical journals, such as the British Medical Journal (BMJ) and the New England Journal of Medicine (NEJM) have been engaging in the intersemiotic translation of selected written abstracts respectively since 2018 and 2019. This specialised practice suggests that intersemiotic translation increasingly “forms the basis for explaining and circulating ideas in society and culture” (O’Halloran et al. 2016: 203).

Besides exploiting intersemiotic translation to promote the dissemination of original scientific research, online medical journals are essentially using “multimodality as a resource for translation” (O’Sullivan 2013: 11, original italics) to meet the needs of their specialist readership. These are primarily found to include medical professionals’ preference for infographics over traditional text-only abstracts as they enhance information retention and reduce cognitive load (Martin et al. 2019). In response to this preference, the BMJ and NEJM are creating visual abstracts to help their readers overcome the mentally-taxing challenge of retaining complex knowledge under the routine conditions of professional time constraints. Generally, visual abstracts are designed to include concise texts and colourful still images which ensure the rapid assimilation of elaborate information in an aesthetically appealing manner.

It is argued that investigating intersemiotic translation in visual abstracts offers the opportunity to reconfigure “the study of translation in a more complex and multidimensional semiotic paradigm” (Troqe 2015: 38). Accordingly, intersemiotic translation is herein considered as a process of “resemiotisation” (Iedema 2003) which can be examined in-depth through “an integrative platform based on the SF metafunctional principle […] and intersemiotic mechanisms and systems (content and expression strata) […] to capture the expansion of meaning […]” (O’Halloran 2008: 443). In this sense, Lim’s (2004: 222-224) Integrative Multisemiotic Model (IMM) offers a valid support for a detailed analysis of intersemiotic translation in visual abstracts as it conceptualises three planes across language and visual images: 1. the Expression plane (typography for language, graphics for images) as the interface between the reader/viewer and the text; 2. the Content plane (lexicogrammatical and discourse semantics strata for language; visual grammar and discourse semantics strata for images); 3) the Context plane (register and genre) where meaning is located within the context of situation and culture. As Lim (2004: 223) further specifies, “the systems of meaning in the Expression and Content plane for language and visual images are seen to be organized metafunctionally […].”

Taking this stance thus allows to better account for the transformative dynamics occurring in medical research abstracts driven by “digital resemiotisation”, i.e. “the ways in which a meaning-making practice is transformed when moved from an offline setting to an online environment” (Poulsen 2022: 70). “Intersemiotic translation as resemiotisation” offers a useful “analytical means” to focus on “how semiotics are translated from one into the other as social processes unfold”, and to question “why these semiotics (rather than others) are mobilized” (Iedema 2003: 29). Moreover, such analytical means helps reveal the semiotic awareness required from specialised translators in making ‘knowledge-based’ choices at the different levels of meaning.

Although multimodal issues have become increasingly important in Translation Studies, scant research has been conducted so far on the intersemiotic translation process of realising visual abstracts (Stahl-Timmins et al. 2019). This paper attempts to fill this void with a twofold purpose in mind. Firstly, it seeks to analyse how meanings are construed across the expression, content and context planes of visual abstracts. Secondly, the study examines visual abstracts in the light of ‘intersemiotic translation as resemiotisation’ for specialist needs in the BMJ and NEJM as a case study. Since these issues appear to be less often addressed in Translation Studies, the article strives to make a contribution towards the development of multimodal translation studies. As Tuominen et al. (2018: 5) have also highlighted, addressing “questions such as the interplay of verbal and non-verbal modes in the construction of meaning” is one way in which “multimodal translation studies […] demonstrate its distinct identity.”

2. Theoretical Framework

The present study is based on the metafunctional principle in Halliday’s (1978) theory of Systemic Functional Linguistics. The principle was extended by Kress and van Leeuwen (1996) to include all other semiotic systems. The social semiotic approach to translation, in fact, considers all modes to operate as meaning-making systems since metafunctions as “a higher order of meaning […] are in play in communication whatever mode is used” (Jewitt 2006: 19, original italics). Accordingly, each semiotic mode serves the ideational, the interpersonal and the textual metafunctions in order to operate as a full system of communication (Kress and van Leeuwen 1996: 40-42). These three metafunctions organise the elements of semiotic modes through which they realise their meanings. These can be succinctly defined as follows:

ideational meaning […] consists of experiential meaning to structure experience of the world and logical meaning to make logical connections in the world; interpersonal meaning to enact social relations and create a stance towards the world; textual meaning to organise experiential, logical and interpersonal meanings (O’Halloran et al. 2016: 204, original italics).

While the Systemic Functional approach is not new to multimodal translation studies, it has not yet been applied to the analysis of intersemiotic translation in medical visual abstracts to the best of the author’s knowledge. The approach appears promising as it supports the investigation of the conditions that “reveal which particular metafunctional meaning is likely to emerge from choices within systems” (Lim 2004: 232) across the expression, content and context planes (Liu and O’Halloran 2009: 371). Furthermore, the metafunctional principle “provides a basis for examining the functionalities of semiotic resources and for analysing the ways in which semiotic choices interact in multimodal discourses to fulfil particular objectives” (O’Halloran 2008: 444).

On these grounds, it is worth looking briefly at the potential functionalities of semiotic resources in making meaning in visual abstracts. Different compositional meanings can be realised in the layout of visual abstracts to translate original linguistic meanings. Semiotic choices can be made from both verbal and non-verbal systems. For instance, images, icons, numbers, graphs, as well as colour (Kress and van Leeuwen 2002) and typography (e.g. font style, size) (van Leeuwen 2005a) have meaning-making potential on the expression plane and significantly affect the content-context plane. The representational and interactive meanings they construe are connected to one another through the three interrelated systems of information value, salience and framing (Kress and van Leeuwen 1996: 177). As a key feature of semiotic space, information value can be identified according to the arrangement of elements along the horizontal and/or vertical axis of compositional zones, as well as in the centre-margin layout of these composite visuals. Horizontally, “Given”, or familiar information is situated on the left, while “New”, or unknown information lies on the right; vertically, general (“ideal”) information is placed at the top, while specific (“real”) information is positioned at the bottom; more meaningful information may also be situated in the “centre”, whereas less relevant content is arranged at the margins of multimodal texts.

Regardless of the position, “salience” helps establish “a hierarchy of importance among the elements, selecting some as […] more worthy of attention than others” (Kress and van Leeuwen 1996: 201).

Compositional elements can further acquire meaning through framing. Units of information can be strongly framed by lines to indicate “segregation”, or differentiation of meaning, by empty spaces to mark “separation” of meanings (van Leeuwen 2005b: 15-17), or even by different colours as a framing device (Kress and van Leeuwen 2002). The absence of framing, instead, signals major connectedness among compositional elements.

On the interrelated expression-content planes of visual abstracts, representational and interactive meanings of the subject matter (experiential) and of expert-to-expert interaction (interpersonal) can be translated both across the verbal-verbal and verbal-non-verbal strata. Original experiential meanings of the research experience may be translated lexico-semantically through sense relations, such as repetition, synonymy, antonymy, hyponymy, meronymy, and collocation (Royce 2007: 103). They may also be resemiotised by expanding their original significance. By extending the systemic functional “system of expansion” to multimodality, it is possible to note that this can be accomplished in different ways. Meanings can be remediated by drawing on the two subsytems of “expansion by elaboration” and “expansion by extension” and their related subtypes (Halliday and Matthiessen 2004: 543-544). Elaboration is realised when meaning is resemiotised in further detail through exemplification or clarification; extension may occur either through “addition” by adding something new, or through “variation”, when meaning is changed by other semiotic resources. Changes occur when original meanings are either replaced (replacive type) or omitted (subtractive type).

Similarly, interpersonal meanings may be translated both intra- and intersemiotically. Verbal meanings arising mainly from the system of Mood may be translated to retain or change original interpersonal meanings (e.g. address via commands, statements, offers, questions). Instead, the “intersemiotic MOOD, or the ways that both the modes address the viewers/readers” allows the translator to either maintain “identical forms of address” or decide to create relations of “intersemiotic reinforcement of address” (Royce 2007: 69-70).

While retaining and changing meaning poses challenges to all types of translation processes, the resemiotisation of abstracts raises specific concerns. It must be borne in mind that written abstracts present a vertical layout which “asks that the reader follow the pre-given complexities of the syntactic reading path” (Kress 2010: 58). They are structured according to the conventional scientific IMRaD format which encompasses four main sections: Introduction, Methods, Results and Discussion. Conversely, visual abstracts may have different formatting styles as no ubiquitous standard exists so far. Moreover, written abstracts generally tend to mirror the original content of research articles without adding any further information; by contrast, visual abstracts are likely to undergo intersemiotic expansion since they condense complex textual content into concise visual summaries. In addition, while both types of abstracts serve the common purpose of concisely reporting research findings within the scientific community, the visual abstract is also meant to meet the expert viewer’s precise needs, as aforementioned.

Against this backdrop, the first aim of this study is to analyse which original meanings are resemiotised on the expression, content and context planes of visual abstracts. It is worth noting that intersemiotic translation cannot be interpreted context-independently since “Intersemiotic Cohesion can be regarded as the ongoing process of contextualization” (Liu and O’Halloran 2009: 385, original italics). In this regard, Lim emphasises the importance of considering “contextualizing relations […] whenever two modalities operate in a multimodal text”: when “the meaning of one modality seems to ‘reflect’ the meaning of the other through some type of convergence, the two resources share co-contextualizing relations”; when “the meaning of one modality seems to be at odds with or unrelated to the other, their semantic relationship is one that diverges”, whereby “the resources share re-contextualizing relations” (2004: 239, original italics).

On the whole, it appears evident that, although “the expression stratum for visual imagery is designed to parallel the expression stratum for language” (O’Halloran 2008: 451), the two systems are different in nature.

Different meanings can be construed also on the content plane through the “metaphorical shifts [which] occur where the functional status of elements is not preserved and new elements are introduced […]” (O’Halloran 2008: 453). Furthermore, “re-contextualizing relations” are likely to determine changes, for instance, in the context of situation (register) stratum and in the higher stratum of the context of culture (genre) (Martin 1992). In this latter case, the research abstract has conventionally been considered a “part-genre of research articles” (Swales 2004: 239). Nowadays, however, it appears to be a specialist genre in its own right, especially owing to its new abstracting function of facilitating online information retrieval. The visual abstract, instead, is likely to show “a propensity for innovation, exploitation and manipulation” (Bhatia 1997: 367) due to the creative intersemiotic translation process it undergoes. Accordingly, the second main aim of this article is to examine how meanings are remediated to meet the expert viewer’s precise needs.

3. Methodology

3.1 Materials

A total of twenty visual abstracts were collected and used as sample materials for this study. Half of these were taken from the BMJ website (www.bmj.com/content/bmj-visual-abstracts) and the other half from the NEJM website (www.nejm.org/multimedia/visual-abstracts?page=4#DO-list) (see Appendix) to ensure increased data representativeness and yield more robust findings. The selection was based on the following two conditions targeting present research purposes: 1) only visual abstracts accompanied by their original written texts were considered. Consequently, any stand-alone visual samples were discarded; 2) visual abstracts dealing with different pathologies and covering the entire timespan of the journal’s intersemiotic translation practice were selected to ensure a richer range of dynamics. Based on this latter criterion, all selected samples covered a range of different diseases and were published by the BMJ between 2018 and 2021 and by the NEJM between 2019 and 2021. Samples were then divided into two separate sets in ascending chronological order and respectively labelled BMJ#1-10 and NEJM#1-10 for further analytical identification.

3.2 Systemic functional-multimodal discourse analysis (SF-MDA)

A systemic functional-multimodal discourse analysis (SF-MDA) was performed on the sample texts. The SF-MDA approach is situated within Systemic Functional Theory which privileges meaning making (Halliday 1978). While the systemic principle was initially applied only to language, “much of multimodality is itself based closely on […] Systemic Functional Linguistics (SFL)” (Ledin and Machin 2019: 497). The systemic principle has, in fact, been adapted in multimodality to refer to:

the inextricable relationship between multimodal semiotic resources and society in SF-MDA [to] describ[e] the underlying organization of semiotic resources which enable the resources to be used for different purposes. […] The options in the systems represent the paradigmatic choices from which selections are made in multimodal texts (O’Halloran and Lim 2014: 138).

Instead, the functional principle refers to:

the meaning potential of the various semiotic resources […] metafunctionally in SF-MDA. The metafunctional organization of meanings is particularly useful because it provides a common set of fundamental principles to compare semiotic resources and the meanings which arise when semiotic choices integrate in multimodal texts. That is, the organization of metafunctional meanings offers a unifying platform for studying semiotic resources and their inter-semiotic relations (O’Halloran and Lim 2014: 140).

As Lim (2011: 8) further points out, “the approach presents a rigorous and detailed analysis of the choices made in the text derived from a system network of meaning options on the different ranks & scales and across the various metafunctions”. This fair practice of detailed analysis justifies the small number of texts chosen for the present research. Concerns about the possible limitations of the systemic functional method mainly regard the procedure of conducting separate analyses on the different semiotic modes rather than using a combined approach (Tuominen et al. 2018). It is argued that SF-MDA investigates semiotic resources also “inter-dependently, in relation with the meanings made with the other semiotic resources” (Lim 2011: 88), and that the present analysis is performed on the expression, content and context planes of visual abstracts bearing in mind their interrelatedness.

Moreover, the fine-grained analysis is circumscribed to the variables of resemiotised meanings in order to pinpoint their significant clustering within and across modes. Based on the empirical evidence provided from the sample texts, the discussion on the SF-MDA results in Section 4 thus refrains from formulating any irrefutable generalisations about intersemiotic translation and multimodal meaning construction.

3.3. Procedure

The analysis was carried out through the following steps. The expression plane of each visual abstract was first examined by looking at how original metafunctional meanings were translated through different semiotic systems based on the most relevant choices represented. Compositional elements were coded according to the systems of “information value, salience and framing” (Kress and van Leeuwen 1996). The ways in which significant elements recreate original metafunctional meanings was then considered. This step was followed by the analysis of the content plane where semiotic choices were examined for possible patterns of metafunctional meanings both at the verbal and non-verbal strata. Lexico-semantic relations indicating meaning retention/change were identified at the verbal stratum; intersemiotic metafunctional meanings were coded according to the systemic functional subsytems of expansion by elaboration or by extension and their related subtypes described in Section 2. The systemic functional systems of linguistic mood and intersemiotic mood were drawn upon to semantically interpret the intersemiotic translation of interpersonal meanings at both verbal and non-verbal levels. Based on all these data findings, significant “contextualizing relations” (Lim 2004) were identified on the contextual plane to seek whether they affected the contextual strata of register and genre.

4. Results and discussion

In this section, the results of the SF-MDA analysis are reported for each of the three planes of visual abstracts.

4.1 Intersemiotic translation on the expression plane

All BMJ samples present a vertical format like their corresponding written abstracts. The salient “typographical meaning” (van Leeuwen 2005a) of abstract titles is maintained through the use of the same bold font style and larger font size. The choice of colour is found to significantly affect original metafunctional meanings, although differences are recorded between the visual abstracts published in 2018 and those issued between 2019-2021. The earlier samples (BMJ#1-3) present an identical grey background against which “colour differentiation” (Kress and van Leeuwen 2002) is used to distinguish different content (Figure 1).

Figure 1. Colour Differentiation for Different Content Meaning (BMJ#2)
(source: C. E. Firanescu et al., BMJ 2018;361:k1551; CC-BY-NC)¹.

As in Figure 1, three main colours are used for the structural organisation: blue hue is chosen for the section referring to ‘research participants’; magenta and green-cyan hues are employed in the flowcharts which summarise medical ‘interventions’; white is selected for a third section on ‘primary outcomes’. Colour differentiation thus contributes to the resemiotisation of the original IMRaD format. The resulting tripartite division appears to resemble the infographic structure: introduction, key message, and conclusion (Krum 2014).

Semantically, the three specific colours also seem to be intentionally picked out to highlight experiential meanings which are generally implicit in written abstracts. The blue hue as a colder colour of the spectrum seems to suggest scientific reliability, given the association of “the blue end [of the colour scale] with cold, calm, distance” (Kress and van Leeuwen 2002: 357); the “hybrid” magenta and green-cyan hues recall the “postmodern idea” that “hybridity is positively valued” (Kress and van Leeuwen 2002: 356). Hence, these cyan hues are adopted to highlight positive experiential meanings of medical interventions. The “de-saturated” white signals “neutralized feeling” (Kress and van Leeuwen 2002: 356) so as to connote primary outcomes with a sense of objectivity. In other words, colour differentiation translates the original underlying experiential meaning of positive, reliable and objective scientific research. This further suggests that, as a new means of knowledge dissemination, visual abstracts need to construct their credibility more overtly than written abstracts do since these have long been acknowledged by the scientific community.

Framing is also a key meaning-making device in the 2018 abstracts since all the coloured blocks of information are framed and separated from one another via empty spaces to reinforce the distinctiveness of experiential meanings (see Figure 1). The empty spaces, however, are all coloured by the common grey background, which thus acts as a semiotic cohesive device connecting the separate content blocks into one textual unit. Furthermore, the compositional elements are arranged in a non-sequential order. This encourages a non-linear reading/viewing path, thus allowing the reader/viewer to construe an individual interpretation of the multimodal text based on single interests. Hence, this arrangement creates an ‘additive’ relationship with the original written abstracts by adding interpersonal meaning which is missing on their expression plane.

A dynamic change occurs, however, in all the 2019-2021 abstracts (BMJ#4-10). Their different structural layout together with the use of new semiotic resources signal a diverse intersemiotic translation process.

Although the layout continues to be displayed in the same vertical format, it is now designed to include five new sections - summary, study design, population, comparison and outcomes (Figure 2). These vaguely recall the conceptual construct of the IMRaD format formerly missing in the 2018 samples.

Figure 2. The New BMJ Structural Layout (BMJ#7)
(source: C. Welsh et al., BMJ 2020;368:m336; CC-BY-NC).

All section headings are given typographical meaning through the use of bold larger font-size characters, thus replicating the same conventional practice. Section headings are horizontally positioned on the left side, thereby maintaining the same value of familiar information found in the monomodal texts. The new content placed on the right acquires major information value through the use of images. These resemiotise complex experiential meanings through ‘exemplification’ as a meaning-making device belonging to the system of ‘expansion by elaboration’ (see Section 2). Visual exemplifications allow the rapid retention of elaborate information and thus reduce cognitive load.

Interpersonal meaning is further construed through a combination of new meaning-making resources. Among these, isotypes, or pictograms of people or things, acquire major importance when placed before the four section headings as they enact the textual metafunction of organising interpersonal meaning (see Figure 2). These isotypes, in fact, visually ease the viewer/reader in following the flow of information by offering an alternative semiotic vocabulary. The initial summary heading is, instead, preceded by an open quotation mark. This typographical resource metaphorically translates an ‘open’ dialogue between the author and the reader/viewer.

As for colour change, the blue hue is now used to give prominence to the isotypes and verbal headings and to replace the grey background of former abstracts. Magenta and green-cyan hues are used mainly in the comparison section to highlight positive experiential meanings of alternative medical interventions. In this case, framing is essentially achieved “by discontinuities of colour” (Kress and van Leeuwen 1996: 204). The same colour is used to “create links and coherence between parts” (Ledin and Machin 2020: 91), for instance, between the comparison section and the related outcomes in Figure 2. The predominance of the ‘colder’ blue hue seems to underline the scientific validity and reliability of visual abstracts and therefore carries additional interpersonal meaning.

In the NEJM set, sample abstracts present a horizontal rather than a vertical format (Figure 3).

Figure 3. The NEJM Structural Layout (NEJM#7)
(source: A.L. Klein et al., 10.1056/NEJMoa2027892; Massachusetts Medical Society, 2021)².

This design suggests that they are conducive to social media and thus implies that they strive to disseminate research across different digital environments. All samples present a less rich combination of semiotic choices mainly sourced from the visual, verbal and numeral systems. They do not, in fact, display flowcharts or isotypes as the BMJ abstracts do. Typographical features (bold, larger font) are not only used for abstract titles, but also to give salience to keywords and numeral data highlighting important research-based (experiential) meanings. Contents are arranged in framed blocks, which are connected to one another by thin white lines, thus indicating weak disconnectedness. Generally, the first block placed at the top left organises information about ‘the cohort of patients’ involved in the research. Here, larger anatomical images are captioned by enlarged numerals indicating cohort numbers and by smaller brief scripts about the patients’ pathological conditions. The other top blocks arrange information about alternative therapeutic interventions. Here, salience is visually attributed to pharmaceutical images (e.g. vial, syringe) and to the related drug/treatment names to convey specialised or ‘real’ research meanings, whereas the verbal resources beneath provide more ‘general’ information and serve the textual metafunction of orienting the reader/viewer. The remaining framed blocks positioned beneath in Figure 3 disseminate research results with percentage numbers underlining important findings.
This particular layout allows the readers/viewers to determine their own reading path as in the case of the 2018 BMJ abstracts. This flexibility is further ensured by the absence of abstract section headings.

The interpersonal metafunction is also privileged by ‘colour modulation’ as a key semiotic feature of all NEJM abstracts. As van Leeuwen (2016: 110) explains, “modulation is the scale from flat, plain colour to varied tints and shades.” NEJM abstracts are characterised by the use of a rich variety of pastel shades to represent verbal, visual and numerical compositional elements. Semantically, these modulated colours are meant to convey the “perceptual truth” of the medical abstracts as opposed to “the truth of flat colour [which] is an abstract truth” (Kress and van Leeuwen 2002: 357). Besides making the NEJM texts aesthetically appealing, colour modulation thus metaphorically translates the empirical truth of scientific content just as colour differentiation in the BMJ texts claims scientific validity and reliability. Since modulation is related to modality (Kress and van Leeuwen 1996, 2002), it also contributes to construing interpersonal meaning through the intersemiotic mood relation of “reinforcement of address” (Royce 2007: 70).

Based on these findings, resemiotisation on the expression plane operates mainly through the compositional order, the use of framing, colour and new semiotic resources. In both 2018 BMJ and NEJM abstracts, the compositional order recreates the original IMRaD structure. In the BMJ 2019-2021 samples, the attempt to align the new format with the original IMRaD structure is disrupted by the subdivision in five abstract sections with different headings in place of the original four.

Framing is also found to differ: strong frames in the 2018 abstracts emphasise the distinctiveness of experiential meanings; weaker ones in the NEJM samples indicate major textual cohesion; colour discontinuity as a framing device in the 2019-2021 samples appears to give prominence to more specific experiential meanings, while using these to simultaneously construe interpersonal meanings.

Colour differentiation in the 2018 abstracts, colour choice in the 2019-2021 samples and colour modulation in the NEJM ones all serve the textual metafunction of orientation. However, “the system of ORIENTATION is expanded by the addition of a systemic contrast in MANIFESTATION between ‘explicit’ and ‘implicit’” (Halliday and Matthiessen 2004: 626, original capitalisation). Experiential meanings of research validity and reliability and of the empirical truth of scientific content are rendered more explicitly through the semiotic code of colour. As a result, “interpersonal projection [expansion] […] is made explicit […] by ‘co-opting’ ideational [experiential] resources to do interpersonal service” (Halliday and Matthiessen 2004: 626).

Finally, the changes created in most BMJ abstracts by the addition of new semiotic resources — the isotype and the typographical symbol of an open quotation mark — also provide evidence of the construal of new interpersonal meaning which was originally missing.

Overall, the expression plane appears to be resemiotised to better highlight key experiential meanings and to skilfully exploit these either to remediate original implicit interpersonal meanings or to add missing ones.

4.2 Intersemiotic translation on the content plane

At the lexicogrammar stratum, patterns of lexical-lexical and lexical-visual ties were found to respectively mediate intramodal and intermodal cohesion between the written and visual abstracts. At the discourse stratum, verbal ties were primarily realised through the sense relations of lexical repetition and synonymy in the BMJ samples; lexical repetition was mainly used in the NEJM texts to reproduce original keywords. In both cases, these relations maintain experiential, or topical meanings across the two text types. In sample BMJ#9, for instance, the original title is significantly modified (Figure 4).

Figure 4. The sense relations of lexical repetition and synonymy at the discourse stratum
(source: BMJ 2020;370:m2516; CC-BY-NC).

Both repetition and synonymy operate to retain meanings: the lexical items diagnostic accuracy, covid-19, serological tests maintain the exact same wording used in the original text. The lexical item antibody testing, i.e. the testing of blood serum to detect the presence of antibodies against a specific antigen, is also scientifically called serological test. Hence, it stands in a sense relation of synonymy with the original term serological tests.
However, intrasemiotic expansion also occurs here since serological tests is translated as three serological tests. This basic verbal mechanism of expansion by extension through addition becomes more elaborate at the intersemiotic level (Figure 5).

Figure 5. An intersemiotic additive relation
(source: BMJ 2020;370:m2516; CC-BY-NC).

Content on the serological tests is presented in three separate information blocks and semiotic resources on the expression plane (frames and colour differentiation) help give more salience to the experiential meanings which have been added. The original concept is also expanded by the addition of isotypes representing the different diagnostic tools used in the three types of tests. The resulting “intersemiotic additive relations” (Liu and O’Halloran 2009) created by these semiotic resources can be classified under the category of “positive addition” within the system of expansion by extension (Halliday and Matthiessen 2004: 543). They, in fact, positively add new experiential meanings to the original one to ease the reader’s comprehension.

Moreover, expansion by extensionoperates not only through addition, but also concurrentlythrough variation at the discourse stratum. For example, the original subtitle in Figure 4 undergoes subtractive variation. It is, in fact, omitted in the visual abstract where the choice of privileging more significant experiential meaning (three serological tests) is made.
In its replacive form, expansion by extension through variation was particularly found to operate at the verbal stratum of the latest BMJ samples (#4-10). Original IMRaD section headings are replaced (see Figure 2) and content meanings are recreated by the new structural order. For instance, the content of the original introductory section in BMJ#4 (Figure 6) is replaced at the beginning of the visual abstract with content taken from the original conclusions section.

This replacive variation is understood to serve the new purpose of enabling the readership to immediately gain a quick overview of the published research findings. Content variation also operates in combination with the‘additive’ open quotation mark on the expression plane to reinforce interpersonal meaning at the intersemiotic level.

Figure 6. Expansion by extension through variation: replacive form at the discourse stratum
(source: BMJ 2019;364:l236; CC-BY-NC).

Replacive variation was found to function also through “the system of MOOD” as “an interpersonal system” (Halliday and Matthiessen 2004: 33).
For instance, the indicative mood in the original title in BMJ#8 (Figure 7) is changed to the interrogative mood to strengthen address via negotiation in dialogue. This shows how “interpersonal projection […] has thus been realized as if it was an ideational [experiential] projection” (Halliday and Matthiessen 2004: 630).

Figure 7. Variation through the system of MOOD for interpersonal meaning
(source: BMJ 2020;369:m696; CC-BY-NC).

The combined mechanisms of subtractive variation and intersemiotic expansion by elaboration were found to characterise all NEJM abstracts.
In NEJM#7 (see Figure 3), for instance, the original clause Rilonacept led to rapid resolution is first changed through subtractive variation, resulting in the omission of led to rapid resolution. The remaining keyword Rilonacept is given greater value to emphasise its importance in treating recurrent pericarditis.

The loss of important verbal information appears to be somewhat compensated concurrently by the mechanism of intersemiotic expansion by elaboration. This type of relation implies that “some element is re-presented” by means of an “elaborating group” which “particularize[s]” meaning through “exposition, or the ‘i.e.’ relation” (Halliday and Matthiessen 2004: 540). In Figure 3, enlarged coloured icons in the upper frames operate as the elaborating group to attract the viewer’s attention. While the icons of a rilonacept vial and a syringe denote the “‘i.e.’ relation” typical of elaboration through exposition, the actual illustrations compensate for the original meaning in a simplistic manner. Salience is, however, attributed to the drug’s name in the upper central block typographically and especially through the use of the red hue. This colour choice appears to be extremely significant, given that “in therapeutic applications […], patients could be […] stimulated by red” (Kress and van Leeuwen 2002: 354; 357), and thereby metaphorically accept the rapid resolution of Rilonacept therapy. While it is left to the viewer to infer this kind of relationship, this example shows how verbal subtractive variation and intersemiotic elaboration through exposition work together to translate essential original experiential meanings and adapt them to the visual format.

Based on these findings, the content of BMJ abstracts is mainly resemiotised through verbal subtractive variation and intersemiotic addition which simultaneously omit less relevant original experiential meanings and expand more significant ones. However, replacive variation recreates content in recent BMJ abstracts due to their different structural organisation. Experiential meanings are essentially realised to serve the interpersonal metafunction. For instance, Figure 7 shows how the interrogative mood, as part of “an interpersonal system”, is exploited to establish author-reader/viewer interaction since it “involves selecting an element that indicates the kind of answer required” (Halliday and Matthiessen 2004: 49, 102).

In the case of NEJM texts, significant changes stem also from the use of verbal subtractive variation, although it is here combined with intersemiotic elaboration through exposition (see Figure 3) instead of expansion by extensionas in the BMJ abstracts (see Figure 5). This kind of combination has the effect of noticeably reducing former experiential meanings to keywords/expressions in favour of visual representations which are meant to compensate for verbal loss. However, the simple intersemiotic resources deployed do not reflect the strategy of compensation. The low information density resulting in all the NEJM visual abstracts rather suggests that the translator has chosen to mediate the highlights of each research study. This clearly reflects the ultimate aim of boosting memorability to meet this new need of the journal’s readership.

4.3 Intersemiotic translation on the context plane

Register and genre as the higher strata of meaning in the visual abstracts were examined at this level. The dynamics on the expression and content planes were found to significantly affect the register of the visual abstracts. In particular, the joint mechanism of verbal subtractive variation and expansion by extension or elaboration seem to create “re-contextualizing” relations (Lim 2004) leading to register variation.

Changes in the formal professional register of written abstracts are produced through “contractions and ‘fast language’” (Giménez-Moreno 2011: 19), as shown in Figure 8 where the verbless clause on the left contracts the original statement below (NEMJ#8) and fast language, characterised by keywords and low information density, is used on the right to translate the original statement beneath (BMJ#2).

Figure 8. Variation of the original formal professional register through contraction and fast language
(sources: F. Comradie et al., 10.1056/NEJMoa1901814; Massachusetts Medical Society, 2020; BMJ 2018;361:k1551; CC-BY-NC).

These variations suggest that space constraint as a contextual factor of visual abstracts affects the original formal register. While the core amount of distinctive specialised meanings is still provided, a more neutral register springing from the use of concise language is now employed. This move is taken further in the BMJ#4-10 samples through intersemiotic expansion. For instance, the use of the open quotation mark (Figure 2) suggests “the closeness” of informal registers (cf. Giménez-Moreno 2011: 19).

The effect of intersemiotic elaboration through exposition in the NEJM visual abstracts produces a greater shift away from the original formal register. The ‘fast language’ of keywords together with low information density produced through exposition appear to show a major inclination towards the use of a professional neutral register (see Figure 3). As a result, this register is now merged with a “social register” which is greatly determined by the choice of nonverbal resources acting as “the clustering of semantic features according to situation type” (Halliday 1978: 111). Its occurrence can be justified by the online context of visual abstracts since digital environments may be easily considered “social open settings” (Giménez-Moreno 2011: 18).
Hence, verbal and nonverbal choices are coordinated by register variation to retain scientific content, enact interpersonal relationship and organise the visual abstracts according to their online social context.

Since “register function[s] as the expression form of genre” (Martin 1992: 495), the move from one register variety to another further determines a genre shift at the highest stratum of the context plane. Montalt Resurrecció and González Davies (2014: 163) point out that “genre shifts as part of the translation process normally occur from more to less specialized genres, and not the other way round.” Thus, it appears that “the code elements [are] selected in such a way that the target-text effect corresponds to the intended target-text functions” (Nord 1997: 93). From this functional perspective, major changes are observed in the generic structure of written abstracts. In the NEMJ and the 2018 BJM visual abstracts, the conventional IMRaD format is modified by a new three-part organisation: a more concise title which is meant to appeal to the expert readership; key content whose visual representation is intended to favour comprehension and retention, and a closure with a hyperlink to the full research article as a follow-up.

Hence, the genre-shifting strategies of simplifying the structural organisation, synthesising verbal information, expanding key content visually and easing comprehension all determine changes governed by the new functional purpose of visual abstracts. This “transgeneric translation” (Montalt Resurrecció and González Davies 2014: 123) results in the “mixing of generic norms” (Bhatia 2010: 35) of abstracts with those of the tripartite structural organisation of infographics.

The 2019-2021 BMJ samples also make use of the above genre-shifting strategies. Here, semiotic resources are additionally chosen to facilitate intersemiotic mood relations as these texts are further modified by the “particular semantic features of an […] interpersonal kind” (Halliday 1978: 145). This is manifested through the embedding of the speech genre, which thus results in the ‘bending’ of the norms of the written abstract genre. On the whole, transgeneric translation appears to be grounded in the appropriation of generic resources through ‘interdiscursivity’:

Interdiscursivity […] refers to more innovative attempts to create various forms of hybrid and relatively novel constructs by appropriating or exploiting established conventions or resources associated with other genres and practices. Interdiscursivity thus accounts for a variety of discursive processes and professional practices, often resulting in ‘mixing’, ‘embedding’, and ‘bending’ of generic norms in professional contexts (Bhatia 2010: 35).

5. Conclusions

The present study has explored the relatively new practice of producing visual abstracts in online medical journals to gain an understanding of how intersemiotic translation operates in this professional context. The SF-MDA analysis examined the most significant semiotic resources and meaning-making systems at work in these multimodal texts, revealing the ways in which they simultaneously operate to retain and change meanings at the expression, content and context levels. In addressing the first research question, the study shows that the choice of particular verbal and visual resources and their semantic relations is made to ensure that original key experiential meanings are mapped onto different semiotic elements on the interrelated expression and content planes. However, experiential meanings were also found to function as a means for privileging new interpersonal meanings.

As for the second research question, findings show that the expression plane was purposed to the resemiotisation of original implicit interpersonal meanings or to the creation of new ones through expansion. The intrinsic empirical truth value, as well as the scientific validity and reliability of original abstracts were emphasised, thus also allowing visual abstracts to gain their own credibility. Moreover, replacive variation produced changes in content meanings; subtractive variation was not compensated by other semiotic resources as the ultimate aim was to boost memorability through low information density. The system of mood was particularly exploited intersemiotically to enhance author-reader/viewer interaction.

The dynamic process of “exploitation and manipulation” (Bhatia 1997: 367) was found to continue through recurrent “re-contextualizing” relations (Lim 2004) leading to register variation and to significant genre shifts through transgeneric translation, thus foregrounding the new communicative purpose of visual abstracts as a key genre-defining element.

While noticeable meaning-making differences were observed in the BMJ and the NEJM sample sets, they were all found to privilege the interpersonal purposes of allowing quick access to research-based knowledge online, facilitating information retention and reducing cognitive load.

On the whole, the social semiotic approach to translation taken in this article has highlighted “the nature of specific clusters or types of translation” (Tymoczko 2007: 111), such as intersemiotic translation in medical research abstracts, the translator’s “transformational creativity” (Aguiar et al. 2015) and the importance of reconfiguring the study of translation in a multidimensional semiotic paradigm. The meaning-generating and dynamic nature of visual abstracts has also helped underline the relevance of multimodality in translation and the fact that intersemiotic translation has implications that go well beyond translation proper. In the rapidly evolving semiotic landscape, this multimodal genre deserves further investigation beyond the two cases considered once the practice of intersemiotic translation as resemiotisation gains ground in online journals in other fields of knowledge.

References

Aguiar, Daniella, Pedro Atã and João Queiroz (2015). “Intersemiotic translation and transformational creativity.” Punctum 1(2), 11-21.
Bhatia, Vijay K. (1997). “The power and politics of genre.” World Englishes 16(3), 359-371.
Bhatia, Vijay K. (2010). “Interdiscursivity in professional communication.” Discourse & Communication 21(1), 32-50.
Giménez-Moreno, Rosa (2011). “Register variation in electronic business correspondence.” International Journal of English Studies 11(1), 15-34.
Halliday, Michael A.K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Edward Arnold.
Halliday, Michael A.K. and Christian M.I.M. Matthiessen (2004). An Introduction to Functional Grammar (3rd ed.). London: Hodder Arnold.
Iedema, Rick (2003). “Multimodality, resemiotisation: extending the analysis of discourse as multi-semiotic practice.” Visual Communication 2(1), 29-57.
Jewitt, Carey (2006). Technology, Literacy and Learning: A multimodal approach. London/New York: Routledge.
Kress, Gunther (1998). “Visual and verbal modes of representation in electronically mediated communication: the potentials of new forms of texts.” Ilana Snyder (ed.) (1998). Page to Screen: Taking Literacy into the Electronic Era. London/New York: Routledge, 53-79.
Kress, Gunther (2010). Multimodality: A social semiotic approach to contemporary communication. London/New York: Routledge.
Kress, Gunther and Theo van Leeuwen (1996). Reading Images: The Grammar of Visual Design. London: Routledge.
Kress, Gunther and Theo van Leeuwen (2002). “Colour as a semiotic mode - notes for a grammar of colour.” Visual Communication 1(3), 343-369.
Krum, Randy (2014). Cool Infographics: Effective Communication with Data Visualization and Design. Indianapolis, IN: John Wiley & Sons.
Ledin, Per and David Machin (2019). “Doing critical discourse studies with multimodality: from metafunctions to materiality.” Critical Discourse Studies 16(5), 497-513.
Ledin, Per and David Machin (2020). Introduction to Multimodal Analysis (2nd ed.). London: Bloomsbury.
Lim, Fei Victor (2004). “Developing an integrative multi-semiotic model.” Kay O’Halloran (ed.) (2004). Multimodal Discourse Analysis: Systemic Functional Perspective. London: Continuum, 220-246.
Lim, Fei Victor (2011). A systemic functional-multimodal discourse analysis approach to pedagogic discourse. PhD Thesis. National University of Singapore.
Liu, Yu and Kay O’Halloran (2009). “Intersemiotic texture: analyzing cohesive devices between language and images.” Social Semiotics 19(4), 367-388.
Martin, James R. (1992). English Text: System and structure. Amsterdam/ Philadelphia: John Benjamins.
Martin, Lynsey J. et al. (2019). “Exploring the role of infographics for summarizing medical literature.” Health Professions Education 5(1), 48-57.
Montalt Resurrecció Vicent and Maria González-Davies (2014). Medical Translation Step by Step: Learning by Drafting. London/New York: Routledge.
Nord, Christiane (1997). Translating as a Purposeful Activity: Functionalist Approaches Explained. Manchester: St Jerome.
O’Halloran, Kay (2008). “Systemic functional-multimodal discourse analysis (SF-MDA): constructing ideational meaning using language and visual imagery.” Visual Communication 7(4), 443-475.
O’Halloran, Kay and Fei Victor Lim (2014). “Systemic Functional Multimodal Discourse Analysis.” Sigrid Norris and Carmen Daniela Maier (eds) (2014). Interactions, Images and Texts: A Reader in Multimodality. Boston/Berlin: De Gruyter Mouton.
O’Halloran, Kay, Sabine Tan and Peter Wignell (2016). “Intersemiotic translation as resemiotisation: a multimodal perspective.” Signata 7, 199-229.
O’Sullivan, Carol (2013). “Introduction: Multimodality as challenge and resource for translation.” JoSTrans, The Journal of Specialised Translation 20, 2-14. https://jostrans.org/issue20/art_osullivan.pdf.
Poulsen, Søren Vigild (2022). “Website interactivity as representation of social actions? Developing a social semiotic discourse approach to interaction design.” Ilaria Moschini and Maria Grazia Sindoni (eds) (2022). Mediation and Multimodal Meaning Making in Digital Environments. London: Routledge, 66-82.
Royce, Terry (2007). “Intersemiotic complementarity: a framework for multimodal discourse analysis.” Terry Royce and Wendy Bowcher (eds) (2007). New Directions in the Analysis of Multimodal Discourse. Mahwah, NJ: Lawrence Erlbaum Associates, 63-109.
Stahl-Timmins, Will, Jonathan Black and Paul Simpson (2019). “Pragmatic evaluation of The BMJ’s visual abstracts.” Information Design Journal 25(1), 101-109.
Swales, John (2004). Research Genres: Explorations and Applications. Cambridge: Cambridge University Press.
Troqe, Rovena (2015). “On the concept of translation: A perspective based on Greimassian semiotics.” Semiotica 204, 33 - 59.
Tuominen, Tiina, Catalina Jiménez Hurtado and Anne Ketola (2018). “Why methods matter: approaching multimodality in translation research.” Linguistica Antverpiensia: New Series – Themes in Translation Studies 17, 1-21.
Tymoczko, Maria (2007). Enlarging Translation, Empowering Translators. Manchester: St. Jerome.
van Leeuwen, Theo (2005a). “Typographic meaning.” Visual Communication 4, 137–143.
van Leeuwen, Theo (2005b). Introducing Social Semiotics. London/New York: Routledge.
van Leeuwen, Theo (2016). “A social semiotic theory of synaesthesia?: a discussion paper.” Hermes-Journal of Language and Communication in Business 55, 105-119.
van Leeuwen, Theo (2017). “Multimodal literacy.” Viden om Literacy 21, 4-11.

Appendix – Abstract Sources

All sources were consulted 23.03.2021.

BMJ (2018-2021)

NEJM (2019-2021)

Biography

Anna Franca Plastina is Associate Professor of English Language and Translation at the University of Calabria, Italy. Her interests include multimodal translation and discourse analysis of specialised language in the new media. She coordinates the inter-university CLAVIER research unit at the University of Calabria. Her most recent publications include the chapter on Healthy Pic Hashtagging in Twitter: The Role of Infographics in #AntibioticGuardian (Routledge Studies in Multimodality Book Series, 2022).
ORCID: http://orcid.org/0000-0002-8259-5161
E-mail: annafranca.plastina@unical.it

Notes

Note 1:

BMJ images herein are published under the CC-BY-NC license which permits their non-commercial use. Sources have been cited according to the BMJ.
Return to this point in the text

Note 2:
NEJM images are reproduced without need for formal permission for research purposes in accordance with the policy of the Massachusetts Medical Society, Publisher of the New England Journal of Medicine, and the Fair Use policy (for culture, education, research) (Section 107 of the Copyright Act).
Return to this point in the text