Benchmarking Machine Translation Efficacy for Teaching EAP Reading Comprehension Skills

Mirzaeian, V.; Maghsoudi, M.

doi:10.22061/tej.2023.9892.2913

Document Type : Original Research Paper-English Issue

Authors

V. Mirzaeian ¹
M. Maghsoudi ²

¹ Department of English, Faculty of Literature, Alzahra University, Tehran, Iran

² Department of English Language Teaching, Farhangian University, Tehran, Iran

https://doi.org/10.22061/tej.2023.9892.2913

Abstract

Background and Objectives: Although Machine Translation (MT) is extensively researched within the field of Artificial Intelligence (AI) and translation studies, few studies have attempted to implement MT output in foreign language teaching (FLT). One potential aspect of using MT in FLT refers to the implementation of MT output for reading comprehension. Considering the existing gap in the body of research on this issue, the present study aimed to investigate whether MT output is qualified enough to be used as an aid in EAP reading comprehension courses. More specifically, this study intended to benchmark the efficacy of MT output for EAP reading comprehension courses based on the data obtained from testing its comprehensibility and probing the students’ perceptions. To achieve the objectives of the study, MT was operationally defined as quality assessment in terms of output efficacy, a combination of usability and comprehensibility, which mirrors the ultimate goal of MT use in EAP reading comprehension courses, from the users' or target readers’ standpoint. Within this perspective, the current research was an attempt to assess the quality of MT output in terms of comprehensibility and the degree to which MT output might be comprehensible to the EAP students participating in this study.
Materials and Methods: The participants of the study, 140 Iranian undergraduate university students majoring in the field of education at Farhangian University, Iran, were selected based on simple random sampling. Oxford Quick Placement Test was used to homogenize them in terms of English proficiency. Two versions of a reliable reading comprehension test, human translation (HT) and Machine Translation (MT), were given to. This test included 25 multiple-choice items, assessing the participants' literal comprehension of information stated in the passage as well as higher-order comprehension that required making inferences and conclusions. In particular, the items measured textual coherence, inference, reference, scanning, skimming, and word-meaning inference. To test the reliability of the tests, the KR-21 formula was applied and the results showed that both HT test (.83) and MT test (.78) were reliable. To investigate the perceptions of the participants on the efficacy of the MT output they encountered on the test, semi-structured interviews were conducted with some of the participants in Persian.
Findings: With reference to the results of non-parametric tests such as Spearman’s rho, and Mann-Whitney Tests, and considering the observed effect sizes (Cohen’s d), it was revealed that, generally, the efficacy of MT output is comparable to that of HT. Moreover, in terms of reading comprehension sub-skills, the qualities of the two translations were comparable with regard to scanning, and inference, but not skimming and reference. Furthermore, the findings from the interview indicated that the students perceive MT to be a seminal aid for their EAP reading comprehension activities despite the minor problems that exist in the output such as morpho-syntactic errors or inappropriate lexical equivalents.
Conclusions: The present study confirmed the fact that the efficacy of MT output is target-reader-dependent and text-dependent since it is determined both by the characteristics of the readers, such as their disciplines, and text features, as demonstrated by the significant differences in comprehension levels of the same readers measured by the same questions for HT and MT output. Accordingly, this study shed limelight on comprehensibility as a criterion of MT output efficacy; that is to say, it has to be reminded that MT quality needs to be defined as a context-bound and target-reader-specific concept.

Keywords

Main Subjects

Machine-translation

COPYRIGHTS
© 2024 The Author(s). This is an open-access article distributed under the terms and conditions of the Creative Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) (https://creativecommons.org/licenses/by-nc/4.0/)

References

[1] Tudor, I. Using translation in ESP. ELT Journal, 1987;41(4): 268-273. https://doi.org/10.1093/elt/41.4.268

[2] Plastina, A. F. Mental translation in the ESP classroom: a case study on processing reading comprehension. Mental Translation in the ESP Classroom, 2006; 1000-1013.

[3] Groves, M., & Mundt, K. Friend or foe? Google Translate in language for academic purposes. English for Specific Purposes, 2015; 37: 112-121.

[4] Bahri, H., & Mahadi, T. Google Translate as a supplementary tool for learning Malay: A case study at Universiti Sains Malaysia. Advances in Language and Literary Studies, 2016; 7(3): 161–167. http://dx.doi.org/10.5755/j01.sal.0.21.1469

[5] Lee, S. M. The impact of using machine translation on EFL students’ writing. Computer Assisted Language Learning, 2020; 33(3): 157-175.

https://doi.org/10.1080/09588221.2018.1553186

[6] Tsai, S. C. Using google translate in EFL drafts: a preliminary investigation. Computer Assisted Language Learning, 2019; 32(5-6): 510-526. DOI:10.1080/09588221.2018.1527361

[7] Suhono, S., Zuniati, M., Pratiwi, W., & Hasyim, U. A. A. Clarifying Google Translate Problems Of Indonesia-English Translation Of Abstract Scientific Writing. Proceedings of the 2nd Workshop on Multidisciplinary and Applications (WMA). 2020; Padang, Indonesia.

http://dx.doi.org/10.4108/eai.24-1-2018.2292399

[8] Knowles, C. L. Investigating instructor perceptions of online machine translation and second language acquisition within most commonly taught language courses. [dissertation]. The University of Memphis; 2016.

[9] Rushwan, I. M. H. The role of translation in developing ESP learners’ reading comprehension skills-A case study of medical students at Najran University-ksa. International Journal of Applied Linguistics and English Literature, 2017; 6(3): 243-253.

https://doi:10.7575/aiac.ijalel.v.6n.3p.243

[10] Mesbah, M., & Faghani, M. Task-based and grammar translation teaching methods in teaching reading comprehension to nursing students: An action research. Aula Orientalis, 2015; 1: 319-25.

[11] Avand, A. Q. Using translation and reading comprehension of ESP learners. The Asian ESP Journal, 2009; 5(1): 44-60.

[12] Kavaliauskien?, G. Proficiency in reading, writing and translation skills: ESP aspect. Vertimo Studijos, 2009; 2(2): 171-184.

[13] Malekan, F., & Hajimohammadi, R. The Relationship between Iranian ESP Learners' Translation Ability and Resilience in Reading Comprehension. International Journal of Education and Literacy Studies, 2017; 5(2): 47-52.

[14] Pham, C. T. K. Reading comprehension and translation performance of English linguistics students of Hung Vuong University: A correlational study. International Journal of English Language and Translation Studies, 2017; 5: 79-85.

[15] Tavakoli, M., Shafiei, S., & Hatam, A. H. The relationship between translation tests and reading comprehension: a case of Iranian University students. Iranian Journal of Applied Language Studies, 2012; 4(1): 193-211.

https://doi:10.22111/IJALS.2012.1353

[16] Laviosa, S., & Cleverton, V. Learning by translating: A contrastive methodology for ESP learning and translation. Scripta Manent, 2015; 2(1): 3-12.

[17] Leonardi, V. Teaching business English through translation. Journal of Universal Language, 2009; 10(1): 139-153. https://doi:10.22425/jul.2009.10.1.139

[18] Mažeikiene. VTranslation as a method in teaching ESP: An inductive thematic analysis of literature. Journal of Teaching English for Specific and Academic Purposes .2019; 6(3): 513-523.

https://doi.org/10.22190/JTESAP1803513M

[19] Novita, L., & Mustafha, B. Translation as a Pedagogical Device for Improving Students’ Reading Comprehension in the ESP text. In Proceedings of the 1st International Conference on Business, Law And Pedagogy, ICBLP 2019. European Alliance for Innovation (EAI).

http://dx.doi.org/10.4108/eai.13-2-2019.2286060

[20] Levy, M., & Steel, C. Language learner perspectives on the functionality and use of electronic language dictionaries. ReCALL: the Journal of EUROCALL, 2015; 27(2): 177. https://doi:10.1017/S095834401400038X

[21] Lee, S. M., & Briggs, N. Effects of using machine translation to mediate the revision process of Korean university students’ academic writing. ReCALL, 2021; 33(1): 18-33.

https://doi:10.1017/S0958344020000191

[22] Nino, A. Evaluating the use of machine translation post-editing in the foreign language class. Computer Assisted Language Learning, 2008; 21(1): 29–49.

http://doi.org/10.1080/ 09588220701865482

[23] Melby, A. K. The translation profession and the computer. CALICO Journal, 2013; 1(1): 55-57.

https://doi.org/10.1558/cj.v1i1.55-57

[24] Garcia, I., & Pena, M. I. Machine translation-assisted language learning: writing for beginners. Computer Assisted Language Learning, 2011; 24(5): 471-487.

https://doi:10.1080/09588221.2011.582687

[25] Chung, E. S., & Ahn, S. The effect of using machine translation on linguistic features in L2 writing across proficiency levels and text genres. Computer Assisted Language Learning, 2021; 5(3): 1-26. https://doi:10.1080/09588221.2020.1871029

[26] Chang, C. K., & Hsu, C. K. A mobile-assisted synchronously collaborative translation–annotation system for English as a foreign language (EFL) reading comprehension. Computer Assisted Language Learning, 2011; 24(2): 155-180. https://doi:10.1080/09588221.2010.536952

[27] Arnold, D., & Balkan, L. Machine Translation. 2001; USA: Blackwell Publishers.

[28] Alhaisoni, An investigation of Saudi EFL university students’ attitudes towards the use of google translate. International Journal of English Language Education, 2017; 5(1): 72-82. https://doi:10.5296/ijele.v5i1.10696

[29] Siregar, R. Exploring the undergraduate student's perception on translation - A preliminary step to teach translation in EFL classes. English Language Teaching, 2018; 11(9): 90-101. DOI:10.5539/elt.v11n9p90

[30] Shterionov, D., Superbo, R., Nagle, P., Casanellas, L., O’dowd, T., & Way, A. Human versus automatic quality evaluation of NMT and PBSMT. Machine Translation, 2018; 32(3): 217-235.

[31] Taleghani, M., Pazouki, E., & Ghahraman, V. The Correlation of Machine Translation Evaluation Metrics with Human Judgement on Persian Language. Journal of Language and Translation, 2019; 9(3): 43-55.

https://doi: 20.1001.1.20088590.2019.9.3.4.6

[32] Bahdanau, D., Cho, K., & Bengio, Y. Neural machine translation by jointly learning to align and translate. 2014; Retrieved online on February 23, 2020 from arXiv preprint arXiv:1409.0473.

[33] Napitupulu, S. Analyzing Indonesian-English abstracts translation in view of translation errors by google translate. International Journal of English Language and Linguistics Research, 2017; 5(2): 15-23.

[34] Salah Habeeba, & Muhammed Investigating the effectiveness of google translate among Iraqi students. International Journal of Innovation, Creativity and Change, 2019; 7(8): 112-122.

[35] Cetiner, C., &Isisag, K. Undergraduate level translation students’ attitudes towards machine translation post-editing training. International Journal of Languages’ Education and Teaching, 2019; 7(1): 110-120. https://doi:10.18298/ijlet.3242

[36] Koehn, P., & Knowles, R. Six challenges for neural machine translation. 2017; Retrieved online on February 23, 2020 from arXiv preprint arXiv:1706.03872.

[37] Chang, C., & Yamada, M. Translation tasks for learning collocations: Effects of machine translation plus post-editing and sight translation. English Teaching and Learning, 2020; 3(5): 87–101. http://doi.org/ 10.1007/s42321-020-00059-x

[38] Toral, A., & Sánchez-Cartagena, V. M. A multifaceted evaluation of neural versus phrase-based machine translation for 9 language directions. 2017; Retrieved online on February 23, 2020 from arXiv preprint arXiv:1701.02901.

[39] Jimenez, C. The role of translation technologies in Spanish language learning. Journal of Spanish Language Teaching, 2018; 4(4): 141-160.

http://doi.org/10.1080/23247797.2017.1408949

[40] Abdelaal, N., & Alazzawie, A. Machine Translation: The case of Arabic-English translation of news texts. Theory and Practice in Language Studies, 2020; 10(4): 408–418. http://doi.org/10.17507/tpls.1004.09

[41] Roturier, J. An investigation into the impact of controlled English rules on the comprehensibility, usefulness and acceptability of machine-translated technical documentation for French and German users. [dissertation]. Dublin City University; 2006.

[42] Alderson, C. Reading Assessment. 2000; New York: Cambridge University Press.

[43] Lewkowicz, J. A. Authenticity in language testing: some outstanding questions. Language testing, 2000; 17(1): 43-64.

[44] Bachman, L. F., & Palmer, A. S. Language Testing in Practice: Designing and Developing Useful Language Tests. 1996; Oxford: Oxford University Press.

[45] Bentivogli, L., Bisazza, A., Cettolo, M., & Federico, M. Neural versus phrase-based machine translation quality: a case study. 2016; Retrieved online on February 23, 2020 from arXiv preprint arXiv:1608.04631.

[46] Doherty, S. Investigating the effects of controlled language on the reading and comprehension of machine translated texts: A mixed-methods approach [dissertation]. Dublin City University; 2012.

[47] Anuyahong, B. A study of English reading strategies and reading motivation used by Thai-Nichi institute of technology students. Automotive Engineering, 2011; 1(14): 21-30.

[48] Becirovic, S., Brdarevic-Celjo, A., & Sinanovic, J. The Use of Metacognitive Reading Strategies among Students at International Burch University: A Case Study. European Journal of Contemporary Education, 2017; 6(4): 645-655. https://doi:10.13187/ejced.2017.4.645

[49] Ganji, M., Yarahmadzehi, N., & Sasani, N. English Major Students’ Awareness of Metacognitive Reading Strategies: Gender and Academic Level in Focus. Issues in Language Teaching, 2018; 7(2): 91-119. https://doi.org/10.22054/ilt.2019.46700.428

[50] Karnal, A. R., & Pereira, V. V. Reading strategies in a L2: A study on machine translation. Reading Matrix: An International Online Journal, 2015; 15(2): 69-80.

[51] Brantmeier, C. Effects of reader's knowledge, text type, and test type on L1 and L2 reading comprehension in Spanish. The Modern Language Journal, 2005; 89(1), 37-53. https://doi.org/10.1111/j.0026-7902.2005.00264.x

[52] Chen, C. M., & Lin, Y. J. Effects of different text display types on reading comprehension, sustained attention and cognitive load in mobile reading contexts. Interactive Learning Environments, 2016; 24(3): 553-571.

https://doi:10.1080/10494820.2014.891526

[53] Coelho, C. L. G., & Correa, J. Reading comprehension: Cognitive abilities and types of text. Psico, 2017; 48(1): 40-49.

[54] Matheson, I. Unpacking Reading Comprehension by Text Type: An Examination of Reading Strategy Use and Cognitive Functioning in Poor and Typically-Achieving Comprehenders [dissertation]. Queen’s University, Kingston; 2018.

[55] Uysal, P. K., & Bilge, H. An investigation on the relationship between reading fluency and level of reading comprehension according to the type of texts. International Electronic Journal of Elementary Education, 2019; 11(2): 161-172.

https://doi:10.26822/iejee.2019248590