The development of ICT and AI has brought about significant changes in diagnostic tests that measure English writing ability. Numerous practical systems already exist for administering writing tests in certification examinations. Similarly, in entrance examinations, automated scoring systems are being introduced sequentially. Automated scoring systems are also gaining attention in the field of education. Historically, educational institutions have often avoided writing tests due to the human and time resources required for scoring. The implementation of an automated scoring system carries the advantage of reducing the human and time costs. However, various studies that have endeavored to introduce automated scoring reveal that the computer's assessment does not invariably align with human evaluation. Consequently, neither educators nor learners consistently place trust in the evaluations and feedback furnished by the automated scoring system.
We developed an automated scoring system for evaluating the quality of content in writing tests on a specific theme. This system was implemented in a required English course for first-year junior college students. Utilizing data from past writing tests and teacher evaluations, we input the information into a cloud-based AI system. Through supervised machine learning, the system was able to score the texts at four distinct levels. Subsequently, writing tests for the required English course in Academic Years (AY) 2022 and 2023 were evaluated, and the concordance rate between the automated scoring system and the teacher's evaluation was examined. Students utilized the system in class and participated in a questionnaire survey, the responses to which were analyzed through text mining.

Authors:
Kaoru Mita, Jissen Women's Junior College, Japan
Atsuko Shimoda, Jissen Women's Junior College, Japan

About the Presenter(s)
Ms Kaoru Mita is a University Professor/Principal Lecturer at Jissen Women's Junior College in Japan

See this presentation on the full schedule – Wednesday Schedule

Bookmark