The increasing adoption of large language models (LLMs) in educational contexts poses a challenge to maintaining the robustness of online assessment results. The advent of post-plagiarism, driven by substantial coverage of AI-based assistants, has the potential to significantly impact the fairness of online learning. This study presents a novel approach to open-book online assessment design, implemented in an introductory AI course for non-STEM undergraduates. The approach is based on the principle of allowing learners to use LLMs while prohibiting them from copying questions and answers indiscriminately. Specifically, hidden context was incorporated into an HTML-based, multiple-choice test to emulate the unreliable responses characteristic of LLMs. These hallucinations were exhibited in incorrect submissions, particularly when minimal attention was given to AI-guided outputs, facilitating critical thinking learning outcomes. Learners who successfully resolved the hidden context constraints tended to disregard question formulations offering a choice between true and false statements. This resulted in an approximately normal distribution of grades for a sample of over 600 learners. This study establishes a basis for further discussion on developing online assessments and implementing LLM-resistant tests to ensure fair and reliable grading standards, approaching its limitations, and reflecting for future use in a broader educational context.

Authors:
Andrei Ternikov, HSE University, Russia

---

Conference Comments & Feedback

Place a comment using your LinkedIn profile

Comments

Share on activity feed

Post with LinkedIn

Powered by WP LinkPress

Share this Presentation

Share this...

Facebook

Pinterest

Twitter

Linkedin

Posted by James Alexander Gordon

Last updated: 2023-02-23 23:45:00

All Posts