About the workshop

Building on the success of the First Workshop on the Automatic Evaluation of Learning and Assessment Content, which was held at AIED 2024 and attracted more than 80 participants, this workshop will focus on approaches for automatically evaluating learning and assessment content, offering an opportunity to discuss common challenges, share best practices, and promising new research directions.

Topics of Interest

Topics of interest include but are not limited to:

• Question evaluation (e.g., in terms of the pedagogical criteria listed above: alignment to the learning objectives, factual accuracy, language level, cognitive validity, etc.).

• Estimation of question statistics (e.g., difficulty, discrimination, response time, etc.).

• Evaluation of distractors in Multiple Choice Questions.

• Evaluation of reading passages in reading comprehension questions.

• Evaluation of lectures and course material.

• Evaluation of learning paths (e.g., in terms of prerequisites and topics taught before a specific exam).

• Evaluation of educational recommendation systems (e.g., personalised curricula).

• Evaluation of hints and scaffolding questions, as well as their adaptation to different students.

• Evaluation of automatically generated feedback provided to students.

• Evaluation of techniques for automated scoring.

• Evaluation of pedagogical alignment of LLMs.

• Evaluation of the ethical implications of using open-weight and commercial LLMs in education.

• Evaluation of bias in educational content and LLM outputs.

Human-in-the-loop approaches are welcome, provided that there is also an automated component in the evaluation and there is a focus on the scalability of the proposed approach. Papers on generation are also very welcome, as long as there is an extensive focus on the evaluation step.

Submission Guidelines

We invite papers in two different categories:

• Research papers: We are accepting short papers (5 pages, excluding references) and long papers (10 pages, excluding references), formatted according to the workshop style (using either the LaTeX template or the DOCX template). 

• Ongoing work: We also accept extended abstracts (max 2 pages), to showcase work in progress and preliminary results. Papers should be formatted according to the workshop style (using either the LaTeX template or the DOCX template). 

Submissions should contain mostly novel work, but there can be some overlap between the submission and work submitted elsewhere (e.g., summaries, focus on the evaluation phase of a broader work). Each of the submissions will be reviewed by the members of the Program Committee.

Submissions in the research papers category may be archival or non-archival, based on the wish of the authors. All archival papers will be in the proceedings volume, which will be submitted for publication to CEUR Workshop Proceedings. 

Submissions in the ongoing work (i.e., extended abstracts) will be non-archival. All non-archival submissions may be submitted to any venue in the future, except for another EvalLac.

Register your interest here !

Important Dates

• Submission deadline: May 25, 2025

• Notification of acceptance: June 15, 2025

• Camera ready: June 22, 2025

• Workshop: July 26, 2025

Organising Committee

• General chair: Luca Benedetto (University of Cambridge)
• Program chairs: George Dueñas (National Pedagogical University), Diana Galván Sosa (University of Cambridge)
• Publicity chairs: Gabrielle Gaudeau (University of Cambridge), Shiva Taslimipoor (University of Cambridge)
• Advisors: Andrew Caines (University of Cambridge), Anastassia Loukina (Grammarly, Inc.), Torsten Zesch (FernUniversit ̈at in Hagen)

Invited Speakers

• Guanliang Chen, Monash University 

Publication

All archival papers will be in a proceedings volume by CEUR Workshop Proceedings

Venue

The workshop will be hybrid. It will be held (in-person) at the Conference Venues of the University of Palermo (Italy).

Note that registration to AIED (at least a "Full Day Workshop") will be required. 

Contact

All questions about submissions should be emailed to luca <dot> benedetto <at> cl.cam.ac.uk