A Hybrid Approach to Zero-Shot Cross-Corpus Essay Scoring via Rubric-Prompted LLM Agents and Deterministic Linguistic Features

A Hybrid Approach to Zero-Shot Cross-Corpus Essay Scoring via Rubric-Prompted LLM Agents and Deterministic Linguistic Features

Chikake T.M. (ISTOK MIPT, Dolgoprudny, Moscow Region, Russia)
Mihailov A.A. (ISTOK MIPT, Dolgoprudny, Moscow Region, Russia)

Abstract

Automated Essay Scoring systems depend on scored training data for every new corpus. We investigate whether this dependency can be reduced by combining two complementary scoring paradigms: rubric-prompted large language model agents that evaluate essays without any training examples, and deterministic linguistic features that capture vocabulary, morphology, and syntax independently of any rubric. On the ASAP 2.0 benchmark (17,307 essays scored 1–6), a hybrid Random Forest regressor fusing both paradigms achieves Quadratic Weighted Kappa of 0.765 on the full corpus, approaching the supervised state of the art (0.841). Features alone reach 0.719 without neural networks. A greedy ensemble of 15 zero-shot agents scores 0.650 with no training at all. Population-matched rubrics significantly outperform cross-population rubrics in zero-shot mode, confirming that rubric design, rather than model size, determines scoring quality. We formalise these findings as an assessment portability spectrum: deterministic features transfer most reliably, followed by population-matched and then cross-population rubrics.

Keywords

automated essay scoring; zero-shot evaluation; cross-corpus transfer; linguistic features; LLM agents; ASAP; assessment portability.

Edition

Proceedings of the Institute for System Programming, vol. 38, issue 4, part 2, 2026, pp. 257-266

ISSN 2220-6426 (Online), ISSN 2079-8156 (Print).

DOI: 10.15514/ISPRAS-2026-38(4)-30

For citation

Chikake T.M., Mihailov A.A. A Hybrid Approach to Zero-Shot Cross-Corpus Essay Scoring via Rubric-Prompted LLM Agents and Deterministic Linguistic Features. Proceedings of the Institute for System Programming, vol. 38, issue 4, part 2, 2026, pp. 257-266 DOI: 10.15514/ISPRAS-2026-38(4)-30.

Full text of the paper in pdf Back to the contents of the volume