Acuity Insights Inc.

Published purpose

We propose a new approach to automated test scoring that combines recent advances in large language models (LLMs) with traditional clear-box ML approaches. We will use an LLM-as-a-judge framework in association with a rubric of scoring elements (developed in consultation with subject matter experts) to classify responses on the basis of these scoring elements. The LLM will also provide its reasoning for these decisions. We will then construct clear-box ML models that combine these scoring elements extracted from the responses into a single numeric score. This method will allow us to not only score a response, but also to identify precisely how individual elements of the rubric contributed to the score and why each element of the rubric was.