Skip to main content
@shmVirus

publications

LLMs as On-Demand Teaching Assistants: Evidence from a Semester-Long Deployment Across Three CS Courses

Sabbir Hosen Mamun, Yuki Tanaka, Helena R. Novak

Preprint2025Under review at ICSE 2026
PreprintCite

Abstract

We report findings from a semester-long deployment of a retrieval-augmented LLM teaching assistant across three undergraduate CS courses (n = 412 students). The assistant answered 87% of student queries without instructor intervention, maintained factual accuracy above 92% on a held-out evaluation set constructed by course staff, and reduced after-hours instructor workload by an estimated 4.2 hours per week. Students reported high satisfaction (μ = 4.1 / 5) but expressed concern about over-reliance and reduced contact with instructors. We identify three failure modes — hallucinated API documentation, context collapse on multi-file debugging questions, and inconsistent enforcement of course-specific constraints — and discuss mitigation strategies. Implications for instructor workload, academic integrity policy, and pedagogical design are discussed.

Cite this work

@misc{Mamun2025LLMs,
  title = {LLMs as On-Demand Teaching Assistants: Evidence from a Semester-Long Deployment Across Three CS Courses},
  author = {Sabbir Hosen Mamun and Yuki Tanaka and Helena R. Novak},
  year = {2025},
  howpublished = {\url{https://arxiv.org/abs/2502.12345}},
  note = {Preprint},
  abstract = {We report findings from a semester-long deployment of a retrieval-augmented LLM teaching assistant across three undergraduate CS courses (n = 412 students). The assistant answered 87% of student queries without instructor intervention, maintained factual accuracy above 92% on a held-out evaluation set constructed by course staff, and reduced after-hours instructor workload by an estimated 4.2 hours per week. Students reported high satisfaction (μ = 4.1 / 5) but expressed concern about over-reliance and reduced contact with instructors. We identify three failure modes — hallucinated API documentation, context collapse on multi-file debugging questions, and inconsistent enforcement of course-specific constraints — and discuss mitigation strategies. Implications for instructor workload, academic integrity policy, and pedagogical design are discussed.},
}