Preprint
Large Language Models

LLMs on University-Level Physics Coding

W. Yeadon, Alex Peach, Craig P. Testrow
March 25, 2024Scientific Reports38 citations
arXivPDFDOI

38

Citations

1

Influential Citations

Scientific Reports

Venue

2024

Year

Abstract

This study evaluates the performance of ChatGPT variants, GPT-3.5 and GPT-4, both with and without prompt engineering, against solely student work and a mixed category containing both student and GPT-4 contributions in university-level physics coding assignments using the Python language. Comparing 50 student submissions to 50 AI-generated submissions across different categories, and marked blindly by three independent markers, we amassed n=300\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n = 300$$\end{document} data points. Students averaged 91.9% (SE:0.4), surpassing the highest performing AI submission category, GPT-4 with prompt engineering, which scored 81.1% (SE:0.8)—a statistically significant difference (p = 2.482×10-10\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$2.482 \times 10^{-10}$$\end{document}). Prompt engineering significantly improved scores for both GPT-4 (p = 1.661×10-4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$1.661 \times 10^{-4}$$\end{document}) and GPT-3.5 (p = 4.967×10-9\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$4.967 \times 10^{-9}$$\end{document}). Additionally, the blinded markers were tasked with guessing the authorship of the submissions on a four-point Likert scale from ‘Definitely AI’ to ‘Definitely Human’. They accurately identified the authorship, with 92.1% of the work categorized as ‘Definitely Human’ being human-authored. Simplifying this to a binary ‘AI’ or ‘Human’ categorization resulted in an average accuracy rate of 85.3%. These findings suggest that while AI-generated work closely approaches the quality of university students’ work, it often remains detectable by human evaluators.

Analysis

Why This Paper Matters

This paper addresses a critical question in AI-assisted education: can large language models (LLMs) like ChatGPT produce university-level physics code that is indistinguishable from student work? As AI tools become ubiquitous, understanding their capabilities and limitations in authentic academic tasks is essential for educators, policymakers, and AI developers. The study's rigorous blind evaluation design provides concrete evidence that, despite rapid advances, current LLMs still lag behind top students and remain detectable by human markers.

The findings have immediate implications for academic integrity. With 85.3% binary detection accuracy, the paper suggests that human evaluators can reliably flag AI-generated code, though the 14.7% misclassification rate indicates room for improvement in detection methods. This work also informs the design of AI-assisted learning tools, showing that prompt engineering can boost performance but not to the level of skilled students.

Technical Contributions

  • Comparative evaluation framework: The study systematically compares four AI conditions (GPT-3.5, GPT-4, with/without prompt engineering) against student and mixed submissions, providing a benchmark for AI coding in physics.
  • Prompt engineering impact: Quantifies the benefit of prompt engineering for coding tasks, with GPT-4 improving from 73.8% to 81.1% and GPT-3.5 from 49.2% to 73.8%.
  • Human detection methodology: Uses a four-point Likert scale for authorship attribution, enabling nuanced analysis of detection confidence and accuracy.
  • Statistical rigor: Employs p-values and standard errors to establish significance of differences between groups.

Results

  • Student performance: Average 91.9% (SE:0.4), highest among all categories.
  • Best AI performance: GPT-4 with prompt engineering scored 81.1% (SE:0.8), significantly lower than students (p=2.482e-10).
  • Prompt engineering effect: Significant improvement for GPT-4 (p=1.661e-4) and GPT-3.5 (p=4.967e-9).
  • Detection accuracy: 92.1% of work marked 'Definitely Human' was human-authored; binary AI/Human classification accuracy was 85.3%.

Significance

This research provides a realistic assessment of LLM capabilities in a domain requiring both physics knowledge and coding skill. It demonstrates that while AI can produce passable code, it has not yet reached the level of top university students. The high detection accuracy suggests that current AI-generated code has identifiable stylistic or quality differences. For the AI field, this work highlights the need for further improvements in reasoning and domain-specific coding, and for education, it offers evidence-based guidance on AI use policies and detection strategies.

large-language-modelsphysics-educationcoding-assignmentsprompt-engineeringgpt-4gpt-3.5human-evaluation
Prompting

Related Papers