Artificial Intelligence (AI) has emerged as a transformative tool in education, notably in facilitating automated exam grading. This study focuses on Automatic Short Answer Grading (ASAG) via ChatGPT-4, a widely accessible and versatile general-purpose generative AI model. We compare the grading outcomes from ChatGPT with those adjudicated by human evaluators within the health science domain. An evaluative framework was deployed to gauge the GPT-4 model’s concordance with an expert educator’s scoring. Human scores were compared to those offered by ChatGPT with different versions of prompts, specifically with 10 examples, 25 examples, and a grading rubric, employing a scoring metric that spans from 0 to 10 points, allowing for decimal values, without any model fine-tuning or parameter modulation. Our findings show that rubrics markedly enhance score alignment with an educator’s evaluative benchmarks, registering intraclass correlation coefficients surpassing 0.8, thus nearly mirroring human judgment. These results suggest that there is ample scope for increasing the effectiveness of ASAG using Large Language Models (LLM) such as ChatGPT. However, it is imperative to recognize that the operability of these systems is not yet fully reliable and stable, making human supervision necessary. The integration of expert supervision ensures both the accuracy and pedagogical validity of these automated tools.