War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education

Analysis

Why This Paper Matters

This paper arrives at a critical moment when higher education is grappling with the rapid proliferation of AI chatbots. By systematically testing major English and Chinese chatbots—ChatGPT (GPT-3.5 and GPT-4), Bing Chat, Bard, and Ernie—on multi-disciplinary academic tasks, the authors cut through the hype to reveal a sobering reality: none of these models perform at an A or B level. The finding that Bing Chat and Bard score F-grade averages is particularly striking, given the publicized claims of their capabilities. This matters because institutions and educators need evidence-based assessments to inform policy, assessment redesign, and teaching strategies, rather than relying on vendor marketing or anecdotal reports.

Furthermore, the paper bridges a gap in the literature by including Chinese-language chatbots (Ernie) and reviewing both English and Chinese academic sources, offering a more global perspective. The four categories of recommendations—for faculty (assessment and teaching/learning), students, and institutions—provide actionable guidance that can be immediately applied. This makes the paper a valuable resource for decision-makers in higher education.

Technical Contributions

• Multi-disciplinary test design: The authors created a test covering diverse academic disciplines (e.g., humanities, sciences, engineering) to evaluate chatbot performance holistically, rather than focusing on narrow benchmarks.
• Comparative methodology: They systematically compared chatbots using a consistent set of prompts and grading criteria, enabling direct performance comparisons across models.
• Grading framework: The use of letter grades (A through F) provides an intuitive, education-relevant metric that highlights the gap between current AI capabilities and academic standards.
• Literature review: The paper synthesizes recent Chinese and English academic literature on chatbots in education, providing context for the experimental results.

Results

• GPT-4 and GPT-3.5 achieved the highest scores, but still fell short of A or B grades.
• Bing Chat and Bard performed at an F-grade average, indicating significant deficiencies in accuracy, reasoning, and relevance for higher education tasks.
• No chatbot demonstrated consistent, reliable performance across all disciplines tested.
• The results contradict sensationalist claims that AI is ready to replace or significantly augment academic work at a high level.

Significance

This paper has broad implications for the AI field and higher education. It provides a reality check for the exaggerated narratives surrounding large language models, emphasizing that current chatbots are not yet intelligent enough for high-stakes academic use. The findings encourage more rigorous evaluation of AI tools before adoption, and the recommendations offer a roadmap for responsible integration. For AI practitioners, the paper underscores the need for continued improvement in reasoning, factual accuracy, and domain-specific knowledge. It also highlights the importance of multilingual and cross-cultural evaluation, as the inclusion of Chinese chatbots reveals performance disparities that may affect global adoption.