Tracing student thinking in Socratic LLM dialogues: what 121 chatbot conversations reveal

Tracing student thinking in Socratic LLM dialogues: what 121 chatbot conversations reveal

11/12/2026 - 14:07

How can we make students’ reasoning visible when they work with an AI chatbot? Most research on AI in education relies on self-reported usefulness or motivation. A new study in Trends in Higher Education, by Qiqi Zhou, Carlos Pereira Santos, two colleagues from BUas, and Tianyu Chen, a Master's student of Imagineering, instead analyses what students actually say in conversation with an LLM-based chatbot.
  • Onderzoek

Study design 

Between February and May 2025, CB students in seven project-based research courses, from year 1 to year 4, were invited to use Riffbot.ai, a chatbot designed to encourage reflection through Socratic dialogues. It asks questions, guiding students from basic description towards self-examination and new thought. 

Of the 115 students enrolled, 93 chose to try it, generating 121 dialogues. The team analysed the 72 conversations in detail. 

Findings  

  • Students engaged with AI questioning in different ways. Some sought direct answers, while others used the questions to clarify, compare, or reconsider their ideas. 
  • Deep inquiry was possible, but not automatic. Students sometimes developed explanations and reflections, but conversations could also remain descriptive or repetitive. 
  • Thinking did not follow a straight line. Students stalled, looped back, asked for answers, or changed direction. 
  • Student feedback was mixed. Some appreciated the opportunity to explore ideas through questioning. Others wanted more flexibility and direct guidance. 

In one data-driven decision-making project, a student moved from uncertainty about available data to reformulating the research question around events whose attendee profiles matched the client’s target group.  

Implications for design 

The authors highlight the importance of designing AI interactions as meaningful experiences, rather than simply sequences of questions. Effective Socratic chatbots need to respond to users' ideas, recognize when guidance is needed, and offer flexibility between questioning, brainstorming, and practical support. This creates opportunities to explore more adaptive, human-centred conversational experiences that encourage creativity, reflection, and active engagement. 

Limitations 

The study involved one applied university, had no control group or learning-outcome measure, and so cannot establish learning gains. Selecting longer dialogues may favour sustained users, and the underlying model was proprietary and not accessible to the researchers. These limitations also highlight the importance of testing and evaluating AI experiences with real users, rather than assuming that an intended interaction design produces the desired experience. 

Future directions 

The research opens opportunities for media design, Imagineering, and human-centred AI innovation. Future studies could explore how different conversational designs, such as Socratic questioning, direct assistance, and creative brainstorming, shape user experiences, engagement, and reflection. Comparing AI interactions with human guidance, testing new AI models, and involving users in iterative design could help develop more responsive and meaningful AI-supported experiences.  

The article is open access: https://www.mdpi.com/2813-4346/5/4/128