Your Students Use AI. But Can You Actually Measure Their AI Skills?
A student says they are confident using artificial intelligence. They know how to write prompts, generate summaries and use AI tools to complete assignments. But can they identify an inaccurate answer? Can they recognise biased information or explain when an AI-generated recommendation should not be trusted?
These are very different abilities, and colleges need to understand the difference.
As AI becomes more common in higher education, institutions face a new challenge. Teaching students to use AI tools is one thing. Determining whether they can use those tools critically, responsibly and effectively is another.
What Does Research Say About AI Literacy Assessment?
A 2026 systematic review published in Sustainability examined 31 studies on instruments used to assess AI literacy in higher education. The review found that 28 of the 31 studies primarily used self-report questionnaires. Performance-based and combined assessment approaches were less common.
The researchers also found that only eight studies focused specifically on generative AI literacy, while the remaining studies examined AI literacy more broadly. These findings highlight a gap between assessing what students believe they can do and directly evaluating what they can demonstrate.
Read the systematic review on AI literacy assessment.
This does not mean self-assessment is ineffective. Questionnaires can help educators understand students' confidence, attitudes and perceived learning needs. However, confidence alone cannot establish whether a student can apply their knowledge successfully.
The important question is whether colleges are measuring actual capabilities or simply asking students how capable they feel.
Confidence and Competence Are Not the Same
Imagine two students completing a questionnaire about their AI skills. The first regularly uses AI tools and rates their abilities highly. The second has less experience and gives themselves a moderate rating. Now, both students receive the same task: review an AI-generated report, verify its claims and identify unsupported conclusions. The first student might accept the report without checking its sources. The second might question several claims and explain why additional evidence is needed.
This is an illustrative example, not a finding directly reported by the study. It shows why perceived ability and demonstrated performance should not automatically be treated as equivalent. For colleges, the distinction matters because assessment results often inform decisions about student development. If an institution wants to know whether students can evaluate AI-generated information, asking them to rate their confidence may not provide enough evidence.
A practical task can reveal a different dimension of their ability.
What Should an AI Literacy Assessment Measure?
AI literacy involves more than knowing how to enter a prompt. Students need to understand the limitations of AI tools and recognise how those limitations affect their work.
A useful assessment approach should consider several capabilities.
Evaluating AI-generated information: Can students verify claims, identify inaccuracies and distinguish evidence from unsupported statements?
Applying AI to practical problems: Can students use AI appropriately to complete a task while demonstrating their own understanding and reasoning?
Recognising ethical concerns: Do students understand privacy, confidentiality, bias and academic integrity when using AI tools?
Making independent decisions: Can students explain why they accepted, rejected or modified an AI-generated recommendation?
Recognising limitations: Do students understand when AI may be unreliable or when another approach would be more appropriate?
The relative importance of these capabilities depends on the student's discipline and intended career. An engineering student and a marketing student may use different tools, but both need to evaluate information and exercise sound judgement.
How Colleges Can Assess AI Skills More Effectively
A practical starting point is to combine self-assessment with activities that allow students to demonstrate their abilities.
Colleges can begin by asking students about their familiarity with AI, their confidence in using it and the challenges they encounter. They can then introduce practical exercises relevant to students' subjects.
For example, a student might evaluate an AI-generated explanation, verify its sources and explain which conclusions are reliable. Another might use AI to explore a problem and then justify the final solution independently.
Educators can assess the reasoning behind the answer, not just the quality of the final output.
Combining these approaches provides a broader picture of student capability. It can also reveal where students' confidence differs from their demonstrated performance.
The findings can then guide targeted support. Some students may need to develop basic AI literacy, while others may benefit from more advanced practice in evaluating evidence or making decisions.
What This Means for Student Employability
AI literacy is becoming relevant to career preparation, but completing an AI course does not automatically demonstrate workplace competence.
A graduate may know several tools and still struggle to evaluate information, solve unfamiliar problems or decide whether an AI-generated recommendation is appropriate.
Colleges can address this by identifying the capabilities relevant to students' intended careers and providing opportunities to demonstrate those capabilities before graduation.
For placement officers, this creates an opportunity to work with academic departments on practical assessment and student development. Instead of relying entirely on course completion or self-reported familiarity, institutions can gather evidence of what students can actually do.
This is also where a broader understanding of student strengths and development needs becomes useful.
ChangeBegins focuses on talent intelligence and assessment, helping bring the conversation back to understanding individual capabilities and career fit. The broader principle is that assessment should produce information that supports better decisions, not simply another score.
The Real Question Colleges Need to Ask
The research does not suggest that colleges should abandon questionnaires or replace every assessment with a practical examination. It highlights the importance of choosing assessment methods that match the capabilities being measured.
Self-reported confidence can provide useful information. Practical tasks can reveal how students apply their knowledge. Used thoughtfully, these approaches can complement each other.
As AI becomes more integrated into education and work, colleges need a clearer understanding of what their students can do, where they need support and how their capabilities relate to future career requirements.
The question is not just whether students feel ready to use AI. It is whether they can demonstrate the judgement and skills needed to use it well.
