Mortanas AcademyHow do you implement Evaluate an AI Education Pilot?
Predefine the comparison, success thresholds, risk guardrails, evidence sources and scale decision before the pilot begins.
Use proportionate learning data to diagnose skills, support cohorts and evaluate programmes without turning proxies into unfair decisions.
Implementation in six steps
Record the learner starting point, the task they must complete and an observable success criterion. Choose a learning outcome tied to performance rather than content volume.
Predefine the comparison, success thresholds, risk guardrails, evidence sources and scale decision before the pilot begins. Give learning materials a source, version, permission status and review date; constrain the AI from inventing content without evidence.
Start with a small, representative learner cohort. Include normal progress, misconceptions, accessibility needs and exceptions that require support.
Define qualified educator review and a clear appeal route for feedback, assessment decisions, sensitive data, academic integrity and learner-impacting recommendations.
Create a pre-registered pilot evaluation plan after the pilot. Retain prompts, sources, educator corrections, learner feedback and the accountable owner in a traceable record.
Track learning gain, transfer and risk threshold as the primary measure. Confirm that results hold across learner groups, transfer to the real task and do not hide increased educator workload.
Worked example
In a worked learning pilot, the team follows 12 learners with different starting levels through a baseline task and a transfer task. The AI-assisted activity uses the same verified evidence pack, while an educator flags incorrect, ambiguous and inaccessible outputs. The team completes a pre-registered pilot evaluation plan. It expands use only when learning gain, transfer and risk threshold, learning gain, equity, educator correction burden and safety guardrails are all acceptable.
Working output
A pre-registered pilot evaluation plan
Primary measure
Learning gain, transfer and risk threshold
Measurement, quality and risk
Signals to track together
- learning gain, transfer and risk threshold
- Human correction time and number of changes
- Exception-routing accuracy
- Source and decision traceability
- User or customer impact
Risks to monitor
- Plausible but incorrect output without source or context
- Use of personal, confidential or contractual data without permission
- Hidden uncertainty or exceptions inside an automated decision
- Higher human correction burden despite faster generation
- Missed differences in region, sector or version
Language, region and scope note
This education guide is designed for English and Turkish learning environments focused on Türkiye and the United Kingdom. Language selection does not determine legal jurisdiction. Current institutional policies, local rules and qualified educator review should be checked for learner data, children and vulnerable groups, accessibility, automated assessment, academic integrity and workplace training.
Sources and verification
Check the current version of primary sources and your implementation context. Commercial outcomes are not guaranteed.
Frequently asked questions
How do you implement Evaluate an AI Education Pilot?
Predefine the comparison, success thresholds, risk guardrails, evidence sources and scale decision before the pilot begins.
What working output should be created?
Create a pre-registered pilot evaluation plan with a named owner and review date so implementation remains traceable.
How should success be measured?
Use learning gain, transfer and risk threshold as the primary measure, together with quality, correction burden and risk signals.
Does this guide replace professional advice?
No. It is educational; legal, security, financial, medical or regulated decisions require appropriately qualified review.