Scenario-Based Assessment With AI

What scenario-based assessment is#

Scenario-based assessment places the student inside a situation and asks what they would do. A nurse faces a patient whose symptoms do not fit a pattern. A project manager learns that a key supplier has failed. A teacher meets a parent who disagrees with a grade. The student gathers information, makes decisions and explains them, and the assessor judges the quality of that reasoning.

It is a form of authentic assessment because the task resembles the work. It is also a form of performance assessment, which means the evidence is what the student does or says, not what they select from a list. For background on the wider idea, see what authentic assessment is.

Why use AI for it#

Scenario assessments have always been costly. Observing a role-play takes a person for every student. A branching case study can only anticipate the paths its author wrote. AI helps in two ways.

• It can play the counterpart. The AI holds a character and responds to what the student says, so the student can ask unplanned questions.
• It leaves a record. The transcript shows the questions the student asked, the information they gathered and the reasons they gave, which makes consistent scoring possible after the fact.

AI does not remove the need for design. A weak scenario with an AI is still a weak scenario.

Design the evidence first#

Start with what you need to be able to say about a student, then design the scenario so it produces that evidence.

1. Name the outcome. For example: "prioritizes competing needs and justifies the order."
2. Decide what behavior would show it. The student asks a question that uncovers the key fact. The student states the priority. The student explains why.
3. Build decision points that force the behavior. If you want to see prioritization, the scenario must include more demands than the student can address at once.
4. Plan hidden information. The key fact appears only if the student asks the right question. This reveals more than a scenario that hands everything over.
5. Define the end. The student reaches a decision, a time limit passes or a number of turns is used.
6. Write the rubric before the scenario is final. If you cannot say how you would score a behavior, the scenario cannot produce evidence for it.

An example#

Outcome: a student can triage competing demands in an incident and justify the order.

• Situation: A storm has closed two roads. The student is the operations lead for a county emergency office and has three requests from the field.
• AI role: The incident commander, who asks for a recommendation and presses the student on trade-offs.
• Hidden information: One request concerns a facility with vulnerable residents, which appears only if the student asks about who is there.
• End: The student gives a final recommendation and the commander accepts or challenges it.
• Rubric: Gathers key information; ranks requests with a stated rationale; anticipates consequences; communicates clearly under pressure.

The same structure works for a patient interview, a client meeting or a parent conference.

Scoring#

Score the transcript, not an impression.

• Use an analytic rubric with four to six criteria and observable descriptors. The oral assessment rubric guide gives a template you can adapt to scenarios.
• Tie each score to a moment in the transcript.
• Score criteria separately before forming an overall judgment.
• If the AI drafts scores, check that each cites evidence you can find, and change those you disagree with.
• If more than one person scores, calibrate on a few sessions first.

Reliability and fairness#

Scenario assessments have particular risks.

• Different students get different conversations. Because the AI responds to each student, the evidence is not identical. Reduce the problem with a fixed set of required information, a consistent completion rule and a rubric that scores the behaviors, not the path.
• Variation in the AI. The same prompt can produce different responses. Test the scenario repeatedly and keep the character's constraints specific.
• Bias in the character. Read transcripts for stereotypes and adjust the persona.
• Access. Provide a text option, extra time and an alternative for students who cannot use the format.
• Practice. Offer a practice scenario on a different case so students learn the format without learning the answer.
• Integrity. If the stakes are high, consider supervised sessions, and ask follow-up questions about the student's decisions afterward.

Using more than one scenario#

One scenario is a small sample of a student's ability. For a high-stakes decision, use several scenarios or combine the scenario with other evidence, such as a written reflection or an oral defense of a related artifact. See AI oral exams.

From scenario to LMS#

If your assessment sits in an LMS, ask the tool how the activity launches from the course and how scores return to the gradebook. Our guide to using AI with Canvas for assessment lists the questions.

Where Rocketproof.ai fits#

Rocketproof.ai supports scenario-based assessment. The instructor writes the scenario or asks the built-in assistant to draft it, defines the persona and goal, and attaches a rubric. A scenario can be graded, and the transcript records the decisions the student made and the reasons they gave. The instructor reviews that evidence before a grade is released or passed back to the LMS. Branching scenarios, in which the next situation depends on the student's decision, are available as well as open conversation. The AI simulations page gives examples.