Wizard of Oz prototyping is a test in which the user gets the output of an automated product while a hidden person performs the function. It measures whether that function earns its place before you invest in the sensors, the electronics and the software that would make it real.
What Wizard of Oz prototyping validates comes down to one thing, the value of the function and never its technical feasibility. What goes on test is not the machine, it is the promise. If the person in front of the unit believes it detects, doses or sorts on its own, their reaction measures the value of that function rather than the quality of a control system that does not yet exist. That is why the technique belongs to the ideation and concepts stage, where an idea is dropped in days, and not to the engineering stage, where dropping it costs a whole project.
Deciding which functions deserve to be automated is a product development decision taken in design and innovation, before any sensor specification exists. When that decision goes wrong, the error does not show up in the test, it shows up with the firmware written and the tooling already machined.
IDEATION AND CONCEPTS
Are you about to automate a function without knowing whether the user will reach for it? Take a look at our work in ideation and concepts
What Wizard of Oz prototyping is and which part of the test is real
In a Wizard of Oz test the interface is real and the mechanism is fake. The user presses, speaks or drops a part in, and on the other side an operator they cannot see does the work the machine would do, with the same output and in a comparable time.
The name comes from the scene where the wizard turns out to be a man pulling levers behind a curtain. That curtain is the only thing separating a valid test from a pointless deception, because the moment users sense somebody is deciding for them they stop behaving like users and start behaving like examiners.
Wizard of Oz and Concierge, the difference is what the user knows
The difference between Wizard of Oz and Concierge is what the user knows, not how the test is built. In Wizard of Oz they believe they are interacting with an automated product, while in Concierge they know a person is delivering the service.
That distinction changes what each test lets you read. Wizard of Oz gives you a spontaneous reaction, because nobody forgives a machine what they forgive an operator who is visibly trying. Concierge, by contrast, lets you ask questions mid session and explore several versions of the function in a single visit, at the cost of losing that clean reaction.
For a function the user will operate unaided, Wizard of Oz is the right setup. For a function whose criterion is not defined, Concierge pays off more, because what you are after is not a score, it is the rule an expert would apply.
What gets simulated and what has to be authentic in a Wizard of Oz test
The only thing simulated is the mechanism. Everything the user perceives has to be authentic, because that is where the data comes from, and that includes the interface, the format of the answer, the delay and the errors.
Delay is the parameter that ruins the most tests. A hidden operator answers immediately, without the latency of a capture, a computation and a decision, so they deliver an experience the real technology will not be able to match. The rule that keeps the test honest is to cap the simulator at the performance the final system could actually reach, which forces you to estimate the plausible response time first, which is the prior job of technology selection and testing.
The same applies to failures. An automated system gets things wrong, and a hidden operator who never does is selling a reliability that does not exist. Writing into the script a share of errors consistent with the candidate technology turns the session into a test of the function rather than a sales demonstration.
The validity of a Wizard of Oz test depends on the simulator being limited to what the real technology could do. A hidden operator faster and more accurate than the final system does not validate the function, it measures the user’s enthusiasm for a capability nobody is going to deliver.
Which decisions Wizard of Oz prototyping closes and which it leaves open
Wizard of Oz prototyping closes questions of value and use, and closes no question of technical feasibility. Mixing the two groups is where projects that beautifully automate a function nobody uses come from.
Value of the function, how often it gets used and willingness to pay
The test answers whether the user reaches for the function when it is in front of them, how often they would use it and in what order they expect the steps to happen. These are answers about observed behaviour and not stated opinion, and that is why they are worth having.
ISO 9241-210 sets out the requirements and recommendations for human-centred design of interactive systems, and one of its principles is that the design is refined through evaluation with users rather than by the development team’s judgement alone. Wizard of Oz is an inexpensive way of applying that principle while the system still cannot be built, and it sits alongside the rest of the user studies that come before a scope decision.
It also tells you something about price, though with care. A function users ask for when you take it away has demonstrated value, and one you have to remind them exists does not, whatever they ticked on a questionnaire. The in-depth interviews run to validate an extraction solution show the same principle applied one stage earlier, when what you are testing is the need and not yet the function.
Technical feasibility, cycle time and accuracy, what the simulation does not measure
No technical conclusion about the automated system comes out of a Wizard of Oz test, because there is no system in the test. The hidden operator proves the function is useful, never that it can be built.
The questions left untouched are the ones that decide the budget. Whether a sensor detects the defect with the part in motion, whether the assembly holds the line’s cycle time, how many false positives it will generate and what the necessary electronics cost are matters for technical feasibility analysis and for testing, and they get answered with the real sensor and the real part in front of you. The computer vision system built to classify delivery notes automatically is an example of that second stretch, the one that starts once you know the function deserves to exist.
The table separates one kind of question from the other, with what it takes to close each one and the price of closing it too early.
| Project question | What the Wizard of Oz test answers | What it takes to close it | What happens if you close it on Wizard of Oz |
|---|---|---|---|
| Does the operator use the function when it is in front of them? | Answers it in full | Nothing more, the real interface is enough | Nothing, this is its home ground |
| In what order do they expect the steps to happen? | Answers it by observing the session | Nothing more if the script does not lead | Nothing, provided you did not suggest the order |
| Does the sensor detect the defect with the part in motion? | Does not answer it | A test with the real sensor and part | You write a specification around an unproven capability |
| Does it hold the line’s cycle time? | Does not answer it | A rig with the final actuation | You promise a rate the unit will not deliver |
| How many false positives will it produce? | Does not answer it | A run of conforming and non-conforming parts | You size the rework station blind |
| What do the required electronics cost? | Hints at the scope, gives no cost | An architecture and a bill of materials | You budget around an undefined function |
The first two rows are settled in an afternoon with a curtain and a script. The next four need material, instrumentation and laboratory time, and that split is exactly what makes Wizard of Oz a profitable tool.
TELL US ABOUT YOUR CASE
Not sure whether the function you want to automate is worth the electronics it needs? Send us the function and the profile of whoever will operate it and we will propose the test setup and the hidden operator script
The cost of mistaking desirability for feasibility
The mistake is not paid for in the test, it is paid for in the next stage. A Wizard of Oz test that is well built and badly read authorises a development the technology cannot sustain, or kills a function that would have worked.
Automating a function the user never gets round to using
The most expensive false positive is the function that delights in the session and is abandoned on the shop floor. It happens when the hidden operator has handled with no visible effort a case the real system handles badly, and the user has rated an imaginary performance.
The bill arrives in the automation budget, the longest lead item in the project and the least reversible. A chosen sensor, a frame designed around it and software written against its output form a block that is hard to undo, and redoing it means going back to the architecture rather than adjusting a parameter.
Dropping a good function because the simulation was badly built
The false negative is quieter and therefore more common. The function gets dropped because the session went badly, and the session went badly because of the setup, not the idea.
The usual causes can be anticipated. A hidden operator applying a different criterion each time, an answer format nobody understands, a curtain that shows, a script that asks instead of observing, and out comes a rejection that describes the experiment and not the product. Before burying a function it is worth checking whether the setup represented it, in the same way that before dropping a design over a failed test it is worth checking whether the part broke or the process that made it broke, a problem the low fidelity prototype raises in the physical world.
A rejection in a Wizard of Oz test is only data about the function if the simulator applied the same criterion in every repetition. If the hidden operator improvised, what you measured is the operator’s variability and not the usefulness of the function.
How to build a Wizard of Oz test that supports a decision
A Wizard of Oz test is an experiment, and as an experiment it needs a hypothesis, a criterion and a record. Without those three it is a demonstration with an audience.
The script fixes what the hidden operator does for every possible input, so that the simulator’s response is reproducible. It is the algorithm that does not yet exist, written in plain language, and drafting it reveals the decisions the automated system will have to make earlier than any test would.
The technical specification ISO/TS 20282-2 defines a user-based test method for measuring the usability of consumer products in terms of effectiveness, efficiency and satisfaction. It is written for a final measurement rather than an exploratory test, but it is a good guide to what deserves recording, tasks completed, times, errors and the exact point where the user stops. What is not recorded in the moment cannot be recovered afterwards, because the session never repeats under the same conditions.
That leaves consent, which is not a formality. Participants are told they are testing a prototype and that the session is being recorded, and the only thing held back is which part of the system is automated. At the end the full setup is explained to them, which is what separates an experiment from a deception and also lets you ask what they would have done differently had they known.
From Wizard of Oz to testing the real automation
The output of a Wizard of Oz test is a specification, not a verdict. What it delivers is the function worth automating, the criterion it has to apply and the error level the user tolerates, and that is precisely the input for the next stage.
With that brief in hand, the proof of concept stops being an open exploration and becomes a bounded verification, because you know what the sensor has to demonstrate and within what margin. This is the point where the error tolerance recorded in the session turns into a measurable threshold and the list of cases in the script turns into a test series.
The order matters more than the speed. A one-week Wizard of Oz that fixes the criterion saves months of development against the wrong criterion, and skipping it does not bring the project forward, it brings the spending forward.
Wizard of Oz prototyping does not replace testing, it decides whether testing is worth it
Wizard of Oz prototyping separates two questions projects tend to answer together, whether the function adds value and whether the function can be built. The first is answered with a curtain and a script, the second only with the sensor, the part and the instrument in front of you.
Put in that order, each stage pays for itself. The session drops ideas for the price of a morning and leaves in writing the criterion for judging the technology, and the test verifies one specific capability instead of exploring open ground.
If you have a function you intend to automate and you still do not know whether the user will reach for it, send the description of the function, the profile of whoever will operate it and the technology you have in mind, and you will get back the test setup, the hidden operator’s script and the tests that would be needed afterwards to confirm that the technology gets there. Talk to the product development team before you write the sensor specification, not after.
Frequently asked questions
Is it legitimate for the user not to know there is a person behind it?
It is legitimate when there is informed consent and a debrief. Participants are told before they start that they will be testing a prototype and that the session is recorded, so they know they are taking part in an experiment, and the only thing held back is which part of the system runs on its own. At the end the full setup is described to them and they are asked what they would have done differently, often the most useful data of the session. What is not acceptable is using the setup in a sales demonstration or an investor presentation while presenting it as a finished product.
Does Wizard of Oz prototyping work for an industrial product or only for software?
It works for any product whose main function is an automated decision, and in industry that covers inspection, sorting, dosing and condition-based stop or start. The physical setup differs, because you have to hide an operator and not just some code, but the logic is the same, the user puts the part in and gets back a verdict issued by a person. For products whose main function is mechanical rather than a decision, the technique adds little and the route is a minimum viable prototype that reproduces the real behaviour.
What drives the cost of a Wizard of Oz test?
Cost is driven by the number of sessions, the fidelity of the interface the user sees and the effort of hiding the mechanism. A test with a tablet and an operator in the next room is cheap, and one needing a believable enclosure, a prepared space and several user profiles brought in moves up an order. The item that surprises is not the mock-up, it is recruitment, because participants have to be the people who will use the product and not whoever is available. Against the cost of automating the wrong function, the test is a small fraction of the budget.
How many users does the test need to be worth running?
Deciding whether a function adds value takes few participants, because the data you are after is qualitative and usage patterns repeat early. What you cannot do with that sample size is estimate acceptance percentages or project demand, which need a planned sample and a different method. The practical rule is to stop when two consecutive sessions produce no new behaviour, and to pair the test with in-depth interviews when you need the reason behind what you observed.
How long does it take to go from Wizard of Oz to a prototype with the real automation?
The usual lead time runs from one to four weeks depending on the technology involved, with an urgent option of 24 to 72 hours for critical cases. The typical split is short on turning the script into a specification, medium on setting up the test and variable on obtaining representative parts, which is what governs the calendar. Send the function and the candidate technology and you will get the test plan with its lead time, along with the criterion the sensor will have to meet.
