Restating The Wearingly Obvious: You Can't Test What Doesn't Exist

“We are not testing the advertising since we do not have,

and cannot have, any such machine.”

Alan Hedges

Eradicating the language of testing

Used prudently, intelligently and sensitively, research can provide us with valuable learnings and inputs in the development of creative solutions.

The body of academic and industry literature on the subject of whether and how research (both qualitative and quantitative) improves the development of creative work is vast.  And life is surely too short to review all the pros and cons. But if there were one word I wish we could strike from our collective vocabularies, it is the pernicious notion that you can ‘test’ rough ideas.

I’d like to see it eradicated for the simple reason that it allows people - overtly or covertly - to abdicate their personal judgement and taste (assuming they have either). In other words it allows people to pass the buck.

Over thirty years ago in response to a commission from the UK’s Institute of Practitioners in Advertising, Alan Hedges wrote his book Testing To Destruction. Hedges’ book is a seminal advertising text. Which means, inevitably, that it is neglected by marketingland. For those with extremely Important Jobs or limited attention spans, it’s one of the shortest books ever written about the subject of advertising and advertising research.

Hedges passionately believed that we should all strike the word “testing” from our advertising research vocabulary:

“We too often speak of testing advertising (a term which should be struck from all our vocabularies) as if we were submitting the piece of film or print to a testing machine (which happens to be made up of consumers) which will accept or reject it; just like the quality control process at the end of a production line which rejects items are over or under weight, or whatever it may.”

Hedges was writing in 1974. Over thirty years ago. Methodologies and thinking have moved on somewhat since Hedges was writing. And yet why have so few learnt so little? Why do swathes of marketingland resist evolution so stubbornly?

You would have all thought we'd all moved on. So however bizarre it might seem, let's re-state the obvious for the hard of thinking.

You are researching stimulus material, not creative solutions

The inescapable but wearingly overlooked truth is that until you actually have finished work you are not testing creative solutions. 

You aren’t exploring responses to advertising. You are exploring responses to stimulus material. You are researching creative hypotheses.

You are evoking incomplete and therefore unreliable responses

This leads to the other inescapable but wearingly overlooked truth. Rough, incomplete ideas evoke rough, incomplete responses from people.

Researchers talk about interpreting and analyzing people’s responses. But that is not enough. Responses must be evaluated against the context of the stimulus material - and against the as yet unrealized executional possibilities of the finished product.

All responses, judgements and conclusions about how people respond to incomplete rough work should therefore be caveated: “Depending on how it’s done.”

Would you, for example, have committed to spending $210 million on this?

Would you have approved this?

The chances are that you would have responded ".... depending on how it's done."

The responses we elicit from people in research studies are a direct reflection of the rough nature of the work. And as such, they are an unreliable predictor of what will happen.

You simply cannot make confident and absolute predictions as to people’s responses and the effectiveness of advertising until is finished, and exposed in real-life.

So whatever else you might think you’re doing, you’re not ‘testing’ anything.

Research cannot claim to have a monopoly on the truth

Given the inherent and inescapable uncertainties any research is therefore as authoritative and as fallible as the individual judgements and opinions of those involved in the advertising process.

Research can be but another input. And it can be a valuable and helpful one. But to elevate it to decision-maker, milestone or arbiter is to overestimate its fallibility.

Living with uncertainty

There is no methodology in the world that can remove the element of uncertainty from the development of creative work. However much some might wish that to be the case.

All creative decisions ultimately involve a leap of faith. And that requires more than simple courage. It requires people with taste and vision. People with the imagination and sensitivity to think ahead and be able to picture what has not yet been created. 

If you are unable to grasp these simple truths, if you slavishly believe rough responses to rough work to be The Truth, and treat them as directives for creative development, if you are uncomfortable with the notion of taking a leap of faith... then you are not one of those people. And you are very probably in fact, in the way. 

The finished product

Here, by the way is Frank Gehry’s finished product: