What would refute ACM Framework: how a brand method is tested
A brand framework is tested by writing down in advance which result would contradict it. ACM Framework has four conditions that would bring it down, all of them about measurement, and five that would narrow its claims, including the link with performance.
Dumitru Talmazan
I once tried on a shirt in a shop in the city centre. In the fitting room it looked impeccable: straight shoulders, the waist where it should be, warm light. I bought it on the spot. At home, in the hallway mirror, it was a different shirt. Only then did I understand that the fitting-room mirror hadn't been made to show me how I looked, but to help me buy.
01How is a brand framework tested?
You write down in advance which result would contradict you. With the Archetypal Coherence Matrix (ACM Framework), the first tests are about measurement, not sales: first you check whether the mirror is accurate, and only then what it's good for.
A framework that can't fail any test is like the fitting-room mirror. It makes you feel good, but it tells you nothing true. Every brand looks great in it, every campaign seems to have worked. That's why ACM Framework wrote down in advance the conditions under which it would have to admit that it doesn't work.
02Why does the mirror have to be accurate first?
Because ACM Framework is a diagnostic instrument. It doesn't promise that you'll sell more, but that it will show you where the chain breaks, with a figure you can check and repeat.
A distorted mirror is useless even if it makes you feel good. An accurate mirror stays useful even if it predicts nothing. In services marketing there's an instrument that has lived for forty years in exactly this position: critics showed that it doesn't predict better than others, and its authors kept it because it shows where the problem is. ACM Framework starts from the same honesty.
First you test whether the mirror is accurate. Only after that, whether it's useful too.
03What would bring ACM Framework down?
Four conditions. If any one of them fails, ACM Framework is no longer a measurement and no rewording can save it: reliability, stability, variation and distinctiveness.
- Reliability: two coders reading the same material with the same guide must arrive at similar profiles. Otherwise the Does perspective can't be measured, and almost everything falls with it.
- Stability: the same brand, measured again shortly afterwards, without having changed anything, must come out the same. If the profile moves on its own, any gap is noise.
- Variation: the four perspectives must differ, and differ from one brand to another. If all brands come out with the same gaps, the instrument tells nothing apart.
- Distinctiveness: coherence across channels must be something other than a measure that already exists. Otherwise we've just given a new name to an old thing.
The order is not up for negotiation. These four are tested before any question about sales or preference. An instrument that doesn't measure consistently isn't worth testing for prediction.
04What happens if two coders don't agree?
Nothing is reported from that material, and the threshold isn't moved to make the result come out. The training is repeated on other materials and the coding is done again.
The threshold was written before any real material was coded: on average less than 15 points of difference, and no material above 30. At least a fifth of the materials, but no fewer than twenty, are read by two people. If the second attempt still doesn't get below the threshold, that isn't hidden. It's the first refutation condition itself, come true.
05Can an AI model do the coding instead of people?
It can help, but it doesn't replace people. The model gets exactly the same guide as a human coder, and the main analysis stays on the people's codes. The model comes in as a check: if the result also holds on its codes, the conclusion is more solid.
For every coding done by the model, we keep the model's name, the version of the instruction and the justification it gave. Agreement between the model and people is calculated on the same materials as agreement between people, so the two can be compared directly. If the model gets as close to people as people are to each other, it can extend the coding to more materials. But the threshold for that is written beforehand, not found afterwards.
In the fitting room, this would mean checking the new mirror by putting it next to the old one, on the same person, before hanging it in every shop.
06What would narrow what ACM Framework can claim?
Five more conditions. If they fail, the instrument remains an accurate mirror, but what can be said on its basis gets narrower.
For example, if the 100-point profile says no more than the simple name of the dominant archetype, it's more honest to count just one archetype. If the two regimes of the gaps don't differ in the data, their rule is dropped. If the weights change nothing compared with a simple average, they're removed.
This is also where the link with performance sits. If the index doesn't connect with preference or trust, ACM Framework remains an accurate mirror that hasn't yet proved what it's good for. That's unpleasant, but not fatal. And for the link to be credible, preference is measured in a second wave, a few weeks after the first, not in the same questionnaire.
07What is a definition and what is a claim?
A definition is a calculation convention and can't be refuted, only shown to be useless. A claim says something about the world, and every claim must have written down which result would show it to be wrong.
That an archetype is present when it goes above an equal share is a definition. That the direction in which a channel deviates matters is a claim. The difference looks technical, but it helps you know what you can challenge: definitions are judged by their usefulness, claims by evidence. A claim that can't be tested with the existing design doesn't remain a claim. Either it becomes a convention, or it's declared a hypothesis for the next piece of research.
That shirt ended up at the back of the wardrobe. The fitting-room mirror hadn't exactly lied, it had just shown me what it took to make me buy. From a brand instrument it's worth asking for the opposite: to say in advance which result would contradict it. If it can't, you're still in the fitting room.
About the example in this article. The fitting room and the shirt are imaginary. The two levels of refutation, the agreement thresholds, the rules for coding with a model and the distinction between a definition and a claim come from the published ACM Framework instrument.