Don't use embeddings for audience research; use reasoning models instead, especially at scale

The hard part isn't knowing this, I assume you already do, but doing it at scale. Which is what I did. And I learned something I want to share.

This article is about the technical choice underneath that scaling problem, and why the obvious answer is the wrong one. Because embeddings might not always be all you need ;)

Embeddings might just be the most powerful tools in modern NLP. They translate words, sentences, or whole texts into numerical vectors that capture whether two pieces of text are about the same subject. For topic clustering, taxonomy matching, semantic search: often exactly the right call. It's awesome, and useful.

Their power lies in something specific: they're trained to treat paraphrases as equivalent. "How do I make pasta?" and "Spaghetti preparation instructions" sit close to each other in vector space, even though they're phrased totally differently. That's exactly what you want for search: users should find answers regardless of how they phrase their question.

But that same property makes embeddings unsuitable for audience research at scale. And that IS something you definitely should be aware of.

Audience research is not topic detection. It doesn't ask "what is this question about", it asks "what emotional state is the person in who's asking it".

That's a different signal, and it doesn't live in the content of the question, but in the form.

Look at this sequence:

To an embedding model: six questions about one topic. High mutual similarity. Same cluster.

To a human reading them one by one: six different points in a consumer journey, possibly even for a single person. Might be in different points in time, or maybe even 15 minutes apart. The first is orienting. The second has the concept but wants it applied. The third is comparing strategies. The fourth feels doubt about fit. The fifth is further along, looking at concrete tooling. The sixth is close to a decision.

That's an audience research goldmine that embedding clustering flattens completely.

Google itself doesn't treat all queries equally. In late 2024, Mark Williams-Cook and his team discovered an endpoint vulnerability that revealed Google assigns nearly every query to one of eight "Refined Query Semantic Classes" (RQ-classes): SHORT_FACT, OTHER, COMPARISON, CONSEQUENCE, REASON, DEFINITION, INSTRUCTION, BOOL¹.

This classification helps Google decide which SERP features to surface and how to weigh results. The classes are derived not from topic, but from question form: the structure of how the question is asked.

Google is doing exactly what audience research needs to do: read form, not just meaning. They have to, because their job is matching results to query type, not just to topic. A bool for example, shouldn't result in a map or a knowledge panel, but in a simple yes or no. Simple example of course, but you get my point.

The issue for us SEOs though is that Google's eight classes are mechanistic. They tell you what kind of question is being asked, but not what state the asker is in. A "Comparative" query can come from someone idly browsing alternatives or from someone in a panicked end-of-quarter decision. The form tells you they're comparing; it doesn't tell you what's driving them.

So for audience research, you need both layers: the mechanical (what kind of question) and the affective (what mental state). The mechanical layer is doable with relatively simple rules. The affective layer is where things get interesting.

The form of a question carries a surprising amount of signal beyond Google's eight classes. Some concrete patterns that show up in every market (please note these are just examples):

These aren't just phrasings; they're linguistic signals of someone weighing a possible negative scenario. That's fear, or at least risk-aversion. A question starting with "what if" reveals something a "what is" question doesn't, regardless of whether they're about the same topic.

Someone asking this way isn't looking for a factual answer. They're looking for social validation, a reference point against which to measure their own choice. That's a different buying phase than someone typing "best CRM".

The same user, on the same day, on the same topic, can use any of these phrasings depending on their mental state at the moment. An audience research method that ignores this nuance misses exactly where strategic value lives.

This may sound subjective, but it isn't. The link between linguistic form and mental state is one of the better-studied things in psycholinguistics.

Carol Kuhlthau's Information Search Process model maps directly onto what audience research needs to detect². It identifies six stages of information seeking (initiation, selection, exploration, formulation, collection, presentation), and each stage has its own cognitive, affective, and physical signature. Early stages carry uncertainty and apprehension; later stages carry confidence and direction. The signal is in the language used at each stage. The model emerged from Kuhlthau's empirical work in the 1980s and has been validated across diverse user groups for more than three decades.

James Pennebaker's Linguistic Inquiry and Word Count (LIWC) framework has cataloged the linguistic categories that map to psychological states since the early 1990s³. It maps words to categories like cognitive mechanisms, certainty, tentativeness, causation, and affective tone, and has been used in hundreds of peer-reviewed studies across psychology, communication, and computational linguistics.

The principle is not contested: linguistic form encodes psychological state, and this can be measured systematically.

But I do need to make a strong caveat here: the examples I've given above are just that: examples. They are meant to tie my theoretical explanation (no doubt fully flawed) to examples you know. It really depends on the context if these examples are applicable to a given situation.

That said: that's where reasoning models come in.

I want to be careful, because this is where it would be easy to overstate things.

The traditional tools in this field are dictionary-based. LIWC counts words against fixed categories. Carefully validated, language-specific, decades of psychometric work behind them. These remain the standard in academic psycholinguistics, and they are excellent for what they do.

The problem with dictionary-based tools, for the specific use case of audience research on commercial SEO data, is twofold. First, the categories are designed for general psychological assessment, not for fine-grained buying-phase analysis. Second, adapting them to new languages or domains is significant work. LIWC has been ported to Dutch, but the validation is non-trivial.

What changed in the last eighteen months: reasoning LLMs became commercially available⁴. These are not the same thing as the tools psycholinguists use. They are a different technique that happens to be useful for a similar kind of problem. And hey, I'm an opportunistic SEO-nerd, so I see an opportunity!

The advantage of reasoning LLMs for this specific use case is practical, not theoretical. You define your framework in a prompt (cognitive phase, affective markers, behavioural intent) with explicit linguistic markers per category. The model reads each query against that framework and produces a structured classification with the reasoning that led to it. No training data required. No labeling investment. Framework adjustable per project or per market. Auditable per query.

This is not what traditional psycholinguists use, and I would not want to pretend otherwise. It builds on the same principle (form encodes state) but uses a different technique that fits a different operational reality: tens of thousands of queries per market, multiple markets in multiple languages, business cases that need answers in weeks rather than research cycles in years.

Recent research has begun validating this approach though! Reasoning-enhanced LLM classification matches or outperforms traditional methods on complex linguistic phenomena involving negation, conditionals, and contextual nuance, exactly the markers that matter for buying-phase analysis⁵.

The interesting position is in the middle: rigorous about the underlying principle (it's established science), modest about the technique (it's a new application, not a new theory).

With Skåut I analyse large volumes of People-Also-Ask queries per market that surface in search results. The earlier approach was to cluster those questions through embedding similarity and derive "audience segments" from the clusters.

That approach produced useful topic information, but no in-depth audience information. It said "your market asks a lot about X", not "your market feels Y about X".

The new approach classifies queries along several dimensions at once: where in the search process the question fits (Kuhlthau-style cognitive phase), what emotional charge it carries (affective markers), and what the asker is concretely trying to do (behavioural intent). I layer this on top of mechanical classification akin to Google's RQ-classes. The result is a much richer picture of what moves a market at a given moment.

A (hypothetical) example: in an analysis of a B2B SaaS market, more than half of the People-Also-Ask queries around a specific product category carried risk-aversion signals ("what if", "safety", "risks"), while the content of the two market leaders almost exclusively used hope-and-growth language. That's a direct strategic gap: the market is looking for reassurance; the competition is offering ambition. Whoever publishes risk-mitigation content first wins that conversation.

That's a conclusion you don't get from embedding clustering. You simply don't. A missed opportunity!

Three takeaways:

This article stands on conversations and work by others.

A particular thanks to Dorron Shapow, whose writing on what he calls Search User Optimisation and the layers underneath search queries was a real source of inspiration for how I think about the affective dimension of audience research. We come at this from different angles (me looking directly towards scalability, which is something we've discussed and maybe don't even agree upon), but his insistence that there is more to a question than its answer has shaped my own framing.

Thanks also to Arnout Hellemans for sparring on these ideas over time. He will let you folks know more soon.

And credit where it is due to Mark Williams-Cook, both for the work referenced in the footnotes and for his broader focus on People-Also-Ask as a serious data source. The tool he built (AlsoAsked) is part of why this kind of question-level analysis is feasible at all in commercial SEO contexts.

I love talking about this. Let me know if you do too.