Feed on

How should you do a neuromarketing test? I’m increasingly being asked whether the scales from the Emotiv EPOC Affective™ Suite system can be used to assess cognitive and emotional responses in e.g. customers. After all, it would be really appealing if we could use a full box set with responses such as Engagement, Meditation, Frustration and Excitement. I also see that some new up and coming companies use this system more or less from the box. After all, who wouldn’t just pay $5,000 for a neuromarketing study rather than the more expensive studies that require whole teams and specialists?

I use the Emotiv system myself for many of my studies. For recording EEG it works well. But when it comes to their emotional scales, the truth is that they are, at best, just a black box with many unknowns. Quite realistically, it’s more like a can of worms.

From one of my own data, it’s easy to check whether the Emotiv scales are distinct. They are produced very nicely through the export function in iMotions’ Attention Tool – the best scalable neuromarketing suite I know and can think of. Best of all, it allows export of time-synced raw EEG data, along with data from eye-tracking, GSR, facial coding and much more.

So how do the Emotiv scales fare? Let’s take the example from “Meditation” and “Frustration”. After all, we should expect that scores for “Meditation” would be quite different from “Frustration” scores, right? Well, the truth is that these scales are highly correlated:


So unless you think that meditation is really about frustration (it may be to some…), you should be skeptical towards the Emotiv scales. Some of the other scales seem to fare better, such as the Engagement score. My own studies so far suggest that the Engagement score is related to working memory load, but this is indeed still a heterogenous construct, and much too premature to draw any conclusions. I do not yet know how specific Engagement scores are to working memory…and should it then be called “Engagement”?

The basic problem remains, however, that the Emotiv system is still a black box solution, and more or less impossible to determine how the scales are made. And to be honest, I would not trust a 22-year old graduate who has taken a course or two on neuroscience, nor any company using the Emotiv solution up front. Just in the same way I would not trust the same persons to perform my bypass surgery or repair my car. And yes, you may call it ad hominem attacks, but in the tradition of science this is how it is. When you speak bullshit, you are called a bullshitter.

As you can guess, I am not a proponent of black boxing, particularly not in neuromarketing where we should be able to converge on the same solutions. Quite the contrary. I simply do not understand the need for secrecy among neuromarketing companies. The science is already out there, so why make up new scales? It opens up the possibility of cheating, snake oil production and what is less. Think about the strategic blunders that may be made based on erroneous and unscientific hand waving.

If you want to do neuromarketing studies, make sure you do it right from the beginning. There is already too much hype and BS in this industry, so let’s start being self-corrective.


13 Responses to “Can you use the Emotiv scales for anything?”

  1. Jake Stauch says:

    Thank you for this Thomas! I agree that Emotiv’s affective suite indicators are highly suspect, and I know of few professionals that would be comfortable using them. But you are mistaken is assuming that this is what NeuroSpire does. We analyze the raw EEG signal, looking at frontal asymmetry as an indicator of emotional approach versus withdrawal according to the Davidson model, which is consistent with dozens of peer-reviewed papers in the field, as well as being a methodology of choice for nearly every EEG neuromarketing company in existence.

    We do not use, nor do we advocate the use of, Emotiv’s out-of-the box detections. I see that the Popular Science article may have misled you, but I’d appreciate it if you’d amend this post in recognition of this misunderstanding. And always feel free to reach out by email to chat further!

  2. tzramsoy says:

    Hi Jake

    So I stand corrected, at least in part ;-)
    It seemed quite clear from the video that you were using the Emotiv suite, so I must have misunderstood that? Still, as you know, prefrontal asymmetry can mean a lot of things, not necessarily approach/avoidance. It is known, for example, that prefrontal asymmetry itself is related both to approach/avoidance (Davidson) but also work memory load. While these two measures may be overlapping, they are certainly not identical, and you cannot infer just from an asymmetry score that you are assessing motivation (and not working memory load).

    Moreover, I will not accept that $5,000 can buy you a full suite of neuromarketing measures. It can only be done if the assessment is reserved for tests and questions that are easy to scale, and where you have benchmarked the measures you are using. I know it is not part of the tradition in marketing nor neuromarketing to provide the validation methods, but I’d still like to hear if you have any materials on this? This is, after all, something that should be based on solid science. If you claim that a measurement is an actual assessment of, say, approach/avoidance, you will need to demonstrate that the setup you are using actually predicts this behaviour. If not, you are only using reverse inference, which one can only do if there is a 1:1 relationship between the neuroimaging measure and cognitive/emotional function.

  3. Steve Genco says:

    Hi Thomas, just wanted to say how much I liked your post. You bring up the important issue of discriminant validity … if 2 measures are labeled to measure different things, do they in fact do so? On the flip side there is convergent validity … if you say your metric measures A, does it correlate with other, independent measures of A?
    Convergence would seem to be the key validation metric for measures like Jake’s. Approach/avoidance is indeed well-established in the literature (although there are also trait/state issues that need to be addressed) but I would expect a smart provider to be able to show me something like this: “here is my measure for 25 people looking at a picture of a hot fudge sundae, and here it is for 25 people looking at a picture of a dirty toilet. Any questions?”
    This, in my opinion, is the only way out of the black box dilemma. If you can persuade me with convincing validity (and reliability) measures, I can tolerate a black box metric. For example, I don’t have clue about Google’s Page Rank algorithm or Coke’s secret formula, but I know the first gets me to the results I’m looking for and the second tastes great (and the same) every time, so why should I care about the formula? Every vendor should want to provide these kinds of validation results. But in general, I agree with you that black box measures are probably counter-productive in neuromarketing. Unless a vendor can prove better performance than “known” measures (and exclusivity in producing that better performance) there is little reason to spend money on a black-box result.
    A little sidebar on the Emotiv measures. I tested the Emotiv suite with some market research stimuli (IAPS pictures) back in 2009 and my experience was that they didn’t correlate very well with traditional EEG and EMG measures (this was back before Emotiv opened up the software to deliver raw EEG). But I think, to be fair, these metrics were originally developed as inputs for video game designers (the headset was originally marketed as a hands-free controller for video games) so the games could adjust their parameters in real time based on players’ cognitive states (most importantly, overwhelmed or bored). My guess was that the measures were meant to track trends over longer periods of time, so were not really well-suited for measuring second-by-second bursts, or in the test we performed, 5 seconds staring at an IAPS image.

  4. tzramsoy says:

    Hi Steve
    Yes, exactly to the point: create good cases that highlight these points. Even better: have cases where you can show scores for products that people eventually choose and the same metrics for those they don’t choose.

    Point taken on the black box issue, but I think that for neuromarketing, there is simply not a tradition for this industry to do double blind tests in the rigid way you see in the pharmaceutical industry. So it will be all black black, and even for demos of predictive value, it will be darn hard to spot if people are cherry picking results.

    I completely agree on the Emotiv note. They have made the metrics for other purposes, and I still think of it as a toy-ish solution. Go figure what I’ll end up choosing, but there are some really good solutions out there, too.

    BTW, I will start blogging about neuromarketing for GreenBook, so check that channel out soon, too ;-)

  5. Hi.. I know this is kinda off topic but I was wondering if you knew where I could get a captcha plugin for my comment form?? I’m using the same blog platform as yours and I’m having trouble finding one??? Thanks a lot! Ajax Roofing Contractors, 15-75 Bayly St. W., #323, Ajax, Ontario, L1S 7L7, Canada, 289-275-1500

  6. [...] the same theme, I’d like to direct your attention to this post from the BrainEthics blog entitled ‘Can you use the Emotiv scales for anything?’ where [...]

  7. Thelma says:

    If you compare the price of this weight loss medicine with the price of other weight loss medicines, you will find that the price of Garcinia
    Cambogia is much cheaper. Being a full-time, or part-time,
    writer is fun, rewarding and can be very lucrative if you play it right.

  8. Ernestine says:

    Superbe article : comme d’hab

  9. US b2b list says:

    Hi, Neat post. There’s an issu with your web ite in web explorer, might test this?

    IE nonetheless is the marketplace chief aand a large portion of folks will omit your wonderful writing due
    to this problem.

    my web-site … US b2b list

  10. Un profond merci à l’admin de ce site internet

  11. Jе suis venu sur votre poste par hasard : je ne le regrette pas du tout !

Leave a Reply

Sharing Buttons by Linksku