Reading the Fossil Record: Why Data and Machine Learning Tell Us Less Than We Think

A few years back I read a post on Retail Wire that pondered whether retailers will need traditional research once mobile tracking is “in place”. The question reveals a very common flaw in how people think about research. And That flaw continues today amid all the AI and machine learning hype.

Developing conclusions from mobile data is the equivalent of scientists reading the fossil record. When I was a kid, scientists had been observing the fossil record for hundreds of years. So, they really thought they knew what the truth was. Dinosaurs were reptiles, they had reptilian skin, they were cold blooded, lived isolated lives, and modern day lizards are their direct descendants.

Fast forward to 2017. My kids learned that some (many) dinosaurs had feathers, that some (many) were herd animals, they were pretty fast moving, that some lived in family based units, some hunted together, and that birds essentially evolved from dinosaurs.

The original scientists weren’t bad at their jobs. In fact, they were brilliant. The problem was all they had was observed data. They created solid, grand theories from the observed facts they knew.

With Observed Data You Never Know What You Don’t Know. Paleontologists erred in their theories because there were thousands and millions of fossil truths they couldn’t see – they hadn’t yet been discovered or analyzed.

Mobile data puts us in a similar spot. So does purchase history, shopper behavior, browsing behavior online, phone call data, and almost every other type of data the modern AI theories are based on.

Ethnographic observers are in a similar bind as are direct marketers who rely purely on response. No matter how hard we work, observation misses more knowledge about human consumers than it captures. And without that knowledge we easily mis-lead ourselves into error.

Too often behavioral data is a type of marketer Rorshach test. Marketers and companies often project onto data finding the things that help individual careers (this is always a risk with research – it’s just a bit worse with data). Or, they project the results of the latest session with a highly paid consultant. Or are the answers they think bosses or shareholders want to hear.

What I’ve observed in corporate data digs is that the least likely thing these data expeditions find are actionable consumer truths. And quite often what I hear as major learnings are things we already knew or were simple to deduce from other information.

The big miss:  BEHAVIORAL DATA NEVER TELLS US “WHY”. The key to consumer behavior is understanding motivation – why someone makes that choice, takes that action, buys that product.

Amazon doesn’t know why. So my Amazon recommendations are incredibly meaningless – and they have 20 years worth of purchase history for me.

No amount of statistics, clever analysis, or merging of data sets can get past this error. They may help, but the limitation of observed data always remains.

Wise companies will continue to rely first and foremost on data that helps us see motivation because motivation is the key to changing profit in big ways. in modern research, its not just mobile that lacks insight into motivation. True “ethnographic research” is purely observational and is quite weak at discovering things that drive sales. (Perhaps that’s why so many firms claim to do ethnographic research but really do in-home one-on-one interviews).

To get to motivation, you have to use qualitative research of some form. It has to be executed by professionals. And it has to be interpreted with all the best care to avoid similar theoretical jumps to the errors noted above. But somehow, I find the challenges in qualitative data much more evident where the errors in things like mobile data are dramatically more insidious.

Data IS important. And companies need to use it. But let’s stop getting carried away. Data can only be valuable when combined with traditional marketing research as well as wisdom and insightful analysis.

Only then will we be following Deming’s admonition to remember that “Information is not knowledge. Let’s not confuse the two.”

Copyright 2017 – Doug Garnett – All Rights Reserved

Comments are closed on this post.