THE CURIOUS MARKETER. Small lies, big lies and big data

The most insightful book I have read this year is Everybody Lies , written by Seth Stephens-Davidowitz. This is a book every thoughtful marketer should read, because it argues that much of what we have understood about human behaviour may actually be incorrect. As marketers, understanding how consumers think and behave is critical to our profession. Therefore, it is somewhat startling to read in this book that the conventional methods through which we figured out consumer attitudes and behaviours may well be faulty, because everybody lies.

The author, a data scientist with a keen nose for human nature, tells us that lying is intrinsic to human nature. In his own words, “People lie about how many drinks they have had on the way home. They lie about how often they go to the gym, how much those new shoes cost, whether they read that book. They say they will be in touch when they won’t. They say they love you when they don’t. They say they are happy while in the dumps. They say they like women when they really like men. People lie to friends. They lie to bosses. They lie to kids. They lie to parents. They lie to doctors. They lie to husbands. They lie to wives. They lie to themselves.”

And so, if people can lie to their loved ones or to themselves, they can surely lie when responding to questionnaires and surveys. Therefore, if surveys of some kind or the other are an important source of information and insights for marketers, are we getting it all wrong? Seth points out that lying played a big role in the failure of surveys and polls to predict Donald Trump’s victory in 2016. In India too, we know of several poll surveys that have similarly predicted entirely incorrect results. Many of us in the marketing profession are painfully aware of surveys which predict that lots of people will line up to buy a new product – but eventually, the new product does very poorly, because there are very few consumers actually wanting to buy. Clearly, in some situations, what people say is not what they will necessarily do.

The author does not debunk surveys altogether. He points out that under some conditions, surveys can indeed elicit honest answers. For instance, he highlights how people are likely to be more honest if they are alone while answering a survey, than if there are others around. Also, internet surveys, where a person is alone with his or her computer, are likely to be better at drawing out truthful responses than in-person surveys. He points out, however, that any survey method will elicit a significant number of lies or misrepresentations, because, as he points out, “people have no incentive to tell surveys the truth”.

Given this reality of flawed surveys and everybody telling lies ever so often, how best can marketers and other researchers learn the real truth about people’s thoughts, attitudes and preferences? Seth Stephens-Davidowitz’s hypothesis is that perhaps the only consistently authentic source of such information is the internet and big data, because many online sites get people to tell the truth about themselves, in a manner that they would not do anywhere else.

Consider Google searches, for instance. If a person is suffering from depression or loneliness or some other secret or potentially embarrassing problem, he or she is unlikely to admit it to another person or to an in-person survey, and particularly so in the Indian context where many of us do not like to speak about these illnesses. But this person does have an incentive to search on Google, in the privacy of his or her own room, for symptoms and for possible products, solutions, doctors or treatments which can help address these problems.

The author tells a superb story about how Netflix, in its early days, learnt about our lying habits, and then used online big data to discover the truth. At that time, Netflix permitted its users to create a queue of movies they wanted to watch sometime in the future, and Netflix would then duly remind users from time to time to watch these same movies. However, users would rarely respond to such reminders. Because, as it turned out, plenty of these movies which people put in the queue were aspirational films based on noble literature, or well-regarded documentaries, and such. On the other hand, what people actually wanted to watch over a weekend were comedies, action thrillers and romance films (from my own weekend movie watching habits, I completely agree.).

Netflix discovered this truth when it built a model based on the millions of online clicks, searches and views of similar customers. So the company stopped asking customers what they wanted to see, because people were lying to themselves when they answered such questions. Instead, it put its trust in online data, and began recommending movies that were based on analysis of such data. The happy result was that customers visited Netflix more frequently, and watched more movies.

How best can marketers use online data, including Google search data, to learn the truth about their customers? The author strongly recommends the use of Google Trends, which has been the source of much of his own research data. This is a tool provided by Google which, in its essence, tells users how frequently a word or set of words has been searched for in different locations at different times. While this tool only permits comparison of the relative frequency of searches, and does not reveal any absolute numbers or other details, it is an invaluable source of truthful data, because when people privately seek knowledge online using a search engine, there is an inherent truthfulness in their behaviour, and hence in the big data trail that they leave behind, even at an aggregate level.

In the book, Seth Stephens-Davidowitz powerfully articulates his view of the four invaluable powers of big data derived from the internet, in a world where everybody lies. Big data offers up new types of data which were not earlier available. Big data provides honest data. It allows us to zoom in on small subsets of people and understand them better. And big data also permits us to do many small and controlled experiments very rapidly.

All these powers of big data and the internet are very relevant to marketers. This book is written in a very engaging style, with lots of anecdotes and stories centred on what can be learnt about human nature and behaviour, if we obtain the right sets of big data, and ask the right questions of such data. It also lays out some of the ethical dilemmas in this area, and what we should not do. But most importantly, it highlights how there is a digital goldmine out there, which is waiting to be tapped, and which reveals the truth far more powerfully than ever before. If you want to know how to discover the real truth about your consumers, I strongly recommend you read Everybody Lies . It is a ground-breaking and fascinating book.

Harish Bhat is Brand Custodian, Tata Sons, and author of The Curious Marketer. These are his personal views. He can be reached on bhatharish@hotmail.com.