LONDON, United Kingdom — For thousands of years, people have been gathering and analysing data to understand and organise the world. But in the analogue age, collecting and processing data was costly and time-consuming, meaning that statisticians and other information analysts were generally limited to working with small data samples.

Digitisation set the stage for a transformative shift. From Google search queries and GPS signals to transaction records and social media updates, today’s information society emits an estimated 2.5 quintillion bytes of data every day. At the same time, increased computing power, cheap storage and new data-crunching technologies have given us the ability to analyse a far larger volume of information than ever before, extracting insight and creating new forms of value in ways that stand to radically change the way consumers, businesses and governments operate and interact.

Fundamentally, 'big data' is about making predictions. Indeed, it's now possible to leverage large amounts of messy, real-world data to build predictive models that can find patterns, establish correlations and infer probabilities with enough accuracy to help us do things like pick the best moment to buy a plane ticket, foresee the spread of deadly infections or identify emerging fashion trends.

To find out more, BoF spoke with Viktor Mayer-Schönberger, professor of internet governance and regulation at Oxford University, and Kenneth Cukier, data editor at The Economist, about the power of big data and what it means for the business of fashion.

How do you define big data?

Big data describes our recent ability to collect and analyse much more data about a particular issue than ever before to gain new insights and offer innovative products and services. It will affect all sectors of our economy and all aspects of our existence, from business to healthcare to education.

To give a sense of how much more data we have these days, consider that when a new major telescope facility begins operations, it tends to collect as much information in the first week as was collected in the entire history of astronomy up until that point. And then it does it again, and again, and again, week in and week out, until a new telescope goes online and we see another step-change in the amount of information that is gathered. Or, think of biotechnology. It took ten years and $1 billion to sequence the human genome in 2003. Today, a single lab can do that in two to three days at a cost of less than $3,000.

How does big data change the way we understand the world?

We currently understand the world through hypotheses — ideas on how exactly a piece of the world works — that we have tested against a small amount of data because collecting data has been so costly and time-consuming. That, of course, limits us in how many ideas we test, as we often have to re-collect the data for a slightly different idea. It makes understanding hard and slow. In contrast, with big data, we can have algorithms test hundreds of millions of hypotheses automatically against an often large dataset.

For example, Google was able to track the outbreak of the seasonal flu using search terms alone. They didn't do that by guessing which terms would best correlate with the flu. They get 3 billion queries a day. Trying to think through what the right terms might be (fever? sneeze? cough medicine?) would be a fool's errand. Instead, Google took the 50 million most common search terms and ran them through 450 million mathematical models to determine which search queries could be used to predict flu outbreaks.

It meant giving up our innate preference for causality and placing our trust in correlations — that is, knowing what, but not why. We don't know if Google's model works because someone goes online when they're feeling ill — they may be overhearing sniffles in the cubicle next to them. We don't know and we don't need to care. To track the flu with searches, correlation was good enough.

What are the implications for business? In your book, you describe data as "the oil of the information economy." Can you unpack that?

Data has always been useful for businesses. It enables economic transactions and helps supply meet demand. After all, price is data, as much as certain product qualities and transaction terms. But until recently, data was seen largely as the lubricant that greases the machine of commerce.

In the age of big data, data itself becomes the good that’s being traded. This shift happens as we realise that the value of data is not exhausted when it’s used for the purpose it was collected. Rather, we can use data for novel, additional purposes that nobody thought about when it was collected.

Who would have thought, for instance, that search terms sent to Google can be repurposed to predict the spread of the flu?

Another example is Farecast, which was acquired and became part of Microsoft's Bing Travel. Farecast told people whether the price they were quoted for an airplane ticket was likely to go up or go down, empowering consumers by letting them know if they should buy right away or wait. It worked by crunching data on previous airfares. In fact, it processed 200 billion flight price records that amounted to almost every seat, on every plane, for every route, every day for an entire year across all commercial aviation in the United States. That's a lot of data with which to base it's prediction. Farecast saved travellers a lot of money and Microsoft eventually bought the company for over $100 million.

But the key is this: Farecast's brilliance was to take information generated for one purpose — selling tickets — and apply it for another. The data had become a raw material, a vital economic input. That's what we mean when we say that data is the oil of the information economy.

Big data is often discussed in the context of technology companies like Google and Microsoft. How can retailers harness the power of big data? Who is doing this well?

Internet companies are some of the first to use big data because they almost viscerally understand the importance of data. Their businesses are often founded on it. But many others can harness the success of big data. What’s crucial is being able to either collect or access relevant data easily.

Take large retail chains: through loyalty cards and other methods they are able to collect a staggering amount of information about what people buy — brands, sizes, types, colours, styles — and when and where they buy. This data is used for transaction and payment, restocking and inventory management, and, in the best of cases, for sales promotions and coupons. But much more could be done, for instance, by analysing and optimising what products are being displayed next to each other, or close to the check-out counter, and when.

To cite one example, Walmart discovered that before a major hurricane, not only did sales of storm supplies spike, but so did sales of Poptarts. Who would have thought? But by seeing the correlation, they could act on it by placing Poptarts at the front of the stores next to the flashlights and batteries, thus making shopping easier for customers and boosting sales.

In trend-driven product categories like fashion, accurately predicting consumer demand is a complex matter. Historical sales data never results in consistently better commercial decisions, while traditional forecasting tools are slow and unscientific. Can big data help?

So far, predicting consumer demand for fashion has been the domain of self-styled ‘experts,’ focus groups and relatively unsophisticated models based on ‘small’ data. Collecting and then analysing actual preferences from potential customers was just too costly and hard to do. This is changing.

As we collect and analyse far more data about people's interactions, individual preferences will become much better known, more comprehensively and in greater detail than ever before. That provides valuable insights for the fashion industry, from what products might perform best, in general, down to what will likely sell well in which store locations, what products are successful when placed next to each other and how to optimise retail experiences.

How else can big data give fashion companies a competitive edge? How can big data analysis inform activities like identifying creative talent, product development and marketing?

Big data will be used to predict customer preferences. Of course, this does not mean that innovative design and original ideas are being replaced by numbers. Rather, the numbers can help designers identify in which direction to go, where to push harder and how to excel in satisfying customers.

And the predictive insights of big data are not limited to understanding customers only. It will enable brands to pick the most promising creative talent earlier and with a better success rate.

Marketing and advertising will become more efficient, too. For instance, advertisers today rarely know how well billboards are working, because we have little actual data about how many people look at these ads. As we collect data about the human gaze — just think of Google Glass — we'll be improving advertising, too.

In a sector as subjective and seemingly unpredictable as fashion, what are the limitations of big data? Is there still room for intuition?

Absolutely. Intuition is central. After all, analysing customer preferences would not easily have revealed that people wanted to buy cars before they were invented; they might have, instead, just wanted a faster horse, to paraphrase Henry Ford. But in the age of big data, intuition will not compete with data about what customer preferences are. Rather, human intuition will be needed precisely because data can never tell the full story and surprise and serendipity are central to human nature.

How might big data change the fashion industry in the years to come?

Every aspect of the business will change, from what colour will be in next season to how to make clothing that fits different body types and how to optimise supply chains.

Viktor Mayer-Schönberger and Kenneth Cukier are the authors of Big Data: A Revolution That Will Transform How We Live, Work and Think, published by Eamon Dolan/Houghton Mifflin Harcourt.