Imagine computer system that can tell when customer is annoyed. system smart enough to know what triggered their dissatisfaction and able to estimate the likelihood of the customer defecting from your business to rival.
The system knows how much the customer has spent in the past and can accurately predict their future worth to your organisation. It knows if this customer brings in the high-value, profitable business you need to boost your bottom line or if they are marginal but influential when it comes to other customers. It can distinguish between someone who is usually loyal or the kind of customer who casually flits from business relationship to business relationship.
It also knows what you can afford and what the customer might accept as an incentive not to take their business elsewhere. The system can even tell your employees when is the best time to contact the customer to save the account.
This may sound like magic, but Deloitte’s Wellington-based consulting partner and leader of the company’s enterprise information management team Thorsten Engel has developed predictive systems able to offer just these insights. Computer systems that predict customer behaviour with this kind of spooky accuracy are but one example of ‘big data’.
Predictions, insights
The words ‘prediction’ and ‘insight’ come up lot when talking about big data. It is the name given to set of technologies that are changing the way the world turns information into knowledge. Like many ideas that come from the technology industry, big data is part marketing term, part buzz-word and part shorthand for series of trends that pave the way to new understanding of the world. It’s better way of making decisions.
Nate Silver, blogger working for the New York Times, famously used big data to correctly predict Barack Obama would win the US presidential election when many opinion polls pointed in the opposite direction. Physicists at CERN used big data to process results from the Large Hadron Collider to determine they found the missing Higgs Boson. And companies regularly use big data to analyse huge piles of seemingly useless data to optimise their supply chains, find new opportunities and, as the example above shows, shore up customer relationships.
In the past, managing and storing data was expensive so much of it was thrown away. And it was difficult to combine information locked up in corporate computer systems with data from other sources. Things are changing fast. Today, it costs about $20 to store terabyte – that’s 1000 gigbytes, roughly the amount of storage in laptop. By 2020, it will cost the price of latte, say $5, to store petabyte of data – that’s million gigabytes.
Commodity hardware makes big data possible; software like Hadoop, new kind of database that can process streams of data in real time – makes it practical.
Think of big data as way of sifting through hundreds of haystacks to find golden needles. Make that sift ‘quickly’ and ‘repeatedly’ through your own haystacks and the publicly available ones in your neighbourhood, to find more and more golden needles before competitor gets there.
Beyond everyday data
Purists define big data in terms of dealing with collections of data that are so large and complex they can’t be manipulated using everyday database management software and analytical tools. One definition says the term applies to any data project that takes computing professionals out of their comfort zone. But these are not the only definitions.
Deloitte’s Engel emphasises big data isn’t one single, simple idea. He says: “It is bundle of concepts. It usually refers to sifting through large number of data points and coming up with non-obvious conclusions.” Central to this is the idea of “jumbling disparate data sets together to get new insights”. Matt Lythe who manages Eagle Technology’s Graphical Information Systems (GIS) business, says big data is, “A way to access intelligence from plethora of information.”
This year’s model
Not surprisingly big data is on everyone’s lips. Gartner, an American technology trend research company, says after few years of experimentation and early-adopter success, 2013 is the year when large organisations around the world will invest in big data. The analyst firm says 42 percent of the IT leaders it interviewed either already have big data projects or will have them by the end of the year.
It’s not just large companies. You don’t need to be big to use big data techniques, you just need access to vast amounts of digital information. And it doesn’t need to be your information. Some of the solutions Deloitte has been involved in can listen in to millions of social media messages on services like Twitter and Facebook to pick up clues about disgruntled customers thinking of moving their accounts. That information is freely available to everyone.
Gen-i project marketing manager for networked ICT products, David Reiss, says companies can get big data results by using their existing tools in innovative ways. Reiss says the techniques, tools and thinking used to deal with huge datasets can all scale down to level where they are useful for mainstream New Zealand businesses. Like most other local technology specialists, Reiss is keen to emphasise big data is as much state of mind as it is about petabytes or exabytes – that’s billion gigabytes. He says even smaller businesses need to make sense of their data.
Business intelligence plus
In some ways big data resembles earlier ways of sifting information from raw data. It uses some ideas from business intelligence (BI), analytics and data-mining, but the way it employs those ideas and the information it delivers make for radical departure.
Gen-I’s Reiss says there’s an important way big data differs from established analytical approaches such as BI. He says typically, business intelligence develops specific report to order, usually for CEO or senior manager. In other words, BI sets out with clear agenda to find something specific.
But often executives don’t know what they will find when they employ big data techniques – it often uncovers unexpected insights.
To illustrate this Reiss talks of project his company ran with logistics organisation that installed fleet-management system in its trucks. Once the system collected enough data the information was sifted for those un-obvious golden nuggets. One discovery was that the organisation’s trucks spent large part of their time off the road network. This meant the organisation could pay lower road user charges based on the time actually spent on the national roads rather than on dirt tracks – an instant but unanticipated financial windfall.
Geoff Beynon, country manager for SAS, an analytical software specialist, highlights another difference. He compares BI with looking in the rear view mirror. He says you can do that with big data, but you can also get predictive insights that give you the power to drive business in new directions.
Call me loyal
This is what happens at one SAS customer, Loyalty New Zealand, the organisation behind Fly Buys cards. Beynon says the card transaction information picked up by point-of-sale terminals means Loyalty New Zealand has huge, quickly growing dataset on member behaviour. It uses SAS software to profile, segment and analyse the data to build complete understanding of customers, so it can make meaningful targeted offers based on what it has learnt about customer behaviour.
He jokes this means Fly Buys isn’t going to offer member toaster if it knows that person recently purchased one. More seriously he says pattern of buying paper nappies, baby clothes and high chairs indicates there is new family member somewhere close to the card-holder. That could be an opportunity to sell more baby products, but there’s also potential to sell appropriate financial services or get-away-from-it weekend break.
Beynon says Loyalty New Zealand uses big data so it can make i
