The Evolution of Big Data in the Service Provider Domain

As I have spoken with analysts and pundits, it's clear that big data remains a "hot topic" within the service provider industry. What I find interesting is that while much has changed within the service provider domain; essentially, much has remained the same over the past 25 years within the expectations of their big data initiatives.

If the last part of that sentence has you thinking "this guy is nuts!" Let me explain. I think that some of the most impressive early, big data work was undertaken to support the development of MCI's Friends and Family calling plan. Through the work of carrier economists and researchers at the University of Arizona, they were able to answer an age-old question of the time: "How many people did the typical telephone subscriber frequently call?” Or, in more modern terms, what was the social network enabled and supported by the telephone! It was a first attempt to understand social networking in the context of a technology enabler, and monetize that knowledge through a new service offering.

At that time, it required spinning through months of billing records that were housed on reel tape and running on mainframe computers that were slower than the laptop in your backpack. With months of time, they were able to achieve their answer. And, it was strikingly simple; three numbers constituted the bulk of the average customer's calling circle! For years afterwards, we would continue to see if that pattern had changed… it didn't and the phone companies disbanded the departments and reassigned the economists to other areas -- at just the wrong time because big changes lay ahead!

What changed? Everything!
The most significant technological change is that compute and storage have radically changed. Gone are the days of reel tape and in its place we have Hadoop databases. Data that used to take months to groom is now available in minutes. This increase in speed and flexibility is what makes it possible for Facebook to know when your relationship is going south or for Google to know where you traveled or for your phone to know how fast you were travelling. Clearly, processing a lot of data is no longer the challenge.

Simpler world of known end-points and QoS by design
More importantly, the change that occurred in the service provider's business model had a profound impact on their big data efforts. When the RBOCs were participating in the National Telecommunications Demand Study, the world was much simpler. All a subscriber could purchase from an operator was plain old telephone service (POTS). The operator knew where the call originated and terminated because users were charged per call. There was no need to worry about Quality of Service (QoS) as the network was engineered to run at 99.999% reliability.

Moving away from certainty
With the introduction of high-speed internet service, it all changed! The network had become a communications platform. For the first time, operators would offer bulk service for a flat price. There was no longer a need to know the details of each session; instead, it was important to understand "how much" had been consumed. Since there was no knowing of where the traffic was going and it was a data network, it was offered as "best effort service."

Welcome the competition
With the emergence of broadband (IP) services, operators saw the emergence of competitors. For the first time, consumers would have a choice of service providers; mobile, cable and fixed. The upshot of competition is that we saw lower prices, bundles and, most importantly, innovation. Services quickly expanded to include voice, chat/IM, video and a slew of over-the-top traffic.

Exponential power of online social networks
While the telephone encouraged users to be social, it was by its nature a person-to-person communications vehicle. The age of broadband ushered in digital social networks that allowed users to become even more social.

Now, our communications circles expanded beyond three, one-to-one sessions to instantaneous communications with family, BFFs, super close friends, close friends, casual friends , acquaintances, followers, and the list goes on. Gone are the days when word of poor customer experience would propagate with a person telling nine to 15 people.. Now, with a post on Facebook the word is instantaneously transmitted to 200-1,500 friends or via Twitter to potentially millions of followers.

Lost in translation
Unfortunately, what was lost in the rush to provide new services and keep up with the competition is exactly what we tirelessly mined to no avail before all the change occurred -- session-level detail. With bulk billing, we lost the visibility to what users were doing at a micro level. With IP networking over non-deterministic paths, we could no longer take network reliability at five-nines as a given. Perhaps worst of all, users had options if services were not meeting expectations.

What hasn't changed
While much has changed over the past 25 years, arguably, there are many aspects of a carrier's big data work that remain the same. There is still a desire to understand consumer behavior. Millennial behavior is radically different from generation X/Y and baby boomer behavior. What are subscribers doing? As their IP packets traverse the network, where are they going? What was their experience? Are there opportunities to enrich the user experience based on demographic, psychographic, geo-location, and session-level knowledge? Can we learn when subscribers are about to churn and programmatically entice them to stay rather than having customer care rank the value of the customer as they are headed for the door and offer them enticements to stay? It's the ability to launch new services with a prior knowledge of take rates and targeted marketing profiles.

Back to the challenge of carrier big data
To get back to the point of big data, answering the hard questions about subscribers and how they use the network, so that operators are profitable, subscribers are happy, and regulators and shareholders are satisfied, we need to get back to basics. We need to know what is happening on the network; all the time, with all the traffic. To do this, operators need to leverage packet-level visibility. With brand reputation on the line, it is no wonder that big data remains a "hot topic" in the industry.

To those who worry about privacy, payload-level details can easily be masked to maintain privacy, but the insight this level of granular detail provides is absolutely critical for the long run success of service providers. As I've told many an industry analyst and pundit, the real challenge with big data is coming up with the questions to ask. What do you think?