Big Data: defin­i­tion and examples

Seemingly om­ni­present in the business and tech world and pas­sion­ately debated, Big Data has had a profound impact on today’s in­creas­ingly digitised economy. While advocates of the business model un­der­score the different ap­plic­a­tions of this amassed data, more and more critics of this practice are voicing their concerns about privacy issues. Rev­el­a­tions from whis­tleblowers like Edward Snowden have shed new light on the sur­veil­lance practices of gov­ern­ment agencies, like the NSA, leaving many users wary of pro­grammes and ap­plic­a­tions so­li­cit­ing them to share their personal in­form­a­tion. For this reason, many citizens have developed a sense of caution and skep­ti­cism regarding the term, ‘Big Data’. But a closer look reveals that there’s much more to the practice than what’s described in newspaper and magazine headlines.

How ‘big’ is Big Data?

The term ‘Big Data’ doesn’t refer to any certain amount of data. There’s no clear line in­dic­at­ing when a quantity of data should be clas­si­fied as ‘Big Data’. Typically, however, the term refers to quant­it­ies of data that are so large that they can no longer be measured in gigabytes.

How is Big Data accrued?

Data volume has reached immense pro­por­tions: it only took 10 minutes in 2014 to collect the same amount of data that humans produced from the beginning of mankind up until 2002. According to some prognoses, this vast sum of data is only going to increase, doubling every two years. The flood of data has mostly been brought on through the in­creas­ing di­git­al­isa­tion of daily life. Big Data is created by stringing together different data sources, like:

• Mobile internet use

• Social media

• Geotar­get­ing

• Cloud computing

• Vital data meas­ure­ments

• Media streaming

Big Data doesn’t solely refer to collected data; the use of this in­form­a­tion and its analyses is also part of this defin­i­tion. The goal here is to find patterns and re­la­tion­ships and to put these into the right contexts. One of the biggest chal­lenges in doing so isn’t just having to work with the enormous volumes of data; data speed and the diversity of such in­form­a­tion also play a role. Data con­tinu­ously flows into an un­struc­tured pool. There, it needs to be gathered, saved, and processed, all in real-time, if possible. In order to properly interpret this data and put it into context, a soph­ist­ic­ated data in­fra­struc­ture is needed.

According to its defin­i­tion, the data volumes that Big Data works with are so extensive that normal software and hardware simply aren’t able to cope with such a large quant­it­ies of in­form­a­tion. When dealing with such sizeable scales, special technical re­quire­ments are demanded of the applied software. Only with the help of special frame­works can data be analysed. The software needs to be able to process as many data sets as quickly as possible and fur­ther­more has to be capable of rapidly importing large quant­it­ies of data. What’s more, the software should be able to provide users with the data quant­it­ies in real time and, if necessary, have the ability to sim­ul­tan­eously answer multiple database requests.

Hadoop is a popular open source solution, but due to its complex im­ple­ment­a­tion, using it often requires the support of a data scientist. For­tu­nately, cloud computing options are available that don’t require such expertise.

There’s a diverse range of ap­plic­a­tions for Big Data that meet the demands of many fields and topics. Even very simple, everyday ap­plic­a­tions that are known to internet users worldwide are based on this tech­no­logy. A popular ap­plic­a­tion of Big Data that can be observed on the sites of large online retailers is the small box that informs you which ad­di­tion­al products other customers who’ve viewed your current item of interest also went on to purchase. These re­com­mend­a­tions arise by eval­u­at­ing the pur­chas­ing data of other customers.

Further areas that profit from Big Data:

• Medical research: by eval­u­at­ing large sums of data, doctors can develop the best-possible therapy solutions and plans for their patients.

• Man­u­fac­tur­ing:  companies can monitor the data of their machines and hence increase the ef­fi­ciency and sus­tain­ab­il­ity of their pro­duc­tion.

• Business: Big Data helps companies get to know their customers and allows them to better match their offers to their desires.

• Energy: in order to tailor energy con­sump­tion to in­di­vidu­al use, the usage rates need to be known. Collected user data ensures a more sus­tain­able energy supply in the long run.

• Marketing: in terms of marketing, Big Data is often used in re­tar­get­ing efforts. The goal here is to improve customer re­la­tion­ships.

• Fighting crime: gov­ern­ments and defense de­part­ments around the world are using Big Data to support their efforts in the fight against terrorism.

Much of the criticism around Big Data en­com­passes data pro­tec­tion topics. Large datasets offer potential for companies and brands; thanks to Big Data, marketing measures can be more easily adjusted. However, for targeting efforts, the applied data can be used to create a precise user profile. Data pro­tec­tion activists and civil liberty or­gan­isa­tions view these measures as an intrusion into in­di­vidu­al’s privacy. 

Another con­tro­ver­sial issue is the virtual complete control some companies have over such data. Where there’s Big Data, there’s often big money to be found. All of this potential for vast profits creates an en­vir­on­ment where big players like Google call the shots. This monopoly of power over which large search engine providers preside is widely cri­ti­cised. Without clear rules and reg­u­la­tions for pro­tect­ing data and an­onymising it, there’s no way of being com­pletely able to rule out abuse.

Despite these cri­ti­cisms, Big Data can be very useful, if the tech­no­logy is correctly im­ple­men­ted. Important progress in cancer research would not have been possible if it weren’t for the power of Big Data. Data gathered from power supply and traffic systems are evaluated and used to optimise existing struc­tures. But despite all the potential offered to the dis­cip­lines of medicine, traffic control, and the business world, there are still plenty of ethical questions that need to be dealt with. Pre­dict­ing certain events, such as the prob­ab­il­ity that an in­di­vidu­al will develop a certain illness, is, at least for many, an un­set­tling prospect. A sound strategy needs to be developed in order to make sure that the rights of private in­di­vidu­als are respected, while not losing sight of the goal of a Big Data project.