What is data mining? Analysis methods for big data

The term ‘data mining’ refers to the targeted analysis of large datasets to uncover new, po­ten­tially valuable in­form­a­tion. We’ll explain the term in more detail and outline relevant ana­lyt­ic­al methods.

What is data mining?

Data mining is the process of trans­form­ing data into mean­ing­ful insights by employing spe­cial­ised tools to extract relevant in­form­a­tion. But why is it called data mining? To better un­der­stand what data mining means, it’s helpful to first break down the metaphor. Let’s take, for example, online tracking tools. These are every­where, gathering an over­whelm­ing amount of data from visitors. While the data at first may seem useless, with data mining, it’s possible to extract mean­ing­ful in­form­a­tion from these mountains of data. Unlike tra­di­tion­al mining, data mining uses stat­ist­ic­al methods to uncover patterns, trends and re­la­tion­ships.

Data mining is typically discussed in the context of big data. This refers to data sets so vast that they can no longer be processed manually, requiring computer-assisted analysis. Data mining methods can, in principle, be applied to data of any scale. The insights derived from data mining can inform the strategic direction of online business and guide marketing decisions. As a result, data mining has a wide range of ap­plic­a­tions.

Ap­plic­a­tions of data mining

Data mining offers the pos­sib­il­ity to optimise e-commerce using a sci­entif­ic approach. Here, large data sets build the basis for ex­plan­a­tions and prognoses. Stat­ist­ic­ally processed and clearly visu­al­ised, they allow online store owners to identify factors for a suc­cess­ful online business and to model their online store marketing strategies. Data mining is used in this process to:

• Divide markets into segments
• Analyse shopping basket data
• Create consumer profiles
• Calculate product prices
• Set up prognoses on contract periods
• Analyse demand
• Identify errors in the pur­chas­ing process

How does data mining work?

Data mining is part of the Knowledge Discovery in Databases (KDD) process, which includes the following steps:

• Define ob­ject­ives: First, specific questions that the data analysis aims to answer need to be es­tab­lished. This helps to identify relevant data and suitable analysis methods more ef­fect­ively.
• Data pre­pro­cessing: The quality of the in­form­a­tion derived from data mining depends heavily on the quality of the data found­a­tion. Relevant data should be cleaned before analysis to remove du­plic­ates, outliers and other dis­tor­tions. It may also be necessary to convert the cleaned data into the format required by the analysis method.
• Data analysis: This is the stage where the actual math­em­at­ic­al data analysis takes place. The analysis tech­niques used here depend heavily on the defined ob­ject­ives and the char­ac­ter­ist­ics of the data. Both tra­di­tion­al data analysis al­gorithms and newer al­gorithms based on neural networks and deep learning can be applied.
• In­ter­pret­a­tion of results: Finally, the results of the analysis are evaluated. If the results are clear and in­sight­ful, they may reveal new cor­rel­a­tions and provide insights that can influence future business strategies.

Data mining methods

Many methods have been developed to identify important re­la­tion­ships, patterns and trends in data, enabling the ex­trac­tion of valuable business insights from large data sets. These methods can also be used for stat­ist­ic­al processes.

• Outlier detection: Extreme values that stand out from the rest of data are known as outliers. In data mining, outlier detection is used to identify atypical data sets. In practice, these data mining methods can, for example, reveal credit card fraud by exposing sus­pi­cious trans­ac­tions.
• Cluster analysis: A cluster refers to a grouping of objects based on sim­il­ar­ity re­la­tion­ships among the group members. The goal of this ana­lyt­ic­al method is to segment un­struc­tured data. To achieve this, al­gorithms like K-Nearest Neighbor (KNN) are used, which search through large data sets for sim­il­ar­ity patterns to identify new clusters. If a data set cannot be assigned to any cluster, it can be in­ter­preted as an outlier. A classic use case for cluster analysis is identi­fy­ing visitor groups.
• Clas­si­fic­a­tion: While cluster analysis primarily focuses on identi­fy­ing new groups, clas­si­fic­a­tion uses pre­defined cat­egor­ies. Data points are placed into cat­egor­ies by matching their traits with other data points in the dataset. A decision tree is a common method for auto­mat­ic­ally clas­si­fy­ing data. For each node, a char­ac­ter­ist­ic of the object is evaluated, and its presence or absence de­term­ines which node is chosen next. This process can be used in e-commerce to divide customers into different segments.
• As­so­ci­ation analysis: As­so­ci­ation analysis seeks to uncover re­la­tion­ships within datasets that can be expressed as inference rules. In e-commerce, this data mining approach can reveal cor­rel­a­tions between products in shopping baskets, with patterns like ‘if product A is purchased, product B is likely to be purchased as well’.
• Re­gres­sion analysis: Re­gres­sion analyses help create models that explain dependent variables through various in­de­pend­ent variables. In practice, this means that the prognosis for a product’s sales per­form­ance can be created by cor­rel­at­ing the product price and the average customer income level in a re­gres­sion model.

What are the limits of data mining?

In data mining, stat­ist­ic­al pro­ced­ures are employed that make it possible to carry out a fun­da­ment­ally objective analysis of available data sets. The rather sub­ject­ive nature of selecting an analysis method as well as the various al­gorithms and para­met­ers can, however, lead to distorted results, re­gard­less of one’s in­ten­tions. Such effects can be avoided by out­sourcing data mining processes to external service providers.

Finally, it’s important to note that data mining only offers results in the form of patterns and cross-con­nec­tions. Answers can only first be obtained when the analysis results are in­ter­preted with regards to previous questions and goals.