In-Memory Databases

Digital com­mu­nic­a­tion generates large amounts of data. This is a great op­por­tun­ity for companies that work with big data. However, the more data a company has to work with, the greater the challenge becomes to recognise con­nectiv­ity and patterns. IT solutions and systems are ever more in demand to support companies in eval­u­at­ing the huge amounts of in­form­a­tion they receive. Data analysis using tra­di­tion­al databases is no longer suf­fi­cient to store, retrieve, and process extremely large col­lec­tions of data. When classic databases reach their limits, in-memory databases can be of use.

An in-memory database (sometimes ab­bre­vi­ated to db) is based on a database man­age­ment system that stores its data col­lec­tions directly in the working memory of one or more computers. Using RAM has a key advantage in that in-memory databases have sig­ni­fic­antly faster access speeds. Stored data is then available very quickly when needed.

In-memory dbs store large amounts of data and provide a range of analysis results. But how exactly does big data storage work, and what kind of tech­no­logy makes big data storage possible?

When storing data with an in-memory database, a dis­tinc­tion is made between column-oriented and row-oriented data storage, whereby some database systems use both methods of data storage. Row-oriented databases arrange the collected data records together in one row. For example, if the values “name, city, and country” are stored, the data would be arranged as follows: name 1, city 1, country 1, name 2, city 2, country 2. In a column-based storage, the data is assigned to its cor­res­pond­ing cat­egor­ies in this way: name 1, name 2, city 1, city 2, country 1, country 2.

The storage format for column-based data storage is called columnar format. By storing data with identical values together, the system minimises the number of existing data. Storage space and trans­mis­sion times are therefore reduced. The analysis per­form­ance of in-memory databases has also improved over time, seeing as only specific columns need to be analysed, and not all of them. This form of data eval­u­ation is called columnar pro­jec­tion.

The concept of in-memory databases is nothing new. The found­a­tions of database tech­no­lo­gies were developed as early as the mid-1980s. However, the IT systems back then did not have the required pro­cessing capacity, so that earlier concepts of in-memory databases could not be used. Modern computer ar­chi­tec­tures such as data ware­hous­ing, 64-bit tech­no­logy, and multi-core pro­cessors finally made it possible for in-memory databases to be put to use.

• In-memory databases usually belong to a data warehouse. These database systems collect and compress data from various sources, save it for the long term, and prepare it for analysis.
• With 64-bit tech­no­logy, it is possible to increase the capacity of the main memory up to the terabyte range. As a result, in-memory DBs have grown in size.
• With multi-core pro­cessors, multiple processor cores work in one chip, resulting in better pro­cessing per­form­ance and higher data per­form­ance. The data per­form­ance shows the net data volume trans­ferred.

Recurring, identical processes occur during the running of in-memory databases. An in-memory database backs up data in the following way:

1. Start the database: when the database is started, the system loads the entire dataset from the hard disk into the working memory. This means that no data has to be loaded while the database is running.
2. Re­ad­just­ing data: the database will review and adjust data fre­quently, so if data changes it remains up to date.
3. Trans­ac­tion log backups: current changes are recorded in trans­ac­tion logs. If an error occurs, the database can be restored to the time before the error occurred. This process is called "roll­for­ward".
4. Data pro­cessing: data is processed according to the ACID principle (atomicity, con­sist­ency, isolation, and dur­ab­il­ity), as it is in tra­di­tion­al databases. The acronym ACID describes the ideal prop­er­ties of processes in database man­age­ment systems.
5. Database rep­lic­a­tion: this step con­tinu­ously copies data from the database to a computer or server as a backup.

The advantage of in-memory databases is that accessing data is much faster than with a tra­di­tion­al database, although this is also the root of the greatest dis­ad­vant­age of in-memory databases, because it is not possible to per­man­ently store data on the RAM. Here is a com­par­is­on of the ad­vant­ages and dis­ad­vant­ages of in-memory databases.

The biggest advantage of using in-memory databases is the sig­ni­fic­antly higher access speeds resulting from the use of RAM. This also leads to a quicker data analysis. However, it’s not only the reduced fetch time that optimises data analysis. In-memory DBs make the eval­u­ation of struc­tured and un­struc­tured data possible from any system. Until now, companies and software solutions have been faced with the challenge of storing and pro­cessing large amounts of un­struc­tured data, such as texts, images, or audio and video files.

By using dis­trib­uted data in­fra­struc­tures, un­struc­tured data can be stored in an in-memory database, in which several pro­cessing units (computers, pro­cessors, etc.) work on a common task in parallel and dis­trib­ute it to different server clusters. This results in both a higher storage capacity, faster pro­cessing, and better transfer speed of the un­struc­tured data.

The use of an RAM means faster access on the one hand, but also brings with it a key dis­ad­vant­age: the data stored is only temporary. If the computer system should crash, all tem­por­ar­ily-stored data would be lost. To combat this, the following methods have been es­tab­lished:

• Snapshot files: there are specific moments, such as in routine intervals or before shutting the system down, when the current version of the database is saved. An important criticism of this is that all data added after the last so-called snapshot would be lost in the case of a crash – depending on how long each interval is, this could be a lot of data.
• Trans­ac­tion protocol securing: noting change in trans­ac­tion logs is an in­teg­rated process used as a means of security. Used in com­bin­a­tion with the snapshot process, the trans­ac­tion protocol can help restore a system after a crash.
• Rep­lic­a­tion: in-memory databases already largely have the function of storing an exact copy of the database on a con­ven­tion­al hard disk. In the event of a failure, the stored database can be accessed.
• Non-volatile RAM memory: a non-volatile RAM memory is able to keep files available for retrieval even after the system has been restarted.

Another dis­ad­vant­age due to RAM use is that the computer itself no longer has as much RAM available. Grid computing can be a solution to this lim­it­a­tion. Grid computing connects many different computers, and for a computer to connect to this link, spe­cial­ised software must be installed on the computer. By merging the unused ca­pa­cit­ies, a virtual, high-per­form­ance computer is created.

A database is generally un­der­stood to be a col­lec­tion of in­form­a­tion that is available elec­tron­ic­ally. Tra­di­tion­al databases only store struc­tured data. Struc­tured data means clearly organised and defined data fields in concrete data records. The data records are arranged in tables, where each data field rep­res­ents a different attribute and is named ac­cord­ingly. The big data movement pushed this model to its limits – its weak­nesses lie in the storage and pro­cessing of large amounts of data. Another weakness is its lack of ad­apt­ab­il­ity. Un­struc­tured data, such as images and documents in natural language, cannot be stored and evaluated.

Once you have had a look at the ad­vant­ages and dis­ad­vant­ages of in-memory databases, and made a com­par­is­on to tra­di­tion­al databases, you can consider which database man­age­ment system (DBMS) is suitable for your company. If you work with big data, the decision has already been made for you – only an in-memory DB would suit your needs. However, the in-memory database may also be the right choice in other cases.

An in-memory database is the right DBMS for you, if:

• You have a lot of data collected
• You need fast and frequent access to your data
• Your existing database man­age­ment systems or servers are over­loaded
• Data per­sist­ence is not your highest priority
• Loss of data (or at least the pos­sib­il­ity of this) is workable for you

Among the best-known in-memory databases are SAP HANA, and Oracle TimesTen. If your company is looking for en­ter­prise software with a wide range of functions, the SAP and Oracle solutions are the most common. Both database man­age­ment systems have the highest possible per­form­ance. In the following we will look at what dis­tin­guishes them and what their practical ap­plic­a­tion in a company would be.

The SAP HANA (high per­form­ance analytic appliance) in-memory database is a com­bin­a­tion of hardware and software. The software was specially developed by SAP, while the hardware (server) comes from ten different man­u­fac­tur­ers. Unlike other in-memory databases, SAP HANA doesn’t store data tem­por­ar­ily, but instead stores it on the working memory per­man­ently, and saves the data using trans­ac­tion logs.

Trans­ac­tion and analysis pro­cessing in a common database makes it possible to process in­form­a­tion in real time. SAP HANA can be im­ple­men­ted on an en­ter­prise server as well as in a cloud, and so reduces the potential chal­lenges to your company's IT structure. In addition, costs for previous data man­age­ment methods are minimised.

The Oracle database has a lot in common with SAP. Data pro­cessing is also done in real time and the ap­plic­a­tion can be executed via a server, or as a cloud service. In contrast to the SAP database, the Oracle TimesTen software and hardware both originate from Oracle itself.

As such, TimesTen is a purely “Oracle” ap­plic­a­tion. The resulting advantage for the user is that in case of an error, you can act in­tern­ally and your company is not dependent on different hardware and software companies. Oracle does not store the collected data ex­clus­ively in memory – data that doesn’t require high per­form­ance pro­cessing can be stored on the hard disk or a flash disk.

The functions of SAP HANA and Oracle TimesTen are pretty much the same. Both databases have good benefits to offer:

• Faster data pro­cessing
• Company re­ori­ent­a­tion by means of in­nov­at­ive ap­plic­a­tions
• Increased flex­ib­il­ity, activity, and ad­apt­ab­il­ity

The following table sum­mar­ises the dif­fer­ences and sim­il­ar­it­ies between the two in-memory databases:

As di­git­al­isa­tion pro­gresses, the already huge amounts of data will continue to grow. The de­velopers of in-memory databases are now faced with having to con­stantly develop previous systems further. The following tasks need to be looked at:

• The col­lec­tion of data from a growing number of sources
• IT structure sim­pli­fic­a­tion, and the reduction of response times and analysis speeds at the same time
• How to gain further insights from data analysis, and support companies in their decision-making processes
• The de­vel­op­ment of ap­plic­a­tions that are even more geared to the chal­lenges of constant digital change

In-memory databases have es­tab­lished them­selves as a suc­cess­ful form of tech­no­logy for storing and pro­cessing data. They enable companies that have to handle large amounts of data to analyse big data as quickly as possible and access it at any time. However, the effective use of in-memory databases is only possible if the data storage is not ex­clus­ively an in-memory database, but only if stand­ard­ised systems for data backup are in­teg­rated into the in-memory database processes.