The search and analytics engine  Elast­ic­search is one of the best open source solutions for indexing and struc­tur­ing large databases. However, it’s possible to gain valuable insights in later analysis of raw data if it’s visu­al­ised in a clear and easily un­der­stand­able form. The visu­al­isa­tion tool, Kibana, has been developed es­pe­cially for present­ing Elast­ic­search data and will be discussed in this tutorial.

What is Kibana?

Kibana is an ex­tens­ible web interface for visually rep­res­ent­ing collected data. Together with Elast­ic­search and the data pro­cessing tool Logstash, it forms the so-called ELK stack (also called Elastic Stack). This open-source suite enables users to collect data from different server sources (and in any format), arrange it, and prepare it for ana­lyt­ic­al purposes. In addition to the ability to visualise the data processed by Logstash and Elast­ic­search, Kibana also offers automatic real-time analysis, a very flexible search algorithm, and different data view types (his­to­grams, graphs, pie charts, etc.). In the dashboard, the in­di­vidu­al in­ter­act­ive visu­al­isa­tions can then be combined to form a dynamic overall picture that can be filtered and searched.

Mz8avM1oOKQ.jpg To display this video, third-party cookies are required. You can access and change your cookie settings here.

As a web-based ap­plic­a­tion written in JavaS­cript, Kibana can be used across platforms. Costs only arise if you use the hosting service, Elastic Cloud, offered by the developer. This paid service allows you to implement and organise a secure Elast­ic­search Kibana cluster on Amazon or Google without having to provide your own resources.

Kibana tutorial: First steps with the visu­al­isa­tion tool

Kibana offers a huge range of functions that can be used to display prepared database stocks. However, before you can filter and visualise the in­form­a­tion in the dashboard so that the desired key values can easily be viewed, analysed, and evaluated in the long term, you have a good bit of work ahead of you. With this Kibana tutorial we want to make it easier for you to get started with the powerful web interface. This article explains how to install Kibana correctly, how to create your own dashboard, and how to integrate existing data into Elastic’s visu­al­isa­tion tool.

Step 1: How to get Kibana up and running

Since Kibana is designed to display data that has been indexed using Elast­ic­search, you will first need to install the search and analytics engine. The cor­res­pond­ing packages for Windows, macOS, and Linux can be found in the Elast­ic­search download center. The pre­requis­ite is that you have a current Java runtime en­vir­on­ment (64-bit) installed.

Kibana itself is also available as cross-platform software for Windows, macOS, and Linux (RPM, DEB). Since the ap­plic­a­tion is based on the JavaS­cript runtime en­vir­on­ment Node.js, the various in­stall­a­tion packages also contain the necessary Node.js binaries used to run the visu­al­isa­tion tool – sep­ar­ately main­tained versions are not supported. The different packages (ZIP com­pressed) can be found like Elast­ic­search on the Elastic website.

Note

Linux and Mac users can also install Kibana from the Elastic re­pos­it­ory using the apt and yum package managers. Detailed in­struc­tions can be found in the online manuals.

Once you have unpacked the Kibana package, run the bin/kibana (macOS, Linux) or bin\kibana.bat (Windows) file to get the Kibana server up and running.

Image: Windows 10 command line: Kibana server initialisation
Ini­tial­ising the Kibana server takes a few seconds.

You then access the Kibana backend via the address "http://localhost:5601" in your browser (Elast­ic­search must already be running for this to work).

Image: Kibana: home screen
The Kibana user interface.

Step 2: Feed Kibana with data

To be able to take a closer look at the Kibana dashboard and its functions in this tutorial, the ap­plic­a­tion must first be supplied with data. On the Elastic website, there are three free down­load­able database samples, which we use here for testing purposes. The three databases listed above are "shakespeare.json" (database of the complete works of William Shakespeare), "accounts.zip" (set of fic­ti­tious accounts), and "logs.jsonl.gz" (set of randomly generated log files).

Download and unzip the three files (account and log file) and save them to the location of your choice.

Image: Database samples for Kibana on elastic.co
To download the sample databases, you can also simply click on the re­spect­ive links – the download will then start auto­mat­ic­ally.

Before you submit the data, it is necessary to create mappings for the Shakespeare and server log database fields. These mappings divide the documents in the index into logical groups and also specify the prop­er­ties of the fields, such as their search­ab­il­ity. The ap­pro­pri­ate tool for con­fig­ur­ing the mappings is the console, which can be found in the Kibana interface under the menu items "Dev Tools" à "Console."

Image: Kibana: Console plugin
The Kibana console plugin provides an interface that allows you to interact directly with Elast­ic­search’s REST API.

Now insert the following mappings in sequence via PUT request:

PUT /shakespeare
{
 "mappings": {
  "doc": {
   "properties": {
    "speaker": {"type": "keyword"},
    "play_name": {"type": "keyword"},
    "line_id": {"type": "integer"},
    "speech_number": {"type": "integer"}
   }
  }
 }
}
PUT /logstash-2015.05.18
{
    "mappings": {
        "log": {
            "properties": {
                "geo": {
                    "properties": {
                        "coordinates": {
                            "type": "geo_point"
                        }
                    }
                }
            }
        }
    }
}
PUT /logstash-2015.05.19
{
    "mappings": {
        "log": {
            "properties": {
                "geo": {
                    "properties": {
                        "coordinates": {
                            "type": "geo_point"
                        }
                    }
                }
            }
        }
    }
}
PUT /logstash-2015.05.20
{
    "mappings": {
        "log": {
            "properties": {
                "geo": {
                    "properties": {
                        "coordinates": {
                            "type": "geo_point"
                        }
                    }
                }
            }
        }
    }
}
Image: Kibana console: inputting database mappings
By clicking on the green triangle symbol, you can display the answer to the request entered in the Kibana console.

Now use the bulk API of Elast­ic­search to load the data records via curl in the terminal. In Windows, use the Power­Shell with Invoke-Rest­Meth­od instead (code example below):

curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/doc/_bulk?pretty' --data-binary @shakespeare_6.0.json
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/_bulk?pretty' --data-binary @logs.jsonl
Invoke-RestMethod "http://localhost:9200/bank/account/_bulk?pretty" -Method Post -ContentType 'application/x-ndjson' -InFile "accounts.json"
Invoke-RestMethod "http://localhost:9200/shakespeare/doc/_bulk?pretty" -Method Post -ContentType 'application/x-ndjson' -InFile "shakespeare_6.0.json"
Invoke-RestMethod "http://localhost:9200/_bulk?pretty" -Method Post -ContentType 'application/x-ndjson' -InFile "logs.jsonl"
Image: Windows Powershell, which displays the loading of the Kibana data set
Make sure that you execute the commands for loading the Kibana data sets in the directory where the three sample databases are located.
Note

Depending on the computing power, inputting the data sets can take several minutes.

Switch back to the Kibana console to verify the success of the loading process with the following GET request:

GET /_cat/indices?v

If the data is in­teg­rated as planned, the output looks like this:

Image: Kibana console: output of the index request
The health status "yellow" indicates that Elast­ic­search has assigned all primary shards (parts of a data index), but not all replicas (back-ups of the shards).

Step 3: Defining a first index pattern

In order for Kibana to know which data it should process, you must create cor­res­pond­ing patterns for the indices "Shakespeare," "bank," and "logstash." You can define the former as follows:

  1. Open the menu "Man­age­ment" and click on "Index Patterns." When you create the first index pattern, the "Create index pattern" page will open auto­mat­ic­ally. Al­tern­at­ively, you can access it using the button with this written on it.
  2. Enter "shakes*" into the field "Index pattern" and click on "Next step."
  3. Since no special con­fig­ur­a­tion is required for this pattern, skip the next setup step and complete the pattern creation directly by clicking "Create index pattern."
Image: Kibana interface: menu for creating the index pattern
A newly created index pattern must match at least one index that contains existing data. Otherwise, you cannot continue the setup process.

Repeat the steps for the pattern "ba*" which is auto­mat­ic­ally assigned to the index "bank."

Finally, define an index pattern called "logstash*" for the three server log indexes. With this pattern, however, you do not skip the con­fig­ur­a­tion menu, but select the entry "@timestamp" in the dropdown menu "Time Filter field name," as these records contain time series data. Then click on "Create index pattern" as for the two previous patterns.

Image: Kibana: time filter configuration menu for the logstash* pattern
Time series data, such as server logs, are data records with data that is or has been con­tinu­ously recorded.

Step 4: Browse inserted datasets

After you feed your Kibana server with records, you can now start an Elast­ic­search search query to search these records and filter the results. To do this, go to the “Discover” menu in Kibana and select the index pattern for your search using the small triangle icon in the left menu bar. As part of this Kibana dashboard tutorial, we have chosen the account record (ba*):

Image: Kibana: list of the entries of a fictitious data set
Kibana lists the in­di­vidu­al entries of the selected record, with the ba* record con­tain­ing typical bank account cat­egor­ies such as balance, name, age, etc.

As a test, you can filter the bank account record to see only accounts that meet certain criteria. For example, to spe­cific­ally search for accounts that have a balance of over £47,500 and belong to people over 38 years of age, type the following command in the search box:

balance:>47500 AND age:>38

Discover then returns the entries for the four accounts 97, 177, 878, and 916 that cor­res­pond to the selected criteria.

Image: Kibana: results of data set filtering
With the button "Add a filter" you can easily define your own filters for the selected database.

With the button "Add a filter" you can easily define your own filters for the selected database.

Step 5: Visualise data

With the pre­par­a­tions made so far in this Kibana tutorial, you are now able to visualise the im­ple­men­ted data to bring your dashboard to life. Here is where an example of a pie chart for the bank account database is created. On the one hand, this diagram should show which pro­por­tion of the total of 1,000 accounts falls into certain bank account balance ranges, and on the other hand, how the age-specific dis­tri­bu­tion within these divisions turns out.

In the first step, open the "Visualize" menu and click on "Create a visu­al­isa­tion" to get a list of the available visu­al­iz­a­tion types. Then select the "Pie" option.

Image: Kibana: selection menu for the visualisation types
The visu­al­isa­tion types available in Kibana include heat maps, bar charts, and area charts.

At the beginning, you will only see a simple circle that sum­mar­ises all entries of the database, since no cat­egor­ies have been defined yet. These are also called "buckets" in Kibana and can be created under the menu item of the same name.

To first define the in­di­vidu­al account balance divisions, click on "Split Slices" and select "Range" from the "Ag­greg­a­tion" drop-down menu:

Image: Kibana: definition window for diagram categories
You can use the "Range" type to create in­di­vidu­al value ranges for selected fields of a data record.

Under "Field" search for the entry "balance" and click on it, then click on the "Add Range" button four times to define the following six bank account balance cat­egor­ies:

0 999
1000 2999
3000 6999
7000 14999
15000 30999
31000 50000

Then click “Apply Changes” (triangle symbol), and the pie chart will show the dis­tri­bu­tion of accounts according to the defined account balance divisions.

Image: Kibana: pie chart with defined categories
Kibana auto­mat­ic­ally creates a legend to the created diagram, whose position (right, left, top, bottom) can be adjusted in the options.

In the second step, you add another ring to the diagram that visu­al­ises the dis­tri­bu­tion of age classes for the in­di­vidu­al account balance areas. To do this, click on "Add sub-buckets," then on "Split Slices" again and select "Terms" from the drop-down menu. Search under "Field" for the entry "age" and accept the changes via "Apply changes."

Image: Kibana: pie chart with defined categories and subcategories
If you move the mouse over the in­di­vidu­al com­pon­ents of a diagram, Kibana will show you further tool tips unless you have de­ac­tiv­ated this in the options.

You can now save the visu­al­isa­tion very easily using the "Save" button located in the upper menu bar.

Step 6: Or­gan­ising the dashboard

The Kibana Dashboard is covered briefly in this tutorial, so you'll create your first test dashboard using the search and visu­al­isa­tion you saved in steps 4 and 5. To do this, select the dashboard in the page nav­ig­a­tion and then click "Create new dashboard" and then "Add." Kibana will now auto­mat­ic­ally list all saved visu­al­isa­tions or Elast­ic­search searches:

Image: Kibana dashboard: menu for adding new panels
The search function makes it easy to find the desired dashboard com­pon­ents, even with a large number of visu­al­isa­tions or Elast­ic­search search results.

Left-click to add the account balance visu­al­iz­a­tion and the sample search result to the dashboard, whereupon you can view both in separate panels in the dashboard:

Image: Kibana dashboard: tutorial test example
The "Share" button allows you to share your Kibana dashboard with other users or integrate it into your website at any time.

You can now modify the panels, for example by resizing them or changing their position. It is also possible to display a visu­al­isa­tion or search result on the entire screen, or delete it from the dashboard. With many visu­al­isa­tions, you can also use “Inspect” to display ad­di­tion­al in­form­a­tion about the un­der­ly­ing data and queries.

Note

If you remove a panel from the Kibana dashboard, the saved visu­al­isa­tion or search will be preserved.

Go to Main Menu