What is Journ­alctl and what can it do?

Journ­alctl is an efficient solution for managing and analysing system logs in Linux en­vir­on­ments. This tool can be used for mon­it­or­ing system activ­it­ies, troubleshoot­ing and real-time log tracking, providing crucial insights for ef­fect­ively dia­gnos­ing system problems.

What is Journ­alctl?

journalctl is a powerful utility for querying and dis­play­ing event logs or logfiles under Linux. It’s a central component of the system and service man­age­ment suite systemd, which comes with many modern Linux dis­tri­bu­tions such as Ubuntu, Fedora and Arch Linux. The name ‘journ­alctl’ is a mix of ‘journal’ (log) and ‘ctl’ (control), which refers to the fact that the command is used to control and analyse logs.

journalctl sim­pli­fies accessing the system log, which is managed by systemd-journald. The system log is a cent­ral­ised col­lec­tion of messages and events that occur during the operation of a Linux system. Unlike tra­di­tion­al text-based logs, journalctl provides a struc­tured and efficient way to search, filter and display log data in real time. This can help diagnose problems or monitor system status.

For this purpose, Journ­alctl saves the log files in binary format, because binary log files are more compact and efficient than their text-based equi­val­ents. Given the large amount of logged data, this enables a faster and more efficient search for specific events or in­form­a­tion within the log files. With their struc­tured and encrypted nature, binary log files also provide enhanced security compared to text-based logs, since their format makes ma­nip­u­lat­ing log data more chal­len­ging.

How to adjust storage space for log files

journalctl can be used to limit and configure storage space that log files occupy on the hard disk. This is done via the systemd-journald service settings. The con­fig­ur­a­tion settings are stored in the file /etc/systemd/journald.conf. Here, you’ll find the following entries:

• SystemMaxUse: Limits the storage space for logs in the system directory
• RuntimeMaxUse: Limits the storage space for logs in the temporary directory

To set the memory limit, add the following lines or change them as required:

[Journal] 
SystemMaxUse=50M 
RuntimeMaxUse=50M

The values from this example (50M) can be modified as required. You can also use different units such as K (kilobytes), M (megabytes), G (gigabytes) or T (ter­ra­bytes). Once you’ve edited the con­fig­ur­a­tion file, systemd-journald must be restarted for the changes to take effect:

sudo systemctl restart systemd-journald

Following these steps, systemd-journald will be con­figured to restrict disk space for log files. Make sure the limit you choose can ad­equately store essential log data. At the same time, it’s important to avoid using an excessive amount of disk space. Keep in mind that older logs are auto­mat­ic­ally removed once the limit of allotted disk space has been reached.

Assess disk space usage

Before adjusting the storage space for log files, check how much storage space is currently occupied by the journal. To do this, use --disk-usage:

journalctl --disk-usage

Here’s an example of what the output looks like:

Journals take up 8.0M on disk.

Delete old log entries

If Journal takes up too much memory, you can delete old log entries. There are two ways to do so:

Use --vacuum-size to reduce the size of your journal by spe­cify­ing the size. With this method, old entries are removed until the total journal storage space occupied on the hard disk has reached the desired size.

sudo journalctl --vacuum-size=1G

Al­tern­at­ively, you can shrink the journal by setting a retention period using the --vacuum-time option. Entries older than the specified time will be deleted. To retain log entries from the past year, you can use the following command:

sudo journalctl --vacuum-time=1years

What can Journ­alctl do?

journalctl offers powerful filtering cap­ab­il­it­ies, allowing users to sift through log entries based on various criteria. This feature enables targeted searches for pertinent in­form­a­tion, fa­cil­it­at­ing quicker problem iden­ti­fic­a­tion. Here are some commonly used journalctl filter options:

Show logs

Executing the command journalctl will display log entries for the current system in reverse chro­no­lo­gic­al order. Use journalctl -f or journalctl --follow to display the entries in real time. New entries are auto­mat­ic­ally displayed in the order in which they arrive.

Depending on how long systemd has been running on your system, you’ll probably be shown an un­man­age­able amount of data, which can be tens or hundreds of thousands of lines long. To find what you’re looking for more quickly, you can filter the logs further using various Linux commands.

How to filter by time

Using journalctl, logs can be filtered according to a specific point in time so they can be searched more ef­fi­ciently for relevant in­form­a­tion.

Filter by boot process

Filtering logs by the boot process proves in­valu­able for examining system events at precise times or during boot scenarios. This targeted approach stream­lines problem diagnosis by focusing log entries on specific system statuses or con­fig­ur­a­tions.

1. Current boot: With journalctl -b all log entries are displayed that have been collected since the last reboot.
2. Past boots: Use the -b option followed by a number to display the log entries for a specific previous boot. For example, if you enter journalctl -b 1, the previous boot will be displayed.
3. List of all available boot processes: The command journalctl --list-boots displays a list of available boots with their re­spect­ive IDs. You can use the desired boot ID to display the logs for a specific boot.

While the saving of past boot processes is activated by default on some Linux dis­tri­bu­tions, users may need to first activate it on others. To do this, create a directory where the log will be saved by entering sudo mkdir -p /var/log/journal. Al­tern­at­ively, you can edit the journal con­fig­ur­a­tion file with sudo nano /etc/systemd/journald.conf. Then set the option Storage= under [Journal] to persistent to activate per­sist­ent logging:

. . . 
[Journal] 
Storage=persistent

Filter by time window

Sometimes it may be necessary to display log entries for a specific time period. journalctl offers the options --since and --until, which can be used to restrict the entries to a specific period. To do this, use the time format YYYY-MM-DD HH:MM:SS. The command to display all log entries between January 1, 2023 at 12:00 and January 2, 2023 at 12:00 looks like this:

journalctl --since "2023-01-01 12:00:00" --until "2023-01-02 12:00:00"

The com­bin­a­tion of the two commands can also filter for a shorter period of time:

journalctl --since 09:00 --until "1 hour ago"

Al­tern­at­ively, you also omit parts of the format. For instance, if you want to display all entries starting at a certain point in time:

journalctl --since "2023-11-16 15:25:00"

journalctl also re­cog­nises relative values such as yesterday, today or tomorrow. To display entries from yesterday, enter the following:

journalctl --since yesterday

How to filter by message char­ac­ter­ist­ics

Using journalctl to filter log entries based on im­port­ance or contents is also extremely useful as it allows you to search for relevant in­form­a­tion and focus on specific aspects of the system logs. This enables efficient error diagnosis, early detection of security problems and rapid per­form­ance mon­it­or­ing, among other things.

Filter by priority

To filter logs with journalctl by message im­port­ance, you can use the priority cat­egor­ies for log entries. To do this, you can use either the priority name or its cor­res­pond­ing numerical value. The lower the number, the more important the message:

• 0: emerg (emergency)
• 1: alert (alarm)
• 2: crit (critical)
• 3: err (error)
• 4: warning (warning)
• 5: notice (note)
• 6: info (in­form­a­tion)
• 7: debug (troubleshoot­ing)

Messages with a certain priority can be filtered using the -p option. For example, the following command only displays log entries with the priority ‘err’ (error) and higher:

journalctl -p err

Filter by unit

Filtering logs by unit is useful for focusing on specific services or processes. These can be filtered using the -u option. For example, to display log entries for the Apache web server, enter the following:

journalctl -u apache2

The search can be further refined using relative time values. To determine if the service has already been performed today, you can enter the following:

journalctl -u apache2 --since today

journalctl can merge data records from different units. For instance, if your Nginx process is linked to a PHP FPM unit, their entries can be merged chro­no­lo­gic­ally. The command for this is:

journalctl -u nginx.service -u php-fpm.service --since today-u apache2 --since today

Filter by process, user or group ID

Journ­alctl can filter log entries by process, user or group ID. If you have the exact PID of the process you want to search for, you can use the _PID option to filter entries. For example, if the PID is 8088, the command would look like this:

journalctl _PID=8088

Al­tern­at­ively, you can use the _UID or _GID filters to display all entries logged by a par­tic­u­lar user or group. For example, if your web server is named ‘www-data’, you can find the user ID by doing the following:

id -u www-data
33

The journal entries can be filtered using the ID:

journalctl _UID=33 --since today

To determine which group IDs entries have been created for, you can use the -F option. This displays all the values that have been saved for the Group ID field:

journalctl -F _GID

Here’s an example of the output:

32
99
102
133
81
84
100
0
124
87

Filter by component

Filtering by component is useful for con­cen­trat­ing on specific ap­plic­a­tions, services or processes. The component field is typically used by various services or software com­pon­ents to dis­tin­guish specific in­form­a­tion in the logs. This filtering makes it possible to narrow log entries down to a par­tic­u­lar component, ap­plic­a­tion or service unit. For instance, to filter entries con­tain­ing the ex­ecut­able file bash, enter the following command:

journalctl /usr/bin/bash

Display kernel messages

Filtering log entries for kernel messages with journalctl is an effective way to analyse in­form­a­tion about kernel operation in a Linux system. Kernel messages can provide in­dic­a­tions of hardware problems, driver conflicts or other system events.

Kernel messages that can be found in the dmesg output can also be filtered from the journal. They can be displayed with the flags -k or–dmesg `:

journalctl -k

The kernel messages of current boot process are displayed by default. You can filter for messages from an al­tern­at­ive boot using the pre­vi­ously mentioned boot selection flags. If you, for example, want to view the kernel messages of the last five boot processes, enter:

journalctl -k -b -5

Changing the journal display in Journ­alctl

Cus­tom­ising the display in journalctl lets users search through logs more precisely and extract in­form­a­tion swiftly. Users can tailor the display to show log data over a specific period or in real time to fa­cil­it­ate rapid iden­ti­fic­a­tion of system errors and issues.

Short­en­ing or expanding the output

You can customise how journalctl displays data by short­en­ing or expanding the output. By default, journalctl displays the entire entry in the pager and runs them on the right-hand side of the screen. The output can be shortened with the --no-full option:

journalctl --no-full

You can expand the display using the -a flag:

journalctl -a

Set journalctl to standard output

In journalctl, log output is by default displayed using a pager like less. This lets users view output in­cre­ment­ally, allowing them to navigate through lengthy log files more easily. However, there are occasions where dis­play­ing the standard output of logs is necessary. Here’s how to do it:

journalctl --no-pager

Set output formats

journalctl also offers options for cus­tom­ising the output format of logs. To do this, you can use the -o option with the re­spect­ive format iden­ti­fi­er. For instance, to output log entries in JSON format, enter the following code:

journalctl -b -u nginx -o json

Here’s the output:

{ "__CURSOR" : "s=13a21661cf4948289c63075db6c25c00;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635", "__REALTIME_TIMESTAMP" : "1422990364739502", "__MONOTONIC_TIMESTAMP" : "27200938", "_BOOT_ID" : "81b58db8fd9046ab9f847ddb82a2fa2d", "PRIORITY" : "6", "_UID" : "0", "_GID" : "0", "_CAP_EFFECTIVE" : "3fffffffff", "_MACHINE_ID" : "752737531a9d1a9c1e3cb52a4ab967ee", "_HOSTNAME" : "desktop", "SYSLOG_FACILITY" : "3", "CODE_FILE" : "src/core/unit.c", "CODE_LINE" : "1402", "CODE_FUNCTION" :

The following formats can be used in Journ­alctl:

• cat: Displays only the message field
• export: Binary format suitable for trans­fer­ring or saving
• json: Standard JSON with one entry per line
• json-pretty: JSON formatted for better read­ab­ilit
• json-sse: Wrapped JSON formatted output that allows adding events sent by the server
• short: Standard syslog style output
• short-iso: Standard format for dis­play­ing ISO-8601 wallclock timestamps
• short-monotonic: Standard format with monotonic timestamps
• short-precise: Standard format with mi­cro­second precision
• verbose: Displays every journal field available for the re­spect­ive entry

How does journalctl conduct active process mon­it­or­ing?

During active process mon­it­or­ing with journalctl, the command-line program tail is used to track logs in real time and display the most recent entries. This makes it easier to monitor system events in real time and react to problems quickly.

How to display current logs

The -n option can be used to display a specific number of data records. It works in exactly the same way as tail -n. To display the last 10 entries, use the following command:

journalctl -n

You can set the number of entries as well, e.g. to 20:

journalctl -n 20