Journal Logging – Introduction
In Spark units, the logging system has undergone some changes.
Previously, log files such as maillog and Apache error_log (among others) were handled by numerous daemons and processes such as syslog, which commonly placed the logs under the /var/log/ folder.
With the introduction of systemd suite for central management, the event logging process was centralized to be handled by a single daemon called journald.
Journald is a sophisticated and advanced utility that collects and handles kernel messages, initrd, services, and other logs, in a single centralized place.
Systemd Journal can be used alongside syslog or can replace it.
Journal Usage
Display full log
The journal log is displayed within a pager (usually via less utility) for simple browsing. It includes messages from the early boot process, kernel, initrd as well as application standard error and out.
journalctl-- Logs begin at Thu 1970-01-01 00:00:04 UTC, end at Thu 2015-12-24 09:12:51 UTC. -- Jan 01 00:00:04 spark systemd-journal[112]: Runtime journal is using 5.7M (max allowed 46.2M). Jan 01 00:00:04 spark systemd-journal[112]: Runtime journal is using 5.7M (max allowed 46.2M). Jan 01 00:00:04 spark kernel: Booting Linux on physical CPU 0xf00 Jan 01 00:00:04 spark kernel: Initializing cgroup subsys cpuset Jan 01 00:00:04 spark kernel: Initializing cgroup subsys cpu Jan 01 00:00:04 spark kernel: Initializing cgroup subsys cpuacct
Display filtered log by time and date
Filter the journal log by arbitrary date and time using –since and –until to restrict displayed entries. The absolute time format can be used as follows: YYYY-MM-DD HH:MM:SS
For example, to display entries since December 9th, 2015 at 18:30, type:
journalctl --since="2015-12-09 18:30:00"
To display entries between time windows, type:
journalctl --since "2015-11-30" --until "2015-12-01 03:00"
Note that if any part of the time or/and date is omitted, it will be replaced by a default value. The default value for date is the current date.
The time, if omitted, will be replaced by “00:00:00”. Populating the seconds field is optional.
Display log filtered by service (unit)
Filter the journal by service such as postfix, dnsmasq, apache and other services. For example, to display logs from SSH log, type:
journalctl -u ssh
To display logs from both Apache and postfix together, type:
journalctl -u apache2 -u postfix
Note that it is possible to combine date and time filtering along with filtering by service.
For example, to display logs only from postfix for today, type:
journalctl -u postfix --since <current day>
or
journalctl -u postfix --since today
To display kernel log, type:
journalctl -k
Display log messages filtered by priority
Filter the journal log by priority using -p, such as errors (err), warnings (warning) or other levels of priority. For example, to filter the display by errors, type:
journalctl -perr
To filter the display by warnings, type:
journalctl -pwarning
Note that -p can also be used with the following common keywords: err, warning, notice, info, debug
Display only a certain amount of log entries
For example, to display the 10 most recent entries, same as tail -n, type:
journalctl -n
Note that it is possible to specify the number of entries to display by typing:
journalctl -n 40
Follow log or a specific service using -f option, same as tail -f To follow the full log, type:
journalctl -f
To follow a specific service, type:
journalctl -u postfix -f
To follow a filtered log by priority of errors, type:
journalctl -perr -f
The most common journal usage is presented in this article, for further details, refer to journalctl manual:
man journalctl
Persistent logging
Log storage
By default Systemd Journal stores its data in memory. After the system reboots the journal log data will be flushed.
To preserve the journal logs for debugging purposes it is necessary to create a journal folder under /var/log/ to allow journal to store data on disk and perform Systemd Journal for the changes to take effect.
To create the folder, type:
sudo mkdir -p /var/log/journal
Then restart Systemd journal, by typing:
systemctl restart systemd-journald
Note – journal data can take a fair amount of storage and I/O processing that can affect your system.
Remember to remove /var/log/journal folder when you are done debugging and it is no longer necessary.
To remove the journal log folder, type:
sudo rm -rf /var/log/journal/
Then restart Systemd journal.
Display log since system boot
If persistent logging is enabled it is possible to view journal log messages by boot.
For example, to display logs since the last boot, type:
journalctl -b
Note that it is possible to have a value to indicate the number of boots back. For example, to display the log from 3 boots ago, type:
journalctl -b -3
journal logically separates the log entries by boot processes, using the line: — Reboot —