Linux - Viewing Log Messages

Posted: | Last updated: | 8 minute read

Inception: You will learn to view various types of logs in Linux Operating System. You will learn some cool tricks to view logs, search a string inside a log file etc.

Linux Logging

In this article you will learn the basic of logging in Linux operating system.

Logs are awesome, because when you have any kind of problem or even you don’t have a problem and you want to understand what is going on or you want to understand your system. Logs will show you some very important things.

So, its a good idea to know, how to see logs in Linux system.

There are couples of ways to view the logs. Someone these require root privileges to access the logs. Actually different logs have different log permissions.

syslog contains a lot of information about your system. As this file is too big, so to view syslog file you can use following utilities.

less: Using less command you can view content of syslog file one screen at a time.

zcat: Using zcat command you can view zipped file content.

zmore: Using zmore command you can view file content in page, without decompressing the file.

zgrep: Using zgrep command you can Search a String inside a compressed file.

grep: Using grep command you can find all occurrences of a string or filer the log file.

tail: Using tail command you can output the last few lines of any file.

head: Using head command you can very beginning of any text file.

Type of Logs in Linux System

You can find system generated logs inside /var/log folder.

You can find alot about the operations of a system by viewing the logs. Most of the logs are grouped in one of the following four categories in Linux system.

  1. System Logs
  2. Events Logs
  3. Application Logs
  4. Service Logs

System Logs in Linux

Logs generated by OS is called system logs, and Linux stores system logs inside a special directory called /var/log.

Linux Operating system writes logs into /var/log/syslog. Following is the example of it:

root@debian10:/home/vagrant# cat /var/log/syslog
April 13 00:00:14 debian10 rsyslogd:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="397" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
April 13 00:00:14 debian10 systemd[1]: logrotate.service: Succeeded.
April 13 00:00:14 debian10 systemd[1]: Started Rotate log files.
April 13 00:05:01 debian10 CRON[11850]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 00:15:01 debian10 CRON[11855]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 00:17:01 debian10 CRON[11858]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
April 13 00:25:01 debian10 CRON[11861]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 01:17:01 debian10 CRON[11879]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
April 13 01:25:01 debian10 CRON[11882]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 02:17:01 debian10 CRON[11901]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
April 13 02:25:01 debian10 CRON[11904]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 03:17:01 debian10 CRON[11922]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
April 13 03:25:01 debian10 CRON[11925]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 04:17:01 debian10 CRON[11944]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
April 13 04:25:01 debian10 CRON[11947]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 05:17:01 debian10 CRON[11966]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
April 13 05:25:01 debian10 CRON[11969]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 05:35:01 debian10 CRON[11972]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 06:17:01 debian10 CRON[11987]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
April 13 06:25:01 debian10 CRON[11991]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 06:25:01 debian10 CRON[11992]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
April 13 06:29:54 debian10 systemd[1]: Starting Daily apt upgrade and clean activities...
April 13 06:29:54 debian10 systemd[1]: apt-daily-upgrade.service: Succeeded.
April 13 06:29:54 debian10 systemd[1]: Started Daily apt upgrade and clean activities.
April 13 06:35:01 debian10 CRON[12071]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 07:15:01 debian10 CRON[12084]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 07:17:01 debian10 CRON[12087]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
April 13 07:25:01 debian10 CRON[12134]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 07:35:01 debian10 CRON[12137]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
April 13 07:45:01 debian10 CRON[12140]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
root@debian10:/home/vagrant#

System booting logs

System boot logs are mainly divided into 2 parts. First come from the kernal like (loading drivers, detecting partitions, etc.), Second type come from Service starting up like ([OK] Starting Apache.. ect.) and these logs are writtern in different files depending upon Linux distribution.

The kernel messages are stored in /var/log/kern.log. Example is given below.

root@debian10:~# cat /var/log/kern.log
April 13 10:36:17 debian10 kernel: [151900.678178] e1000: eth0 NIC Link is Down
April 13 10:36:23 debian10 kernel: [151906.828591] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
April 13 11:10:42 debian10 kernel: [153965.995161] e1000: eth0 NIC Link is Down
April 13 11:10:44 debian10 kernel: [153968.092826] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
April 13 14:41:34 debian10 kernel: [166618.387972] e1000: eth0 NIC Link is Down
April 13 14:41:39 debian10 kernel: [166622.451783] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
April 13 14:41:41 debian10 kernel: [166624.458858] e1000: eth0 NIC Link is Down
April 13 14:41:43 debian10 kernel: [166626.491122] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
April 13 17:37:47 debian10 kernel: [177190.889930] e1000: eth0 NIC Link is Down
April 13 17:37:49 debian10 kernel: [177192.907377] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
April 13 17:37:51 debian10 kernel: [177194.926951] e1000: eth0 NIC Link is Down
April 13 17:37:53 debian10 kernel: [177196.944597] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
April 13 19:34:41 debian10 kernel: [184204.913644] e1000: eth0 NIC Link is Down
April 13 19:34:43 debian10 kernel: [184206.922786] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
April 13 12:41:59 debian10 kernel: [245843.023043] e1000: eth0 NIC Link is Down
April 13 12:42:01 debian10 kernel: [245845.031533] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
April 13 12:56:28 debian10 kernel: [246711.537776] e1000: eth0 NIC Link is Down
April 13 12:56:30 debian10 kernel: [246713.606363] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
root@debian10:~#

Mostly you will find one or more logs file in different Operating Systems.

Few common log files are:

  1. /var/log/boot.log: System Boot logs, which stores system boot time logs.
  2. /var/log/auth.log: Auth Logs, it stores all successful and failure login logs.
  3. /var/log/apache2/: Apache server logs. This directory contains all access, error, and other_vhosts_access logs.
  4. /var/log/mysqld.log: MySQL Database server logs.
  5. /var/log/debug: Debug logs are debugging detailed messages.
  6. /var/log/daemon.log: Daemon logs are also called system logs, these are logs messages of events and services, which runs operating system.
  7. /var/log/maillog: Mail Server logs.
  8. /var/log/kern.log: Kernel log stores information about the Linux Kernel.
  9. /var/log/yum.log: This is yum command logs.

System Logging Daemon

Deamon is program which runs in backgroud for a certain purpose, and essential for something. The syslogd is the daemon which runs for collecting System logs.

The syslogd configuration file is /etc/syslog.conf.

Following are few interesting User logs

Get current logged-in user

To view the current logged-in user, execute the who command.

vagrant@debian10:~$ who
vagrant  pts/0        April 13 23:36 (10.0.2.2)
vagrant@debian10:~$

Get History of logged-in user

Use last reboot command to view the history of logged-in users.

vagrant@debian10:~$ last reboot
reboot   system boot  4.19.0-9-amd64   Sat April 12 16:24   still running
reboot   system boot  4.19.0-9-amd64   Fri April 11 18:51 - 19:14  (00:23)

wtmp begins Wed Jul 29 18:51:06 2020
vagrant@debian10:~$

Get Login time of each user

Use lastlog command to view the last login time of each user.

vagrant@debian10:~$ lastlog
Username         Port     From             Latest
root                                       **Never logged in**
daemon                                     **Never logged in**
bin                                        **Never logged in**
sys                                        **Never logged in**
sync                                       **Never logged in**
games                                      **Never logged in**
man                                        **Never logged in**
lp                                         **Never logged in**
mail                                       **Never logged in**
news                                       **Never logged in**
uucp                                       **Never logged in**
proxy                                      **Never logged in**
www-data                                   **Never logged in**
backup                                     **Never logged in**
list                                       **Never logged in**
irc                                        **Never logged in**
gnats                                      **Never logged in**
nobody                                     **Never logged in**
_apt                                       **Never logged in**
systemd-timesync                           **Never logged in**
systemd-network                            **Never logged in**
systemd-resolve                            **Never logged in**
messagebus                                 **Never logged in**
sshd                                       **Never logged in**
vagrant          pts/0    10.0.2.2         Sun April 13 23:36:04 +0000 2020
systemd-coredump                           **Never logged in**
memcache                                   **Never logged in**
postfix                                    **Never logged in**
vboxadd                                    **Never logged in**

Systemd Logging daemon: journalctl

Any service managed by Systemd or lets say you operate it through systemctl command then its log can be views by journalctl utility.

For example, we have see apache2 is managed by systemctl hence you can view its logs like below.

vagrant@debian10:~$ sudo journalctl -fu apache2
-- Logs begin at Fri 2020-07-31 16:24:46 UTC. --
April 01 20:58:57 debian10.localdomain systemd[1]: Started The Apache HTTP Server.
April 02 00:01:14 debian10.localdomain systemd[1]: Reloading The Apache HTTP Server.
April 02 00:01:14 debian10.localdomain apachectl[11013]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using debian10.localdomain. Set the 'ServerName' directive globally to suppress this message
April 02 00:01:14 debian10.localdomain systemd[1]: Reloaded The Apache HTTP Server.
April 03 00:00:14 debian10.localdomain systemd[1]: Reloading The Apache HTTP Server.
April 03 00:00:14 debian10.localdomain apachectl[11782]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using debian10.localdomain. Set the 'ServerName' directive globally to suppress this message
April 03 00:00:14 debian10.localdomain systemd[1]: Reloaded The Apache HTTP Server.
April 04 00:00:14 debian10.localdomain systemd[1]: Reloading The Apache HTTP Server.
April 04 00:00:14 debian10.localdomain apachectl[12706]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using debian10.localdomain. Set the 'ServerName' directive globally to suppress this message
April 04 00:00:14 debian10.localdomain systemd[1]: Reloaded The Apache HTTP Server.

Log Rotation

There are services and daemons which continuesly keep writing logs messages to log file. Log file sizes are continuesly increasing. There is concept of logrotation linux to breakdown the log file after a certain time and size.

Log rotation configuration save the older logs in a compressed file and keep writing logs. This is a big topic to understand Log Rotation, Configuration of Log Rotation /etc/logrotate.conf, etc. in future articles.

Conclusion: Now, you have a good understanding of Log messages in Linux System of various components like System Logs, Running Service logs, etc. This will help a lot in understanding the behaviour of any service running in Linux System.

Tags:

Categories:

Updated: