22nd May 2021

Using GoAccess with Hiawatha Web-Server

GoAccess is a remarkable analyzer for your log-files written by the web-server. For example, GoAccess can read and analyze the log-files from Apache web-server. In the same vein, after some configuration, it can also read and analyze the log-files from Hiawatha web-server. The Hiawatha web-server is used for this blog. I have written on Hiawatha here: Set-Up Hiawatha Web-Server, and Set-Up "Let's Encrypt" for Hiawatha Web-Server.

Below is the required configuration such that GoAccess can read Hiawatha log-files:

date-format	%a %d %b %Y
time-format	%T %z
log-format	%h|%d %t|%s|%b|%r|%R|%u|Host: %v|%^|%^|%^|%^|%^|%^|%^|%^|%^|%^

This should be placed in file $HOME/.goaccessrc.

First go to the log-file directory: cd /var/log/hiawatha. Running goaccess is thus:

goaccess access.log

Usually you will have multiple log-files, the majority of which will be zipped. GoAccess can read from stdin, therefore you can feed the unzipped log-files to GoAccess. For example:

zcat access.*.gz | goaccess access.log access.log.1 > /srv/http/goaccess.html

The HTML output looks something like this for OS distribution: GoAccess Web Output: OS distribution Overall info in HTML format looks like this: GoAccess Web Output Overall View I was quite surprised how much traffic the web-server handles, even before this blog was hosted on Hiawatha. Much of this traffic seems to stem from bots or is just junk. See Filtering Bots and Crawlers from Access.log to see the percentage of bots on this website.

GoAccess can either produce HTML reports, or can show the results directly in your terminal. Below is the output from st or xterm:

 Dashboard - Overall Analyzed Requests (09/May/2021 - 22/May/2021)                            [Active Panel: Requests]

  Total Requests  10782 Unique Visitors  1031 Requested Files 1119 Referrers  199
  Valid Requests  10576 Init. Proc. Time 1s   Static Files    138  Log Size   3.40 MiB
  Failed Requests 206   Excl. IP Hits    0    Not Found       2218 Tx. Amount 192.51 MiB
  Log Source      access.log; access.log.1

 > 2 - Requested Files (URLs)                                                                         Total: 366/1119

 Hits     h% Vis.     v% Tx. Amount Mtd  Proto    Data
 ---- ------ ---- ------ ---------- ---- -------- ----
  893  8.44%  589 57.13%   1.06 MiB GET  HTTP/1.1 /
  191  1.81%  155 15.03%   1.59 MiB GET  HTTP/1.1 /blog/index.html
  112  1.06%   16  1.55%  24.81 KiB HEAD HTTP/1.1 /
  101  0.95%   16  1.55% 929.90 KiB POST HTTP/1.1 /build/search.php
   68  0.64%   58  5.63% 557.15 KiB GET  HTTP/1.1 /build/blog/index.html
   54  0.51%   30  2.91% 186.07 KiB GET  HTTP/1.1 /feed.xml
   47  0.44%    6  0.58%  10.97 KiB HEAD HTTP/1.1 /blog/index.html

   3 - Static Requests                                                                                 Total: 138/138

 Hits     h% Vis.     v% Tx. Amount Mtd  Proto    Data
 ---- ------ ---- ------ ---------- ---- -------- ----
  225  2.13%   83  8.05% 531.72 KiB GET  HTTP/1.1 /favicon.ico
  140  1.32%  133 12.90%  38.95 KiB GET  HTTP/1.1 /robots.txt
   10  0.09%    9  0.87% 168.17 KiB GET  HTTP/1.1 /build/img/jpilot-plugin.png
   10  0.09%    9  0.87% 593.29 KiB GET  HTTP/1.1 /build/img/jpilot-search.png
    8  0.08%    6  0.58%   1.67 KiB HEAD HTTP/1.0 /robots.txt
    4  0.04%    2  0.19%  58.77 KiB GET  HTTP/1.1 /img/PerfRyzenIntelARM1.png
    4  0.04%    1  0.10%   2.03 MiB GET  HTTP/1.1 /build/img/IMG_20140413_124816.jpg

   4 - Not Found URLs (404s)                                                                          Total: 366/2218

 Hits     h% Vis.     v% Tx. Amount Mtd  Proto    Data
 ---- ------ ---- ------ ---------- ---- -------- ----
   59  0.56%    0  0.00%  54.00 KiB GET  HTTP/1.1 /wp-login.php
   54  0.51%    0  0.00%  50.73 KiB GET  HTTP/1.1 /phpmyadmin/
   32  0.30%    0  0.00%  29.24 KiB POST HTTP/1.1 /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php
   30  0.28%    0  0.00%  28.18 KiB GET  HTTP/1.1 /ads.txt
   29  0.27%    0  0.00%  26.48 KiB GET  HTTP/1.1 /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php
   27  0.26%    0  0.00%  25.26 KiB GET  HTTP/1.1 /_ignition/execute-solution
   23  0.22%    0  0.00%  21.50 KiB GET  HTTP/1.1 /wp-content/plugins/wp-file-manager/readme.txt

   5 - Visitor Hostnames and IPs                                                                       Total: 366/830

 [?] Help [Enter] Exp. Panel                     0 - Sat May 22 18:41:56 2021                     [q]uit GoAccess 1.4.6

Pressing Tab-key will navigate through the various panels.

There is a short mentioning in Wikipedia as well: GoAccess. The man-page provides a wealth of information.