Webserver reporting: Difference between revisions

(12 intermediate revisions by the same user not shown)

Line 1:

== Install ==

{{Feature

~~We install~~ the Tokyo Cabinet ~~version~~ for ~~data~~ persistence

|image=Report.png

|imgdesc=Real-time dashboards

|title=

}}

{{#set:feature description = we add real-time visibility into the web server activity. }}

{{#set:feature notes = Unlike Google Analytics, this reporting includes bot traffic, and is not mined by your provider. }}

{{#set:feature tests = There may be a report at public_html/report.html }}

{{#set:feature examples = [{{SERVER}}/public_html/report.html public_html/report.html] }}

QualityBox 34.x features [https://wiki.freephile.org/public_html/report.html real-time web analytics]. Now you can see exactly what's happening, both bots and regular traffic, instantly. The analytics works from the command-line for administrators and also as a single page real-time web app for easy reporting/sharing in the web browser.

==Install==

Webserver reporting is made available through the [https://goaccess.io GoAccess] project. It's available by default in your QualityBox. You can disable it in your public.yml. In order to get reports, your QualityBox administrator must set them up for you.

==Useful commands==

'''Checking for Tokyo Cabinet'''

The Tokyo Cabinet on-disk b-tree library is used for persistence (only available in Debian). You can check for it with:<syntaxhighlight lang="bash">

goaccess -s

</syntaxhighlight>Output:

using Tokyo Cabinet on-disk B+ Tree

'''Generating reports'''

''nb. Either run as root, or use sudo to be able to read the log file.'' Reports are not generated by default. The system administrator can run real-time reports on-demand (visible in the console) and can also setup HTML reporting.

Meza uses a custom log format (nicknamed 'combined') that is NOT the standard 'combined' format. We supply a customized configuration file (<code>/etc/goaccess/my.goaccess.conf</code>) so that you can report on your logs without any changes to your Web Server logging.

~~echo~~ "~~deb http~~://~~deb~~.~~goaccess~~.io/ $(~~lsb_release~~ -cs) ~~main~~" ~~| sudo tee~~ -a /etc/~~apt~~/~~sources~~.~~list~~.d/goaccess.~~list~~

# use default config file (will NOT work with default QualityBox due to the log format)

~~wget~~ -O - https://~~deb~~.~~goaccess~~.io/~~gnugpg~~.key ~~| sudo apt~~-~~key add~~ -

tail -1000 /var/log/apache2/access_log | goaccess -p /etc/goaccess/goaccess.conf -

# use custom config file

tail -1000 /var/log/apache2/access_log | goaccess -p /etc/goaccess/my.goaccess.conf -

# use no config file (minimum args)

tail -1000 /var/log/apache2/access_log | goaccess --log-format '~h{, } %^ %e [%d:%t %z] %D "%r" %s "%R" "%u" %^ %b' --time-format "%T" --date-format "%d/%b/%Y" -

# real-time stats generation

sudo tail -f -n +0 /var/log/httpd/access_log | sudo goaccess -p /etc/goaccess/my.goaccess.conf -o /opt/htdocs/public_html/report.html --real-time-html --ws-url=wss://wiki.freephile.org:443 --addr=127.0.0.1 --origin=https://wiki.freephile.org --ssl-cert=/etc/letsencrypt/live/wiki.freephile.org/cert.pem --ssl-key=/etc/letsencrypt/live/wiki.freephile.org/privkey.pem -

# report (console) most recent log (with querystring)

sudo goaccess /var/log/httpd/access_log --log-format '~h{, } %^ %e [%d:%t %z] %D "%r" %s "%R" "%u" %^ %b' --time-format "%T" --date-format "%d/%b/%Y"

# report (console) two minute time span

sed -n '/17:45:00/,/17:47:00/ p' /var/log/apache2/access_log | goaccess --log-format '~h{, } %^ %e [%d:%t %z] %D "%r" %s "%R" "%u" %^ %b' --time-format "%T" --date-format "%d/%b/%Y" -

# Be 'nice', daemonize, persist data (Debian only)

nice -n 19 goaccess -f /var/log/apache2/access_log -p /etc/goaccess/my.goaccess.conf -o /opt/htdocs/wiki/public_html/report.html --real-time-html --addr=127.0.0.1 --ws-url=wss://wiki.freephile.org:443 --addr=127.0.0.1 --origin=https://wiki.freephile.org --ssl-cert=/etc/letsencrypt/live/wiki.freephile.org/cert.pem --ssl-key=/etc/letsencrypt/live/wiki.freephile.org/privkey.pem --daemonize --keep-db-files --load-from-disk

# Persist data locally, while also generating real-time reporting

# --db-path=/opt/data-meza/logs/ is not really a good location bc goaccess will generate a ton of .db files there.

/usr/local/bin/goaccess /var/log/httpd/access_log -p /etc/goaccess/my.goaccess.conf --persist --restore --db-path=/opt/data-meza/logs/ -o /opt/htdocs/public_html/report.html --real-time-html --addr=127.0.0.1 --ws-url=wss://wiki.freephile.org:443 --addr=127.0.0.1 --origin=https://wiki.freephile.org --ssl-cert=/etc/letsencrypt/live/wiki.freephile.org/cert.pem --ssl-key=/etc/letsencrypt/live/wiki.freephile.org/privkey.pem --daemonize

~~sudo apt-get update~~

~~sudo apt-get install goaccess-tcb geoip-bin~~

</syntaxhighlight>

==ToDo==

There are [https://blog.silverbucket.net/post/31927044856/3-ways-to-configure-haproxy-for-websockets 3 ways of configuring HAProxy for websockets]:

#Proxy based on subdomain

#Proxy based on URI

#Proxy using Websocket detection

We use this last approach, but should add at least subdomain processing to enable subdomain mode in QualityBox

[[Category:Apache]]

@@ Line 1: / Line 1: @@
-== Install ==
+{{Feature
-We install the Tokyo Cabinet version for data persistence
+|image=Report.png
+|imgdesc=Real-time dashboards
+|title=
+}}
+{{#set:feature description = we add real-time visibility into the web server activity. }}
+{{#set:feature notes = Unlike Google Analytics, this reporting includes bot traffic, and is not mined by your provider. }}
+{{#set:feature tests = There may be a report at public_html/report.html }}
+{{#set:feature examples = [{{SERVER}}/public_html/report.html public_html/report.html] }}
+QualityBox 34.x features [https://wiki.freephile.org/public_html/report.html real-time web analytics]. Now you can see exactly what's happening, both bots and regular traffic, instantly. The analytics works from the command-line for administrators and also as a single page real-time web app for easy reporting/sharing in the web browser.
+==Install==
+Webserver reporting is made available through the [https://goaccess.io GoAccess] project. It's available by default in your QualityBox. You can disable it in your public.yml.  In order to get reports, your QualityBox administrator must set them up for you.
+==Useful commands==
+'''Checking for Tokyo Cabinet'''
+The Tokyo Cabinet on-disk b-tree library is used for persistence (only available in Debian). You can check for it with:<syntaxhighlight lang="bash">
+goaccess -s
+</syntaxhighlight>Output:
+ using Tokyo Cabinet on-disk B+ Tree
+'''Generating reports'''
+''nb. Either run as root, or use sudo to be able to read the log file.''  Reports are not generated by default. The system administrator can run real-time reports on-demand (visible in the console) and can also setup HTML reporting.
+Meza uses a custom log format (nicknamed 'combined') that is NOT the standard 'combined' format.  We supply a customized configuration file (<code>/etc/goaccess/my.goaccess.conf</code>) so that you can report on your logs without any changes to your Web Server logging.
 <syntaxhighlight lang="bash">
-echo "deb http://deb.goaccess.io/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/goaccess.list
+# use default config file (will NOT work with default QualityBox due to the log format)
-wget -O - https://deb.goaccess.io/gnugpg.key | sudo apt-key add -
+tail -1000 /var/log/apache2/access_log | goaccess -p /etc/goaccess/goaccess.conf -
+# use custom config file
+tail -1000 /var/log/apache2/access_log | goaccess -p /etc/goaccess/my.goaccess.conf -
+# use no config file (minimum args)
+tail -1000 /var/log/apache2/access_log | goaccess --log-format '~h{, } %^ %e [%d:%t %z] %D "%r" %s "%R" "%u" %^ %b' --time-format "%T" --date-format "%d/%b/%Y" -
+# real-time stats generation
+sudo tail -f -n +0 /var/log/httpd/access_log | sudo goaccess -p /etc/goaccess/my.goaccess.conf -o /opt/htdocs/public_html/report.html --real-time-html --ws-url=wss://wiki.freephile.org:443 --addr=127.0.0.1 --origin=https://wiki.freephile.org --ssl-cert=/etc/letsencrypt/live/wiki.freephile.org/cert.pem --ssl-key=/etc/letsencrypt/live/wiki.freephile.org/privkey.pem -
+# report (console) most recent log (with querystring)
+sudo goaccess /var/log/httpd/access_log --log-format '~h{, } %^ %e [%d:%t %z] %D "%r" %s "%R" "%u" %^ %b' --time-format "%T" --date-format "%d/%b/%Y"
+# report (console) two minute time span
+sed -n '/17:45:00/,/17:47:00/ p' /var/log/apache2/access_log | goaccess  --log-format '~h{, } %^ %e [%d:%t %z] %D "%r" %s "%R" "%u" %^ %b' --time-format "%T" --date-format "%d/%b/%Y" -
+# Be 'nice', daemonize, persist data (Debian only)
+nice -n 19 goaccess -f /var/log/apache2/access_log -p /etc/goaccess/my.goaccess.conf -o /opt/htdocs/wiki/public_html/report.html --real-time-html --addr=127.0.0.1 --ws-url=wss://wiki.freephile.org:443 --addr=127.0.0.1 --origin=https://wiki.freephile.org --ssl-cert=/etc/letsencrypt/live/wiki.freephile.org/cert.pem --ssl-key=/etc/letsencrypt/live/wiki.freephile.org/privkey.pem --daemonize --keep-db-files --load-from-disk
+# Persist data locally, while also generating real-time reporting
+# --db-path=/opt/data-meza/logs/ is not really a good location bc goaccess will generate a ton of .db files there.
+/usr/local/bin/goaccess /var/log/httpd/access_log -p /etc/goaccess/my.goaccess.conf --persist --restore --db-path=/opt/data-meza/logs/ -o /opt/htdocs/public_html/report.html --real-time-html --addr=127.0.0.1 --ws-url=wss://wiki.freephile.org:443 --addr=127.0.0.1 --origin=https://wiki.freephile.org --ssl-cert=/etc/letsencrypt/live/wiki.freephile.org/cert.pem --ssl-key=/etc/letsencrypt/live/wiki.freephile.org/privkey.pem --daemonize
-sudo apt-get update
-sudo apt-get install goaccess-tcb geoip-bin
 </syntaxhighlight>
+==ToDo==
+There are [https://blog.silverbucket.net/post/31927044856/3-ways-to-configure-haproxy-for-websockets 3 ways of configuring HAProxy for websockets]:
+#Proxy based on subdomain
+#Proxy based on URI
+#Proxy using Websocket detection
+We use this last approach, but should add at least subdomain processing to enable subdomain mode in QualityBox
 [[Category:Apache]]