Netdata

Netdata is one of the QualityBox dashboards.

See this website Live

Configuration

System Locations

Depending on how you install netdata, it will be distributed in the normal system locations such as

   - the daemon     at /usr/sbin/netdata
   - config files   in /etc/netdata
   - web files      in /usr/share/netdata
   - plugins        in /usr/libexec/netdata
   - cache files    in /var/cache/netdata
   - db files       in /var/lib/netdata
   - log files      in /var/log/netdata
   - pid file       at /var/run/netdata.pid
   - logrotate file at /etc/logrotate.d/netdata

Or, if you use

bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh)

to install, you'll get all of netdata installed into /opt/netdata

Host Modifications

A Netdata role is available in the 32.x branch of Meza

Otherwise, you have to make room in HAProxy for netdata:

HAProxy

frontend netdata 
        bind *:20000 ssl crt /etc/haproxy/certs/wiki.freephile.org.pem
        mode http 
        default_backend netdata-back 
 
backend netdata-back 
        server nd1 127.0.0.1:19999

Kernel

You have kernel memory de-duper (called Kernel Same-page Merging, or KSM) available, but it is not currently enabled.

Memory de-duplication instructions

To enable it run:

   echo 1 >/sys/kernel/mm/ksm/run
   echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs

If you enable it, you will save 40-60% of netdata memory.

Ports

netdata by default listens on all IPs on port 19999. We add a rule to firewalld to allow 20000 and then pass that port through to the backend in haproxy config.

 http://this.machine.ip:20000/ => http://127.0.0.1:19999

Start/Stop

To stop netdata run: systemctl stop netdata

To start netdata run: systemctl start netdata

To reload configuration: killall -USR2 netdata

Notifications

The default configuration will send messages to 'root' so be sure to either edit the conf sudo vim /etc/netdata/health_alarm_notify.conf, or set vim /etc/aliases && newaliases

Turn off alarm

     to: silent # silence notification; still see on website
enabled: no     # disable alarm

more details in the netdata docs.

Issues

You'll probably receive alarms for 'tcp listen drops'. This is likely bot-related (sending INVALID packets) and NOT due to your application dropping legitimate packets. There is a good discussion on how to identify the source of the problem and how to mitigate or resolve it Issue #3234Issue #3826 TLDR; increase the threshold to 1 (/etc/netdata/health.d/tcp_listen.conf) so you don't get bogus alerts.

Also, you should modify your firewall to drop invalid packets before they're either counted (by netstats) or dropped (by the kernel).

iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
ip6tables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
ip6tables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

Following the advice from NASA at https://wiki.earthdata.nasa.gov/display/HDD/SOMAXCONN, I increased my somaxconn kernel parameter to 1024 from 128

 cat /proc/sys/net/core/somaxconn
 128
 sysctl -w net.core.somaxconn=1024

Updates

Netdata will update itself, and puts a script into cron: ln -s /root/netdata/netdata-updater.sh /etc/cron.daily/netdata-updater