Difference between revisions of "Netdata"

From Freephile Wiki
Jump to navigation Jump to search
(link dashboards; one-line install in /opt)
Line 1: Line 1:
Netdata is one of the QualityBox dashboards.
+
Netdata is one of the [https://discourse.equality-tech.com/t/dashboards-in-qualitybox/107 QualityBox dashboards].
  
 
See this website Live http://wiki.freephile.org:20000/
 
See this website Live http://wiki.freephile.org:20000/
Line 5: Line 5:
  
 
== System Locations ==
 
== System Locations ==
 +
Depending on how you install netdata, it will be distributed in the normal system locations such as
 
<pre>
 
<pre>
 
   - the daemon    at /usr/sbin/netdata
 
   - the daemon    at /usr/sbin/netdata
Line 16: Line 17:
 
   - logrotate file at /etc/logrotate.d/netdata
 
   - logrotate file at /etc/logrotate.d/netdata
 
</pre>
 
</pre>
 +
 +
Or, if you use
 +
<pre>bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh)</pre>
 +
to install, you'll get all of netdata installed into <code>/opt/netdata</code>
  
  
 
== Host Modifications ==
 
== Host Modifications ==
  
 +
A Netdata role is available in [https://github.com/enterprisemediawiki/meza/blob/32.x/src/roles/netdata/tasks/main.yml the 32.x branch of Meza]
 +
 +
 
=== HAProxy ===
 
=== HAProxy ===
 
<source lang="python">
 
<source lang="python">
Line 63: Line 71:
 
== Issues ==
 
== Issues ==
  
You'll probably receive alarms for 'tcp listen drops'. This is likely bot-related (sending INVALID packets) and NOT due to your application dropping legitimate packets. There is a good discussion on how to identify the source of the problem and how to mitigate or resolve it [https://github.com/firehol/netdata/issues/3234 Issue #3234] [https://github.com/firehol/netdata/issues/3826 Issue #3826] TLDR; increase the threshold to 1 (<code>/etc/netdata/health.d/tcp_listen.conf</code>) so you don't get bogus alerts.   
+
You'll probably receive alarms for 'tcp listen drops'. This is likely bot-related (sending INVALID packets) and NOT due to your application dropping legitimate packets. There is a good discussion on how to identify the source of the problem and how to mitigate or resolve it [https://github.com/firehol/netdata/issues/3234 Issue #3234][https://github.com/firehol/netdata/issues/3826 Issue #3826] TLDR; increase the threshold to 1 (<code>/etc/netdata/health.d/tcp_listen.conf</code>) so you don't get bogus alerts.   
  
 
Also, you should modify your firewall to drop invalid packets before they're either counted (by netstats) or dropped (by the kernel).
 
Also, you should modify your firewall to drop invalid packets before they're either counted (by netstats) or dropped (by the kernel).

Revision as of 12:35, 17 February 2020

Netdata is one of the QualityBox dashboards.

See this website Live http://wiki.freephile.org:20000/

System Locations[edit | edit source]

Depending on how you install netdata, it will be distributed in the normal system locations such as

   - the daemon     at /usr/sbin/netdata
   - config files   in /etc/netdata
   - web files      in /usr/share/netdata
   - plugins        in /usr/libexec/netdata
   - cache files    in /var/cache/netdata
   - db files       in /var/lib/netdata
   - log files      in /var/log/netdata
   - pid file       at /var/run/netdata.pid
   - logrotate file at /etc/logrotate.d/netdata

Or, if you use

bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh)

to install, you'll get all of netdata installed into /opt/netdata


Host Modifications[edit | edit source]

A Netdata role is available in the 32.x branch of Meza


HAProxy[edit | edit source]

frontend netdata 
        bind *:20000 ssl crt /etc/haproxy/certs/wiki.freephile.org.pem
        mode http 
        default_backend netdata-back 
 
backend netdata-back 
        server nd1 127.0.0.1:19999

Kernel[edit | edit source]

You have kernel memory de-duper (called Kernel Same-page Merging, or KSM) available, but it is not currently enabled.

Memory de-duplication instructions

To enable it run:

   echo 1 >/sys/kernel/mm/ksm/run
   echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs

If you enable it, you will save 40-60% of netdata memory.


Ports[edit | edit source]

netdata by default listens on all IPs on port 19999. We add a rule to firewalld to allow 20000 and then pass that port through to the backend in haproxy config.

 http://this.machine.ip:20000/ => http://127.0.0.1:19999

Start/Stop[edit | edit source]

To stop netdata run: systemctl stop netdata

To start netdata run: systemctl start netdata

Installation extras[edit | edit source]

The configuration will send messages to 'root' so be sure to either edit the conf sudo vim /etc/netdata/health_alarm_notify.conf, or set vim /etc/aliases && newaliases

Issues[edit | edit source]

You'll probably receive alarms for 'tcp listen drops'. This is likely bot-related (sending INVALID packets) and NOT due to your application dropping legitimate packets. There is a good discussion on how to identify the source of the problem and how to mitigate or resolve it Issue #3234Issue #3826 TLDR; increase the threshold to 1 (/etc/netdata/health.d/tcp_listen.conf) so you don't get bogus alerts.

Also, you should modify your firewall to drop invalid packets before they're either counted (by netstats) or dropped (by the kernel).

iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
ip6tables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
ip6tables -A INPUT -m tcp -p tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP

Following the advice from NASA at https://wiki.earthdata.nasa.gov/display/HDD/SOMAXCONN, I increased my somaxconn kernel parameter to 1024 from 128

 cat /proc/sys/net/core/somaxconn
 128
 sysctl -w net.core.somaxconn=1024

TCP State diagram


Updates[edit | edit source]

Netdata will update itself, and puts a script into cron: ln -s /root/netdata/netdata-updater.sh /etc/cron.daily/netdata-updater