Plugin: python.d.plugin Module: changefinder
This collector uses the Python changefinder library to perform online changepoint detection on your Netdata charts and/or dimensions.
Instead of this collector just collecting data, it also does some computation on the data it collects to return a changepoint score for each chart or dimension you configure it to work on. This is an online machine learning algorithm so there is no batch step to train the model, instead it evolves over time as more data arrives. That makes this particular algorithm quite cheap to compute at each step of data collection (see the notes section below for more details) and it should scale fairly well to work on lots of charts or hosts (if running on a parent node for example).
n_score_samples
) for the collector to ‘settle’ into it’stypical behaviour in terms of the trained models and scores you will see in the normal running of your node. Mainly
this is because it can take a while to build up a proper distribution of previous scores in over to convert the raw
score returned by the ChangeFinder algorithm into a percentile based on the most recent n_score_samples
that have
already been produced. So when you first turn the collector on, it will have a lot of flags in the beginning and then
should ‘settle down’ once it has built up enough history. This is a typical characteristic of online machine learning
approaches which need some initial window of time before they can be useful.
netdata.runtime_changefinder
) of ~30ms.apps.mem
) being continually used by the python.d.plugin
under default configuration.This collector is supported on all platforms.
This collector supports collecting metrics from multiple instances of this integration, including remote instances.
By default this collector will work over all system.*
charts.
The default configuration for this integration does not impose any limits on data collection.
The default configuration for this integration is not expected to impose a significant performance impact on the system.
This collector will only work with Python 3 and requires the packages below be installed.
# become netdata user
sudo su -s /bin/bash netdata
# install required packages for the netdata user
pip3 install --user numpy==1.19.5 changefinder==0.03 scipy==1.5.4
Note: if you need to tell Netdata to use Python 3 then you can pass the below command in the python plugin section
of your netdata.conf
file.
[ plugin:python.d ]
# update every = 1
command options = -ppython3
The configuration file name for this integration is python.d/changefinder.conf
.
You can edit the configuration file using the edit-config
script from the
Netdata config directory.
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config python.d/changefinder.conf
There are 2 sections:
The following options can be defined globally: priority, penalty, autodetection_retry, update_every, but can also be defined per JOB to override the global values.
Additionally, the following collapsed table contains all the options that can be configured inside a JOB definition.
Every configuration JOB starts with a job_name
value which will appear in the dashboard, unless a name
parameter is specified.
Name | Description | Default | Required |
---|---|---|---|
charts_regex | what charts to pull data for - A regex like system\..*/ or system\..*/apps.cpu/apps.mem etc. |
system..* | yes |
charts_to_exclude | charts to exclude, useful if you would like to exclude some specific charts. note: should be a ‘,’ separated string like ‘chart.name,chart.name’. | no | |
mode | get ChangeFinder scores ‘per_dim’ or ‘per_chart’. | per_chart | yes |
cf_r | default parameters that can be passed to the changefinder library. | 0.5 | no |
cf_order | default parameters that can be passed to the changefinder library. | 1 | no |
cf_smooth | default parameters that can be passed to the changefinder library. | 15 | no |
cf_threshold | the percentile above which scores will be flagged. | 99 | no |
n_score_samples | the number of recent scores to use when calculating the percentile of the changefinder score. | 14400 | no |
show_scores | set to true if you also want to chart the percentile scores in addition to the flags. (mainly useful for debugging or if you want to dive deeper on how the scores are evolving over time) | no | no |
Default configuration.
local:
name: 'local'
host: '127.0.0.1:19999'
charts_regex: 'system\..*'
charts_to_exclude: ''
mode: 'per_chart'
cf_r: 0.5
cf_order: 1
cf_smooth: 15
cf_threshold: 99
n_score_samples: 14400
show_scores: false
Metrics grouped by scope.
The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.
This scope has no labels.
Metrics:
Metric | Dimensions | Unit |
---|---|---|
changefinder.scores | a dimension per chart | score |
changefinder.flags | a dimension per chart | flag |
There are no alerts configured by default for this integration.
To troubleshoot issues with the changefinder
collector, run the python.d.plugin
with the debug option enabled. The output
should give you clues as to why the collector isn’t working.
Navigate to the plugins.d
directory, usually at /usr/libexec/netdata/plugins.d/
. If that’s not the case on
your system, open netdata.conf
and look for the plugins
setting under [directories]
.
cd /usr/libexec/netdata/plugins.d/
Switch to the netdata
user.
sudo -u netdata -s
Run the python.d.plugin
to debug the collector:
./python.d.plugin changefinder debug trace
If you’re encountering problems with the changefinder
collector, follow these steps to retrieve logs and identify potential issues:
Use the following command to view logs generated since the last Netdata service restart:
journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep changefinder
Locate the collector log file, typically at /var/log/netdata/collector.log
, and use grep
to filter for collector’s name:
grep changefinder /var/log/netdata/collector.log
Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.
If your Netdata runs in a Docker container named “netdata” (replace if different), use this command:
docker logs netdata 2>&1 | grep changefinder