The UFM Telemetry platform provides network validation tools to monitor network performance and conditions, capturing and streaming rich real-time network telemetry information, application workload usage to an on-premise or cloud-based database for further analysis.
UFM telemetry is packaged in a docker image that should be loaded and deployed on a Linux machine with docker installed (as a prerequisite). This chapter describes how to deploy UFM telemetry on a Linux machine.
To deploy the UFM telemetry:
1. Connect to the Linux machine via SSH.
2. Ensure the docker is installed on the Linux machine.
Run:[root@r-ufm145 ~]# docker –version
3. Start the docker service.
Run:[root@r-ufm145 ~]# sudo service docker start
4. Pull the image.
Run:[root@r-ufm145 ~]# docker pull mellanox/ufm-telemetry:rhel7.3_x86_64_ofed5.1-2.3.7_release_1.6_latest
5. Create the default .ini files and place them in the local directory mapped to /config in the container and initialize the container configuration.
Run:[root@r-ufm145 ~]# docker run -v /tmp/config:/config --rm -d mellanox/ufm-telemetry:rhel7.3_x86_64_ofed5.1-2.3.7_release_1.6_latest /get_collectx_configs.sh "sample_rate=300;hca=mlx5_0;cable_info_schedule=1/00:00,3/00:00,5/00:00"
This collects port counter data every 5 minutes, uses HCA mlx5_0, and collects cable info on the 1st, 3rd, and 5th day of the week at midnight.
Where: sample_rate: Frequency of collecting port counters
hca: Card to use
* cable_info_schedule: Time of collecting cable info data (optional)
7. Verify that UFM Telemetry is running: a. Ensure the UFM telemetry container is up. Run: [root@r-ufm145 ~]# docker ps
b. If the container name exists, access the shell of the container. Run: [root@r-ufm145 ~]# sudo docker exec -it ufm-telemetry bash
c. Run "ps -fade" and verify that the list of running processes includes agx, clx, supervisord, agx_manager.py, agx_server.py, launch_ibdiagnet.py, launch_retention.py, launch_compression.py, launch_cableinfo.py.
[root@r-ufm145 workspace]# ps -fade
d. Review your configurations under "/config/launch_ibdiagnet_config.ini".
8. To view the UFM telemetry configuration files.
Run:
[root@ r-ufm145 ~]# ls -l /config/
-rw-r--r-- 1 3478 101 396 Jul 15 21:04 clx_config.ini
-rw-r--r-- 1 3478 101 2987 Jul 15 21:04 collectx.ini
-rw-r--r-- 1 3478 101 4257 Jul 15 21:04 launch_ibdiagnet_config.ini
-rw-r--r-- 1 3478 101 1912 Jul 16 12:03 supervisord.conf
Note; User can change any value in these files and they will be applied automatically without any requirements for reboot or restart
UFM Telemetry adds the ability to stream to multiple destinations using Fluent Bit. The streaming
implementation can stream to any Fluent Bit export plugin, with the "Forward" plugin being
particularly useful as it allows sending data to a customer-maintained Fluent Bit or FluentD instance
which the customer can then configure as based on their requirements.
3. Configure/create export files *.exp in export directory /config/fluent_bit_configs/ and set
"enable=1" for plugins you want to run. Please see details in the "Export Files" section.
4. Enable FluentBit export by setting "plugin_env_FLUENT_BIT_EXPORT_ENABLE=1" in /config/
launch_ibdiagnet_config.ini.
Run: [root@r-ufm workspace]# vi /telemetry.config/launch_ibdiagnet_config.ini
…
[fluentbit_export]
plugin_env_FLUENT_BIT_EXPORT_ENABLE=1
plugin_env_FLUENT_BIT_CONFIG_DIR=/telemetry.config/fluent_bit_configs
plugin_env_LD_LIBRARY_PATH=/opt/mellanox/collectx/lib
...
5. Run destination programs that will receive data. See more details in the "Examples" section.
6. See the data on the receiving side.
Note:Ibdiagnet will collect and export data periodically as configured by launch_ibdiagnet_config.ini file
using the sample_rate parameter.
<H$>Export Files
Export destinations are set by configuring .exp files or creating new ones. All export files are placed in
the export configuration folder /config/fluent_bit_configs. The easiest way to start is to use documented
example exp-files for the following plugins:
forward
file
influxdb
stdout
stdout_raw (this plugin is presented only in the Fluent Bit version installed in the UFM Telemetry
docker image)
Note:All plugins are disabled by default. To enable a plugin, set "enable=1".
Export File Configuration Details
Each export destination has the following fields:
name – configuration name
plugin_name – Fluent Bit plugin name
enable – 1 or 0 values to enable/disable this destination
host – the host for Fluent Bit plugin
port – port for Fluent Bit plugin
msgpack_data_layout – the msgpacked data format. Default is "flb_std". The other option is
"custom". See section "Msgpack Data Layout" for details.
plugin_key=val – key-value pairs of Fluent Bit plugin parameter (optional)
counterset/fieldset – file paths (optional). See the details in section "Cset/Fset Filtering".
Data layout can be configured using .exp files by setting "msgpack_data_layout=layout".
Two layouts are available:
"flb_std" data layout is an array of 2 fields: timestamp double value and a plain dictionary (keyvalue
pairs). The standard layout is appropriate for all Fluent Bit plugins. For example:
[timestamp_val, {"timestamp"->ts_val, type=>"counters/events", "source"=>"source_val", "key_1"=>val_1,
"key_2"=>val_2,...}]
"custom" data layout is a dictionary of meta-fields and counter fields. Values are placed into a
separate plain dictionary. Custom data format can be dumped with "stdout_raw" output plugin
of fluent-bit installed or can be forwarded with "forward" output plugin. Counters example:
{"timestamp"=>timestamp_val, "type"=>"counters", "source"=>"source_val", "values"=> {"key_1"=>val_1,
"key_2"=>val_2,...}}
Events example:
{"timestamp"=>timestamp_val, "type"=>"events", "type_name"=>"type_name_val", "source"=>" source_val",
"values"=>{"key_1"=>val_1, "key_2"=>val_2,...}}
Each export file can optionally use one cset and one fset file to filter UFM telemetry counters and
events data. *Cset file contains tokens per line to filter data with "type"="counters". *Fset contains several blocks started with the header line "[event_type_name]" and tokens under
that header. Fset file is used to filter data with "type"="events".
If several tokens are needed to be matched simultaneously use "tok1+tok2+tok3". Exclusive tokens are
available too: line "tok1+tok2-tok3-tok4" will filter names that match both tok1 and tok2 and do not
match tok3 or tok4. Note:For more details see documentation in the files "ufm_enterprise.cset" and "ufm_ enterprise.fset"
inside the UFM Telemetry docker folder "/config/fluent_bit_configs".
Details of /config/fluent_bit_configs/ufm_enterprise.cset: # put tokens on separate lines
# Tokens are the actual name 'fragments' to be matched
# port$ # match names ending with token "port"
# ^port # match names starting with token "port"
# ^port$ # include name that is exact token "port
# port+xmit # match names that contain both tokens "port" and "xmit"
# port-support # match names that contain the token "port" and do not match the "-" token "support"
#
# Tip: To disable counter export put a single token line that fits nothing
# List of available counters:
#
#node_guid
#port_guid
#port_num
#lid
#link_down_counter
#link_error_recovery_counter
#symbol_error_counter
#port_rcv_remote_physical_errors
#port_rcv_errors
#port_xmit_discard
#port_rcv_switch_relay_errors
#excessive_buffer_errors
Details of /config/fluent_bit_configs/ufm_enterprise.fset:
# Put your events here
# Usage:
#
# [type_name_1]
# tokens
# [type_name_2]
# tokens
# [type_name_3]
# tokens
# ...
# Tokens are the actual name 'fragments' to be matched
# port$ # match names ending with token "port"
# ^port # match names starting with token "port"
# ^port$ # include name that is exact token "port
# port+xmit # match names that contain both tokens "port" and "xmit"
# port-support # match names that contain the token "port" and do not match the "-" token "support"
# The next example will export the whole "switch_fan" events and events "CableInfo" filtered with token "port" :
# [switch_fan]
#
# [CableInfo]
# port
# To know which event type names are available use one of these options:
# 1. Check export and find field "type_name"=>"switch_temperature"
# OR
# 2. Open log file "/tmp/ibd/ibdiagnet2_port_counters.log" and find event types are printed to log:
# ...
# [info] type [CableInfo] is type of interest
# [info] type [switch_temperature] is type of interest
# [info] type [switch_fan] is type of interest
# [info] type [switch_general] is type of interest
# ...
# Corner cases:
# 1. Empty fset file will export all events.
# 2. Tokens written above/without [event_type] will be ignored.
# 3. If cannot open fset file, warning will be printed, all event types will be exported.
1.Connect to a remote Linux machine via SSH and ensure docker is installed and started on it. Run: [root@r-ufm ~]# sudo service docker start2.Pull FluentD image. Run: [root@r-ufm ~]# sudo docker pull fluentd 3.Create a configuration file for fluentd container. Run:
[root@r-ufm248 ~]# export fluentd_dir=/tmp/fluentd
[root@r-ufm248 ~]# mkdir -p $ fluentd_dir
[root@r-ufm248 ~]# vim $ fluentd_dir/config.conf #fill it with next configuration
@type forward
bind 0.0.0.0
port 24432