mellanox/ufm-plugin-tfs repository overview

• Login as admin

• Run

  enable
  

  Copy

  config terminal
  

  Copy

• Make sure that UFM is running

  show ufm status
  

  Copy

  • If UFM is down then run it

    ufm start
    

    Copy

• Make sure docker is running

  no docker shutdown
  

  Copy

• Load the latest plugin's docker image

  • In case of HA, load the plugin on the standby node as well.
  • If your appliance is connected to the internet, you could simply run:

    docker pull mellanox/ufm-plugin-tfs
    

    Copy
  • If your appliance is not connected to the internet, you need to load the image offline
    • Use a machine that is connected to the internet to save the docker image

      docker save mellanox/ufm-plugin-tfs:latest | gzip > ufm-plugin-tfs.tar.gz
      

      Copy
    • Move the file to scp shared location that is accessible to the appliance.
    • Fetch the image to the appliance

      image fetch scp://user@hostname/path-to-file/ufm-plugin-tfs.tar.gz
      

      Copy
    • Load the image

      docker load ufm-plugin-tfs.tar.gz
      

      Copy

• Enable & start the plugin

  ufm plugin tfs add
  

  Copy

• Check that the plugin is up and running with

  show ufm plugin
  

  Copy

• Load the latest plugin container

  • In case of HA, load the plugin on the standby node as well;

  • If your machine is connected to the internet, you could simply run:

    docker pull mellanox/ufm-plugin-tfs
    

    Copy
    • If your UFM machine is not connected to the internet, you need to load the image offline

      • Use a machine that is connected to the internet to save the docker image

        docker save mellanox/ufm-plugin-tfs:latest | gzip > ufm-plugin-tfs.tar.gz
        

        Copy

      • Move the file to some shared location that is accessible to the UFM machine

      • Load the image to UFM machine

        docker load -i < /[some-shared-location]/ufm-plugin-tfs.tar.gz
        

        Copy

• Enable & start the plugin

  • UFM bare-metal installation:

  /opt/ufm/scripts/manage_ufm_plugins.sh add -p tfs
  

  Copy
  • UFM docker installation:

  docker exec ufm /opt/ufm/scripts/manage_ufm_plugins.sh add -p tfs
  

  Copy

• Check that plugin is up and running with

  • UFM bare-metal installation:

  /opt/ufm/scripts/manage_ufm_plugins.sh show
  

  Copy
  • UFM docker installation:

  docker exec ufm /opt/ufm/scripts/manage_ufm_plugins.sh show
  

  Copy

• Pull the Fluentd Docker⁠ by running:

  docker pull fluent/fluentd
  

  Copy

• Run the Fluentd docker by running:

  docker run -ti --rm --network host -v /tmp/fluentd:/fluentd/etc fluentd -c /fluentd/etc/fluentd.conf -v
  

  Copy

• We provide fluentd.conf⁠ as a fluentd configurations sample.
• TFS supports FORWARD⁠ and HTTP⁠ protocols for sending the data to the Fluent destination. The FORWARD is the default protocol, to switch to the HTTP protocol, need to enable streaming.compressed_streaming⁠

TFS supports streaming the data into the Fluent destination via the IPv6, you need to make sure that the Fluent is listening to the IPv6 interface by replacing fluentd host address⁠ (bind 0.0.0.0) with (bind ::)

The following REST API is provided to set the plugin's configurations:

cURL Example:

curl --location 'https://<UFM_IP>/ufmRest/plugin/tfs/conf' \
--header 'Content-Type: application/json' \
--data '{
 "fluentd-endpoint": {
     "host": "<FLUENT_RECIEVER_IP>",
     "port": 24224,
     "timeout": 120
 },
 "streaming": {
     "enabled": true,
     "stream_only_new_samples": false
 },
 "ufm-telemetry-endpoint": [
     {
         "host": "127.0.0.1",
         "interval": <COLLECTING_INTERVAL_SECONDS, e.g. 30>,
         "port": <TELEMETRY_HTTP_PORT, e.g. 9001>,
         "url": <TELEMETRY_CSET_URL, e.g. csv/metrics OR csv/cset/converted_enterprise>
     }
 ]
}' -k -u <UFM_USERNAME>:<UFM_PASSWORD>

Full Payload Example:

{
     "ufm-telemetry-endpoint": [{
         "host": "127.0.0.1",
         "url": "csv/metrics",
         "port": 9001,
         "interval": 30,
         "message_tag_name": "high_freq_endpoint"
     }],
     "fluentd-endpoint": {
         "host": "10.209.36.68",
         "port": 24226
     },
     "streaming": {
         "compressed_streaming": false,
         "bulk_streaming": true,
         "enabled": true,
         "stream_only_new_samples": true
     },
     "logs-config": {
         "log_file_backup_count": 5,
         "log_file_max_size": 10485760,
         "logs_file_name": "/log/tfs.log",
         "logs_level": "INFO"
     },
     "meta-fields":{
         "alias_node_description": "node_name",
         "alias_node_guid": "AID",
         "add_type":"csv"
     }
 }

The following REST API is provided to get the current plugin's configurations:

cURL Example:

 curl --location 'https://<UFM_IP>/ufmRest/plugin/tfs/conf' -k -u <UFM_USERNAME>:<UFM_PASSWORD>

Response Example:

{
     "ufm-telemetry-endpoint": [{
         "host": "127.0.0.1",
         "url": "csv/metrics",
         "port": 9001,
         "interval": 30,
         "message_tag_name": "high_freq_endpoint"
     }],
     "fluentd-endpoint": {
         "host": "10.209.36.68",
         "port": 24226
     },
     "streaming": {
         "compressed_streaming": false,
         "bulk_streaming": true,
         "enabled": true,
         "stream_only_new_samples": true
     },
     "logs-config": {
         "log_file_backup_count": 5,
         "log_file_max_size": 10485760,
         "logs_file_name": "/log/tfs.log",
         "logs_level": "INFO"
     },
     "meta-fields":{
         "alias_node_description": "node_name",
         "alias_node_guid": "AID",
         "add_type":"csv"
     }
 }

You can configure the TFS plugin to poll metrics from multiple endpoints. To do this, add the telemetry endpoint configurations using the conf API. Each added endpoint will have its own polling/streaming interval.

Payload example with multiple UFM Telemetry endpoints:

{
     "ufm-telemetry-endpoint": [{
         "host": "127.0.0.1",
         "url": "csv/metrics",
         "port": 9001,
         "interval": 10,
         "message_tag_name": "high_freq_endpoint"
     },{
         "host": "127.0.0.1",
         "url": "csv/metrics",
         "port": 9002,
         "interval": 60,
         "message_tag_name": "low_freq_endpoint"
     }],
     "fluentd-endpoint": {
         "host": "10.209.36.68",
         "port": 24226
     }
 }

The sharding functionality that is built into UFM telemetry, allows for efficient data polling from multiple telemetry metrics endpoints. This feature is particularly useful when dealing with large amounts of data or when operating in a network with limited bandwidth.

⁠How To Utilize Sharding in TFS:

To use the sharding functionality, you need to add specific parameters to the URL of the configured telemetry endpoint. These parameters include num_shards, shard, and sharding_field.

Here is a payload example of how to use these parameters with the TFS configurations payload:

{
     "ufm-telemetry-endpoint": [{
         "host": "127.0.0.1",
         "url": "csv/xcset/ib_basic_debug?num_shards=3&shard=0&sharding_field=port_guid",
         "port": 9002,
         "interval": 120
     },{
         "host": "127.0.0.1",
         "url": "csv/xcset/ib_basic_debug?num_shards=3&shard=1&sharding_field=port_guid",
         "port": 9002,
         "interval": 120
     },{
         "host": "127.0.0.1",
         "url": "csv/xcset/ib_basic_debug?num_shards=3&shard=2&sharding_field=port_guid",
         "port": 9002,
         "interval": 120
     }],
     "fluentd-endpoint": {
         "host": "10.209.36.68",
         "port": 24226
     }
 }

In this example, the telemetry data is divided into three shards (num_shards=3), and each endpoint with a different shard (shard=0, shard=1, shard=2). The sharding_field parameter is used to specify the field on which the data is to be sharded. In the provided example, sharding_field is set to port_guid. This means that the data is divided into shards based on the port_guid field. This field was chosen because it provides a convenient way to divide the data into distinct, non-overlapping shards.

⁠Tuning the Sharding:

For optimal performance, it is recommended to tune the sharding so that a single shard transfers in about 10-15 seconds. This leaves plenty of overhead to avoid the telemetry's server timeout issues. You may need to experiment with the number of shards to achieve this. For instance, if your network is slow, you might need to increase the number of shards.

Meta-fields are custom fields that you can add to each record streamed through TFS. There are two types of meta-fields: Aliases and Constants.

Aliases Aliases allow you to rename an existing field in the record. To create an alias, specify the original field name and the new name you want to use. Note that aliases only work with fields that match the exact name specified.

• Syntax

  alias_originalFieldName=aliasName

• Example If you want to rename the field "node_guid" to "AID", you would use:

  alias_node_guid=AID

Constans Constants let you add a new field with a fixed value to each record.

• Syntax

  add_newFieldName=constantValue

• Example To add a new field named "type" with the value "csv", you would use:

  add_type=csv

Payload configurations example Here’s how you can define these meta-fields in the TFS configuration payload:

{
    "meta-fields": {
        "alias_node_description": "node_name",
        "alias_node_guid": "AID",
        "add_type": "csv"
    }
}

Expected output

{
      "timestamp": "1644411135311315",
      "source_id": "0xe41d2d030003e450",
      "node_guid": "e41d2d030003e450",
      "port_guid": "e41d2d030003e450",
      "port_num": "10",
      "node_description": "SwitchIB Mellanox Technologies",
      "node_name": "SwitchIB Mellanox Technologies",
      "AID": "e41d2d030003e450",
      "type": "csv"
}

• Get the current attributes configurations by the following REST API:

  METHOD: GET

  URL: https://[UFM-IP]/ufmRest/plugin/tfs/conf/attributes

  cURL Example:

   curl --location 'https://<UFM_IP>/ufmRest/plugin/tfs/conf/attributes' -k -u <UFM_USERNAME>:<UFM_PASSWORD>
  

  Copy

  Response Example:

  JSON contains all the attributes and their configurations:

  { 
  "ExcessiveBufferOverrunErrorsExtended": {
      "enabled": true,
      "name": "ExcessiveBufferOverrunErrorsExtended"
   },
  "LinkDownedCounterExtended": {
      "enabled": true,
      "name": "LinkDownedCounterExtended"
   },
    "LinkErrorRecoveryCounterExtended": {
      "enabled": true,
      "name": "LinkErrorRecoveryCounterExtended"
   },
    "LocalLinkIntegrityErrorsExtended": {
      "enabled": true,
      "name": "LocalLinkIntegrityErrorsExtended"
   }
  }
  

  Copy

• Update the streaming attributes configurations by the following REST API:

  METHOD: POST

  URL: https://[UFM-IP]/ufmRest/plugin/tfs/conf/attributes

  cURL Example:

   curl --location 'https://<UFM_IP>/ufmRest/plugin/tfs/conf/attributes' \
   --header 'Content-Type: application/json' \
   --data '{
        "ExcessiveBufferOverrunErrorsExtended": {
           "enabled": true,
           "name": "ExcBuffOverrunErrExt"
       },
       "LinkDownedCounterExtended": {
           "enabled": false
       },
       "LinkErrorRecoveryCounterExtended": {
           "enabled": true,
           "name": "linkErrRecCountExt"
       },
       "LocalLinkIntegrityErrorsExtended": {
           "enabled": true,
           "name": "localLinkIntErrExt"
       }
   }' -k -u <UFM_USERNAME>:<UFM_PASSWORD>       
  

  Copy

• Changes to attribute configurations are applied automatically and will take effect during the next streaming period.

Prometheus HTTP endpoint is provided, that contains metrics about the streaming performance & statistics for the last streaming period.

• Get the streaming performance statistics by the following API:

  METHOD: GET

  URL: https://[UFM-IP]/ufmRest/plugin/tfs/metrics

  Response: Text contains performance metrics for the last streaming interval in Prometheus format:

  # HELP num_of_processed_counters_in_last_msg Number of processed counters/attributes in the last streaming interval
  # TYPE num_of_processed_counters_in_last_msg gauge
  num_of_processed_counters_in_last_msg{endpoint="10.209.36.68:9001/csv/xcset/ib_basic_debug"} 176.0
  num_of_processed_counters_in_last_msg{endpoint="10.209.36.67:9001/csv/xcset/ib_basic_debug"} 189.0
  # HELP num_of_streamed_ports_in_last_msg Number of processed ports in the last streaming interval
  # TYPE num_of_streamed_ports_in_last_msg gauge
  num_of_streamed_ports_in_last_msg{endpoint="10.209.36.68:9001/csv/xcset/ib_basic_debug"} 6.0
  num_of_streamed_ports_in_last_msg{endpoint="10.209.36.67:9001/csv/xcset/ib_basic_debug"} 4.0
  # HELP streaming_time_seconds Time period for last streamed message in seconds
  # TYPE streaming_time_seconds gauge
  streaming_time_seconds{endpoint="10.209.36.68:9001/csv/xcset/ib_basic_debug"} 0.064626
  streaming_time_seconds{endpoint="10.209.36.67:9001/csv/xcset/ib_basic_debug"} 0.025279
  # HELP telemetry_expected_response_size_bytes Expected size of the last received telemetry response in bytes
  # TYPE telemetry_expected_response_size_bytes gauge
  telemetry_expected_response_size_bytes{endpoint="10.209.36.68:9001/csv/xcset/ib_basic_debug"} 5156.0
  telemetry_expected_response_size_bytes{endpoint="10.209.36.67:9001/csv/xcset/ib_basic_debug"} 4726.0
  # HELP telemetry_received_response_size_bytes Actual size of the last received telemetry response in bytes
  # TYPE telemetry_received_response_size_bytes gauge
  telemetry_received_response_size_bytes{endpoint="10.209.36.68:9001/csv/xcset/ib_basic_debug"} 5156.0
  telemetry_received_response_size_bytes{endpoint="10.209.36.67:9001/csv/xcset/ib_basic_debug"} 4726.0
  # HELP telemetry_response_time_seconds Response time of the last telemetry request in seconds
  # TYPE telemetry_response_time_seconds gauge
  telemetry_response_time_seconds{endpoint="10.209.36.68:9001/csv/xcset/ib_basic_debug"} 0.028893
  telemetry_response_time_seconds{endpoint="10.209.36.67:9001/csv/xcset/ib_basic_debug"} 0.07777
  # HELP telemetry_response_process_time_seconds Processing time of the last received telemetry response in seconds
  # TYPE telemetry_response_process_time_seconds gauge
  telemetry_response_process_time_seconds{endpoint="10.209.36.68:9001/csv/xcset/ib_basic_debug"} 0.00455
  telemetry_response_process_time_seconds{endpoint="10.209.36.67:9001/csv/xcset/ib_basic_debug"} 0.003142
  

  Copy

  Attribute	Description
  num_of_streamed_ports_in_last_msg	# of processed ports in the last streaming interval
  num_of_processed_counters_in_last_msg	# of processed counters/attributes in the last streaming interval
  streaming_time_seconds	Time period for last streamed message in seconds
  telemetry_expected_response_size_bytes	Expected size of the last recivied telemetry response in bytes
  telemetry_received_response_size_bytes	Actual size of the last recivied telemetry response in bytes
  telemetry_response_time_seconds	Response time of the last telemetry request in seconds
  telemetry_response_process_time_seconds	Processing time of the last recivied telemetry response in seconds

The below charts present the total processing and streaming time for various sets of ports & counters,it does not include the real telemetry response time for requesting the data: