Increasing server CPU usage by chart if incoming messages include additional properties

[colinl]

Current Behavior

If messages passed to the ui-chart node include significant amounts of additional data then the node red process uses an increasing amount of CPU time as the number of data points on the chart increases.
For the example flow below, running node-red on an Intel i7, initially node-red shows about 2% usage. The flow injects a message each second, and over about 20 mins the CPU slowly increases to around 8%. Stopping the incoming messages immediately drops the CPU usage back to 2%, and releasing them again switches it back to 8%. Obviously, using 6% of an i7 core on just buffering up messages is ridiculous.

It is not a browser related issue as the symptom can be seen even if there is no browser open.

The fact that it increases slowly means that it not to do with rolling the message buffer as it is not necessary for the buffer to be full.

The message is significant but not huge, this is the function generating it

msg = {
    "payload": -433,
    "topic": "M3 (Measured)",
    "mac": "3C1ACCBAA19D",
    "data_type": "int16",
    "property": "battery_power",
    "communication_status": "operational",
    "scale": 1,
    "unit": "W",
    "precision": 1,
    "friendly_name": "Battery Power",
    "device_class": "power",
    "state_class": "measurement",
    "entry_type": "SENSOR_DEFINITIONS",
    "_msgid": "16df34e86e841df2",
    "messageId": "6a3d7abd40cb157a90acc984",
    "modbusRequest": {
        "unitid": 1,
        "fc": 3,
        "address": 30001,
        "quantity": 1,
        "emptyMsgOnFail": true,
        "keepMsgProperties": true,
        "enableDeformedMessages": false,
        "messageId": "6a3d7abd40cb157a90acc984"
    },
    "responseBuffer": {
        "data": [
            65103
        ],
        "buffer": [
            254,
            79
        ]
    },
    "parse_buffer": [
        254,
        79
    ],
    "specification": {
        "options": {
            "byteSwap": false,
            "resultType": "keyvalue",
            "singleResult": true,
            "msgProperty": "translated_payload",
            "setTopic": true
        },
        "items": [
            {
                "name": "value",
                "type": "int16be",
                "offset": 0,
                "length": 1,
                "mask": "",
                "scale": "1",
                "offsetbit": 0,
                "id": 0,
                "value": -433
            }
        ]
    },
    "originalPayload": [
        65103
    ],
    "values": [
        -433
    ],
    "objectResults": {
        "value": {
            "name": "value",
            "type": "int16be",
            "offset": 0,
            "length": 1,
            "mask": "",
            "scale": "1",
            "offsetbit": 0,
            "id": 0,
            "value": -433
        }
    },
    "keyvalues": {
        "value": -433
    },
    "arrayResults": [
        {
            "name": "value",
            "type": "int16be",
            "offset": 0,
            "length": 1,
            "mask": "",
            "scale": "1",
            "offsetbit": 0,
            "id": 0,
            "value": -433
        }
    ],
    "buffer": [
        254,
        79
    ],
    "translated_payload": {
        "value": -433
    },
    "_event": "node:53ebc095797a50fc"
}
msg.payload = Math.floor(Math.random() * (2000 - (-1000) + 1)) + (-1000);   
return msg;

Adding a node in front of the chart which moves just the payload and topic into a new message and passing that to the chart stops the issue from occurring.

The problem was identified by a user in forum post https://discourse.nodered.org/t/feasability-of-migrating-to-dashboard-2-0/101298.

I am investigating to find which part of the code is using up the CPU.

Expected Behavior

No response

Steps To Reproduce

Run this flow and watch the node-red CPU usage over about 20 minutes.

This is a better flow for investigating the issue. It passes in 10 messages per second, which should still be no problem for the server. Running on an Intel i7 the node-red cpu usage climes to 50% in just a couple of minutes. The flow also includes Inject nodes to block or release the incoming messages and a function node (Reduce) which can be inserted to remove the additional data in the message.

[{"id":"c12532a704294d5c","type":"group","z":"371f5f719cb5e2ab","name":"Chart CPU test. Issue #2151","style":{"label":true},"nodes":["893419d21d52a0f3","392c75611b71aaa9","1a80648df0d6abfd","69c0edb80b6c835a","457d9fc3638fa678","331341489f068df6","0642db45d40d5b8f","e8772e058c096989","cc612ff604efa32f","909073dac6635927","33366f32a7a526c7"],"x":54,"y":79,"w":892,"h":222},{"id":"893419d21d52a0f3","type":"inject","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"1","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":170,"y":220,"wires":[["392c75611b71aaa9"]]},{"id":"392c75611b71aaa9","type":"function","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"Create object","func":"msg = {\n    \"payload\": -433,\n    \"topic\": \"M3 (Measured)\",\n    \"mac\": \"3C1ACCBAA19D\",\n    \"data_type\": \"int16\",\n    \"property\": \"battery_power\",\n    \"communication_status\": \"operational\",\n    \"scale\": 1,\n    \"unit\": \"W\",\n    \"precision\": 1,\n    \"friendly_name\": \"Battery Power\",\n    \"device_class\": \"power\",\n    \"state_class\": \"measurement\",\n    \"entry_type\": \"SENSOR_DEFINITIONS\",\n    \"_msgid\": \"16df34e86e841df2\",\n    \"messageId\": \"6a3d7abd40cb157a90acc984\",\n    \"modbusRequest\": {\n        \"unitid\": 1,\n        \"fc\": 3,\n        \"address\": 30001,\n        \"quantity\": 1,\n        \"emptyMsgOnFail\": true,\n        \"keepMsgProperties\": true,\n        \"enableDeformedMessages\": false,\n        \"messageId\": \"6a3d7abd40cb157a90acc984\"\n    },\n    \"responseBuffer\": {\n        \"data\": [\n            65103\n        ],\n        \"buffer\": [\n            254,\n            79\n        ]\n    },\n    \"parse_buffer\": [\n        254,\n        79\n    ],\n    \"specification\": {\n        \"options\": {\n            \"byteSwap\": false,\n            \"resultType\": \"keyvalue\",\n            \"singleResult\": true,\n            \"msgProperty\": \"translated_payload\",\n            \"setTopic\": true\n        },\n        \"items\": [\n            {\n                \"name\": \"value\",\n                \"type\": \"int16be\",\n                \"offset\": 0,\n                \"length\": 1,\n                \"mask\": \"\",\n                \"scale\": \"1\",\n                \"offsetbit\": 0,\n                \"id\": 0,\n                \"value\": -433\n            }\n        ]\n    },\n    \"originalPayload\": [\n        65103\n    ],\n    \"values\": [\n        -433\n    ],\n    \"objectResults\": {\n        \"value\": {\n            \"name\": \"value\",\n            \"type\": \"int16be\",\n            \"offset\": 0,\n            \"length\": 1,\n            \"mask\": \"\",\n            \"scale\": \"1\",\n            \"offsetbit\": 0,\n            \"id\": 0,\n            \"value\": -433\n        }\n    },\n    \"keyvalues\": {\n        \"value\": -433\n    },\n    \"arrayResults\": [\n        {\n            \"name\": \"value\",\n            \"type\": \"int16be\",\n            \"offset\": 0,\n            \"length\": 1,\n            \"mask\": \"\",\n            \"scale\": \"1\",\n            \"offsetbit\": 0,\n            \"id\": 0,\n            \"value\": -433\n        }\n    ],\n    \"buffer\": [\n        254,\n        79\n    ],\n    \"translated_payload\": {\n        \"value\": -433\n    },\n    \"_event\": \"node:53ebc095797a50fc\"\n}\nmsg.payload = Math.floor(Math.random() * (2000 - (-1000) + 1)) + (-1000);   \nfor (let i = 0; i < 10; i++) {\n    msg.payload = Math.floor(Math.random() * (2000 - (-1000) + 1)) + (-1000);   \n    node.send(msg)\n}   \nreturn null;","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":350,"y":220,"wires":[["e8772e058c096989"]]},{"id":"1a80648df0d6abfd","type":"ui-chart","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","group":"4f87bd59a15b847e","name":"","label":"cpu test","order":9007199254740991,"chartType":"line","category":"topic","categoryType":"msg","xAxisLabel":"","xAxisProperty":"","xAxisPropertyType":"timestamp","xAxisType":"time","xAxisFormat":"","xAxisFormatType":"auto","xmin":"","xmax":"","yAxisLabel":"","yAxisProperty":"payload","yAxisPropertyType":"msg","ymin":"","ymax":"","bins":10,"action":"append","stackSeries":false,"pointShape":"circle","pointRadius":"1","showLegend":true,"removeOlder":"10","removeOlderUnit":"60","removeOlderPoints":"","colors":["#0095ff","#ff0000","#ff7f0e","#2ca02c","#a347e1","#d62728","#ff9896","#9467bd","#c5b0d5"],"textColor":["#666666"],"textColorDefault":true,"gridColor":["#e5e5e5"],"gridColorDefault":true,"width":6,"height":"4","className":"","interpolation":"step","x":860,"y":220,"wires":[[]]},{"id":"69c0edb80b6c835a","type":"inject","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"Block","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":350,"y":120,"wires":[["331341489f068df6"]]},{"id":"457d9fc3638fa678","type":"inject","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"Release","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":360,"y":160,"wires":[["0642db45d40d5b8f"]]},{"id":"331341489f068df6","type":"change","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"Set flow.block","rules":[{"t":"set","p":"block","pt":"flow","to":"true","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":120,"wires":[[]]},{"id":"0642db45d40d5b8f","type":"change","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"Clear flow.block","rules":[{"t":"set","p":"block","pt":"flow","to":"false","tot":"bool"}],"action":"","property":"","from":"","to":"","reg":false,"x":520,"y":160,"wires":[[]]},{"id":"e8772e058c096989","type":"switch","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"Block?","property":"block","propertyType":"flow","rules":[{"t":"false"}],"checkall":"true","repair":false,"outputs":1,"x":510,"y":220,"wires":[["cc612ff604efa32f","1a80648df0d6abfd"]]},{"id":"cc612ff604efa32f","type":"debug","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"debug 16","active":false,"tosidebar":true,"console":false,"tostatus":true,"complete":"payload","targetType":"msg","statusVal":"payload","statusType":"auto","x":640,"y":260,"wires":[]},{"id":"909073dac6635927","type":"function","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"reduce","func":"return {payload: msg.payload, topic: msg.topic}","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":670,"y":200,"wires":[["1a80648df0d6abfd"]]},{"id":"33366f32a7a526c7","type":"inject","z":"371f5f719cb5e2ab","g":"c12532a704294d5c","name":"Clear Chart","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[]","payloadType":"json","x":790,"y":120,"wires":[["1a80648df0d6abfd"]]},{"id":"4f87bd59a15b847e","type":"ui-group","name":"test","page":"c6ff182a4185f2f2","width":"6","height":"1","order":1,"showTitle":true,"className":"","visible":"true","disabled":"false","groupType":"default"},{"id":"c6ff182a4185f2f2","type":"ui-page","name":"Test page","ui":"a872b0a9bcc57d7d","path":"/testpage","icon":"home","layout":"grid","theme":"c4cba76c368d7996","breakpoints":[{"name":"Default","px":"0","cols":"3"},{"name":"Tablet","px":"576","cols":"6"},{"name":"Small Desktop","px":"768","cols":"9"},{"name":"Desktop","px":"1024","cols":"12"}],"order":10,"className":"","visible":true,"disabled":false},{"id":"a872b0a9bcc57d7d","type":"ui-base","name":"My Dashboard","path":"/dashboard","appIcon":"","includeClientData":true,"acceptsClientConfig":["ui-notification","ui-control"],"showPathInSidebar":false,"headerContent":"page","navigationStyle":"default","titleBarStyle":"default","showReconnectNotification":true,"notificationDisplayTime":1,"showDisconnectNotification":true,"allowInstall":false},{"id":"c4cba76c368d7996","type":"ui-theme","name":"Test theme","colors":{"surface":"#ffffff","primary":"#0094ce","bgPage":"#eeeeee","groupBg":"#ffffff","groupOutline":"#cccccc"},"sizes":{"density":"default","pagePadding":"12px","groupGap":"12px","groupBorderRadius":"4px","widgetGap":"12px"}},{"id":"2607f0bd5d70bbac","type":"global-config","env":[],"modules":{"@flowfuse/node-red-dashboard":"1.30.2"}}]

Environment

• Dashboard version: 1.30.2 (also checked with 1.28.0)
• Node-RED version: 5.0.0
• Node.js version: 24.18.0
• npm version:
• Platform/OS: Ubunto 26.04 on laptop
• Browser:

Have you provided an initial effort estimate for this issue?

I am not a FlowFuse team member

[colinl]

Example flow added.

[edterbak]

@colinl Following it.
If you need me to test it again. You can ping me.
I'm not that experienced with git and building test branches myself though. That probably limits my usefulness for you.
I am using docker-compose (inside proxmox), so I am able to pull a specific version that route safely.

[colinl]

@Steve-Mcl, or anyone else, I could do with some help analysing this if you have a little time.

The symptom is that the server node-red cpu usage increases as the number of samples in the datastore increases. That immediately said to me that the obvious cause is the filtering to remove old samples when the time or point count limit is reached. So, in ui_chart.js, I completely removed that code by commenting out from onInput everything from line 266

                       if (maxPoints && config.removeOlderPoints) {

to

                        } else if (config.xAxisType === 'category' && config.chartType !== 'histogram') {
                            // for categorical xaxis and types other than histogram then only keep the latest data point for
                            // each category in each series
                            clearOldCategoricalPoints()
                        } 

But that made no difference.

I then thought that it may be to do with getting the data into the data store (cloning messages and so on) so, in data.js append() , after

            data[node.id].push(config.RED.util.cloneMessage(msg))

I added

            data[node.id].pop()

so that the data is pushed onto the store but then immediately popped off again. That does stop the cpu buildup.

So the problem is not getting the messages into the store, nor is it the code that filters the store to remove old data, but I am a loss to know what else might be causing the problem. As far as I can see there is nothing else that does anything significant related to the store.

Do you have any suggestions on what else I can try?