Metric "frostfs_node_writecache_actual_objects_total" shows wrong value #563

New issue

Closed

opened 2023-08-03 16:41:11 +00:00 by anikeev-yadro · 1 comment

anikeev-yadro commented

2023-08-03 16:41:11 +00:00

Member

Expected Behavior

Metric "frostfs_node_writecache_actual_objects_total" should shows actual objects count in writecache.

Current Behavior

Metric "frostfs_node_writecache_actual_objects_total" shows wrong value.

Steps to Reproduce (for bugs)

Put small obects to node
See in logs that WC was flushed objects after PUT

Aug 03 16:26:13 aanikeev-node2 frostfs-node[700593]: 2023-08-03T16:26:13.921Z        info        log/log.go:13        local object storage operation        {"shard_id": "9doad2LeJuMoU5KQvtqK4a", "address": "DJp5f6aDTRWT4rv4g9oC7mrx1zsAMWYxUZY3FGkhvnW7/DqAGT9rQJsjR23CwGEMWYmkYFfa9Jnn5UkcRE8zYWci6", "type": "write-cache", "op": "db PUT"}
Aug 03 16:26:14 aanikeev-node2 frostfs-node[700593]: 2023-08-03T16:26:14.068Z        debug        writecache/flush.go:152        tried to flush items from write-cache        {"shard_id": "UPVhvVR6JWzsDCdWQHhHun", "count": 1, "start": "3MXKEGM4xkxU6Ne6EdhHjwBLtzXtjZ3565jmPrr9WarBpHAHFoT53YfDQYaAEw8rnTGNee1UmC4RCVoN4xtq3FErfAHJePaWPEc8xxdWqwwe797GcZJfuTLptK"}
Aug 03 16:26:14 aanikeev-node2 frostfs-node[700593]: 2023-08-03T16:26:14.667Z        debug        writecache/flush.go:152        tried to flush items from write-cache        {"shard_id": "7XcKSiC77STRYTxwRdzDLu", "count": 1, "start": "XvzbizZck39QRPT7G7QjMRzye3vQgtDi2XXuJdxrJBh4TmhPPJgtKxL89WiWtgrZEjamgV3rAGTNzGw15FuQ6vfFyC2ijfNXD6BTv8fp5Vo99FhTWCaeHgAu"}
Aug 03 16:26:14 aanikeev-node2 frostfs-node[700593]: 2023-08-03T16:26:14.747Z        debug        writecache/flush.go:152        tried to flush items from write-cache        {"shard_id": "9doad2LeJuMoU5KQvtqK4a", "count": 1, "start": "3ReKqV7KEYPytvWStG4QXfm1Fx7NiWQE7yNBMdH7FsMhZbLTGcufU3mYpLuYMejCvMQQuWcSZmgK6bF6A8EkNZDuL1cKxfKiW4s8GcqRtURCy7JwL9eGj77nmT"}

See that metric shows objects in the WC

service@aanikeev-node2:~$ curl -s localhost:6672 | grep frostfs_node_writecache_actual_objects_total
# HELP frostfs_node_writecache_actual_objects_total Actual objects count in writecache
# TYPE frostfs_node_writecache_actual_objects_total gauge
frostfs_node_writecache_actual_objects_total{shard_id="4dMhSsTKiTDbcRisopSU9b",storage="db"} 6
frostfs_node_writecache_actual_objects_total{shard_id="4dMhSsTKiTDbcRisopSU9b",storage="fstree"} 1
frostfs_node_writecache_actual_objects_total{shard_id="BtteeosaNRvWiHCVUCKrWF",storage="db"} 12
frostfs_node_writecache_actual_objects_total{shard_id="BtteeosaNRvWiHCVUCKrWF",storage="fstree"} 1
frostfs_node_writecache_actual_objects_total{shard_id="DH8smfiQV4TcaWQXk2JLqX",storage="db"} 11
frostfs_node_writecache_actual_objects_total{shard_id="DH8smfiQV4TcaWQXk2JLqX",storage="fstree"} 1
frostfs_node_writecache_actual_objects_total{shard_id="L34jnC2SHGdVRPgG8DjLy7",storage="db"} 5
frostfs_node_writecache_actual_objects_total{shard_id="L34jnC2SHGdVRPgG8DjLy7",storage="fstree"} 1
frostfs_node_writecache_actual_objects_total{shard_id="PaUB9Uw3MiP3P3kLKE9zRC",storage="db"} 4
frostfs_node_writecache_actual_objects_total{shard_id="PaUB9Uw3MiP3P3kLKE9zRC",storage="fstree"} 1

4 .Try to flush WC manually

Aug 03 16:27:46 aanikeev-node2 sudo[700621]:     root : TTY=pts/2 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/frostfs-cli --endpoint 127.0.0.1:8091 -w /etc/frostfs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all
5. But metric didn't change values.

## Regression
No

## Version

FrostFS Storage node
Version: v0.0.1-770-gb8052c79
GoVersion: go1.20.5

## Your Environment
Cloud
4 nodes

## Expected Behavior Metric "frostfs_node_writecache_actual_objects_total" should shows actual objects count in writecache. ## Current Behavior Metric "frostfs_node_writecache_actual_objects_total" shows wrong value. ## Steps to Reproduce (for bugs) 1. Put small obects to node 2. See in logs that WC was flushed objects after PUT ``` Aug 03 16:26:13 aanikeev-node2 frostfs-node[700593]: 2023-08-03T16:26:13.921Z info log/log.go:13 local object storage operation {"shard_id": "9doad2LeJuMoU5KQvtqK4a", "address": "DJp5f6aDTRWT4rv4g9oC7mrx1zsAMWYxUZY3FGkhvnW7/DqAGT9rQJsjR23CwGEMWYmkYFfa9Jnn5UkcRE8zYWci6", "type": "write-cache", "op": "db PUT"} Aug 03 16:26:14 aanikeev-node2 frostfs-node[700593]: 2023-08-03T16:26:14.068Z debug writecache/flush.go:152 tried to flush items from write-cache {"shard_id": "UPVhvVR6JWzsDCdWQHhHun", "count": 1, "start": "3MXKEGM4xkxU6Ne6EdhHjwBLtzXtjZ3565jmPrr9WarBpHAHFoT53YfDQYaAEw8rnTGNee1UmC4RCVoN4xtq3FErfAHJePaWPEc8xxdWqwwe797GcZJfuTLptK"} Aug 03 16:26:14 aanikeev-node2 frostfs-node[700593]: 2023-08-03T16:26:14.667Z debug writecache/flush.go:152 tried to flush items from write-cache {"shard_id": "7XcKSiC77STRYTxwRdzDLu", "count": 1, "start": "XvzbizZck39QRPT7G7QjMRzye3vQgtDi2XXuJdxrJBh4TmhPPJgtKxL89WiWtgrZEjamgV3rAGTNzGw15FuQ6vfFyC2ijfNXD6BTv8fp5Vo99FhTWCaeHgAu"} Aug 03 16:26:14 aanikeev-node2 frostfs-node[700593]: 2023-08-03T16:26:14.747Z debug writecache/flush.go:152 tried to flush items from write-cache {"shard_id": "9doad2LeJuMoU5KQvtqK4a", "count": 1, "start": "3ReKqV7KEYPytvWStG4QXfm1Fx7NiWQE7yNBMdH7FsMhZbLTGcufU3mYpLuYMejCvMQQuWcSZmgK6bF6A8EkNZDuL1cKxfKiW4s8GcqRtURCy7JwL9eGj77nmT"} ``` 3. See that metric shows objects in the WC ``` service@aanikeev-node2:~$ curl -s localhost:6672 | grep frostfs_node_writecache_actual_objects_total # HELP frostfs_node_writecache_actual_objects_total Actual objects count in writecache # TYPE frostfs_node_writecache_actual_objects_total gauge frostfs_node_writecache_actual_objects_total{shard_id="4dMhSsTKiTDbcRisopSU9b",storage="db"} 6 frostfs_node_writecache_actual_objects_total{shard_id="4dMhSsTKiTDbcRisopSU9b",storage="fstree"} 1 frostfs_node_writecache_actual_objects_total{shard_id="BtteeosaNRvWiHCVUCKrWF",storage="db"} 12 frostfs_node_writecache_actual_objects_total{shard_id="BtteeosaNRvWiHCVUCKrWF",storage="fstree"} 1 frostfs_node_writecache_actual_objects_total{shard_id="DH8smfiQV4TcaWQXk2JLqX",storage="db"} 11 frostfs_node_writecache_actual_objects_total{shard_id="DH8smfiQV4TcaWQXk2JLqX",storage="fstree"} 1 frostfs_node_writecache_actual_objects_total{shard_id="L34jnC2SHGdVRPgG8DjLy7",storage="db"} 5 frostfs_node_writecache_actual_objects_total{shard_id="L34jnC2SHGdVRPgG8DjLy7",storage="fstree"} 1 frostfs_node_writecache_actual_objects_total{shard_id="PaUB9Uw3MiP3P3kLKE9zRC",storage="db"} 4 frostfs_node_writecache_actual_objects_total{shard_id="PaUB9Uw3MiP3P3kLKE9zRC",storage="fstree"} 1 ``` 4 .Try to flush WC manually ``` Aug 03 16:27:46 aanikeev-node2 sudo[700621]: root : TTY=pts/2 ; PWD=/root ; USER=root ; COMMAND=/usr/bin/frostfs-cli --endpoint 127.0.0.1:8091 -w /etc/frostfs/storage/wallet.json --config /tmp/storage.yaml control shards flush-cache --all 5. But metric didn't change values. ## Regression No ## Version ``` FrostFS Storage node Version: v0.0.1-770-gb8052c79 GoVersion: go1.20.5 ``` ## Your Environment Cloud 4 nodes