TrueCloudLab/frostfs-node

Fork 27

Metric frostfs_node_engine_container_size_bytes appears after container was removed #938

New issue

Closed

opened 2024-01-29 14:37:59 +00:00 by anikeev-yadro · 1 comment

anikeev-yadro commented

2024-01-29 14:37:59 +00:00

Member

Expected Behavior

Storage node does not produce zero-value metric of container size after container removal according to #864

Current Behavior

Metric frostfs_node_engine_container_size_bytes appears after container was removed

Steps to Reproduce (for bugs)

1.Create container

service@loader-165:~$ frostfs-cli  container create --rpc-endpoint '10.78.130.215:8080' --wallet /etc/k6/scenarios/files/wallet.json --await --policy 'REP 2 IN X CBF 2 SELECT 2 FROM * AS X' --timeout '100s'
Enter password >
container ID: 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C
awaiting...
container has been persisted on sidechain

2.Put object

service@loader-165:~$ frostfs-cli  object put --rpc-endpoint '10.78.130.215:8080' --wallet /etc/k6/scenarios/files/wallet.json --timeout '100s' --file /tmp/object.sample200 --cid 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7Xm
Enter password >
 209715200 / 209715200 [============================================================================================================================================================================================
[/tmp/object.sample200] Object successfully stored
  OID: 2qN1CZg5jWmCqdaYfbcp1mq1811xj2tSGqg7cDsskfB4
  CID: 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C

3.Delete container

service@loader-165:~$ frostfs-cli container delete --rpc-endpoint '10.78.130.215:8080' --wallet /etc/k6/scenarios/files/wallet.json --await --cid 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C
Enter password >
container delete method invoked
awaiting...
container has been removed: 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C

4.Metrics have zero size

root@aanikeev-node1:/tmp# curl -s localhost:6672 | grep frostfs_node_engine_container_objects_total |grep 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C
frostfs_node_engine_container_objects_total{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C",shard_id="MrsZE477JHV3eKKsyrweb3",type="logic"} 0
frostfs_node_engine_container_objects_total{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C",shard_id="MrsZE477JHV3eKKsyrweb3",type="phy"} 0
frostfs_node_engine_container_objects_total{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C",shard_id="MrsZE477JHV3eKKsyrweb3",type="user"} 0

root@aanikeev-node1:/tmp# curl -s localhost:6672 | grep frostfs_node_engine_container_size_byte |grep 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C
frostfs_node_engine_container_size_bytes{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C"} 0

5.Tick epoch

root@aanikeev-node1:/tmp# frostfs-adm morph force-new-epoch -c /tmp/config.yaml
Current epoch: 561, increase to 562.
Waiting for transactions to persist...

6.Metric frostfs_node_engine_container_objects_total was removed, but metric frostfs_node_engine_container_size_byte continue appears

root@aanikeev-node1:/tmp# curl -s localhost:6672 | grep frostfs_node_engine_container_size_byte |grep 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C
frostfs_node_engine_container_size_bytes{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C"} 0

root@aanikeev-node1:/tmp# curl -s localhost:6672 | grep frostfs_node_engine_container_objects_total |grep 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C

Regression

Version

FrostFS Storage node
Version: v0.37.0-rc.1-218-gf5160b27
GoVersion: go1.20.1

Your Environment

Virtual
4 modes

## Expected Behavior Storage node does not produce zero-value metric of container size after container removal according to https://git.frostfs.info/TrueCloudLab/frostfs-node/issues/864 ## Current Behavior Metric frostfs_node_engine_container_size_bytes appears after container was removed ## Steps to Reproduce (for bugs) 1.Create container ``` service@loader-165:~$ frostfs-cli container create --rpc-endpoint '10.78.130.215:8080' --wallet /etc/k6/scenarios/files/wallet.json --await --policy 'REP 2 IN X CBF 2 SELECT 2 FROM * AS X' --timeout '100s' Enter password > container ID: 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C awaiting... container has been persisted on sidechain ``` 2.Put object ``` service@loader-165:~$ frostfs-cli object put --rpc-endpoint '10.78.130.215:8080' --wallet /etc/k6/scenarios/files/wallet.json --timeout '100s' --file /tmp/object.sample200 --cid 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7Xm Enter password > 209715200 / 209715200 [============================================================================================================================================================================================ [/tmp/object.sample200] Object successfully stored OID: 2qN1CZg5jWmCqdaYfbcp1mq1811xj2tSGqg7cDsskfB4 CID: 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C ``` 3.Delete container ``` service@loader-165:~$ frostfs-cli container delete --rpc-endpoint '10.78.130.215:8080' --wallet /etc/k6/scenarios/files/wallet.json --await --cid 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C Enter password > container delete method invoked awaiting... container has been removed: 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C ``` 4.Metrics have zero size ``` root@aanikeev-node1:/tmp# curl -s localhost:6672 | grep frostfs_node_engine_container_objects_total |grep 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C frostfs_node_engine_container_objects_total{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C",shard_id="MrsZE477JHV3eKKsyrweb3",type="logic"} 0 frostfs_node_engine_container_objects_total{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C",shard_id="MrsZE477JHV3eKKsyrweb3",type="phy"} 0 frostfs_node_engine_container_objects_total{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C",shard_id="MrsZE477JHV3eKKsyrweb3",type="user"} 0 root@aanikeev-node1:/tmp# curl -s localhost:6672 | grep frostfs_node_engine_container_size_byte |grep 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C frostfs_node_engine_container_size_bytes{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C"} 0 ``` 5.Tick epoch ``` root@aanikeev-node1:/tmp# frostfs-adm morph force-new-epoch -c /tmp/config.yaml Current epoch: 561, increase to 562. Waiting for transactions to persist... ``` 6.Metric frostfs_node_engine_container_objects_total was removed, but metric frostfs_node_engine_container_size_byte continue appears ``` root@aanikeev-node1:/tmp# curl -s localhost:6672 | grep frostfs_node_engine_container_size_byte |grep 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C frostfs_node_engine_container_size_bytes{cid="8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C"} 0 root@aanikeev-node1:/tmp# curl -s localhost:6672 | grep frostfs_node_engine_container_objects_total |grep 8CvW3PsNX3Chnz9239X4SzPDdRjqPU7XmmnodEyDMq6C ``` ## Regression No ## Version ``` FrostFS Storage node Version: v0.37.0-rc.1-218-gf5160b27 GoVersion: go1.20.1 ``` ## Your Environment Virtual 4 modes

anikeev-yadro added the

bug

triage

labels 2024-01-29 14:37:59 +00:00

dstepanov-yadro self-assigned this 2024-01-29 14:49:20 +00:00

fyrchik added

frostfs-node

and removed

triage

labels 2024-01-29 17:06:12 +00:00

fyrchik added this to the v0.38.0 milestone 2024-01-29 17:06:15 +00:00

dstepanov-yadro referenced this issue

2024-01-30 09:45:06 +00:00

Fix container counters metrics removal #941

dstepanov-yadro referenced this issue from a commit

2024-01-30 10:27:25 +00:00

[#938] engine: Fix container count removal

dstepanov-yadro referenced this issue from a commit

2024-01-30 10:27:25 +00:00

[#938] shard: Update only changed counters

dstepanov-yadro commented

2024-01-30 10:55:54 +00:00

Member

Fixed.

Scenario:
1. Policer inhumes objects of removed container, size metric becomes 0
2. GC removes metric
3. GC removes physical objects of the removed container and updates metric (sets size=0), metric appears again

Fixes:

Do not update metrics if value (size or count) is zero
2. Delete size records in DB (doesn't relate to bug, but it is bug anyway).

Fixed. Scenario: 1. Policer inhumes objects of removed container, size metric becomes 0 2. GC removes metric 3. GC removes physical objects of the removed container and updates metric (sets size=0), metric appears again Fixes: 1. Do not update metrics if value (size or count) is zero 2. Delete size records in DB (doesn't relate to bug, but it is bug anyway).

dstepanov-yadro closed this issue

2024-01-30 10:55:54 +00:00