TrueCloudLab/frostfs-node
21
1
Fork 27
Code Issues 104 Pull requests 4 Projects Releases 2 Activity Actions

Add container objects counter #778

Merged
fyrchik merged 2 commits from dstepanov-yadro/frostfs-node:feat/container_objects_total_metric into master 2024-09-04 19:51:04 +00:00
Conversation 1 Commits 2 Files changed 17 +765 -83
dstepanov-yadro commented 2023-11-01 17:55:15 +00:00
Member
Copy link

Add container counter metric.

Object counter metric marked as deprecated as it can be replaced by new metric.

Closes #763

Add container counter metric. Object counter metric marked as deprecated as it can be replaced by new metric. Closes #763
dstepanov-yadro force-pushed feat/container_objects_total_metric from 745255d977 to fca3c3547c 2023-11-02 11:52:58 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from a45d638d7c to bf88665ee3 2023-11-02 13:50:06 +00:00 Compare
dstepanov-yadro reviewed 2023-11-02 13:54:42 +00:00
pkg/local_object_storage/shard/metric_updater.go Outdated
@ -0,0 +27,4 @@
}
}
func (u *metricUpdater) start() {
dstepanov-yadro commented 2023-11-02 13:54:42 +00:00
Author
Member
Copy link

The metric is updated at interval, as it is easier to implement.

The metric is updated at interval, as it is easier to implement.
fyrchik commented 2023-11-06 07:04:24 +00:00
Owner
Copy link

We update it during PUT and DELETE, what is the purpose of this thread?

We update it during PUT and DELETE, what is the purpose of this thread?
dstepanov-yadro commented 2023-11-06 07:20:31 +00:00
Author
Member
Copy link

We update counter in DB during PUT/DELETE, but metric values will be updated with interval.

Now, for example, metric object counter per shard updates on every PUT/DELETE, not with intreval. This lead to such complex return results: https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/metabase/delete.go#L28

We update counter in DB during PUT/DELETE, but metric values will be updated with interval. Now, for example, metric object counter per shard updates on every PUT/DELETE, not with intreval. This lead to such complex return results: https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/metabase/delete.go#L28
fyrchik commented 2023-11-07 07:09:55 +00:00
Owner
Copy link

The reason for such code is mostly that delete is a batch operation.
May be we could make it non-batch or move metrics update closer to counter update?

I am afraid background process will consume more resources than it should, even when the node is idle.

The reason for such code is mostly that `delete` is a batch operation. May be we could make it non-batch or move metrics update closer to counter update? I am afraid background process will consume more resources than it should, even when the node is idle.
dstepanov-yadro commented 2023-11-07 08:17:45 +00:00
Author
Member
Copy link

Ok, fixed.

Ok, fixed.
fyrchik marked this conversation as resolved
dstepanov-yadro changed title from WIP: Add container objects counter to Add container objects counter 2023-11-02 14:07:07 +00:00
dstepanov-yadro requested review from storage-core-committers 2023-11-02 14:07:16 +00:00
dstepanov-yadro requested review from storage-core-developers 2023-11-02 14:07:16 +00:00
fyrchik reviewed 2023-11-06 07:18:32 +00:00
pkg/local_object_storage/metabase/counter.go Outdated
@ -115,0 +250,4 @@
var counterKey []byte
switch typ {
case phy:
counterKey = containerCounterKey(cnrID, true)
fyrchik commented 2023-11-06 07:10:11 +00:00
Owner
Copy link

It is always the same size, we can pre-allocate here.

It is always the same size, we can pre-allocate here.
dstepanov-yadro commented 2023-11-07 09:14:04 +00:00
Author
Member
Copy link

done

done
fyrchik marked this conversation as resolved
pkg/local_object_storage/metabase/counter.go Outdated
@ -176,0 +425,4 @@
if len(buf) != cidSize+3 {
return cid.ID{}, false, fmt.Errorf("invalid key length")
}
typeStr := string(buf[cidSize:])
fyrchik commented 2023-11-06 07:18:18 +00:00
Owner
Copy link

We do this just to compare with 2 constants, can we avoid allocations here if they are present?

We do this just to compare with 2 constants, can we avoid allocations here if they are present?
dstepanov-yadro commented 2023-11-07 09:14:12 +00:00
Author
Member
Copy link

done

done
fyrchik marked this conversation as resolved
pkg/local_object_storage/metabase/put.go Outdated
@ -184,3 +184,3 @@
if !isParent {
err = db.updateCounter(tx, phy, 1, true)
err = db.updateCounter(tx, phy, map[cid.ID]uint64{cnr: 1}, true)
fyrchik commented 2023-11-06 07:08:02 +00:00
Owner
Copy link

Can we avoid creating a map here?

Can we avoid creating a map here?
dstepanov-yadro commented 2023-11-07 10:05:11 +00:00
Author
Member
Copy link

done

done
fyrchik marked this conversation as resolved
pkg/local_object_storage/shard/metrics_test.go Outdated
@ -197,0 +254,4 @@
for _, obj := range oo {
cnr, _ := obj.ContainerID()
expV, expOk := expectedLogCC[cnr.EncodeToString()]
v, ok := mm.getContainerCount(cnr.EncodeToString(), logical)
fyrchik commented 2023-11-06 06:57:45 +00:00
Owner
Copy link

How about mm.checkContainerCount(t, cnr, logical, v) or require.Equal(t, expV, mm.getContainerCount(t, ...)?
To make the function smaller

How about `mm.checkContainerCount(t, cnr, logical, v)` or `require.Equal(t, expV, mm.getContainerCount(t, ...)`? To make the function smaller
dstepanov-yadro commented 2023-11-07 07:20:08 +00:00
Author
Member
Copy link

Done, thx

Done, thx
fyrchik marked this conversation as resolved
pkg/metrics/engine.go Outdated
@ -51,2 +52,2 @@
writeCache: newWriteCacheMetrics(),
mode: newShardIDMode(engineSubsystem, "mode_info", "Shard mode"),
objectCounter: newEngineGaugeVector("objects_total",
"Objects counters per shards. DEPRECATED: Will be deleted in next releasese, use frostfs_node_engine_container_objects_total metric.",
fyrchik commented 2023-11-06 06:56:07 +00:00
Owner
Copy link

Why is it deprecated? It just has different usage: we count object in the container to calculate quotas, we count objects per shards to calculate the amount of space.

Why is it deprecated? It just has different usage: we count object in the container to calculate quotas, we count objects per shards to calculate the amount of space.
dstepanov-yadro commented 2023-11-06 07:21:58 +00:00
Author
Member
Copy link

Deprecated metric (objects per shard) can be calculated from new one (objects per shard per container) with PromQL query. So it looks redundant.

Deprecated metric (objects per shard) can be calculated from new one (objects per shard per container) with PromQL query. So it looks redundant.
fyrchik marked this conversation as resolved
dstepanov-yadro force-pushed feat/container_objects_total_metric from bf88665ee3 to e4ee5f270d 2023-11-07 07:19:45 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from e4ee5f270d to a63a2af7c8 2023-11-07 08:16:44 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from a63a2af7c8 to 7cfb10aba1 2023-11-07 09:12:46 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from 7cfb10aba1 to 9f85720499 2023-11-07 09:13:55 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from 9f85720499 to 06ae40725c 2023-11-07 10:04:34 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from 06ae40725c to 1ca44a00be 2023-11-07 10:14:56 +00:00 Compare
fyrchik approved these changes 2023-11-07 10:36:45 +00:00
pkg/local_object_storage/metabase/counter.go Outdated
@ -79,0 +207,4 @@
return db.updateContainerCounter(tx, typ, delta, inc)
}
func (db *DB) incCounters(tx *bbolt.Tx, cnrID cid.ID) error {
fyrchik commented 2023-11-07 10:32:27 +00:00
Owner
Copy link

In bbolt execution time is constrained by the number of keys we put (as values might be stored on different pages and it takes some time to find a key). How about grouping all container counters in 1 value?

In bbolt execution time is constrained by the _number_ of keys we put (as values might be stored on different pages and it takes some time to find a key). How about grouping all container counters in 1 value?
dstepanov-yadro commented 2023-11-07 12:05:00 +00:00
Author
Member
Copy link

Now both (phy and log) are stored with single key (CID).

Now both (`phy` and `log`) are stored with single key (`CID`).
pkg/local_object_storage/metabase/counter.go Outdated
@ -161,1 +375,3 @@
err = b.Put(objectPhyCounterKey, data)
func isContainerCounterInitialized(tx *bbolt.Tx, containerCounterB *bbolt.Bucket) (bool, error) {
containerCounterInitialized := false
if err := containerCounterB.ForEach(func(_, _ []byte) error {
fyrchik commented 2023-11-07 10:34:37 +00:00
Owner
Copy link

This will break if we later have subbuckets here, but I doubt we will.
Anyway, wlll containerCounterInitialized := containerCounterB.Stats().KeyN != 0 suffice?

This will break if we later have subbuckets here, but I doubt we will. Anyway, wlll `containerCounterInitialized := containerCounterB.Stats().KeyN != 0` suffice?
fyrchik commented 2023-11-07 10:41:57 +00:00
Owner
Copy link

Also, can bucket != nil check serve as a sign that we have initialized everything?

Also, can `bucket != nil` check serve as a sign that we have initialized everything?
dstepanov-yadro commented 2023-11-07 10:43:10 +00:00
Author
Member
Copy link

Stats() will read all bucket pages, but we need only first.

`Stats()` will read all bucket pages, but we need only first.
👍 1
dstepanov-yadro commented 2023-11-07 10:51:43 +00:00
Author
Member
Copy link

Also, can bucket != nil check serve as a sign that we have initialized everything?

Yes, fixed

> Also, can bucket != nil check serve as a sign that we have initialized everything? Yes, fixed
fyrchik marked this conversation as resolved
dstepanov-yadro force-pushed feat/container_objects_total_metric from 1ca44a00be to a761948e1d 2023-11-07 10:51:34 +00:00 Compare
fyrchik commented 2023-11-07 11:06:28 +00:00
Owner
Copy link

Failing test -- #788 , seems unrelated

Failing test -- https://git.frostfs.info/TrueCloudLab/frostfs-node/issues/788 , seems unrelated
acid-ant approved these changes 2023-11-07 11:51:30 +00:00
dstepanov-yadro force-pushed feat/container_objects_total_metric from a761948e1d to d5951edc6c 2023-11-07 11:56:59 +00:00 Compare
fyrchik approved these changes 2023-11-07 12:08:30 +00:00
dstepanov-yadro force-pushed feat/container_objects_total_metric from d5951edc6c to 70ab1ebd54 2023-11-08 09:31:24 +00:00 Compare
fyrchik merged commit 70ab1ebd54 into master 2023-11-08 10:44:05 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
acid-ant
fyrchik
TrueCloudLab/storage-core-developers
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
badger

Badger DB related issues

  
frostfs-adm

   
frostfs-cli

   
frostfs-ir

   
frostfs-lens

   
frostfs-node

   
good first issue

Good for newcomers

  
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#778
No description provided.
Delete branch "dstepanov-yadro/frostfs-node:feat/container_objects_total_metric"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?