TrueCloudLab
/
frostfs-node
19
1
Fork 21
Code Issues 86 Pull Requests 6 Actions Releases Activity

Add container objects counter #778

Merged
fyrchik merged 2 commits from dstepanov-yadro/frostfs-node:feat/container_objects_total_metric into master 2023-11-08 10:44:05 +00:00
Conversation 1 Commits 2 Files Changed 17
dstepanov-yadro commented 2023-11-01 17:55:15 +00:00
Collaborator

Add container counter metric.

Object counter metric marked as deprecated as it can be replaced by new metric.

Closes #763

Add container counter metric. Object counter metric marked as deprecated as it can be replaced by new metric. Closes #763
dstepanov-yadro force-pushed feat/container_objects_total_metric from 745255d977 to fca3c3547c 2023-11-02 11:52:58 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from a45d638d7c to bf88665ee3 2023-11-02 13:50:06 +00:00 Compare
dstepanov-yadro reviewed 2023-11-02 13:54:42 +00:00
pkg/local_object_storage/shard/metric_updater.go Outdated
@ -0,0 +27,4 @@
}
}
func (u *metricUpdater) start() {
dstepanov-yadro commented 2023-11-02 13:54:42 +00:00
Poster
Collaborator

The metric is updated at interval, as it is easier to implement.

The metric is updated at interval, as it is easier to implement.
fyrchik commented 2023-11-06 07:04:24 +00:00
Owner

We update it during PUT and DELETE, what is the purpose of this thread?

We update it during PUT and DELETE, what is the purpose of this thread?
dstepanov-yadro commented 2023-11-06 07:20:31 +00:00
Poster
Collaborator

We update counter in DB during PUT/DELETE, but metric values will be updated with interval.

Now, for example, metric object counter per shard updates on every PUT/DELETE, not with intreval. This lead to such complex return results: https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/metabase/delete.go#L28

We update counter in DB during PUT/DELETE, but metric values will be updated with interval. Now, for example, metric object counter per shard updates on every PUT/DELETE, not with intreval. This lead to such complex return results: https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/metabase/delete.go#L28
fyrchik commented 2023-11-07 07:09:55 +00:00
Owner

The reason for such code is mostly that delete is a batch operation.
May be we could make it non-batch or move metrics update closer to counter update?

I am afraid background process will consume more resources than it should, even when the node is idle.

The reason for such code is mostly that `delete` is a batch operation. May be we could make it non-batch or move metrics update closer to counter update? I am afraid background process will consume more resources than it should, even when the node is idle.
dstepanov-yadro commented 2023-11-07 08:17:45 +00:00
Poster
Collaborator

Ok, fixed.

Ok, fixed.
fyrchik marked this conversation as resolved
dstepanov-yadro changed title from WIP: Add container objects counter to Add container objects counter 2023-11-02 14:07:07 +00:00
dstepanov-yadro requested review from storage-core-committers 2023-11-02 14:07:16 +00:00
dstepanov-yadro requested review from storage-core-developers 2023-11-02 14:07:16 +00:00
fyrchik reviewed 2023-11-06 07:18:32 +00:00
pkg/local_object_storage/metabase/counter.go Outdated
@ -115,0 +250,4 @@
var counterKey []byte
switch typ {
case phy:
counterKey = containerCounterKey(cnrID, true)
fyrchik commented 2023-11-06 07:10:11 +00:00
Owner

It is always the same size, we can pre-allocate here.

It is always the same size, we can pre-allocate here.
dstepanov-yadro commented 2023-11-07 09:14:04 +00:00
Poster
Collaborator

done

done
fyrchik marked this conversation as resolved
pkg/local_object_storage/metabase/counter.go Outdated
@ -176,0 +425,4 @@
if len(buf) != cidSize+3 {
return cid.ID{}, false, fmt.Errorf("invalid key length")
}
typeStr := string(buf[cidSize:])
fyrchik commented 2023-11-06 07:18:18 +00:00
Owner

We do this just to compare with 2 constants, can we avoid allocations here if they are present?

We do this just to compare with 2 constants, can we avoid allocations here if they are present?
dstepanov-yadro commented 2023-11-07 09:14:12 +00:00
Poster
Collaborator

done

done
fyrchik marked this conversation as resolved
pkg/local_object_storage/metabase/put.go Outdated
@ -184,3 +184,3 @@
if !isParent {
err = db.updateCounter(tx, phy, 1, true)
err = db.updateCounter(tx, phy, map[cid.ID]uint64{cnr: 1}, true)
fyrchik commented 2023-11-06 07:08:02 +00:00
Owner

Can we avoid creating a map here?

Can we avoid creating a map here?
dstepanov-yadro commented 2023-11-07 10:05:11 +00:00
Poster
Collaborator

done

done
fyrchik marked this conversation as resolved
pkg/local_object_storage/shard/metrics_test.go Outdated
@ -197,0 +254,4 @@
for _, obj := range oo {
cnr, _ := obj.ContainerID()
expV, expOk := expectedLogCC[cnr.EncodeToString()]
v, ok := mm.getContainerCount(cnr.EncodeToString(), logical)
fyrchik commented 2023-11-06 06:57:45 +00:00
Owner

How about mm.checkContainerCount(t, cnr, logical, v) or require.Equal(t, expV, mm.getContainerCount(t, ...)?
To make the function smaller

How about `mm.checkContainerCount(t, cnr, logical, v)` or `require.Equal(t, expV, mm.getContainerCount(t, ...)`? To make the function smaller
dstepanov-yadro commented 2023-11-07 07:20:08 +00:00
Poster
Collaborator

Done, thx

Done, thx
fyrchik marked this conversation as resolved
pkg/metrics/engine.go Outdated
@ -51,2 +52,2 @@
writeCache: newWriteCacheMetrics(),
mode: newShardIDMode(engineSubsystem, "mode_info", "Shard mode"),
objectCounter: newEngineGaugeVector("objects_total",
"Objects counters per shards. DEPRECATED: Will be deleted in next releasese, use frostfs_node_engine_container_objects_total metric.",
fyrchik commented 2023-11-06 06:56:07 +00:00
Owner

Why is it deprecated? It just has different usage: we count object in the container to calculate quotas, we count objects per shards to calculate the amount of space.

Why is it deprecated? It just has different usage: we count object in the container to calculate quotas, we count objects per shards to calculate the amount of space.
dstepanov-yadro commented 2023-11-06 07:21:58 +00:00
Poster
Collaborator

Deprecated metric (objects per shard) can be calculated from new one (objects per shard per container) with PromQL query. So it looks redundant.

Deprecated metric (objects per shard) can be calculated from new one (objects per shard per container) with PromQL query. So it looks redundant.
fyrchik marked this conversation as resolved
dstepanov-yadro force-pushed feat/container_objects_total_metric from bf88665ee3 to e4ee5f270d 2023-11-07 07:19:45 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from e4ee5f270d to a63a2af7c8 2023-11-07 08:16:44 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from a63a2af7c8 to 7cfb10aba1 2023-11-07 09:12:46 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from 7cfb10aba1 to 9f85720499 2023-11-07 09:13:55 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from 9f85720499 to 06ae40725c 2023-11-07 10:04:34 +00:00 Compare
dstepanov-yadro force-pushed feat/container_objects_total_metric from 06ae40725c to 1ca44a00be 2023-11-07 10:14:56 +00:00 Compare
fyrchik approved these changes 2023-11-07 10:36:45 +00:00
pkg/local_object_storage/metabase/counter.go Outdated
@ -79,0 +207,4 @@
return db.updateContainerCounter(tx, typ, delta, inc)
}
func (db *DB) incCounters(tx *bbolt.Tx, cnrID cid.ID) error {
fyrchik commented 2023-11-07 10:32:27 +00:00
Owner

In bbolt execution time is constrained by the number of keys we put (as values might be stored on different pages and it takes some time to find a key). How about grouping all container counters in 1 value?

In bbolt execution time is constrained by the _number_ of keys we put (as values might be stored on different pages and it takes some time to find a key). How about grouping all container counters in 1 value?
dstepanov-yadro commented 2023-11-07 12:05:00 +00:00
Poster
Collaborator

Now both (phy and log) are stored with single key (CID).

Now both (`phy` and `log`) are stored with single key (`CID`).
pkg/local_object_storage/metabase/counter.go Outdated
@ -161,1 +375,3 @@
err = b.Put(objectPhyCounterKey, data)
func isContainerCounterInitialized(tx *bbolt.Tx, containerCounterB *bbolt.Bucket) (bool, error) {
containerCounterInitialized := false
if err := containerCounterB.ForEach(func(_, _ []byte) error {
fyrchik commented 2023-11-07 10:34:37 +00:00
Owner

This will break if we later have subbuckets here, but I doubt we will.
Anyway, wlll containerCounterInitialized := containerCounterB.Stats().KeyN != 0 suffice?

This will break if we later have subbuckets here, but I doubt we will. Anyway, wlll `containerCounterInitialized := containerCounterB.Stats().KeyN != 0` suffice?
fyrchik commented 2023-11-07 10:41:57 +00:00
Owner

Also, can bucket != nil check serve as a sign that we have initialized everything?

Also, can `bucket != nil` check serve as a sign that we have initialized everything?
dstepanov-yadro commented 2023-11-07 10:43:10 +00:00
Poster
Collaborator

Stats() will read all bucket pages, but we need only first.

`Stats()` will read all bucket pages, but we need only first.
👍 1
dstepanov-yadro commented 2023-11-07 10:51:43 +00:00
Poster
Collaborator

Also, can bucket != nil check serve as a sign that we have initialized everything?

Yes, fixed

> Also, can bucket != nil check serve as a sign that we have initialized everything? Yes, fixed
fyrchik marked this conversation as resolved
dstepanov-yadro force-pushed feat/container_objects_total_metric from 1ca44a00be to a761948e1d 2023-11-07 10:51:34 +00:00 Compare
fyrchik commented 2023-11-07 11:06:28 +00:00
Owner

Failing test -- #788 , seems unrelated

Failing test -- https://git.frostfs.info/TrueCloudLab/frostfs-node/issues/788 , seems unrelated
acid-ant approved these changes 2023-11-07 11:51:30 +00:00
dstepanov-yadro force-pushed feat/container_objects_total_metric from a761948e1d to d5951edc6c 2023-11-07 11:56:59 +00:00 Compare
fyrchik approved these changes 2023-11-07 12:08:30 +00:00
dstepanov-yadro force-pushed feat/container_objects_total_metric from d5951edc6c to 70ab1ebd54 2023-11-08 09:31:24 +00:00 Compare
fyrchik merged commit 70ab1ebd54 into master 2023-11-08 10:44:05 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
acid-ant
fyrchik
TrueCloudLab/storage-core-developers
Labels
Clear labels   
P0

Highest   
P1

High   
P2

Medium   
P3

Low   
badger

Badger DB related issues   
frostfs-adm
   
frostfs-cli
   
frostfs-ir
   
frostfs-lens
   
frostfs-node
   
good first issue

Good for newcomers   
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff   
blocked

The issue or PR is blocked by something   
bug

Something is not working   
config

Configuration format update or breaking change   
discussion

Talking about something in order to reach a decision or to exchange ideas   
documentation

Documentation related   
duplicate

This issue or pull request already exists   
enhancement

New feature   
go

Language features adoption   
help wanted

Extra attention is needed   
internal
   
invalid

Something is wrong   
kludge

This is a kludge and we know it   
observability

Related to metrics, tracing and logs   
perfomance

Perfomance related   
question

More information is needed   
refactoring

Refactoring   
wontfix

This won't be fixed
No Label
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#778
There is no content yet.
Delete Branch "dstepanov-yadro/frostfs-node:feat/container_objects_total_metric"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?