TrueCloudLab
/
frostfs-node
18
1
Fork 20
Code Issues 88 Pull Requests 8 Actions Releases Activity

Fix complex object deletion #348

Merged
fyrchik merged 2 commits from dstepanov-yadro/frostfs-node:fix/complex_object_lifetime into master 2023-05-16 12:44:58 +00:00
Conversation 2 Commits 2 Files Changed 8
dstepanov-yadro commented 2023-05-15 09:08:44 +00:00
Collaborator

To physically delete complex object we need parent with physical objects, not only parent.

Closes #332

To physically delete complex object we need parent with physical objects, not only parent. Closes #332
dstepanov-yadro force-pushed fix/complex_object_lifetime from 0c8f198e3e to 529bde2861 2023-05-15 09:12:15 +00:00 Compare
dstepanov-yadro force-pushed fix/complex_object_lifetime from 529bde2861 to 3e585c4176 2023-05-15 09:13:11 +00:00 Compare
dstepanov-yadro requested review from storage-core-committers 2023-05-15 09:13:45 +00:00
dstepanov-yadro requested review from storage-core-developers 2023-05-15 09:13:45 +00:00
dstepanov-yadro force-pushed fix/complex_object_lifetime from 3e585c4176 to 84694ad465 2023-05-15 09:14:23 +00:00 Compare
acid-ant commented 2023-05-15 09:24:59 +00:00
Collaborator

Please update CHANGELOG.

Please update CHANGELOG.
dstepanov-yadro force-pushed fix/complex_object_lifetime from 84694ad465 to 3bcd9f0fd5 2023-05-15 09:57:25 +00:00 Compare
dstepanov-yadro commented 2023-05-15 09:57:56 +00:00
Poster
Collaborator

Please update CHANGELOG.

Done

> Please update CHANGELOG. Done
aarifullin approved these changes 2023-05-15 11:04:37 +00:00
aarifullin left a comment
Collaborator

LGTM

LGTM
ale64bit reviewed 2023-05-15 12:14:05 +00:00
pkg/local_object_storage/metabase/children.go Outdated
@ -0,0 +25,4 @@
}
result[addr] = []oid.Address{}
bkt := tx.Bucket(parentBucketName(addr.Container(), buffer[:]))
ale64bit commented 2023-05-15 12:14:01 +00:00
Collaborator

why the slicing ([:]) here? it's already a slice.

why the slicing (`[:]`) here? it's already a slice.
dstepanov-yadro commented 2023-05-15 12:52:58 +00:00
Poster
Collaborator

fixed

fixed
ale64bit marked this conversation as resolved
pkg/local_object_storage/metabase/children.go Outdated
@ -0,0 +35,4 @@
return err
}
if len(binObjIDs) == 0 {
ale64bit commented 2023-05-15 12:01:33 +00:00
Collaborator

why the check? the next loop won't do anything anyway.

why the check? the next loop won't do anything anyway.
dstepanov-yadro commented 2023-05-15 12:52:38 +00:00
Poster
Collaborator

done

done
ale64bit marked this conversation as resolved
pkg/local_object_storage/shard/gc_test.go Outdated
@ -115,6 +75,8 @@ func Test_GCDropsLockedExpiredObject(t *testing.T) {
epoch.Value = 105
sh.NotificationChannel() <- shard.EventNewEpoch(epoch.Value)
time.Sleep(1 * time.Second) //wait GC completes
ale64bit commented 2023-05-15 12:07:29 +00:00
Collaborator

this is unfortunate...isn't there another way? (same below)

this is unfortunate...isn't there another way? (same below)
dstepanov-yadro commented 2023-05-15 13:00:14 +00:00
Poster
Collaborator

Looks ugly, I agree. But there are two async processes: one for expired objects collecting, second for physically delete expired objects.

Looks ugly, I agree. But there are two async processes: one for expired objects collecting, second for physically delete expired objects.
ale64bit commented 2023-05-15 16:30:50 +00:00
Collaborator

I'm not sure what to do about it, but it would be nice to look at it some other time.

I'm not sure what to do about it, but it would be nice to look at it some other time.
ale64bit marked this conversation as resolved
pkg/local_object_storage/shard/gc_test.go Outdated
@ -125,0 +180,4 @@
}, 3*time.Second, 1*time.Second, "expired complex object must be deleted on epoch after lock expires")
}
func createAndInitGCTestShard(t *testing.T, epoch *epochState) *shard.Shard {
ale64bit commented 2023-05-15 12:11:08 +00:00
Collaborator

we have already some very similar functionality. Can we avoid the duping?
https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/shard/shard_test.go#L36

we have already some very similar functionality. Can we avoid the duping? https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/shard/shard_test.go#L36
dstepanov-yadro commented 2023-05-15 12:52:29 +00:00
Poster
Collaborator

done

done
ale64bit marked this conversation as resolved
dstepanov-yadro force-pushed fix/complex_object_lifetime from 3bcd9f0fd5 to a0a9455fec 2023-05-15 12:51:50 +00:00 Compare
ale64bit approved these changes 2023-05-15 16:31:03 +00:00
acid-ant approved these changes 2023-05-16 06:35:34 +00:00
fyrchik reviewed 2023-05-16 08:22:27 +00:00
pkg/local_object_storage/shard/gc_test.go Outdated
@ -115,6 +65,8 @@ func Test_GCDropsLockedExpiredObject(t *testing.T) {
epoch.Value = 105
sh.NotificationChannel() <- shard.EventNewEpoch(epoch.Value)
time.Sleep(1 * time.Second) //wait GC completes
fyrchik commented 2023-05-16 08:20:59 +00:00
Owner

Can we do it more reliably? It will break at some point, e.g. with -race flag on CI.

Can we do it more reliably? It will break at some point, e.g. with -race flag on CI.
dstepanov-yadro commented 2023-05-16 08:29:35 +00:00
Poster
Collaborator

It will be too hard. First we need to wait expired objects collection completes. Then we need to wait GC completed.

It will be too hard. First we need to wait expired objects collection completes. Then we need to wait GC completed.
dstepanov-yadro commented 2023-05-16 08:52:06 +00:00
Poster
Collaborator

just dropped this wait

just dropped this wait
fyrchik marked this conversation as resolved
pkg/local_object_storage/shard/shard_test.go Outdated
@ -34,3 +39,3 @@
}
func newCustomShard(t testing.TB, rootPath string, enableWriteCache bool, wcOpts []writecache.Option, bsOpts []blobstor.Option) *shard.Shard {
func newCustomShard(t testing.TB, rootPath string, enableWriteCache bool, wcOpts []writecache.Option, bsOpts []blobstor.Option,
fyrchik commented 2023-05-16 08:22:06 +00:00
Owner

We already have options for writecache and blobstor. What about also having options for a metabase instead?

We already have options for writecache and blobstor. What about also having options for a metabase instead?
dstepanov-yadro commented 2023-05-16 08:52:21 +00:00
Poster
Collaborator

fixed

fixed
fyrchik marked this conversation as resolved
dstepanov-yadro force-pushed fix/complex_object_lifetime from a0a9455fec to 107b6b8780 2023-05-16 08:49:17 +00:00 Compare
dstepanov-yadro force-pushed fix/complex_object_lifetime from 107b6b8780 to 76ee54fddb 2023-05-16 08:51:57 +00:00 Compare
fyrchik approved these changes 2023-05-16 12:44:44 +00:00
fyrchik merged commit 869fcbf591 into master 2023-05-16 12:44:58 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
aarifullin
ale64bit
acid-ant
fyrchik
Labels
Clear labels   
P0

Highest   
P1

High   
P2

Medium   
P3

Low   
badger

Badger DB related issues   
frostfs-adm
   
frostfs-cli
   
frostfs-ir
   
frostfs-lens
   
frostfs-node
   
good first issue

Good for newcomers   
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff   
blocked

The issue or PR is blocked by something   
bug

Something is not working   
config

Configuration format update or breaking change   
discussion

Talking about something in order to reach a decision or to exchange ideas   
documentation

Documentation related   
duplicate

This issue or pull request already exists   
enhancement

New feature   
go

Language features adoption   
help wanted

Extra attention is needed   
internal
   
invalid

Something is wrong   
kludge

This is a kludge and we know it   
observability

Related to metrics, tracing and logs   
perfomance

Perfomance related   
question

More information is needed   
refactoring

Refactoring   
wontfix

This won't be fixed
No Label
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
5 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#348
There is no content yet.
Delete Branch "dstepanov-yadro/frostfs-node:fix/complex_object_lifetime"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?