TrueCloudLab/frostfs-node
21
1
Fork 27
Code Issues 104 Pull requests 4 Projects Releases 2 Activity Actions

Fix complex object deletion #348

Merged
fyrchik merged 2 commits from dstepanov-yadro/frostfs-node:fix/complex_object_lifetime into master 2023-05-16 12:44:58 +00:00
Conversation 2 Commits 2 Files changed 8 +204 -60
dstepanov-yadro commented 2023-05-15 09:08:44 +00:00
Member
Copy link

To physically delete complex object we need parent with physical objects, not only parent.

Closes #332

To physically delete complex object we need parent with physical objects, not only parent. Closes #332
dstepanov-yadro force-pushed fix/complex_object_lifetime from 0c8f198e3e to 529bde2861 2023-05-15 09:12:15 +00:00 Compare
dstepanov-yadro force-pushed fix/complex_object_lifetime from 529bde2861 to 3e585c4176 2023-05-15 09:13:11 +00:00 Compare
dstepanov-yadro requested review from storage-core-committers 2023-05-15 09:13:45 +00:00
dstepanov-yadro requested review from storage-core-developers 2023-05-15 09:13:45 +00:00
dstepanov-yadro force-pushed fix/complex_object_lifetime from 3e585c4176 to 84694ad465 2023-05-15 09:14:23 +00:00 Compare
acid-ant commented 2023-05-15 09:24:59 +00:00
Member
Copy link

Please update CHANGELOG.

Please update CHANGELOG.
dstepanov-yadro force-pushed fix/complex_object_lifetime from 84694ad465 to 3bcd9f0fd5 2023-05-15 09:57:25 +00:00 Compare
dstepanov-yadro commented 2023-05-15 09:57:56 +00:00
Author
Member
Copy link

Please update CHANGELOG.

Done

> Please update CHANGELOG. Done
aarifullin approved these changes 2023-05-15 11:04:37 +00:00
aarifullin left a comment
Member
Copy link

LGTM

LGTM
ale64bit reviewed 2023-05-15 12:14:05 +00:00
pkg/local_object_storage/metabase/children.go Outdated
@ -0,0 +25,4 @@
}
result[addr] = []oid.Address{}
bkt := tx.Bucket(parentBucketName(addr.Container(), buffer[:]))
ale64bit commented 2023-05-15 12:14:01 +00:00
Member
Copy link

why the slicing ([:]) here? it's already a slice.

why the slicing (`[:]`) here? it's already a slice.
dstepanov-yadro commented 2023-05-15 12:52:58 +00:00
Author
Member
Copy link

fixed

fixed
ale64bit marked this conversation as resolved
pkg/local_object_storage/metabase/children.go Outdated
@ -0,0 +35,4 @@
return err
}
if len(binObjIDs) == 0 {
ale64bit commented 2023-05-15 12:01:33 +00:00
Member
Copy link

why the check? the next loop won't do anything anyway.

why the check? the next loop won't do anything anyway.
dstepanov-yadro commented 2023-05-15 12:52:38 +00:00
Author
Member
Copy link

done

done
ale64bit marked this conversation as resolved
pkg/local_object_storage/shard/gc_test.go Outdated
@ -115,6 +75,8 @@ func Test_GCDropsLockedExpiredObject(t *testing.T) {
epoch.Value = 105
sh.NotificationChannel() <- shard.EventNewEpoch(epoch.Value)
time.Sleep(1 * time.Second) //wait GC completes
ale64bit commented 2023-05-15 12:07:29 +00:00
Member
Copy link

this is unfortunate...isn't there another way? (same below)

this is unfortunate...isn't there another way? (same below)
dstepanov-yadro commented 2023-05-15 13:00:14 +00:00
Author
Member
Copy link

Looks ugly, I agree. But there are two async processes: one for expired objects collecting, second for physically delete expired objects.

Looks ugly, I agree. But there are two async processes: one for expired objects collecting, second for physically delete expired objects.
ale64bit commented 2023-05-15 16:30:50 +00:00
Member
Copy link

I'm not sure what to do about it, but it would be nice to look at it some other time.

I'm not sure what to do about it, but it would be nice to look at it some other time.
ale64bit marked this conversation as resolved
pkg/local_object_storage/shard/gc_test.go Outdated
@ -125,0 +180,4 @@
}, 3*time.Second, 1*time.Second, "expired complex object must be deleted on epoch after lock expires")
}
func createAndInitGCTestShard(t *testing.T, epoch *epochState) *shard.Shard {
ale64bit commented 2023-05-15 12:11:08 +00:00
Member
Copy link

we have already some very similar functionality. Can we avoid the duping?
https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/shard/shard_test.go#L36

we have already some very similar functionality. Can we avoid the duping? https://git.frostfs.info/TrueCloudLab/frostfs-node/src/branch/master/pkg/local_object_storage/shard/shard_test.go#L36
dstepanov-yadro commented 2023-05-15 12:52:29 +00:00
Author
Member
Copy link

done

done
ale64bit marked this conversation as resolved
dstepanov-yadro force-pushed fix/complex_object_lifetime from 3bcd9f0fd5 to a0a9455fec 2023-05-15 12:51:50 +00:00 Compare
ale64bit approved these changes 2023-05-15 16:31:03 +00:00
acid-ant approved these changes 2023-05-16 06:35:34 +00:00
fyrchik reviewed 2023-05-16 08:22:27 +00:00
pkg/local_object_storage/shard/gc_test.go Outdated
@ -115,6 +65,8 @@ func Test_GCDropsLockedExpiredObject(t *testing.T) {
epoch.Value = 105
sh.NotificationChannel() <- shard.EventNewEpoch(epoch.Value)
time.Sleep(1 * time.Second) //wait GC completes
fyrchik commented 2023-05-16 08:20:59 +00:00
Owner
Copy link

Can we do it more reliably? It will break at some point, e.g. with -race flag on CI.

Can we do it more reliably? It will break at some point, e.g. with -race flag on CI.
dstepanov-yadro commented 2023-05-16 08:29:35 +00:00
Author
Member
Copy link

It will be too hard. First we need to wait expired objects collection completes. Then we need to wait GC completed.

It will be too hard. First we need to wait expired objects collection completes. Then we need to wait GC completed.
dstepanov-yadro commented 2023-05-16 08:52:06 +00:00
Author
Member
Copy link

just dropped this wait

just dropped this wait
fyrchik marked this conversation as resolved
pkg/local_object_storage/shard/shard_test.go Outdated
@ -34,3 +39,3 @@
}
func newCustomShard(t testing.TB, rootPath string, enableWriteCache bool, wcOpts []writecache.Option, bsOpts []blobstor.Option) *shard.Shard {
func newCustomShard(t testing.TB, rootPath string, enableWriteCache bool, wcOpts []writecache.Option, bsOpts []blobstor.Option,
fyrchik commented 2023-05-16 08:22:06 +00:00
Owner
Copy link

We already have options for writecache and blobstor. What about also having options for a metabase instead?

We already have options for writecache and blobstor. What about also having options for a metabase instead?
dstepanov-yadro commented 2023-05-16 08:52:21 +00:00
Author
Member
Copy link

fixed

fixed
fyrchik marked this conversation as resolved
dstepanov-yadro force-pushed fix/complex_object_lifetime from a0a9455fec to 107b6b8780 2023-05-16 08:49:17 +00:00 Compare
dstepanov-yadro force-pushed fix/complex_object_lifetime from 107b6b8780 to 76ee54fddb 2023-05-16 08:51:57 +00:00 Compare
fyrchik approved these changes 2023-05-16 12:44:44 +00:00
fyrchik merged commit 869fcbf591 into master 2023-05-16 12:44:58 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
aarifullin
ale64bit
acid-ant
fyrchik
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
badger

Badger DB related issues

  
frostfs-adm

   
frostfs-cli

   
frostfs-ir

   
frostfs-lens

   
frostfs-node

   
good first issue

Good for newcomers

  
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
5 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#348
No description provided.
Delete branch "dstepanov-yadro/frostfs-node:fix/complex_object_lifetime"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?