Pilorama migration #960
No reviewers
Labels
No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/frostfs-node#960
Loading…
Reference in a new issue
No description provided.
Delete branch "dstepanov-yadro/frostfs-node:feat/pilorama_migrate"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Extended
frostfs-cli control shards evacuation start
to support pilorama evacuation: added optionscope
to specify evacuation scope:all
,objects
ortrees
.Trees will be evacuated to other shards or to other container node.
Relates #947
38d6200ee8
to7307525f62
7307525f62
to863cb6ca8b
fc161ebd0e
toda99e6d8df
51db28a666
tob64596e1d7
b64596e1d7
to6cfb4da032
6cfb4da032
tod36a41e61f
d36a41e61f
to8adaba09ad
8adaba09ad
to66834db5fe
66834db5fe
to5ab641f211
5ab641f211
to89b1c4f69e
89b1c4f69e
to576ee32bb1
WIP: Pilorama migrationto Pilorama migration576ee32bb1
to043088f016
Nice!
043088f016
to0f55610555
0f55610555
to72d03ba6e8
@ -290,3 +312,1 @@
resp.GetBody().GetTotal(),
resp.GetBody().GetFailed(),
resp.GetBody().GetSkipped()))
sb.WriteString(fmt.Sprintf(" Evacuated %d objects out of %d, failed to evacuate: %d, skipped: %d; evacuated %d trees out of %d, failed to evacuate: %d.",
What's the benefit in outputting both objects in trees if only 1 evacuation is being used (the numbers for other will be 0, I guess)
To have unified output: it could be parsed by tests for example.
@ -301,0 +444,4 @@
}
if success {
e.log.Debug(logs.EngineShardsEvacuationTreeEvacuatedLocal,
zap.String("cid", contTree.CID.EncodeToString()), zap.String("treeID", contTree.TreeID),
We use
shard_id
in labels usually, why is itshardID
here?fixed
@ -301,0 +480,4 @@
prm EvacuateShardPrm, shards []pooledShard, weights []float64, shardsToEvacuate map[string]*shard.Shard,
) (bool, string, error) {
target, found, err := e.findShardToEvacuateTree(ctx, tree, shards, weights, shardsToEvacuate)
if err != nil && !prm.IgnoreErrors {
I believe
!prm.IgnoreErrors
is not needed here: we ignore errors of evacuation, but if we haven't found a suitable shard, there is nothing we can do below, right?right, fixed
@ -396,0 +434,4 @@
require.Nil(t, res, "async evacuation must return nil")
require.NoError(t, err, "first evacuation failed")
require.Eventually(t, func() bool {
Eventually
will eventually fail on CI.Though, it is sometimes inevitable, may we could provide a notify channel in parameters, which we
close
upon finish?Changed test to sync version.
@ -1137,0 +1210,4 @@
err := metaerr.Wrap(t.db.View(func(tx *bbolt.Tx) error {
c := tx.Cursor()
for k, _ := c.Seek(prm.NextPageToken); k != nil; k, _ = c.Next() {
if bytes.Equal(k, dataBucket) || bytes.Equal(k, logBucket) || bytes.Equal(k, prm.NextPageToken) {
buckets can be filtered with a simple
v != nil
value (v is get from bothSeek()
andNext()
))prm.NextPageToken
could be checked only on the first iterationCursor opened on
tx
, so it iterates only buckets.One more variable and one more
if
? Ok, fixed.I've thought about this if outside the loop because iteration is ordered and equality can happen only once.
@ -1191,1 +1191,4 @@
}
func TestForest_ListTrees(t *testing.T) {
prev := TreeListTreesBatchSize
Please, no
We provide parameter struct in
TreeListTrees
, why not include batch size there?Done
@ -66,0 +66,4 @@
// TreeListTrees returns all pairs "containerID:treeID".
TreeListTrees(ctx context.Context, prm TreeListTreesPrm) (*TreeListTreesResult, error)
TreeApplyStream(ctx context.Context, cnr cidSDK.ID, treeID string, source <-chan *Move) error
We should mention in comments that
TreeApplyStream
has specific use and can block other write tx for a pretty long timeDone for boltdb implementation.
@ -101,0 +127,4 @@
},
}
err = tree.SignMessage(req, s.key)
Why do se sign here and not in the tree service?
Just because we form request here too. So tree service gets completed request.
72d03ba6e8
tob2e94e35e3
b2e94e35e3
to6a3513745c
6a3513745c
tof2256cac2b
f2256cac2b
to3298a78c7a
3298a78c7a
to4e6fb83407
4e6fb83407
to72e77e02a9
72e77e02a9
to03f6051c0f
03f6051c0f
to60527abb65