not enough nodes to SELECT from
answer on the dev-env #1448
Labels
No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/frostfs-node#1448
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
It may be a frostfs-cli, frostfs-node or dev-env issue, need to investigate.
Container commands seem not to work. Object search works, though.
Steps to reproduce
dev-env
withmake down clean up
Why does the epoch matter? If I see the network map, container should be built.
I'm not sure, but I think the problem is in this function where we get a network map. I keep working on the problem
func (ac *apeChecker) isContainerKey(pk []byte, cnrID cid.ID, cont *containercore.Container) (bool, error) {
binCnrID := make([]byte, sha256.Size)
cnrID.Encode(binCnrID)
nm, err := netmap.GetLatestNetworkMap(ac.nm)
if err != nil {
return false, err
}
in, err := isContainerNode(nm, pk, binCnrID, cont)
if err != nil {
return false, err
} else if in {
return true, nil
}
// then check previous netmap, this can happen in-between epoch change
// when node migrates data from last epoch container
nm, err = netmap.GetPreviousNetworkMap(ac.nm)
if err != nil {
return false, err
}
return isContainerNode(nm, pk, binCnrID, cont)
}
Suppose we're on the 1st epoch and
pk
isn't in the current network mapnetmap.GetLatestNetworkMap
returns the current network map like the one aboveisContainerNode
returnsfalse
and no errornetmap.GetPreviousNetworkMap
returns an empty network map because of the 0th epochisContainerNode
returnsfalse
and an errornot enough nodes to SELECT from
In general, that error could occur not only on the 1st epoch but also when a network map has had not enough nodes on the previous two epochs (fact checking needed)
I noticed that the problem occurs only when the
-g
flag is usedMost likely because all the wallets you use belong either to node or container owner.
cnrVectors, _ := nm.ContainerNodes(cont.Value.PlacementPolicy(), binCnrID)
func (m NetMap) ContainerNodes(p PlacementPolicy, pivot []byte) ([][]NodeInfo, error) {
In fact, we can successfully get container nodes even when a selector doesn't have enough nodes for a replica, however, in this case the network map mustn't be empty, otherwise an error is returned.
Ref. TrueCloudLab/frostfs-sdk-go#167