Initial EC implementation #205
No reviewers
Labels
No labels
P0
P1
P2
P3
good first issue
pool
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No milestone
No project
No assignees
4 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/frostfs-sdk-go#205
Loading…
Reference in a new issue
No description provided.
Delete branch "fyrchik/frostfs-sdk-go:erasure-code"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Done:
Out of scope:
@ -11,6 +11,7 @@ const (
TypeTombstone
_
TypeLock
TypeECChunk
Will remove, old iteration.
230e6583a4
to5cbd984458
5cbd984458
to47156fd14d
@ -665,0 +682,4 @@
case parser.IEcStmtContext:
res, ok = r.Accept(p).(*netmap.Replica)
default:
continue
This means current statement will be skipped. Why?
We are interested only in EC and REP statements here and in their order.
Previously it was
AllRepStmt
, now we get all children and filter all non-REP and non-EC statements, they will be processed below.@ -913,1 +958,4 @@
}
hasEC = hasEC || p.replicas[i].GetECDataCount() != 0 || p.replicas[i].GetECParityCount() != 0
}
if hasEC && !p.unique {
Also we can check that
X+Y <= Z
forEC X.Y SELECT Z FROM ...
policy.Fixed.
@ -0,0 +42,4 @@
headerLength := 0
for i := range parts {
if parts[i] == nil {
continue
This looks like invalid usage. I think error must be returned.
@ -0,0 +65,4 @@
func (c *Constructor) fillPayload(parts []*objectSDK.Object) {
shards := make([][]byte, len(parts))
for i := range parts {
if parts[i] == nil {
Again, looks like invalid usage.
These functions accept collected parts from other nodes, some of the parts may be nil -- these are the ones we need to reconstruct.
@ -0,0 +5,4 @@
// "crypto/ecdsa"
// objectSDK "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object"
// "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/object/transformer"
Circular dependency, but only for interfaces, will fix.
@ -58,0 +73,4 @@
var err error
enc, err = erasurecode.NewConstructor(int(p.ECParams.DataCount), int(p.ECParams.ParityCount))
if err != nil {
panic(fmt.Errorf("FIXME: %v", err))
Move to
initialize()
?Removed EC from transformer completely.
47156fd14d
to3203051e30
3203051e30
tob7a7717fcb
b7a7717fcb
tocb8fae0c99
cb8fae0c99
tofdb07d39ca
fdb07d39ca
to3640759e12
b04dbf49c7
to47b8c98b38
WIP: initial EC implementationto Initial EC implementation@ -0,0 +31,4 @@
// WriteObject implements the transformer.ObjectWriter interface.
func (t *Target) WriteObject(ctx context.Context, obj *objectSDK.Object) error {
t.c.clear()
Hm, this is already done in
Constructor.Split
. Also this isConstructor
's internal method.true, fixed
@ -0,0 +12,4 @@
c.clear()
var headerLength int
for i := range parts {
Parts need to be sorted by index according this check in
validatePart
:It is important. Need to specify in comment.
But better to create slice copy (it is slice of pointers), sort it and validate against copy.
But this is exactly what
Verify
is intended to catch: it accepts sorted slice of parts (actually it is more of a testing thing, but can also be used to verify reconstruction).47b8c98b38
to02f8c09b72
02f8c09b72
to66e1b3e0c3
66e1b3e0c3
to50e538f27a
@ -0,0 +84,4 @@
// If headerLength is not zero it is asserted to be equal in the ec header.
// Otherwise, new headerLength is returned.
func validatePart(parts []*objectSDK.Object, i int, headerLength int) (int, error) {
ec := parts[i].GetECHeader()
What do you think about adding
i >= 0 && i < len(parts)
check?This is a private method and is always called in a loop in 2 places, I think we are good here.
@ -0,0 +15,4 @@
return nil, err
}
headerShards, err := c.encodeRaw(header)
I've got a question: you separate header and a payload and then marshal the header.
I thought that the number of EC-encoded parts depends on binary size (
header_marshalled_size != payload_marshalled_size
) but here you expect the header shards number is equal to payload shards number. How come?Size is not equal, but the number of shards is equal.
It follows, that the size of chunks are not equal.