[#1] Add frostfs backend #1
No reviewers
Labels
No labels
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
No milestone
No project
No assignees
6 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: TrueCloudLab/rclone#1
Loading…
Add table
Reference in a new issue
No description provided.
Delete branch "KurlesHS/rclone:feature/add-frostfs-support"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
To run the FrostFS backend integration tests, we need to prepare a configuration file with the "TestFrostFS" section, for example:
/rclone.conf
and then start tests
Signed-off-by: Aleksey Kravchenko al.kravchenko@yadro.com
54ad3809d2
to538ac9c9df
538ac9c9df
to387f80f636
ca9abdb92e
to4a647a40c7
4a647a40c7
tod9fbd56598
d9fbd56598
to54c191978d
54c191978d
toa370ad60f9
a370ad60f9
to0e26cadc30
0e26cadc30
to2d25df59c5
2d25df59c5
toaacfcd8a28
aacfcd8a28
to7f403ca2db
7f403ca2db
to7453b0430f
7453b0430f
tobfaad53f9a
bfaad53f9a
tof47ef1e0d3
f47ef1e0d3
to6053fb8cce
WIP: [#1] Add frostfs backendto [#1] Add frostfs backend@ -18,6 +18,7 @@ import (
_ "github.com/rclone/rclone/backend/fichier"
_ "github.com/rclone/rclone/backend/filefabric"
_ "github.com/rclone/rclone/backend/filescom"
_ "github.com/rclone/rclone/backend/frostfs"
I suppose we also should add some docs to readme and other places (see contributing guide https://git.frostfs.info/TrueCloudLab/rclone/src/branch/tcl/master/CONTRIBUTING.md)
Done.
@ -0,0 +18,4 @@
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/acl"
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
Unnecessary empty line
@ -0,0 +19,4 @@
cid "git.frostfs.info/TrueCloudLab/frostfs-sdk-go/container/id"
"git.frostfs.info/TrueCloudLab/frostfs-sdk-go/netmap"
// containerApi "git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/container"
Debug?
@ -0,0 +135,4 @@
}
const (
attrFilePath = "FilePath"
Instead of this const we can use
object.AttributeFilePath
@ -0,0 +151,4 @@
Password string `config:"password"`
PlacementPolicy string `config:"placement_policy"`
BasicACLStr string `config:"basic_acl"`
BasicACL acl.Basic `config:"-"`
We must not use basic acl. Since new versions of frostfs-node doesn't support it.
We should either:
basic_acl
flag and form appropriate APE rules and set if for container (after creation)basic_acl
flag toape_rules
and set these rules for containerI have renamed this parameter as "container creation policy" and implemented its parsing in the corresponding APE rules.
@ -0,0 +285,4 @@
if attr.Key() == object.AttributeFileName {
objInfo.name = attr.Value()
}
if attr.Key() == attrFilePath {
We can use
object.AttributeFilePath
@ -0,0 +469,4 @@
// Hashes returns the supported hash sets.
func (f *Fs) Hashes() hash.Set {
return hash.Set(hash.SHA256)
Can be just
Unfortunately, no, because the
hash.SHA256
variable is of typehash.Type
.@ -0,0 +616,4 @@
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
// When updating an object, the path to it should not change.
src = robject.NewStaticObjectInfo(o.Remote(), src.ModTime(ctx), src.Size(), src.Storable(), nil, src.Fs())
obj, err := o.fs.Put(ctx, in, src, options...)
It seems we can use
Patch
now instead ofPut
andDelete
Done. However, it was not possible to avoid using the "Delete" method as the "Patch" method does not remove the original object.
Patch
may retain object ID, is it possible that we "update" file with the same payload?Do we have any randomization in header?
Yes, it's posible
No, we don't. But I added a check for matching object IDs before and after executing the "Patch" method. If they match, the update process ends.
@ -0,0 +46,4 @@
}
seen[address] = struct{}{}
endpointInfo := endpointInfo{
nitpick: Can we use name that don't collide with type name
For example:
@ -3,1 +3,3 @@
go 1.21
go 1.22
toolchain go1.22.10
Do we really need to fix toolchain?
6053fb8cce
toef2d650f44
[#1] Add frostfs backendto WIP: [#1] Add frostfs backendef2d650f44
to52adedbb76
@ -0,0 +110,4 @@
},
},
},
{
question: shall we support
basic-acl
considering it was dropped fromfrostfs-cli
? Or it's for back-compatibility only?see #1 (comment)
@ -0,0 +119,4 @@
Value: "public-read-write",
Help: "Public container, anyone can read and write",
},
{
Is it a correct value? I haven't found it here
To avoid ambiguity, I renamed this parameter to "container creation policy". #1 (comment)
52adedbb76
to7884f2e622
7884f2e622
to25cf424937
25cf424937
tobb44b4a20c
WIP: [#1] Add frostfs backendto [#1] Add frostfs backend[#1] Add frostfs backendto WIP: [#1] Add frostfs backendbb44b4a20c
to111801b51a
111801b51a
to18a193ef37
18a193ef37
to888a1bb7a1
888a1bb7a1
to4c8b84c93a
4c8b84c93a
to814e1a0d18
83e11d38e3
tod7751246e4
7ad3b9e285
toeafb429201
eafb429201
tobf17a132bb
bf17a132bb
tod2e5429f82
d2e5429f82
to10fdf60eaa
10fdf60eaa
to1dc6754c6d
1dc6754c6d
to436d0fcd6e
WIP: [#1] Add frostfs backendto [#1] Add frostfs backend436d0fcd6e
to802952078e
802952078e
toaca96c9681
aca96c9681
to79813b16f9
Overall looks good to me
@ -0,0 +959,4 @@
if idEr, ok := dirEntry.(fs.IDer); ok {
if err = cnrID.DecodeString(idEr.ID()); err == nil {
f.containerIDCache[containerName] = cnrID.String()
}
That's a really wide code ladder 😃
I did a little refactoring of this section of the code.
@ -0,0 +1123,4 @@
addr.SetObject(id)
prmDelete.SetAddress(addr)
if err = f.pool.DeleteObject(ctx, prmDelete); err != nil {
@dkirillov @mbiryukova is there a room for future optimization by creating tombstone objects manually?
@ -116,3 +116,3 @@
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2-beta.8331.25cf42493.feature/add-frostfs-support")
Is this change autogenerated, by the way?
Yes, the makefile has a separate step for generating documentation, which takes a fairly large piece of information directly from the code.
@ -0,0 +174,4 @@
Option request_timeout.
FrostFS request timeout
Enter a value of type Duration. Press Enter for the default (4s).
Should this default value match default value from properties description below?
Fixed.
79813b16f9
to3a783d209e
New commits pushed, approval review dismissed automatically according to repository settings
New commits pushed, approval review dismissed automatically according to repository settings
3a783d209e
to1652352544
1652352544
tob93e134b5b