forked from TrueCloudLab/rclone
onedrive: add --onedrive-delta flag to enable ListR
Before this change ListR was unconditionally enabled on onedrive. This caused performance problems for some uses, so now the --onedrive-delta flag has to be supplied. Fixes #7362
This commit is contained in:
parent
b5857f0bf8
commit
74d5477fad
3 changed files with 43 additions and 1 deletions
|
@ -324,6 +324,37 @@ the --onedrive-av-override flag, or av_override = true in the config
|
||||||
file.
|
file.
|
||||||
`,
|
`,
|
||||||
Advanced: true,
|
Advanced: true,
|
||||||
|
}, {
|
||||||
|
Name: "delta",
|
||||||
|
Default: false,
|
||||||
|
Help: strings.ReplaceAll(`If set rclone will use delta listing to implement recursive listings.
|
||||||
|
|
||||||
|
If this flag is set the the onedrive backend will advertise |ListR|
|
||||||
|
support for recursive listings.
|
||||||
|
|
||||||
|
Setting this flag speeds up these things greatly:
|
||||||
|
|
||||||
|
rclone lsf -R onedrive:
|
||||||
|
rclone size onedrive:
|
||||||
|
rclone rc vfs/refresh recursive=true
|
||||||
|
|
||||||
|
**However** the delta listing API **only** works at the root of the
|
||||||
|
drive. If you use it not at the root then it recurses from the root
|
||||||
|
and discards all the data that is not under the directory you asked
|
||||||
|
for. So it will be correct but may not be very efficient.
|
||||||
|
|
||||||
|
This is why this flag is not set as the default.
|
||||||
|
|
||||||
|
As a rule of thumb if nearly all of your data is under rclone's root
|
||||||
|
directory (the |root/directory| in |onedrive:root/directory|) then
|
||||||
|
using this flag will be be a big performance win. If your data is
|
||||||
|
mostly not under the root then using this flag will be a big
|
||||||
|
performance loss.
|
||||||
|
|
||||||
|
It is recommended if you are mounting your onedrive at the root
|
||||||
|
(or near the root when using crypt) and using rclone |rc vfs/refresh|.
|
||||||
|
`, "|", "`"),
|
||||||
|
Advanced: true,
|
||||||
}, {
|
}, {
|
||||||
Name: config.ConfigEncoding,
|
Name: config.ConfigEncoding,
|
||||||
Help: config.ConfigEncodingHelp,
|
Help: config.ConfigEncodingHelp,
|
||||||
|
@ -645,6 +676,7 @@ type Options struct {
|
||||||
LinkPassword string `config:"link_password"`
|
LinkPassword string `config:"link_password"`
|
||||||
HashType string `config:"hash_type"`
|
HashType string `config:"hash_type"`
|
||||||
AVOverride bool `config:"av_override"`
|
AVOverride bool `config:"av_override"`
|
||||||
|
Delta bool `config:"delta"`
|
||||||
Enc encoder.MultiEncoder `config:"encoding"`
|
Enc encoder.MultiEncoder `config:"encoding"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -976,6 +1008,11 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
|
||||||
|
|
||||||
f.dirCache = dircache.New(root, rootID, f)
|
f.dirCache = dircache.New(root, rootID, f)
|
||||||
|
|
||||||
|
// ListR only supported if delta set
|
||||||
|
if !f.opt.Delta {
|
||||||
|
f.features.ListR = nil
|
||||||
|
}
|
||||||
|
|
||||||
// Find the current root
|
// Find the current root
|
||||||
err = f.dirCache.FindRoot(ctx, false)
|
err = f.dirCache.FindRoot(ctx, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|
|
@ -189,6 +189,9 @@ This remote supports `--fast-list` which allows you to use fewer
|
||||||
transactions in exchange for more memory. See the [rclone
|
transactions in exchange for more memory. See the [rclone
|
||||||
docs](/docs/#fast-list) for more details.
|
docs](/docs/#fast-list) for more details.
|
||||||
|
|
||||||
|
This must be enabled with the `--onedrive-delta` flag (or `delta =
|
||||||
|
true` in the config file) as it can cause performance degradation.
|
||||||
|
|
||||||
It does this by using the delta listing facilities of OneDrive which
|
It does this by using the delta listing facilities of OneDrive which
|
||||||
returns all the files in the remote very efficiently. This is much
|
returns all the files in the remote very efficiently. This is much
|
||||||
more efficient than listing directories recursively and is Microsoft's
|
more efficient than listing directories recursively and is Microsoft's
|
||||||
|
|
|
@ -493,7 +493,7 @@ upon backend-specific capabilities.
|
||||||
| Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | No |
|
| Memory | No | Yes | No | No | No | Yes | Yes | No | No | No | No |
|
||||||
| Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | Yes | No | No | No |
|
| Microsoft Azure Blob Storage | Yes | Yes | No | No | No | Yes | Yes | Yes | No | No | No |
|
||||||
| Microsoft Azure Files Storage | No | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | Yes |
|
| Microsoft Azure Files Storage | No | Yes | Yes | Yes | No | No | Yes | Yes | No | Yes | Yes |
|
||||||
| Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes |
|
| Microsoft OneDrive | Yes | Yes | Yes | Yes | Yes | Yes ⁵ | No | No | Yes | Yes | Yes |
|
||||||
| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | No | Yes |
|
| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | No | Yes |
|
||||||
| OpenStack Swift | Yes ¹ | Yes | No | No | No | Yes | Yes | No | No | Yes | No |
|
| OpenStack Swift | Yes ¹ | Yes | No | No | No | Yes | Yes | No | No | Yes | No |
|
||||||
| Oracle Object Storage | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | No |
|
| Oracle Object Storage | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | No |
|
||||||
|
@ -527,6 +527,8 @@ purging a directory inside a bucket, files are deleted individually.
|
||||||
|
|
||||||
⁴ Use the `--sftp-copy-is-hardlink` flag to enable.
|
⁴ Use the `--sftp-copy-is-hardlink` flag to enable.
|
||||||
|
|
||||||
|
⁵ Use the `--onedrive-delta` flag to enable.
|
||||||
|
|
||||||
### Purge ###
|
### Purge ###
|
||||||
|
|
||||||
This deletes a directory quicker than just deleting all the files in
|
This deletes a directory quicker than just deleting all the files in
|
||||||
|
|
Loading…
Add table
Reference in a new issue