Windows webdav does an OPTIONS request on the root even when given a
path and if we return 404 here then Windows refuses to use the path.
This patch allows OPTIONS requests only on the root to fix this.
This affects all the HTTP servers.
Before this change it wasn't possible to see where transfers were
going from and to in core/stats and core/transferred.
When use in rclone mount in particular this made interpreting the
stats very hard.
the field `raw` of `oauth2.Token` may be an uncomparable type(often map[string]interface{}), causing `*token != *ts.token` expression to panic(comparing uncomparable type ...).
the semantics of comparing whether two tokens are the same can be achieved by comparing accessToken, refreshToken and expire to avoid panic.
Before this change multi-thread copies using the FTP backend used to error with
551 Error reading file
This was caused by a spurious error being reported which this code silences.
Fixes#7532
See #3942
When f.opt.MaxAge == 0, f.db is never set, however several methods later assume
it is set and attempt to access it, causing an invalid memory address error.
This change fixes the issue in a few spots (there may still be others I haven't
yet encountered.)
Before this change, calling the `oauthutil.NewRenew` func may
cause goroutine leaks.
This change adds a `Shutdown` method to allow the caller to exit
the goroutine to avoid leaks.
Signed-off-by: rkonfj <rkonfj@gmail.com>
In ths security related issue the go1.21.4 stdlib changed the parsing
of volume names on Windows.
https://github.com/golang/go/issues/63713
This had the consequences of breaking the MkdirAll tests which were
looking for specific error messages which changed and using invalid
paths.
In particular under go1.21.3:
filepath.VolumeName(`\\?\C:`) == `\\?\C:`
But under go1.21.4 it is:
filepath.VolumeName(`\\?\C:`) == `\\?`
The path `\\?\C:` isn't actually a valid Windows path. I reported this
as a FYI bug upstream - I'm not expecting it to be fixed.
See: https://github.com/golang/go/issues/64101
makes the following go strings functions available to be used in custom templates; contains, hasPrefix, hasSuffix
added documentation for exported funcs
Before this change, the maximum number of connections was set to 10.
This means that b2 could deadlock while uploading multipart uploads
due to a lock being held longer than it should have been.
Before this change the concurrency used for an upload was rather
inconsistent.
- if size below `--backend-upload-cutoff` (default 200M) do single part upload.
- if size below `--multi-thread-cutoff` (default 256M) or using streaming
uploads (eg `rclone rcat) do multipart upload using
`--backend-upload-concurrency` to set the concurrency used by the uploader.
- otherwise do multipart upload using `--multi-thread-streams` to set the
concurrency.
This change makes the default for the concurrency used be the
`--backend-upload-concurrency`. If `--multi-thread-streams` is set and larger
than the `--backend-upload-concurrency` then that will be used instead.
This means that if the user sets `--backend-upload-concurrency` then it will be
obeyed for all multipart/multi-thread transfers and the user can override them
all with `--multi-thread-streams`.
See: #7056
This makes the memory controls of the s3 backend inoperative and
replaced with the global ones.
--s3-memory-pool-flush-time
--s3-memory-pool-use-mmap
By using the buffered reader this fixes excessive memory use when
uploading large files as it will share memory pages between all
readers.
Fixes#7141
I ( @boukendesho ) have volunteered to maintain the snap package so
this adds it back into the installation instructions.
It will set a `snap` tag visible in `rclone version` so we know where
it came from for support queries.
From the Go docs:
"A `nil` map is equivalent to an empty map. [1]
Therefore, an additional nil check for `opts.ExtraHeaders` before the loop is
unnecessary because `opts.ExtraHeaders` is a `map`.
[1]: https://go.dev/ref/spec#Map_types
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This adds an additional parameter to the creation of each flag. This
specifies one or more flag groups. This **must** be set for global
flags and **must not** be set for local flags.
This causes flags.md to be built with sections to aid comprehension
and it causes the documentation pages for each command (and the
`--help`) to be built showing the flags groups as specified in the
`groups` annotation on the command.
See: https://forum.rclone.org/t/make-docs-for-mortals-not-only-rclone-gurus/39476/
Some changes about test cases:
Because MiddlewareCORS will return early on OPTIONS request,
this middleware should only be used once at NewServer function.
Test cases should pass AllowOrigin config instead of adding
this middleware again.
A new test case was added to test CORS preflight request with
an authenticator. Preflight request should always return 200 OK
regardless of autentications.
Co-authored-by: yuudi <yuudi@users.noreply.github.com>
This introduces a new fs.Option flag, Sensitive and uses this along
with IsPassword to redact the info in the config file for support
purposes.
It adds this flag into backends where appropriate. It was necessary to
add oauthutil.SharedOptions to some backends as they were missing
them.
Fixes#5209
golang.org/x/oauth2/jws is deprecated: this package is not intended for public use and
might be removed in the future. It exists for internal use only. Please switch to another
JWS package or copy this package into your own source tree.
github.com/golang-jwt/jwt/v4 seems to be a good alternative, and was already
an implicit dependency.
Before this fix, it was noticed that the rclone webdav client did not
re-use HTTP connections when it should have been.
This turned out to be because rclone was not draining the HTTP bodies
when it was not expecting a response.
From the Go docs:
> If the returned error is nil, the Response will contain a non-nil
> Body which the user is expected to close. If the Body is not both
> read to EOF and closed, the Client's underlying RoundTripper
> (typically Transport) may not be able to re-use a persistent TCP
> connection to the server for a subsequent "keep-alive" request.
This fixes the problem by draining up to 10MB of data from an HTTP
response if the NoResponse flag is set, or at the end of a JSON or XML
response (which could have some whitespace on the end).
See: https://forum.rclone.org/t/webdav-with-persistent-connections/37024/
