Commit graph

571 commits

Author SHA1 Message Date
Dan Walters
7e2568a312 dlna: cds: don't specify childCount at all when unknown
Basically, solving #3541 with a different approach - bringing in
the upstream upnpav module, and changing ChildCount from int to a
*int to avoid childCount="0" in the XML output when that value is
simply unknown.

Current approach is leading to some recursion issues and according
to the DLNA spec it shouldn't be necessary, anyway.
2020-02-25 08:41:00 +01:00
Nick Craig-Wood
ec1271818f mount2: hide mount2 command for the moment 2020-02-11 14:28:13 +00:00
Nick Craig-Wood
8318020387 Implement mount2 with go-fuse
This passes the tests and works efficiently with the non sequential vfs ReadAt fix.
2020-02-11 14:28:13 +00:00
Nick Craig-Wood
5c869d5bd3 cmd: make stats be printed on non-zero exit code
This also rationalises the exit sequence so that dumping
goroutines/open files happens regardless of exit error state.

See: https://forum.rclone.org/t/transfer-stats-on-non-0-exit/14211
2020-02-10 16:40:43 +00:00
Nick Craig-Wood
a54210a2e4 docs: restore missing mount --daemon docs
This was done as part of ebfeec9fb4 which unfortunately patched
the auto generated files.
2020-02-10 15:29:39 +00:00
Nick Craig-Wood
8b664c3ec5 docs: restore lost spelling fixes
These came from 3d424c6e08 which unfortunately got added the
docs to the auto generated files.
2020-02-10 15:29:39 +00:00
Nick Craig-Wood
feee92c790 docs: restore lost mount share docs
These came from 162fdfe455 which unfortunately added the docs to
the auto generated files.
2020-02-10 15:29:39 +00:00
Nick Craig-Wood
de93852512 docs: restore lost auth proxy logs
These came from f2a789ea98 which unfortunately added the docs to
the auto generated files.
2020-02-10 15:29:39 +00:00
Nick Craig-Wood
dfb710eab7 gendocs: add autogenerated header to all docs 2020-02-10 15:29:39 +00:00
Lars Lehtonen
4ee3c21a9d cmd/serve/ftp: replace deprecated os.SEEK_SET with io.SeekStart 2020-02-06 10:58:34 +00:00
Nick Craig-Wood
f80d98553a dbhashsum: stop it returning UNSUPPORTED on dropbox 2020-01-29 19:49:42 +00:00
Nick Craig-Wood
bfd9f32188 lsjson: add --no-mimetype flag, speed up lsf
Before this changed we unconditionally fetched the MimeType. On Some
backends like s3 and swift this takes an extra transaction which meant
that `lsf` on those backends was needlessly slow.

This adds an internal option so `lsf` can declare whether it wants
MimeTypes or not depending on whether the user asked for them and an
external flag `--no-mimetype` for `lsjson`.

See: https://forum.rclone.org/t/reliably-setup-incremental-updates/14006/8
2020-01-26 16:38:00 +00:00
landall
77e55b8265 hashsum: Add flag --base64 flag - fixes #3663
This flag can be used to output QuickXorHash in the same format as MS
Graph API.
2020-01-20 17:29:58 +00:00
Xiaoxing Ye
ccaca04a5d rcd: move webgui apart; option to disable browser
Fix #3601, #3785
2020-01-20 12:27:55 +00:00
Nick Craig-Wood
7cf8ea354c dedupe: add missing modes to help string 2020-01-19 11:09:45 +00:00
Nick Craig-Wood
422ad38e5b copyurl: add --stdout flag to write to stdout 2020-01-18 11:15:51 +00:00
Nick Craig-Wood
0b7f959433 cmount: when setting dates discard out of range dates
It appears that sometimes Windows/WinFSP/cgofuse sends dates which are
the epoch to rclone.  These dates appear as 1601-01-01 00:00:00 plus
or minus the timezone.

These dates aren't being sent from rclone.

This patch filters dates out before 1601-01-02 so rclone does not
attempt to set them.

See: https://forum.rclone.org/t/bug-corruption-of-modtime-via-vfs-layer/12204
See: https://forum.rclone.org/t/io-error-googleapi-error-403-insufficient-permission-insufficientpermissions/11372
See: https://github.com/billziss-gh/cgofuse/issues/35
2020-01-18 11:13:35 +00:00
Nick Craig-Wood
81002747c5 dedupe: implement keep smallest too
This is to help deduping google docs and their exported versions if
they accidentally get uploaded to the source again.

See: https://forum.rclone.org/t/my-stupidity-or-a-bug/13861
2020-01-17 13:08:37 +00:00
Nick Craig-Wood
b88dec51e5 proxy: replace use of bcrypt with sha256
Unfortunately bcrypt only hashes the first 72 bytes of a given input
which meant that using it on ssh keys which are longer than 72 bytes
was incorrect.

This swaps over to using sha256 which should be adequate for the
purpose of protecting in memory passwords where the unencrypted
password is likely in memory too.
2020-01-15 16:23:57 +00:00
Paul Tinsley
f2a789ea98 serve sftp: Add support for public key with auth proxy - fixes #3572 2020-01-15 16:23:57 +00:00
Nick Craig-Wood
ae340cf7d9 log: factor flags into logflags package - fixes #3792 2020-01-09 13:25:37 +00:00
Nick Craig-Wood
a4bc4daf30 mounttest: fix unreliable tests on Windows CI
The failure is this which is not reproducable locally, only on the CI
servers.

    --- FAIL: TestMount/CacheMode=minimal/TestWriteFileOverwrite (1.01s)
        fs.go:351:
            Error Trace:    fs.go:351
                            write.go:65
            Error:          Received unexpected error:
                            open E:testwrite: The request could not be performed because of an I/O device error.
            Test:           TestMount/CacheMode=minimal/TestWriteFileOverwrite

The corresponding ERROR from the log is this:

    ERROR : IO error: truncate C:\Users\runneradmin\AppData\Local\rclone\vfs\local\C\Users\RUNNER~1\AppData\Local\Temp\rclone298719627\testwrite: Access is denied.

Instead of using ioutil.WriteFile this fix uses an equivalent based on
rclone's lib/file which doesn't set the exclusive flag on
Windows. This allows files to be deleted that are open.  It also
deletes existing files if an error is received and retries.
2020-01-09 11:11:49 +00:00
Nick Craig-Wood
706da80d88 mount: don't build on go1.10 as bazil/fuse no longer supports it 2020-01-08 08:44:02 +00:00
Danil Semelenov
e2773b3b4e Fix completion with an encrypted config
Closes #3767.
2019-11-29 14:48:12 +00:00
Ankur Gupta
75a6c49f87 Fix error counter - fixes #3650
For few commands, RClone counts a error multiple times. This was fixed by
creating a new error type which keeps a flag to remember if the error has
already been counted or not. The CountError function now wraps the original
error eith the above new error type and returns it.
2019-11-18 14:13:02 +00:00
Nick Craig-Wood
c21a4fee58 mount,cmount: make sure we call unmount when exiting 2019-11-11 22:08:52 +00:00
Nick Craig-Wood
9115752679 proxy: reduce the internal bcrypt strength to fix race tests
Before this change the race tests were taking too long.  The bcrypt
function went from about 20ms to 1s under the race detector and this
is called for every transaction on webdav.

This change reduces the bcrypt strength so it takes 1ms non race so
the race tests pass and still has adequate security for in memory only
storage.
2019-11-11 16:20:28 +00:00
Nick Craig-Wood
e0d9314059 mounttest: fix occasionally failing test TestRenameOpenHandle 2019-11-11 16:20:27 +00:00
Nick Craig-Wood
21c6babdbb mount: enable async reads for a 20% speedup
Now that the vfs can cope with out of order reads we can enable the
async read feature for an increase in througput on the local disk of
about 20%.
2019-11-11 16:20:27 +00:00
Xiaoxing Ye
520ddbcceb config: do not open browser on headless if google fs
On google fs (drive, google photos, and google cloud storage), if
headless is selected, do not open browser.

This also supplies a new option "auth-no-open-browser" for authorize
if the user does not want it.

This should fix #3323.
2019-10-30 14:12:42 +00:00
Xiaoxing Ye
6d58d9a86f vendor: change goftp/server url
Closing #3674
2019-10-29 17:41:56 +00:00
Brett Dutro
378a3f4133 mount: replace use of WriteAt with Write for cache mode >= writes and O_APPEND
os.File.WriteAt returns an error if a file was opened with O_APPEND.
This replaces it with os.File.Write if the file was opened with
O_APPEND.
2019-10-26 17:27:52 +01:00
Nick Craig-Wood
59026c4761 mount, cmount: don't pass huge filenames (>4k) to FUSE as it can't cope 2019-10-21 20:51:11 +01:00
Nick Craig-Wood
1b2ffbeca0 cmd: fix environment variables not setting command line flags
Before this fix quite a lot of the commands were ignoring environment
variables intended to set flags.
2019-10-14 17:02:09 +01:00
Nick Craig-Wood
19429083ad cmd: fix spelling of Definition 2019-10-14 17:02:09 +01:00
Nick Craig-Wood
b4b59c53f1 mount: fix "mount_fusefs: -o timeout=: option not supported" on FreeBSD
Before this change `rclone mount` would give this error on FreeBSD

    mount helper error: mount_fusefs: -o timeout=: option not supported

Because the default value for FreeBSD was set to 15m for
--daemon-timeout and that FreeBSD does not support the timeout option.

This change sets the default for --daemon-timeout to 0 on FreeBSD
which fixes the problem.

Fixes #3610
2019-10-13 11:36:51 +01:00
Nick Craig-Wood
e8b92f4853 sftp: fix test failures
This was introduced by 50a3a96e27
2019-10-09 17:43:03 +01:00
Nick Craig-Wood
50a3a96e27 serve sftp: fix crash on unsupported operations (eg Readlink)
Before this change the sftp handler returned a nil error for unknown
operations which meant the server crashed when one was encountered.

In particular the "Readlink" operations was causing problems.

After this change the handler returns ErrSshFxOpUnsupported which
signals to the remote end that we don't support that operation.

See: https://forum.rclone.org/t/rclone-serve-sftp-not-working-in-windows/12209
2019-10-09 16:12:21 +01:00
Dan Walters
8950b586c4 dlna: associate subtitles with all possible media nodes
When there was a .nfo and a .mp4, they were being associated only with
the .nfo.
2019-10-09 11:57:42 +01:00
Brett Dutro
7d0d7e66ca vfs: move writeback of dirty data out of close() method into its own method (FlushWrites) and remove close() call from Flush()
If a file handle is duplicated with dup() and the duplicate handle is
flushed, rclone will go ahead and close the file, making the original
file handle stale. This change removes the close() call from Flush() and
replaces it with FlushWrites() so that the file only gets closed when
Release() is called. The new FlushWrites method takes care of actually
writing the file back to the underlying storage.

Fixes #3381
2019-10-09 10:07:29 +01:00
Nick Craig-Wood
593de059be lib/terminal: factor from cmd/progress, swap Azure/go-ansiterm for mattn/go-colorable 2019-10-08 19:59:47 +01:00
Dan Walters
572d302620 dlna: simplify search method for associating subtitles with media nodes
Seems to be some corner cases that are not being handled, so taking a different
approach that should be a little more robust.

Also, changing resources to be served under a subpath:  We've been serving
media at /res?path=%2Fdir%2Ffilename.mp4; change that to be just /r/dir/filename.mp4.
It's cleaner, easier to reason about, and a necessary first step towards just
serving the resources via httplib anyway.
2019-10-08 07:49:39 +01:00
Nick Craig-Wood
15b1feea9d mount: fix panic on File.Open - Fixes #3595
This problem was introduced in "mount: allow files of unkown size to
be read properly" 0baafb158f by failure to check that the
DirEntry was nil or not.
2019-10-06 19:26:58 +01:00
Dan Walters
6337cc70d3 dlna: support for external srt subtitles
Allows for filename.srt, filename.en.srt, etc., to be automatically associated with video.mp4 (or whatever) when playing over dlna.

This is the "modern" method, which I've verified to work on VLC and in LG webOS 2.  There is a vendor specific mechanism for Samsung that I havn't been able to get working on my F series.

Also made some minor corrections to logging and container IDs.
2019-10-06 12:18:56 +01:00
Fionera
1dc8bcd48c Remove backend dependency from fs/hash 2019-10-01 16:29:58 +01:00
Nick Craig-Wood
3e53376a49 build_csv: fix output of control characters 2019-09-30 22:00:25 +01:00
Fabian Möller
6ba08b8612 info: rewrite invalid character test and reporting 2019-09-30 14:05:49 +01:00
Richard Patel
32d5af8fb6 cmd/rcd: Address ZipSlip vulnerability
Don't create files outside of target
directory while unzipping.

Fixes #3529 reported by Nico Waisman at Semmle Security Team
2019-09-29 11:14:21 +01:00
Fionera
49d6d6425c serve/httplib: Write the template to a buffer to catch render errors
Fixes #3559
2019-09-22 21:31:11 +01:00
Nick Craig-Wood
08b86cc94b mount: skip tests on <= 2 CPUs to avoid lockup in #3154 2019-09-20 11:53:08 +01:00