1288 commits

Author	SHA1	Message	Date
Nick Craig-Wood	5a44bafa4e	fstest: add fs.ErrorCantShareDirectories for backends which can only share files	2019-07-24 20:34:29 +01:00
Nick Craig-Wood	8fdce31700	config: Fix hiding of options from the configurator	2019-07-24 20:34:29 +01:00
yparitcher	8e8b78d7e5	Implement --compare-dest & --copy-dest Fixes #3278	2019-07-22 19:42:29 +01:00
buengese	19ae053168	rcserver: remove _async key from input parameters after parsing so later operations won't get confused - fixes #3346	2019-07-20 19:35:10 +02:00
yparitcher	b4266da4eb	sync: fix SyncSuffix tests #3272	2019-07-03 17:36:22 +01:00
Nick Craig-Wood	9cafeeb4b6	dirtree: make tests more reliable	2019-07-02 16:29:40 +01:00
Nick Craig-Wood	bc70bff125	fs/dirtree: factor DirTree out of fs/walk and add tests	2019-07-02 15:26:55 +01:00
Nick Craig-Wood	387b496d1e	operations: fix tests TestMoveFileBackupDir and TestCopyFileBackupDir again ... Commit 734f504d5f wasn't tested properly and had a typo which caused it not to build :-(	2019-07-02 12:22:29 +01:00
Nick Craig-Wood	734f504d5f	operations: fix tests TestMoveFileBackupDir and TestCopyFileBackupDir ... ..so they don't run on backends which can't move or copy.	2019-07-02 10:46:49 +01:00
Aleksandar Jankovic	cc0800a72e	march: return errors when listing dirs ... Partially fixes #3172	2019-07-01 15:32:46 +01:00
AbelThar	70c6b01f54	fs: Higher units for ETA - fixes #3221	2019-07-01 12:09:19 +01:00
Aleksandar Jankovic	7b2b396d37	context: fix errgroup interaction with context ... Fixes #3307	2019-07-01 11:51:51 +01:00
yparitcher	de14378734	Implement --suffix without --backup-dir for current dir ... Fixes #2801	2019-07-01 10:46:26 +01:00
yparitcher	eea1b6de32	Abstract --Backup-dir checks so can be applied across Sync, Copy, Move	2019-07-01 10:46:26 +01:00
Laura	dde4dd0198	fichier: 1fichier support - fixes #2908 ... This was started by Fionera, finished off by Laura with fixes and more docs from Nick. Co-authored-by: Fionera <fionera@fionera.de> Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>	2019-06-30 18:35:01 +01:00
Laura	6ff7b2eaab	fs: add fs.ErrorCantUploadEmptyFiles ... Any backends which can't upload 0 length files should return this errror.	2019-06-30 18:11:45 +01:00
Laura	38ebdf54be	sync/operations: don't use zero length files in tests ... We now have a backend (fichier) which doesn't support 0 length files. Therefore all 0 length files in the tests have been replaced with length 1. In a future commit we will implement a test for 0 length files.	2019-06-30 18:11:45 +01:00
Nick Craig-Wood	276f8cccf6	rc: return current settings if core/bwlimit called without parameters	2019-06-24 13:22:24 +01:00
Nick Craig-Wood	0ae844d1f8	config: reset environment variables in config file test to fix build	2019-06-22 17:49:23 +01:00
Maran	ba72e62b41	fs/config: Add method to reload configfile from disk ... Fixes #3268	2019-06-19 14:47:54 +01:00
Aleksandar Jankovic	5935cb0a29	jobs: add ability to stop async jobs ... Depends on #3257	2019-06-19 14:17:41 +01:00
Aleksandar Jankovic	f78cd1e043	Add context propagation to rclone ... - Change rclone/fs interfaces to accept context.Context - Update interface implementations to use context.Context - Change top level usage to propagate context to lover level functions Context propagation is needed for stopping transfers and passing other request-scoped values.	2019-06-19 11:59:46 +01:00
Nick Craig-Wood	6a2a075c14	fs/cache: fix locking ... This was causing `fatal error: sync: unlock of unlocked mutex` if a panic ocurred in fsNewFs.	2019-06-19 10:50:59 +01:00
Nick Craig-Wood	4549305fec	Start v1.48.0-DEV development	2019-06-15 18:32:17 +01:00
Nick Craig-Wood	245fed513a	Version v1.48.0	2019-06-15 13:55:41 +01:00
Nick Craig-Wood	52332a4b24	moveto: fix detection of same file name to include the root ... Fixes problem introduced in d2be792d5e	2019-06-15 13:55:41 +01:00
Nick Craig-Wood	75606dcc27	sync: fix tests on union remote	2019-06-15 10:53:36 +01:00
Nick Craig-Wood	f3719fe269	fs/cache: unlock mutex in cache.Get to allow recursive calls ... This fixes the test lockup in the union tests	2019-06-15 10:42:53 +01:00
Gary Kim	d2be792d5e	moveto: fix case-insensitive same remote move	2019-06-15 10:06:01 +01:00
Wojciech Smigielski	2793d4b4cc	remove duplicate code	2019-06-15 10:02:25 +01:00
Wojciech Smigielski	30ac9d920a	enable creating encrypted config through external script invocation - fixes #3127	2019-06-15 10:02:25 +01:00
Aleksandar Jankovic	93207ead9c	rc/jobs: make job expiry timeouts configurable	2019-06-15 09:55:32 +01:00
albertony	976a020a2f	Use rclone.conf from rclone executable directory if already existing	2019-06-12 10:08:00 +01:00
Nick Craig-Wood	903ede52cd	config: make config create/update encrypt passwords where necessary ... Before this change when using "rclone config create" it wasn't possible to add passwords in one go, it was necessary to call "rclone config password" to add the passwords afterwards as "rclone config create" didn't obscure passwords. After this change "rclone config create" and "rclone config update" will obscure passwords as necessary as will the corresponding API calls config/create and config/update. This makes "rclone config password" and its API config/password obsolete, however they will be left for backwards compatibility.	2019-06-10 18:08:55 +01:00
Nick Craig-Wood	f681d32996	rc: Fix serving bucket based objects with --rc-serve ... Before this change serving bucket based objects `[remote:bucket]/path/to/object` would fail with 404 not found. This was because the leading `/` in `/path/to/object` was being passed to NewObject.	2019-06-10 11:59:06 +01:00
Garry McNulty	34f03ce590	operations: ignore negative sizes when calculating total (#3135)	2019-06-09 16:00:41 +01:00
forgems	4b27c6719b	fs: Allow sync of a file and a directory with the same name ... When sorting fs.DirEntries we sort by DirEntry type and when synchronizing files let the directories be before objects, so when the destintation fs doesn't support duplicate names, we will only lose duplicated object instead of whole directory. The enables synchronisation to work with a file and a directory of the same name which is reasonably common on bucket based remotes.	2019-06-09 15:57:05 +01:00
Nick Craig-Wood	454dfd3c9e	rc: Add operations/fsinfo: Return information about the remote ... This returns a information about the remote including Name, Root, Hashes and optional Features.	2019-06-08 09:19:07 +01:00
Nick Craig-Wood	e1cf551ded	fs: add Features.Enabled to return map of enabled features by name	2019-06-08 08:46:53 +01:00
Nick Craig-Wood	1aa65d60e1	lsjson: add IsBucket field for bucket based remote listing of the root	2019-06-07 17:28:15 +01:00
Nick Craig-Wood	ac4c8d8dfc	cmd/providers: add DefaultStr, ValueStr and Type fields ... These fields are auto generated: - DefaultStr - a string rendering of Default - ValueStr - a string rendering of Value - Type - the type of the option	2019-06-05 16:23:42 +01:00
garry415	1124c423ee	fs: add --ignore-case-sync for forced case insensitivity - fixes #2773	2019-06-03 21:12:10 +01:00
Nick Craig-Wood	e7c20e0bce	operations: make move and copy individual files obey --backup-dir ... Before this change, when using rclone copy or move with --backup-dir and the source was a single file, rclone would fail to use the backup directory. This change looks up the backup directory in the Fs cache and uses it as appropriate. This affects any commands which call operations.MoveFile or operations.CopyFile which includes rclone move/moveto/copy/copyto where the source is a single file. Fixes #3219	2019-05-27 16:14:55 +01:00
Nick Craig-Wood	8ee6034b23	Look for Fs in the cache rather than calling NewFs directly ... This will save operations when rclone is used in remote control mode or with the same remote multiple times in the command line.	2019-05-27 16:14:55 +01:00
Nick Craig-Wood	206e1caa99	fs/cache: factor Fs caching from fs/rc into its own package	2019-05-27 16:14:55 +01:00
Nick Craig-Wood	0e23c4542f	sync: fix integrations tests ... 2eb31a4f1d broke the integration tests for remotes which use Copy+Delete as server side Move.	2019-05-12 09:50:20 +01:00
Nick Craig-Wood	edda6d91cd	Use go-homedir to read the home directory more reliably	2019-05-11 23:39:04 +01:00
Nick Craig-Wood	7d70e92664	operations: enable multi threaded downloads - Fixes #2252 ... This implements the --multi-thread-cutoff and --multi-thread-streams flags to control multi thread downloading to the local backend.	2019-05-11 23:35:19 +01:00
Nick Craig-Wood	687cbf3ded	operations: if --ignore-checksum is in effect, don't calculate checksum ... Before this change we calculated the checksum which is potentially time consuming and then ignored the result. After the change we don't calculate the checksum if we are about to ignore it.	2019-05-11 23:35:19 +01:00
Nick Craig-Wood	822483aac5	accounting: enable accounting without passing through the stream #2252 ... This is in preparation for multithreaded downloads	2019-05-11 23:35:19 +01:00
Nick Craig-Wood	2eb31a4f1d	sync: move Transferring into operations.Copy ... This makes the code more consistent with the operations code setting the transfer statistics up.	2019-05-11 23:35:19 +01:00
Nick Craig-Wood	0655738da6	operations: re-work reopen framework so it can take a RangeOption #2252 ... This is in preparation for multipart downloads.	2019-05-11 23:35:19 +01:00
Nick Craig-Wood	7c4fe3eb75	local: define OpenWriterAt interface and test and implement it #2252 ... This will enable multipart downloads in future commits	2019-05-11 23:35:19 +01:00
Stefan Breunig	72721f4c8d	copyurl: honor --no-check-certificate	2019-05-11 17:44:58 +01:00
Peter Berbec	0d511b7878	cmd: implement --stats-one-line-date and --stats-one-line-date-format	2019-05-11 17:39:57 +01:00
Jon	bbe28df800	docs: Fix typo: Dump HTTP bodies -> Dump HTTP headers	2019-05-11 16:40:40 +01:00
Fabian Möller	d04b0b856a	fserrors: use errors.Walk for the wrapped error types	2019-05-01 16:56:08 +01:00
Nick Craig-Wood	577fda059d	rc: fix race in tests	2019-05-01 16:09:50 +01:00
Nick Craig-Wood	9df322e889	tests: make test servers choose a random port to make more reliable ... Tests have been randomly failing with messages like listen tcp 127.0.0.1:51778: bind: address already in use Rework all the test servers so they choose a random free port on startup and use that for the tests to avoid.	2019-05-01 16:09:50 +01:00
Fabian Möller	48c09608ea	fix spelling	2019-04-30 14:12:18 +02:00
Nick Craig-Wood	a57756a05c	lsjson, lsf: support showing the Tier of the object	2019-04-16 13:34:06 +01:00
Nick Craig-Wood	cd0d43fffb	fs: add missing PublicLink to mask ... The enables wrapping file systems to declare that they don't support PublicLink if the underlying fs doesn't.	2019-04-16 13:33:10 +01:00
Nick Craig-Wood	e7fc3dcd31	fs: copy the ID too when we copy a Directory object ... This means that crypt which wraps directory objects will retain the ID of the underlying object.	2019-04-16 13:33:10 +01:00
Nick Craig-Wood	2f21aa86b4	fstest: add tests for coverage of optional methods for wrapping Fs	2019-04-16 13:33:10 +01:00
Nick Craig-Wood	16d8014cbb	build: drop support for go1.8	2019-04-15 21:49:58 +01:00
calisro	8190a81201	lsjson: added EncryptedPath to output - fixes #3094	2019-04-15 18:12:09 +01:00
Nick Craig-Wood	e2a2eb349f	Start v1.47.0-DEV development	2019-04-13 13:08:37 +01:00
Nick Craig-Wood	a0d4fdb2fa	Version v1.47.0	2019-04-13 11:01:58 +01:00
Nick Craig-Wood	a28239f005	filter: Make --files-from traverse as before unless --no-traverse is set ... In c5ac96e9e7 we made --files-from only read the objects specified and don't scan directories. This caused problems with Google drive (very very slow) and B2 (excessive API consumption) so it was decided to make the old behaviour (traversing the directories) the default with --files-from and use the existing --no-traverse flag (which has exactly the right semantics) to enable the new non scanning behaviour. See: https://forum.rclone.org/t/using-files-from-with-drive-hammers-the-api/8726 Fixes #3102 Fixes #3095	2019-04-12 17:16:49 +01:00
calistri	5855714474	lsjson: added --files-only and --dirs-only flags ... Factored common code from lsf/lsjson into operations.ListJSON	2019-04-11 11:43:25 +01:00
Nick Craig-Wood	626f0d1886	copy: account for server side copy bytes and obey --max-transfer	2019-03-25 15:36:38 +00:00
Nick Craig-Wood	2065e73d0b	cmd: implement RetryAfter errors which cause a sleep before a retry ... Use NewRetryAfterError to return an error which will cause a high level retry after the delay specified.	2019-03-25 13:41:34 +00:00
Nick Craig-Wood	0d475958c7	Fix errors discovered with go vet nilness tool	2019-03-18 11:23:00 +00:00
Nick Craig-Wood	14ef4437e5	dedupe: fix bug introduced when converting to use walk.ListR #2902 ... Before the fix we were only de-duping the ListR batches. Afterwards we dedupe everything. This will have the consequence that rclone uses more memory as it will build a map of all the directory names, not just the names in a given directory.	2019-03-17 11:01:20 +00:00
Nick Craig-Wood	2b05bd9a08	rc: implement operations/publiclink the equivalent of rclone link ... Fixes #3042	2019-03-17 09:41:31 +00:00
Nick Craig-Wood	4376019062	dedupe: Use walk.ListR for listing commands. ... This dramatically increases the speed (7x in my tests) of the de-dupe as google drive supports ListR directly and dedupe did not work with `--fast-list`. Fixes #2902	2019-03-16 17:41:12 +00:00
Nick Craig-Wood	d5f2df2f3d	Use walk.ListR for listing operations ... This will increase speed for backends which support ListR and will not have the memory overhead of using --fast-list. It also means that errors are queued until the end so as much of the remote will be listed as possible before returning an error. Commands affected are: - lsf - ls - lsl - lsjson - lsd - md5sum/sha1sum/hashsum - size - delete - cat - settier	2019-03-16 17:41:12 +00:00
Nick Craig-Wood	efd720b533	walk: Implement walk.ListR which will use ListR if at all possible ... It otherwise has the nearly the same interface as walk.Walk which it will fall back to if it can't use ListR. Using walk.ListR will speed up file system operations by default and use much less memory and start immediately compared to if --fast-list had been supplied.	2019-03-16 17:41:12 +00:00
Nick Craig-Wood	047f00a411	filter: Add BoundedRecursion method ... This indicates that the filter set could be satisfied by a bounded directory recursion.	2019-03-16 17:41:12 +00:00
Nick Craig-Wood	28230d93b4	sync: Implement --suffix-keep-extension for use with --suffix - fixes #3032	2019-03-15 14:21:39 +00:00
Nick Craig-Wood	58d9a3e1b5	filter: reload filter when the options are set via the rc - fixes #3018	2019-03-10 13:09:44 +00:00
Nick Craig-Wood	cccadfa7ae	rc: add ability for options blocks to register reload functions	2019-03-10 13:09:44 +00:00
ishuah	1b52f8d2a5	copy/sync/move: add --create-empty-src-dirs flag - fixes #2869	2019-03-10 11:56:38 +00:00
Fionera	55b9a4ed30	Add ServerSideAcrossConfig Flag and check for it. fixes #2728	2019-03-09 16:18:45 +00:00
Nick Craig-Wood	e3bceb9083	operations: fix Overlapping test for Windows native paths	2019-02-28 11:39:32 +00:00
Nick Craig-Wood	5240f9d1e5	sync: fix integration tests to check correct error	2019-02-27 22:05:16 +00:00
Nick Craig-Wood	84c650818e	sync: don't allow syncs on overlapping remotes - fixes #2932	2019-02-26 19:25:52 +00:00
Nick Craig-Wood	c5775cf73d	fserrors: don't panic on uncomparable errors	2019-02-26 15:39:16 +00:00
Fabian Möller	8f4ea77c07	fs: remove unnecessary pacer warning	2019-02-18 08:42:36 +01:00
Fabian Möller	61616ba864	pacer: make pacer more flexible ... Make the pacer package more flexible by extracting the pace calculation functions into a separate interface. This also allows to move features that require the fs package like logging and custom errors into the fs package. Also add a RetryAfterError sentinel error that can be used to signal a desired retry time to the Calculator.	2019-02-16 14:38:07 +00:00
Nick Craig-Wood	0b9d7fec0c	lsf: add 'e' format to show encrypted names and 'o' for original IDs ... This brings it up to par with lsjson. This commit also reworks the framework to use ListJSON internally which removes duplicated code and makes testing easier.	2019-02-14 14:45:35 +00:00
Nick Craig-Wood	240c15883f	accounting: fix total ETA when --stats-unit bits is in effect	2019-02-14 07:56:52 +00:00
Nick Craig-Wood	b167d30420	Add client side TLS/SSL flags --ca-cert/--client-cert/--client-key ... Fixes #2966	2019-02-12 12:26:19 +00:00
Nick Craig-Wood	076d3da825	operations: resume downloads if the reader fails in copy - fixes #2108 ... This puts a shim on the reader opened by Copy so that if an error is returned, the reader is re-opened at the correct seek point. This should make downloading very large files more reliable.	2019-02-12 11:47:57 +00:00
Nick Craig-Wood	d29c545627	Start v1.46-DEV development	2019-02-09 12:21:57 +00:00
Nick Craig-Wood	eb85ecc9c4	Version v1.46	2019-02-09 10:42:57 +00:00
Nick Craig-Wood	8ec55ae20b	Fix broken flag type tests ... Introduced in fc1bf5f931	2019-02-07 16:42:26 +00:00
Nick Craig-Wood	fc1bf5f931	Make flags show up with their proper names, eg SizeSuffix rather than int	2019-02-07 11:57:26 +00:00
Nick Craig-Wood	bdc638530e	walk: make NewDirTree always use ListR #2946 ... This fixes vfs/refresh with recurse=true needing the --fast-list flag	2019-02-04 10:37:27 +00:00
Nick Craig-Wood	2135879dda	lsjson: use exactly the correct number of decimal places in the seconds	2019-02-03 20:03:23 +00:00
Cnly	318d1bb6f9	fs: clarify behaviour of Put() and Upload() for unknown-sized objects	2019-02-03 11:47:57 +00:00
Cnly	b8b53901e8	operations: call Rcat in Copy when size is -1 - #2832	2019-02-03 11:47:57 +00:00
Nick Craig-Wood	6e153781a7	rc: add help to show how to set log level with options/set	2019-02-03 11:47:57 +00:00
Nick Craig-Wood	eb91356e28	fs/asyncreader: optionally user mmap for memory allocation with --use-mmap #2200 ... This replaces the `sync.Pool` allocator with lib/pool. This implements a pool of buffers of up to 64MB which can be re-used but is flushed every 5 seconds. If `--use-mmap` is set then rclone will use mmap for memory allocations which is much better at returning memory to the OS.	2019-02-02 14:35:56 +00:00
Nick Craig-Wood	d966cef14c	build: fix problems found with unconvert	2019-02-02 11:45:07 +00:00
Nick Craig-Wood	97752ca8fb	build: fix problems found with ineffasign linter	2019-02-02 11:45:07 +00:00
Nick Craig-Wood	bbd03f49a4	operations: Implement DirMove for moving a directory #2539 ... This does the equivalent of sync.Move but is specialised for moving files in one backend.	2019-01-27 21:26:56 +00:00
Nick Craig-Wood	a30e80564d	config: when using auto confirm make user interaction configurable ... * drive: don't run teamdrive config if auto confirm set * onedrive: don't run extra config if auto confirm set * make Confirm results customisable by config Fixes #1010	2019-01-18 14:46:26 +00:00
Nick Craig-Wood	5990573ccd	accounting: fix layout of stats - fixes #2910 ... This fixes several things wrong with the layout of the stats. Transfers which haven't started are printed in the same format as those which have so the stats with `--progress` don't show horrible artifacts. Checkers and transfers now get a ": checkers" and ": transfers" label on the end of the stats line. Transfers will have the transfer stats when the transfer has started instead of this. There was a bug in the routine which shortened the file names (it always produces strings 1 too long). This is now fixed with a test. The formatting string was wrong with a fixed width of 45 - this is now replaces with the value of `--stats-file-name-length`. This also meant that there were unecessary leading spaces in the file names. So the default `--stats-file-name-length` was raised to 45 from 40.	2019-01-14 16:12:39 +00:00
qip	f471a7e3f5	fshttp: Add cookie support with cmdline switch --use-cookies ... Cookies are handled by cookiejar in memory with fshttp module through the entire session. One useful scenario is, with HTTP storage system where index server adds authentication cookie while redirecting to CDN for actual files. Also, it can be helpful to reuse fshttp in other storage systems requiring cookie.	2019-01-11 17:35:29 +00:00
Dario Guzik	7782eda88e	check: Add stats showing total files matched.	2019-01-11 14:13:48 +00:00
Nick Craig-Wood	ff72059a94	operations: warn if --checksum is set but there are no hashes available ... Also caveat the help of --checksum Fixes #2903	2019-01-10 11:07:10 +00:00
Denis Skovpen	2d2533a08a	cmd/copyurl: fix checking of --dry-run	2019-01-08 11:28:05 +00:00
Nick Craig-Wood	2d01a65e36	oauthutil: read a fresh token config file before using the refresh token. ... This means that rclone will pick up tokens from concurrently running rclones. This helps for Box which only allows each refresh token to be used once. Without this fix, rclone caches the refresh token at the start of the run, then when the token expires the refresh token may have been used already by a concurrently running rclone. This also will retry the oauth up to 5 times at 1 second intervals. See: https://forum.rclone.org/t/box-token-refresh-timing/8175	2019-01-08 11:01:30 +00:00
Jay	082a7065b1	Use vfsgen for static HTML templates	2018-12-26 15:07:21 +00:00
Nick Craig-Wood	5ee1816a71	filter: parallelise reading of --files-from - fixes #2835 ... Before this change rclone would read the list of files from the files-from parameter and check they existed one at a time. This could take a very long time for lots of files. After this change, rclone will check up to --checkers in parallel.	2018-12-13 13:22:30 +00:00
Nick Craig-Wood	107293c80e	copy,move: restore --no-traverse flag ... The --no-traverse flag was not implemented when the new sync routines (using the march package) was implemented. This re-implements --no-traverse in march by trying to find a match for each object with NewObject rather than from a directory listing.	2018-12-02 20:28:13 +00:00
Nick Craig-Wood	e3c4ebd59a	march: factor calling parameters into a structure	2018-12-02 18:07:26 +00:00
Nick Craig-Wood	01fcad9b9c	rc: fix docs for sync/{sync,copy,move} and operations/{copy,move}file	2018-11-29 11:11:30 +00:00
Nick Craig-Wood	a6c28a5faa	Start v1.45-DEV development	2018-11-24 15:20:24 +00:00
Nick Craig-Wood	d35bd15762	Version v1.45	2018-11-24 13:44:25 +00:00
Peter Kaminski	63a2a935fc	fix typos in original files, per #2727 review request	2018-11-14 22:48:58 +00:00
Nick Craig-Wood	f92beb4e14	fstest: Fix TestPurge causing errors with subsequent tests on azure ... Before this change TestPurge would remove a container and subsequent tests would fail because the container was still being deleted so couldn't be created. This was fixed by introducing an fstest.NewRunIndividual() test runner for TestPurge which causes the test to be run on a new container.	2018-11-14 17:14:02 +00:00
Nick Craig-Wood	16f797a7d7	filter: add --ignore-case flag - fixes #502 ... The --ignore-case flag causes the filtering of file names to be case insensitive. The flag name comes from GNU tar.	2018-11-12 14:29:37 +00:00
Nick Craig-Wood	4ad08794c9	fserrors: add "server closed idle connection" to retriable errors ... This seems to be related to this go issue: https://github.com/golang/go/issues/19943 See: https://forum.rclone.org/t/copy-from-dropbox-to-google-drive-yields-failed-to-copy-failed-to-open-source-object-server-closed-idle-connection-error/7460	2018-11-08 11:12:25 +00:00
Nick Craig-Wood	c6786eeb2d	move: don't create directories with --dry-run - fixes #2676	2018-11-06 13:34:15 +00:00
Nick Craig-Wood	57b85b8155	rc: fix job tests on Windows	2018-11-06 13:03:48 +00:00
Nick Craig-Wood	e6dd121f52	config: add rc operations for config	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	e600217666	config: create config directory on save if it is missing	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	bc17ca7ed9	rc: implement core/obscure	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	1916410316	rc: add core/version and put definitions next to implementations	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	dddfbec92a	cmd/version: factor version number parsing routines into fs/version	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	75a88de55c	rc/rcserver: with --rc-files if auth set, pass on to URL opened ... If `--rc-user` or `--rc-pass` is set then the URL that is opened with `--rc-files` will have the authorization in the URL in the `http://user:pass@localhost/` style.	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	2466f4d152	sync: add rc commands for sync/copy/move	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	39283c8a35	operations: implement operations remote control commands	2018-11-05 15:44:40 +00:00
Nick Craig-Wood	46c2f55545	copyurl: factor code into operations and write test	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	fc2afcbcbd	lsjson: factor internals of lsjson command into operations	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	fa0a9653d2	rc: methods marked as AuthRequired need auth unless --rc-no-auth ... Methods which can read or mutate external storage will require authorisation - enforce this. This can be overidden by `--rc-no-auth`.	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	75e8ea383c	rc: implement rc.PutCachedFs for prefilling the remote cache	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	8c8b58a7de	rc: expire remote cache and fix tests under race detector	2018-11-04 20:42:57 +00:00
Nick Craig-Wood	b961e07c57	rc: ensure rclone fails to start up if the --rc port is in use already	2018-11-04 15:11:51 +00:00
Nick Craig-Wood	89550e7121	rcserver: serve directories as well as files	2018-11-04 15:11:51 +00:00
Nick Craig-Wood	b972dcb0ae	rc: implement options/blocks,get,set and register options	2018-11-03 11:32:00 +00:00
Nick Craig-Wood	0bfa9811f7	rc: factor server code into rcserver and implement serving objects ... If a GET or HEAD request is receivied with a URL parameter of fs then it will be served from that remote.	2018-11-03 11:32:00 +00:00
Nick Craig-Wood	75252e4a89	rc: add --rc-files flag to serve files on the rc http server ... This enables building a browser based UI for rclone	2018-11-03 11:32:00 +00:00
Nick Craig-Wood	2089405e1b	fs/rc: add more infrastructure to help writing rc functions ... - Fs cache for rc commands - Helper functions for parsing the input - Reshape command for manipulating JSON blobs - Background Job starting, control, query and expiry	2018-11-02 17:32:20 +00:00
ssaqua	3d81b75f44	dedupe: check for existing filename before renaming a dupe file	2018-11-02 16:51:52 +00:00
Nick Craig-Wood	948a5d25c2	operations: Fix Purge and Rmdirs when dir is not empty ... Before this change, Purge on the fallback path would try to delete directories starting from the root rather than the dir passed in. Rmdirs would also attempt to delete the root.	2018-10-27 11:51:17 +01:00
Nick Craig-Wood	543e37f662	Require go1.8 for compilation	2018-10-25 17:06:33 +01:00
Nick Craig-Wood	c5ac96e9e7	Make --files-from only read the objects specified and don't scan directories ... Before this change using --files-from would scan all the directories that the files could possibly be in causing rclone to do more work that was necessary. After this change, rclone constructs an in memory tree using the --fast-list mechanism but from all of the files in the --files-from list and without scanning any directories. Any objects that are not found in the --files-from list are ignored silently. This mechanism is used for sync/copy/move (march) and all of the listing commands ls/lsf/md5sum/etc (walk).	2018-10-20 18:13:31 +01:00
Nick Craig-Wood	67703a73de	Start v1.44-DEV development	2018-10-15 12:33:27 +01:00
Nick Craig-Wood	f96ce5674b	Version v1.44	2018-10-15 11:03:08 +01:00
Nick Craig-Wood	ade252f13b	build: fixup code formatting after goimports change	2018-10-13 22:47:12 +01:00
Fabian Möller	57273d364b	fstests: add TestFsPutChunked	2018-10-11 14:47:58 +01:00
Fabian Möller	1a40bceb1d	backend: unify NewFs path handling for wrapping remotes ... Use the same function to join the root paths for the wrapping remotes alias, cache and crypt. The new function fspath.JoinRootPath is equivalent to path.Join, but if the first non empty element starts with "//", this is preserved to allow Windows network path to be used in these remotes.	2018-10-10 17:50:27 +01:00
Nick Craig-Wood	6709084e2f	config: Show URL of backend help page when starting config	2018-10-06 11:47:46 +01:00
Nick Craig-Wood	6b1f915ebc	fs: Implement RegInfo.FileName to return the on disk filename for a backend ... Use it in make_backend_docs.py	2018-10-06 11:47:46 +01:00
Nick Craig-Wood	1557287c64	fs: Make Option.GetValue() public #2541	2018-10-06 11:47:45 +01:00
Nick Craig-Wood	e7e467fb3a	cmd: factor FlagName into fs.Option #2541	2018-10-06 11:47:45 +01:00
Nick Craig-Wood	59e14c25df	vfs: enable rename for nearly all remotes using server side Move or Copy ... Before this change remotes without server side Move (eg swift, s3, gcs) would not be able to rename files. After it means nearly all remotes will be able to rename files on rclone mount with the notable exceptions of b2 and yandex. This changes checks to see if the remote can do Move or Copy then calls `operations.Move` to do the actual move. This will do a server side Move or Copy but won't download and re-upload the file. It also checks to see if the destination exists first which avoids conflicts or duplicates. Fixes #1965 Fixes #2569	2018-09-29 14:56:20 +01:00
Fabian Möller	1eec59e091	fs: update ChangeNotifier interface ... This introduces a channel to the ChangeNotify function, which can be used to update the poll-interval and cleanly exit the polling function.	2018-09-25 14:01:13 +02:00
ssaqua	ae63e4b4f0	list: change debug logs for excluded items	2018-09-24 17:08:35 +01:00
frenos	382a6863b5	rc: add support for OPTIONS and basic CORS - #2575	2018-09-24 17:04:47 +01:00
sandeepkru	e0c5f7ff1b	fs - Remove unreferenced ListTierer optional interface	2018-09-18 13:56:09 +01:00
Joanna Marek	e022ffce0f	accounting: change too long names cutting mechanism - fixes #2490	2018-09-18 10:27:23 +01:00
sandeepkru	7accd30da8	cmd and fs: Added new command settier which performs storage tier changes on ... supported remotes	2018-09-12 21:09:08 +01:00
sandeepkru	5716a58413	fs: Added new optional interfaces SetTierer, GetTierer and ListTierer, these are to ... perform object tier changes on supported remotes	2018-09-12 21:09:08 +01:00
Cédric Connes	1b95ca2852	stats: handle FatalError and NoRetryError when reported to stats	2018-09-07 14:44:50 +01:00
albertony	09d7bd2d40	config: don't create default config dir when user supplies --config ... Avoid creating empty default configuration directory when user supplies path to config file. Fixes #2514.	2018-09-03 15:30:53 +01:00
dcpu	f4aaec9ce5	log: Add --log-format flag - fixes #2424	2018-09-02 18:11:09 +01:00
Nick Craig-Wood	543d75723b	Start v1.43-DEV development	2018-09-01 18:37:48 +01:00
Nick Craig-Wood	20c55a6829	Version v1.43	2018-09-01 12:58:00 +01:00
Nick Craig-Wood	7d74686698	fs/accounting: increase maximum burst size of token bucket ... This stops occasional errors when using --bwlimit which look like this Token bucket error: rate: Wait(n=2255475) exceeds limiter's burst 2097152	2018-08-30 17:24:08 +01:00
Denis	1c578ced1c	cmd: add copyurl command - Fixes #1320	2018-08-30 16:45:41 +01:00
Nick Craig-Wood	de6ec8056f	fs/accounting: fix moving average speed for file stats ... Before this change the moving average for the individual file stats would start at 0 and only converge to the correct value over 15-30 seconds. This change starts the weighting period as 1 and moves it up once per sample which gets the average to a better value instantly.	2018-08-28 22:55:51 +01:00
Nick Craig-Wood	66fe4a2523	fs/accounting: fix stats display which was missing transferring data ... Before this change the total stats were ignoring the in flight checking, transferring and bytes.	2018-08-28 22:21:17 +01:00
Nick Craig-Wood	f5617dadf3	fs/accounting: factor out eta and percent calculations and write tests	2018-08-28 22:21:17 +01:00
Nick Craig-Wood	ca44fb1fba	accounting: fix time to completion estimates ... Previous to this change package used for this github.com/VividCortex/ewma took a 0 average to mean reset the statistics. This happens quite often when transferring files though a buffer. Replace that implementation with a simple home grown one (with about the same constant), without that feature.	2018-08-26 22:00:48 +01:00
Nick Craig-Wood	94950258a4	fs: allow backends to be named using their Name or Prefix #2449 ... This means that, for example Google Cloud Storage can be known as `:gcs:bucket` on the command line, as well as `:google cloud storage:bucket`.	2018-08-26 17:59:31 +01:00
Nick Craig-Wood	8656bd2bb0	fs: Allow on the fly remotes with :backend: syntax - fixes #2449 ... This change allows remotes to be created on the fly without a config file by using the remote type prefixed with a : as the remote name, Eg :s3: to make an s3 remote. This assumes the user is supplying the backend config via command line flags or environment variables.	2018-08-26 17:59:31 +01:00
Nick Craig-Wood	b6db90cc32	cmd: add --progress/-P flag to show progress ... Fixes #2347 Fixes #1210	2018-08-26 17:20:38 +01:00
Nick Craig-Wood	9f671c5dd0	fs: fix tests for *SepList	2018-08-21 10:58:59 +01:00
Alex Chen	c6c74cb869	mountlib: fix mount --daemon not working with encrypted config - fixes #2473 ... This passes the configKey to the child process as an Obscured temporary file with an environment variable to the	2018-08-21 09:41:16 +01:00
Nick Craig-Wood	1b8f4b616c	fs: move CommaSepList and SpaceSepList here from config ... fs can't import config so having them there means they are not usable by rclone core.	2018-08-20 17:52:05 +01:00
Fabian Möller	f818df52b8	config: add List type	2018-08-20 17:38:51 +01:00
Nick Craig-Wood	7712a0e111	fs/asyncreader: skip some tests to work around race detector bug ... The race detector currently detects a race with len(chan) against close(chan). See: https://github.com/golang/go/issues/27070 Skip the tests which trip this bug under the race detector.	2018-08-20 12:34:29 +01:00
Fabian Möller	6dc5aa7454	docs: clearify buffer-size is per transfer/filehandle	2018-08-17 18:11:40 +01:00
Fabian Möller	552eb8e06b	vfs: try to seek buffer on read only files	2018-08-17 18:10:28 +01:00
Nick Craig-Wood	6390dec7db	fs/accounting: add --stats-one-line flag for single line stats	2018-08-17 17:58:00 +01:00
Nick Craig-Wood	80a3db34a8	fs/accounting: show the total progress of the sync in the stats #379	2018-08-17 17:58:00 +01:00
Nick Craig-Wood	cb7a461287	sync: add a buffer for checks, uploads and renames #379 ... --max-backlog controls the queue length. Add statistics for the check/upload/rename queues. This means that checking can complete before the uploads which will give rclone the ability to show exactly what is outstanding.	2018-08-17 17:58:00 +01:00
Andres Alvarez	990919f268	Add disclaimer about generated passwords being stored in an obscured format	2018-08-11 15:07:50 +01:00
Sebastian Bünger	dd3e912731	fs/OpenOptions: Make FixRangeOption clamp range to filesize.	2018-08-10 11:14:34 +01:00
Nick Craig-Wood	e979cd62c1	rc: fix formatting in docs	2018-08-07 21:05:21 +01:00
reddi1	6bfdabab6d	rc: added core/stats to return the stats - fixes #2405	2018-08-07 20:56:40 +01:00
Andres Alvarez	c1f5add049	Add tests for reveal functions	2018-08-07 20:51:50 +01:00
Oleg Kovalov	06c9f76cd2	all: fix go-critic linter suggestions	2018-08-06 21:14:03 +01:00
Fabian Möller	9815b09d90	fs: add multipliers for SizeSuffix	2018-07-29 18:17:05 +01:00
Fabian Möller	9c90b5e77c	stats: use appropriate Lock func's	2018-07-22 11:33:19 +02:00
Nick Craig-Wood	ab78eb13e4	sync: correct help for --delete-during and --delete-after	2018-07-18 19:30:14 +01:00
ishuah	dcc74fa404	move: fix delete-empty-src-dirs flag to delete all empty dirs on move - fixes #2372	2018-07-17 10:34:34 +01:00
Nick Craig-Wood	4d7d240c12	config: Add advanced section to the config editor	2018-07-16 21:20:47 +01:00
Nick Craig-Wood	d046402d80	config: Make sure Required values are entered	2018-07-16 21:20:47 +01:00
Nick Craig-Wood	9bdf465c10	config: make config wizard understand types and defaults	2018-07-16 21:20:47 +01:00
Nick Craig-Wood	f3f48d7d49	Implement new backend config system ... This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet	2018-07-16 21:20:47 +01:00
Nick Craig-Wood	3c89406886	config: Make fs.ConfigFileGet return an exists flag	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	85d09729f2	fs: factor OptionToEnv and ConfigToEnv into fs	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	b3bd2d1c9e	config: add configstruct parser to parse maps into config structures	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	4c586a9264	config: add configmap package to manage config in a generic way	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	1c80e84f8a	fs: Implement Scan method for SizeSuffix and Duration	2018-07-16 08:50:52 +01:00
Nick Craig-Wood	dbb4b2c900	fs/config: don't print errors about --config if supplied - fixes #2397 ... Before this change if the rclone was running in an environment which couldn't find the HOME directory, it would print a warning about supplying a --config flag even if the user had done so.	2018-07-15 12:39:11 +01:00
Nick Craig-Wood	a3d9a38f51	fs/fserrors: make sure Cause never returns nil	2018-07-13 10:31:40 +01:00
Nick Craig-Wood	144c1a04d4	fs: Fix parsing of paths under Windows - fixes #2353 ... Before this copyto would parse windows paths incorrectly. This change moves the parsing code into fspath and makes sure fspath.Split calls fspath.Parse which does the parsing correctly for This also renames fspath.RemoteParse to fspath.Parse for consistency	2018-07-06 23:16:43 +01:00
Nick Craig-Wood	8c0335a176	build: fix for goimports format change ... See https://github.com/golang/go/issues/23709	2018-07-03 22:33:15 +01:00
Nick Craig-Wood	947e10eb2b	config: fix error reading password from piped input - fixes #1308	2018-06-28 11:54:15 +01:00
Nick Craig-Wood	935533e57f	filter: raise --include and --exclude warning to ERROR so it appears without -v	2018-06-22 22:18:55 +01:00
Mateusz	56e1e82005	fs: added weekday schedule into --bwlimit - fixes #1822	2018-06-17 18:38:09 +01:00
Nick Craig-Wood	7110349547	Start v1.42-DEV development	2018-06-16 21:25:58 +01:00
Nick Craig-Wood	a9adb43896	Version v1.42	2018-06-16 18:21:09 +01:00
Nick Craig-Wood	1c75581959	sync: fix TestCopyRedownload after ModifyWindow changes #2310	2018-06-10 17:34:00 +01:00
Filip Bartodziej	ce109ed9c0	log: password prompt output fixed for unix - partially fixes #2220	2018-06-10 12:57:45 +01:00
Nick Craig-Wood	ba43acb6aa	sync: fix TestCopyEmptyDirectories after ModifyWindow changes #2310	2018-06-04 21:41:25 +01:00
ishuah	d758e1908e	copy: create (pseudo copy) empty source directories to destination - fixes #1837	2018-06-04 11:01:14 +01:00
ishuah	737aed8412	Ensure items in srcEmptyDirs are actually empty	2018-06-04 11:01:14 +01:00
Stefan	4009fb67c8	fs: calculate ModifyWindow each time on the fly instead of relying on global state - see #2319, #2328	2018-06-03 20:45:34 +02:00
Nick Craig-Wood	3ef938ebde	lsf: add --absolute flag to add a leading / onto path names	2018-06-03 10:42:34 +01:00
Kasper Byrdal Nielsen	aad75e6720	check: Add one-way argument ... --one-way argument will check that all files on source matches the files on detination, but not the other way. For example files present on destination but not on source will not trigger an error. Fixes: #1526	2018-05-31 09:42:16 +01:00
Nick Craig-Wood	3a2248aa5f	rc: add core/gc to run a garbage collection on demand	2018-05-24 15:10:28 +01:00
Nick Craig-Wood	573ef4c8ee	rc: enable go profiling by default on the --rc port ... This means you can use the pprof tool on a running rclone, eg go tool pprof http://localhost:5572/debug/pprof/heap	2018-05-24 15:10:28 +01:00
Nick Craig-Wood	512f4b4487	Update error checking on fmt.Fprint* after errcheck update ... Now we need to check or ignore errors on fmt.Fprint* explicitly - previously errcheck just ignored them for us.	2018-05-22 09:41:13 +01:00
Nick Craig-Wood	8e625e0bc3	config: add ConfirmWithDefault to change the default on AutoConfig	2018-05-16 09:09:41 +01:00
Nick Craig-Wood	e56be0dfd8	lsf: Add --csv flag for compliant CSV output	2018-05-13 12:18:21 +01:00
Nick Craig-Wood	2a32e2d838	operations: turn ListFormatted into a Format method on ListFormat	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	f77efc7649	lsf: Add 'm' format specifier to show the MimeType	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	aadbcce486	fs: Add MimeTypeDirEntry to return the MimeType of a DirEntry	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	909c3a92d6	lsf: implement 'i' format for showing object ID - fixes #1476	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	826975c341	fs: add Optional ID() method to Object and implement it in backends ... ID() shows the internal ID of the Object if available.	2018-05-13 12:17:55 +01:00
Nick Craig-Wood	e80ae4e09c	build: remove unused struct fields spotted by structcheck	2018-05-05 17:32:41 +01:00
Nick Craig-Wood	1320e84bc2	build: remove unused code spotted by the deadcode linter	2018-05-05 17:32:41 +01:00
Nick Craig-Wood	cb5bd47e61	build: fix errors spotted by ineffassign linter ... These were mostly caused by shadowing err and a good fraction of them will have caused errors not to be propagated properly.	2018-05-05 17:32:41 +01:00
Nick Craig-Wood	f95835d613	fserrors: Look deeper into errors for Fatal/Retry/NoRetry errors. ... Before this change fatal errors which were wrapped in a system error (eg a URLError) were not recognised as fatal errors.	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	be79b47a7a	sync: log when we abandon the sync due to a fatal error	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	be22735609	fs/accounting: fix deadlock on GetBytes ... A deadlock could occur since we have now put a mutex on GetBytes from StatsInfo.String (s.mu) - progress (acc.statmu) and read (acc.statmu) - GetBytes (s.mu). Fix this by giving stringSet its own locking and excluding the call which caused the deadlock from the mutex in StatsInfo.String.	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	1b1b3c13cd	sync: add a test for aborting on max upload	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	5c128272fd	Implement --max-transfer flag to quit transferring at a limit #1655	2018-05-05 12:58:28 +01:00
Nick Craig-Wood	d178233e74	sync,march: check the cancel context on every channel send and receive ... This fixes a deadlock on sync when all the copying channels receive a Fatal Error.	2018-05-05 12:58:28 +01:00
Fabian Möller	9fdf273614	fs: improve ChunkedReader ... - make Close permanent and return errors afterwards - use RangeSeek from the wrapped reader if present - add a limit to chunk growth - correct RangeSeek interface behavior - add tests	2018-05-05 12:49:42 +01:00
Nick Craig-Wood	55674c0bfc	Start v1.41-DEV development	2018-04-28 12:37:55 +01:00
Nick Craig-Wood	e4c380b2a8	Version v1.41	2018-04-28 11:46:27 +01:00
Nick Craig-Wood	74cbdea0ef	Revert "copy: create (pseudo copy) empty source directories to destination" ... Unfortunately this commit attempts to create every directory rather than just the empty ones, so will need re-working. Removing this feature for the 1.41 release This reverts commit 0daced29db.	2018-04-28 10:02:32 +01:00
ishuah	0daced29db	copy: create (pseudo copy) empty source directories to destination - fixes #1837	2018-04-27 16:15:32 +01:00
Richard Yang	a81ec00a8c	dedupe: Add dedupe largest functionality - fixes #2269	2018-04-26 16:21:07 +01:00
Nick Craig-Wood	9f04ce282e	rc: fix setting bwlimit to unlimited	2018-04-26 12:21:29 +01:00
Nick Craig-Wood	764440068e	filter: fix --min-age and --max-age together check ... Somehow in the code reorganisation of 11da2a6c9b the check for --min-age and --max-age got switched around. This commit fixes that and means you can use --min-age and --max-age together.	2018-04-26 09:17:22 +01:00
Nick Craig-Wood	a703216286	filter: take double negatives out of filter flag help	2018-04-26 09:17:13 +01:00
Nick Craig-Wood	9ab2521ef2	rc: autogenerate and tidy the docs and commands ... * Rename rc/pid -> core/pid * Sort the output of `rc list` * Make a script to autogenerate the docs * Tidy docs	2018-04-23 20:57:17 +01:00
Nick Craig-Wood	21a10e58c9	rc: implement core/memstats to print internal memory usage info	2018-04-23 20:49:36 +01:00
Nick Craig-Wood	24980d7123	config: fix typo in error message #2268	2018-04-21 22:49:30 +01:00
Nick Craig-Wood	bcdb7719c6	fs/hash: install QuickXorHash as a supported rclone hash type #2262	2018-04-20 21:02:57 +01:00
Nick Craig-Wood	c51d97c752	hashsum: make generic tool for any hash to produce md5sum like output	2018-04-20 21:02:37 +01:00
Nick Craig-Wood	ef3bcec76c	fs: Extend SizeSuffix to include TB and PB for rclone about	2018-04-17 21:53:42 +01:00
Nick Craig-Wood	1ac6dacf0f	about: complete other providers and re-work internals ... * Implement about for: * local, crypt, cache, drive, swift, hubic, onedrive, pcloud, dropbox * Implement `--json` and `---full` flag for `rclone about` * change About interface to return a Usage structure * Remove operations.About as it is too thin an interface * Implement Integration test Relates to #1138 and #1564	2018-04-17 21:53:27 +01:00
a-roussos	94e277d759	about: add new command 'about' to get quota info from a remote ... Implemented for drive only. Relates to #1138 and #1564.	2018-04-17 21:50:14 +01:00
Nick Craig-Wood	8c3740c2c5	config: Improve the Provider matching to have a negated match #2140 ... This makes it easier to make classes of provider in the config.	2018-04-13 16:06:37 +01:00
Giri Badanahatti	acd5d4377e	config,s3: hierarchical configuration support #2140 ... This introduces a method of making provider specific configuration within a remote. This is useful particularly in s3. This commit does the basic configuration in S3 for IBM COS.	2018-04-13 16:05:35 +01:00
Peter Baumgartner	1db68571fd	s3,swift: Add --use-server-modtime ... `--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.	2018-04-13 13:32:17 +01:00
Nick Craig-Wood	d97fe3b824	fs/operations: make dedupe work with mega ... * factor into its own files * remove assumptions about having a given hash type * make tests work if the remote has no hash	2018-04-13 13:23:55 +01:00
Nick Craig-Wood	dcf8334673	fs: add --dump goroutines and --dump openfiles ... These are developer flags useful for tracking down resource leaks.	2018-04-11 20:55:58 +01:00
Nick Craig-Wood	37be78705d	fs/fshttp: limit MaxIdleConns and MaxIdleConnsPerHost ... Before this change mega (which uses a different host per download) would open too many sockets.	2018-04-11 20:51:28 +01:00
Michael P. Dubner	92b266d361	rc: new call rc/pid - closes #2211	2018-04-09 13:33:04 +01:00
Nick Craig-Wood	cbec59146a	lsf: make sure we use localtime in tests - fixes Box integration tests ... This problem was introduced with eca99b33c0. It seems Box is the only remote which converts time zones, so if you give it a GMT time zone, it returns a PST time zone which represents the same instant.	2018-04-09 11:46:49 +01:00
Nick Craig-Wood	be54fd8f70	Remove builds conditional on go1.7 since that is now guaranteed #2154 ... Old fallback code was deleted and the go1.7 style code inlined where appropriate.	2018-04-07 11:42:55 +01:00
Nick Craig-Wood	e5be471ce0	Use io.SeekStart/End/Current constants now for go1.7+ #2154	2018-04-07 11:42:36 +01:00
Nick Craig-Wood	80588a5a6b	Replace "golang.org/x/net/context" with "context" for go1.7+ #2154	2018-04-07 11:42:08 +01:00
Nick Craig-Wood	67023f0040	Require go1.7 for compilation #2154 ... * Update the travis tests to exclude go1.6 * Update the compile check to require go1.7+ * Update misc go1.6 workarounds marked in the source	2018-04-06 20:18:14 +01:00
Matt Holt	e13f65b953	serve restic: Print actual listener address	2018-04-04 14:56:26 +01:00
Nick Craig-Wood	eca99b33c0	lsd,lsf: make sure all times we output are in local time - fixes #2183 ... Previous to this change times from lsd/lsf were output in whatever timezone they were in whereas times from lsl were converted to localtime.	2018-04-01 15:40:04 +01:00
Stefan	a8267d1628	link: allow creating public link to files and folders - closes #1562	2018-03-29 09:10:19 +02:00
Mateusz Pabian	91b068ad3a	sync: implement --ignore-errors - fixes #642	2018-03-23 22:01:10 +00:00
Nick Craig-Wood	940df88eb2	Start v1.40-DEV development	2018-03-19 14:20:48 +00:00
Nick Craig-Wood	26f1c55987	Version v1.40	2018-03-19 10:06:13 +00:00
Stefan	86892467d9	config: load config file only on first access (closes #1659, closes #2096) (#2147)	2018-03-17 12:36:30 +01:00
Nick Craig-Wood	4295428a0f	fs/accounting: add remote control of bwlimit	2018-03-17 10:34:02 +00:00
Nick Craig-Wood	5bf639048f	sync: log an error that --track-renames doesn't work with sync or move ... Fixes #2008	2018-03-17 10:34:02 +00:00
remusb	97b48cf988	rc: add support for Go 1.6	2018-03-14 22:58:20 +02:00
Nick Craig-Wood	86e5a35491	Implement Remote Control for rclone #2111 ... This implements a remote control protocol activated with the --rc flag and a new command `rclone rc` to use that interface. Still to do * docs - need finishing * tests	2018-03-14 22:58:20 +02:00
Nick Craig-Wood	dfd0f4c5a4	sync: when using --backup-dir don't delete files if we can't set their modtime ... This is a problem when syncing a file which just needed its modtime set with dropbox which can't set the mod time of a file without re-uploading it. Before this change we would delete the file, then the server side move would fail moving the file to the backup-dir because it no longer existed. After this change the destination file is moved to the backup-dir instead of being deleted and the new file is uploaded. Fixes #2134	2018-03-13 16:05:06 +00:00
Nick Craig-Wood	0c9dc006c5	fs: make display of default values of --min-age/--max-age be off - Fixes #2121	2018-03-13 09:06:07 +00:00
Remus Bunduc	70f07fd3ac	fs: add ChangeNotify and backend support for it (#2094) ... * fs: rename DirChangeNotify to ChangeNotify * cache: switch to ChangeNotify * ChangeNotify: keep order of notifications	2018-03-08 22:03:34 +02:00
Nick Craig-Wood	e0eb666dbf	fs/walk: fix new golint warning about unused variables in range	2018-03-02 17:01:58 +00:00
Nick Craig-Wood	abc736df1d	cat: Use RangeOption for limited fetches to make more efficient #1825	2018-02-19 18:00:39 +00:00
Nick Craig-Wood	ab0d06eb16	fs: Make RangeOption mandatory #1825	2018-02-19 18:00:39 +00:00
Mateusz	afc963ed92	config: retry saving the config after failure - fixes #2060	2018-02-19 17:59:27 +00:00
Fabian Möller	451cd6d971	fs: add ChunkedReader	2018-02-19 15:45:24 +00:00
Fabian Möller	a647c54888	fs: add RangeSeeker interface	2018-02-19 15:45:24 +00:00
Nick Craig-Wood	a5c3bcc9c7	fshttp: fix idle timeouts for HTTP connections #2057 ... Now we only nudge on the idle timeout after a successful Read or Write which returns some bytes and no errors.	2018-02-16 10:35:41 +00:00
Bjørn Erik Pedersen	ab8c0a81fa	Add a delete threshold to sync (--max-delete) ... Fixes #959	2018-02-12 11:29:58 +00:00
Nick Craig-Wood	d0d6b83a7a	fs/accounting: rework to enable accounting to work with crypt and b2 ... This removes the old system of part accounting and replaces it with a system of popping off the accounting reader and wrapping up new ones as necessary. This makes it much easier to carry the context down the chain of wrapped readers and get the limiting as near as possible to the output. This makes the accounting more accurate and the bandwidth limiting smoother. Fixes #2029 and Fixes #1443	2018-02-02 15:14:41 +00:00
Nick Craig-Wood	bea02fcf52	fs/accounting: factor into separate files without changing functionality	2018-02-02 15:14:40 +00:00
Nick Craig-Wood	6fb868e00c	config: fix --log-level flag after code reorganization - fixes #2043	2018-02-02 14:07:44 +00:00
Nick Craig-Wood	8df78f2b6d	operations: ignore size of objects when they are < 0 #320 ... This allows google docs to be transferred and checked correctly.	2018-01-31 16:22:05 +00:00
Nick Craig-Wood	2eb5cfb7ad	fs: Formalize the ObjectUnWrapper interface	2018-01-31 16:21:41 +00:00
Nick Craig-Wood	fe52502f19	fs: Adjust RangeOption.Decode to return -1 for read to end ... A Range request can never request 0 bytes however this change was made to make a clearer signal that the limit means read to the end. Add test and more documentation and fixup uses	2018-01-27 14:31:29 +00:00
Nick Craig-Wood	9a73688e3a	fs: Add ParseRangeOption to parse incoming Range: requests	2018-01-27 13:16:37 +00:00
Nick Craig-Wood	bc3ee977f4	fs/hash: move interface assertion to tests so it doesn't pull in spf13/flag	2018-01-26 14:35:18 +00:00
Nick Craig-Wood	a6833b68ca	local: factor RangeOption code to Decode() method and readers.LimitedReadCloser #1825	2018-01-22 19:44:00 +00:00
Nick Craig-Wood	d876392d15	onedrive: Factor code into fs.FixRangeOption	2018-01-22 17:05:00 +00:00
Nick Craig-Wood	84bc4dc142	Clarify RangeOption semantics	2018-01-21 09:51:28 +00:00
Nick Craig-Wood	ee66419a27	fs/fserrors: Add test for error from #1964	2018-01-19 17:07:40 +00:00
Nick Craig-Wood	5ad226ab54	fs: Add dir option to fs.Purge #1891 ... Purge optional interface needs fixing too.	2018-01-19 11:45:50 +00:00
Nick Craig-Wood	97c414f025	config/hash: rename more symbols after factoring into own package	2018-01-18 20:27:52 +00:00
Nick Craig-Wood	71722b5b95	config: factor Obscure and Reveal into its own package	2018-01-18 20:19:55 +00:00
Ernest Borowski	5cce74d630	flags: remove --no-traverse flag because it is obsolete - fixes #1813 ... Signed-off-by: Ernest Borowski <er.borowski@gmail.com>	2018-01-18 11:00:25 +00:00
Iakov Davydov	acd55a8f65	local, fs: --exclude-if-present ignores directories which it doesn't have permission for - fixes #1959	2018-01-16 20:00:16 +00:00
Nick Craig-Wood	11da2a6c9b	Break the fs package up into smaller parts. ... The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.	2018-01-15 17:51:14 +00:00
Nick Craig-Wood	b8b620f5c2	Move all backends into backend directory	2018-01-12 17:07:38 +00:00
Nick Craig-Wood	712e6a8085	lsf: fix integration tests	2018-01-11 13:52:15 +00:00
Will Gunn	5c7e8d5a2b	fs: Add --stats-file-name-length to specify the printed file name length for stats ... Fixes #1206	2018-01-10 20:32:36 +00:00
Nick Craig-Wood	c74c3b37da	lsf: add option to print hashes	2018-01-06 17:53:37 +00:00
Nick Craig-Wood	7c71ee1a5b	fs: fix TestListFormat on remotes which return 0 as dir size not -1	2018-01-06 17:47:42 +00:00
Jakub Tasiemski	0d041602cf	cmd: new command lsf	2018-01-06 14:39:31 +00:00
Nick Craig-Wood	b412c745a1	Start v1.39-DEV development	2017-12-23 13:40:28 +00:00
Nick Craig-Wood	f34a9116d4	Version v1.39	2017-12-23 13:07:45 +00:00
Nick Craig-Wood	7e585cda96	fs: fix TestRmdirsLeaveRoot test	2017-12-14 08:57:28 +00:00
lewapm	9c242edc10	rmdirs: add --leave-root flag - fixes #1874	2017-12-13 10:23:54 +00:00
ishuah	b05e472d2e	stats: condensed transfer output to fit 80x25 terminals	2017-12-09 10:48:36 +03:00
remusb	25b073c767	fs: add Wrap feature for FS to identify their parent FS (#1884)	2017-12-06 17:14:34 +02:00
Nick Craig-Wood	fa4a25a73b	fs: only test one level of cache ... Can't test multiple caches at once as can only have 1 DB open at once	2017-12-04 15:50:59 +00:00
Ernest Borowski	934df67aef	filter: warn the user if he use --include and --exclude together fixes #1764 ... Signed-off-by: Ernest Borowski <er.borowski@gmail.com>	2017-12-04 14:20:01 +00:00
Nick Craig-Wood	4b185355df	fs: rcat - use in memory object and Copy for more reliable transfers	2017-12-03 12:14:15 +00:00
Nick Craig-Wood	7d15c33e42	fs: make Copy and Move return the destination object if possible	2017-12-03 12:14:15 +00:00
Nick Craig-Wood	11332a19a0	fs: make an in memory object for short transfers	2017-12-03 12:14:15 +00:00
Nick Craig-Wood	639e812789	fs: integration tests: add SUMMARY heading for log scraping	2017-11-29 15:55:37 +00:00
Nick Craig-Wood	1c6cad2252	fs: integration tests: add 30 minute timeout per test	2017-11-29 13:51:17 +00:00
ishuah	aab8051f50	move: add --delete-empty-src-dirs flag - fixes #1854	2017-11-28 11:38:19 +03:00
Nick Craig-Wood	fdb01437d8	fs: Allow the http Transport to have an optional filter request function	2017-11-24 09:07:56 +00:00
Nick Craig-Wood	02ffd43572	fs: Save the config before asking for a token - fixes #1220 ... Before this if the client_id/client_secret was edited it would disappear when asking for the new token. This means the post config is done after the user has confirmed the config is OK which can't be helped.	2017-11-23 14:01:32 +00:00
Nick Craig-Wood	e53892f53b	fs,drive,dropbox: Make and use new RepeatableReader variants to lower memory use ... RepeatableReaderSized has a pre-allocated buffer which should help with memory usage - before it grew the buffer. Since we know the size of the chunks, pre-allocating it should be much more efficient. RepeatableReaderBuffer uses the buffer passed in. RepeatableLimit* are convenience funcitions for wrapping a reader in an io.LimitReader and then a RepeatableReader with the same buffer size.	2017-11-23 13:53:46 +00:00
ishuah	6c62fced60	move: fixed root source directories getting deleted after move - fixes #1849	2017-11-23 12:01:35 +03:00
Nick Craig-Wood	159fce0106	fs: fix --cache-dir to have some effect	2017-11-22 17:05:02 +00:00
Nick Craig-Wood	cbe5d7ce64	fs: Remove X-Auth-Token: from headers when dumping for swift	2017-11-21 17:32:07 +00:00
Nick Craig-Wood	1a65a4e769	fs: Add --dump flag, introduce --dump requests, responses and remove --dump-auth, --dump-filters ... Now --dump-flag is written as --dump flag. This is a comma separated list which can contain * headers - HTTP headers as before * bodies - HTTP bodies as before * requests - HTTP request bodies * responses - HTTP response bodies * auth - HTTP auth * filters - Filter rexeps Leave --dump-headers and --dump-bodies for the time being but remove the other --dump-* flags as they aren't used very often.	2017-11-21 17:32:07 +00:00
ishuah	b4aa920a3d	stats: show the amount of data transferred in kb/mb - fixes #1167	2017-11-21 12:40:02 +03:00
Nick Craig-Wood	8951875c21	vfs,mount,cmount,mountlib: allow flags to be overriden by environment variables	2017-11-20 12:46:21 +00:00
Nick Craig-Wood	e946a8eab0	fs: Add CacheDir config variable	2017-11-20 12:00:32 +00:00
Nick Craig-Wood	46947b3b9b	rcat: fix goroutine leak ... This was leaking goroutines in the short file case beause it wasn't calling Close() on the Account object. This became apparent when testing with mount.	2017-11-20 12:00:32 +00:00
Iakov Davydov	7d3abdc463	tests for --exclude-if-present	2017-11-17 21:51:11 +00:00
Iakov Davydov	538246f6c3	support exclude file in --fast-list mode	2017-11-17 21:51:11 +00:00
Iakov Davydov	557dd8f031	ListDirSorted check for excludefile	2017-11-17 21:51:11 +00:00
Iakov Davydov	37aaa19f3a	new option: --exclude-if-present	2017-11-17 21:51:11 +00:00
Iakov Davydov	cef2e3bf83	path -> startPath in walkRDirTree (we need the path package)	2017-11-17 21:51:11 +00:00
Iakov Davydov	a3a436ce16	WalkRDirTree: return error if unknown item type	2017-11-17 21:51:11 +00:00
Iakov Davydov	5d05df3124	ListContainsExcludeFile: checks for exclude file in the list	2017-11-17 21:51:11 +00:00
Iakov Davydov	421ba84e12	DirTree.Prune: deletes several directories	2017-11-17 21:51:11 +00:00
Iakov Davydov	7ae7080824	FileExists check if a file exists	2017-11-17 21:51:11 +00:00
Remus Bunduc	016abf825e	cache: first version	2017-11-15 15:23:21 +00:00
ishuah	aec2265be0	rclone: implement exit codes - #1136	2017-11-15 17:48:37 +03:00
Substantiel	2423fa40e2	config: add password sub command for setting obscured passwords	2017-11-15 14:44:45 +00:00
ishuah	c65734ee69	move: delete source directory after successful move - fixes #1642	2017-11-07 22:21:38 +00:00
Substantiel	7749157596	Add --auto-confirm flag	2017-11-05 21:56:50 +00:00
Nick Craig-Wood	cc7ed13b9b	fs: factor test running code into fstest/run.go	2017-11-03 12:59:58 +00:00
ishuah	10f27e2ff2	allow trailing+leading whitespace for passwords - #1717 ... warn users when they enter passwords with leading/trailing whitespaces Updated config_test.go, removing deprecated test case and updated TestReveal	2017-10-20 15:46:17 +01:00
Nick Craig-Wood	115053930e	Make error messages less crypting when revealing an unobscured password - fixes #1743	2017-10-16 22:03:06 +01:00
Nick Craig-Wood	edfab09eb9	config: add sub commands for full config file management ... Previously config sub commands were manually parsed rather than using cobra. Make config command have the following sub commands: * create Create a new remote with name, type and options. * delete Delete an existing remote <name>. * dump Dump the config file as JSON. * edit Enter an interactive configuration session. * file Show path of configuration file in use. * providers List in JSON format all the providers and options. * show Print (decrypted) config file, or the config for a single remote. * update Update options in an existing remote. The following changes were made to existing commands * listproviders was renamed to providers * listoptions was removed in favour of providing the output in providers * jsonconfig was renamed to create * an optional parameter was added to the show command	2017-10-14 11:50:41 +01:00
thierry	0575623dff	Add config listproviders, listoptions, jsonconfig for automated config ... Addition of a method listing the providers, a method listing the options of a provider and method of manual configuration.	2017-10-13 17:17:36 +01:00
Nick Craig-Wood	fc8b13c993	moveto/copyto: Fix to allow copying to the same name - fixes #1736	2017-10-12 20:45:36 +01:00
Nick Craig-Wood	43ced30f11	fs: Add more errors to retry - fixes #1733	2017-10-10 19:51:02 +01:00
Nick Craig-Wood	3eb7f52e39	fs: Add "unexpected EOF reading trailer" as a retriable error - fixes #1730	2017-10-09 17:29:16 +01:00
Nick Craig-Wood	7f3dc9b5c4	Implement WebDAV remote #580 ... This has special knowledge of Owncloud and Nextcloud to enable more functionality such as mod times.	2017-10-09 16:19:37 +01:00
Nick Craig-Wood	00fe6d95da	fs: fix duplicate files causing spurious copies ... Before this fix duplicate files (on Google Drive) caused the next file to be spuriously copied. `rclone dedupe` worked around the problem.	2017-10-02 16:52:53 +01:00
Nick Craig-Wood	9e9297838f	Implement pcloud remote - #418	2017-10-01 11:37:35 +01:00
Nick Craig-Wood	1c10497b68	Start v1.38-DEV development	2017-09-30 15:16:09 +01:00
Nick Craig-Wood	d96e45ba5b	Version v1.38	2017-09-30 14:20:43 +01:00
Nick Craig-Wood	657b3a674d	fs: fix test_all -clean to run just one cleaning thread per remote	2017-09-30 11:07:09 +01:00
Nick Craig-Wood	2857ed5c35	fs: fix --immutable tests on remotes which don't have modtime	2017-09-28 08:56:30 +01:00
Jacob McNamee	2d8e75cab4	Implement --immutable option	2017-09-24 20:00:00 +01:00
Stefan	85877f3adc	config: add show/file subcommands which print the config/its path (fixes #1086)	2017-09-19 17:59:19 +02:00
Nick Craig-Wood	87335de8a8	fs: fix filename normalization issues in the tests when running on OS X	2017-09-17 15:31:22 +01:00
Stefan Breunig	12405f9f41	fuse: re-use rcat to support uploads for all remotes (fixes #1672)	2017-09-16 22:49:08 +02:00
Stefan	234bfae0d5	b2: implement streaming upload of files with unknown length (see #1614) (closes #1686)	2017-09-16 22:43:48 +02:00
Nick Craig-Wood	4ac9a65049	fs: stop normalizing file names but do a normalized compare in the sync ... This works by using a transform function to transform file names when doing a compare when matching file names in a directory. rclone now UTF-8 normalizes the file names and does a case insensitive compare if the destination remote is case insensitive. This deprecates the --local-no-unicode-normalization flag. Fixes #1477	2017-09-16 19:49:31 +01:00
Nick Craig-Wood	a8e41f081c	fs: re-implement check and cryptcheck using the same traversal as sync ... This makes them 100% consistent with sync and also make them use less memory as they no longer build the whole tree in memory first. Fixes #1657	2017-09-16 19:49:31 +01:00
Nick Craig-Wood	261c7ad9e4	fs: make syncCopyMove use context for go routine cancellation	2017-09-16 19:49:31 +01:00
Nick Craig-Wood	fe96d5cf0a	fs: factor multiple directory traverse out of sync	2017-09-16 19:49:31 +01:00
Nick Craig-Wood	6df12b3f00	fs: improve retriable error detection	2017-09-16 19:48:49 +01:00
Stefan Breunig	9d3f11b493	amazonclouddrive, rcat: ensure rcat integration test passes even with AmazonCloudDrive (fixes: #1680)	2017-09-15 18:09:04 +02:00
Nick Craig-Wood	798502b204	fs: add more errors to be considered temporary errors ... This makes a framework for adding temporary errors identified by syscall number or by error string. Fixes #1660	2017-09-14 18:01:43 +01:00
Stefan Breunig	56dedc49e3	rcat: properly report if the upload fails	2017-09-13 20:21:52 +02:00
Girish Ramakrishnan	4c45cbea18	copy: error out if dst could not be listed	2017-09-12 09:29:44 +01:00
Stefan Breunig	a91448c83a	rcat: honor --dry-run even for small files	2017-09-11 22:28:16 +02:00
Stefan Breunig	80b1f2a494	rcat: configurable small files cutoff and implement proper upload verification	2017-09-11 08:26:53 +02:00
Stefan Breunig	57817397a0	rcat: directly upload small files without streaming them	2017-09-11 08:25:34 +02:00
Nick Craig-Wood	cbea06026a	Make check obey --ignore-size - fixes #1643	2017-09-01 11:20:41 +01:00
Nick Craig-Wood	2c8d6e86cc	fs: fix gofmt	2017-08-31 16:01:19 +01:00
cbruegg	bb6300b032	Fix bwlimit toggle in conjunction with schedules (Fixes #1607)	2017-08-31 15:33:29 +01:00
Stefan Breunig	a122b9fa7a	yandex: implement streaming uploads (see #1614)	2017-08-19 14:07:23 +02:00
Alex McGrath Kraak	2cc9071791	http: add --user-agent option. close #1557	2017-08-17 06:49:27 +02:00
Nick Craig-Wood	265fb8a5e2	fs: Manage empty directories - fixes #100 ... During the sync we collect a list of directories which should be empty and attempt to rmdir them at the end of the sync. If the directories are not empty then the rmdir will fail, logging a message but not erroring the sync.	2017-08-09 21:07:00 +01:00
Nick Craig-Wood	8a1a900733	fstest: use Feature.CanHaveEmptyDirectories to sharpen tests ... Now we actually test whether the directories are present or not, filtering out empty directories in the test using the CanHaveEmptyDirectories flag.	2017-08-09 20:55:08 +01:00
Nick Craig-Wood	20ae7d562b	fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes	2017-08-09 20:55:08 +01:00
Nick Craig-Wood	ec2ea37ad2	fs: Add --disable flag to disable optional features - fixes #1551 ... Eg to disable server side copy use `--disable copy`, to see a list of what you can disable, `--disable help`.	2017-08-07 21:34:45 +01:00
Nick Craig-Wood	bfe812ea6b	dedupe: implement merging of duplicate directories - fixes #1243	2017-08-07 15:36:41 +01:00
Nick Craig-Wood	db1995e63a	Add MergeDirs optional interface and implement it for drive	2017-08-07 15:32:47 +01:00
Nick Craig-Wood	81a2ab599f	fs: add optional ID to fs.Directory and set it in the remotes which care	2017-08-07 15:31:22 +01:00
Stefan Breunig	40f24e0ea3	config: use absolute ConfigPath to ensure newly written config is on the same mount - fixes #1569	2017-08-05 12:13:25 +02:00
Nick Craig-Wood	92d2e1f8d7	azureblob: rework and complete #801 ... * Fixup bitrot (rclone and Azure library) * Implement Copy * Add modtime to metadata under mtime key as RFC3339Nano * Make multipart upload work * Make it pass the integration tests * Fix uploading of zero length blobs * Rename to azureblob as it seems likely we will do azurefile * Add docs	2017-08-04 22:56:16 +01:00
Andrei Dragomir	036fd61a50	Added Azure Blob storage support #801	2017-08-04 22:54:27 +01:00
wuyu	ec5b72f8d5	Add new QingStor remote ... Add new package qingstor to support QingStor API. Add new unit test for its and tested through; But I commented on some tests case because of some of the features of QingStor. Add new docs for it.	2017-08-04 17:25:47 +01:00
Nick Craig-Wood	f682002b84	fs: Make tests create a new bucket rather than purging the old one ... This enables QingStor to pass the tests as it has a 2 minute lockout on deleting the old bucket then creating it again.	2017-08-04 17:09:28 +01:00
Stefan Breunig	28a18303f3	implement rcat – fixes #230, fixes #1001	2017-08-03 21:42:35 +02:00
Nick Craig-Wood	3e3a59768e	fs/test_all: fix after fstest factorisation	2017-08-03 20:01:05 +01:00
Nick Craig-Wood	e01741b557	fs: Cleaning up directories in test is no longer needed ... ..as it is done in the finalise method.	2017-08-03 20:01:05 +01:00
Nick Craig-Wood	71fe046937	fs: Add Find method to DirTree	2017-07-26 16:38:53 +01:00
Nick Craig-Wood	d5ff7104e5	fs: Implement NewDirTree for non --fast-list	2017-07-26 16:38:44 +01:00
Nick Craig-Wood	cd4895690a	fstest: Factor test initialisation into Initialise()	2017-07-26 16:38:33 +01:00
Nick Craig-Wood	1ecf2bcbd5	fs: fix typo in --bind description	2017-07-23 23:08:33 +01:00
Nick Craig-Wood	c3d6cc91ec	Fix --bind flag changes under go1.6 ... Correcting 9f24639568	2017-07-23 22:36:32 +01:00
Nick Craig-Wood	9f24639568	Add --bind flag for choosing the local addr on outgoing connections - fixes #1087 ... Supported by all remotes except FTP.	2017-07-23 16:27:39 +01:00
Nick Craig-Wood	7b81f12dad	box: add docs ... * reorder remotes so they are in alphabetical order by full name everywhere * update CONTRIBUTING doc	2017-07-23 11:32:34 +01:00
Nick Craig-Wood	d279161cee	Implement box storage remote - #97	2017-07-23 11:32:34 +01:00
Nick Craig-Wood	db6009126d	Fix test failure with new stretchr/testify - fixes #1550	2017-07-23 08:59:07 +01:00
Nick Craig-Wood	c8b5ee1e54	Start v1.37-DEV development	2017-07-22 20:43:06 +01:00
Nick Craig-Wood	a73ecec11f	Version v1.37	2017-07-22 20:04:29 +01:00
Nick Craig-Wood	0cf19ef66a	Make ListDirSorted check for subdirectories and write test	2017-07-19 09:36:27 +01:00
Nick Craig-Wood	655891170f	Check in ListDirSorted that the directory entries all belong	2017-07-18 23:39:42 +01:00
Nick Craig-Wood	6f71260acf	Add --tpslimit and --tpslimit-burst to limit transactions per second for HTTP ... This is useful if you are being rate limited or banned by your cloud storage provider.	2017-07-16 17:25:39 +01:00
Nick Craig-Wood	62e28d0a72	Replace token bucket limiter github.com/tsenart/tb with golang.org/x/time/rate ... In tests tsenart/tb has proved inaccurate at low rates.	2017-07-16 16:14:44 +01:00
Nick Craig-Wood	72add5ab27	sync: state whether duplicates are objects are directories	2017-07-08 15:42:18 +01:00
Nick Craig-Wood	69ff009264	Use a stable sort for sorting directory entries ... This is useful if there are duplicates. Assuming the remote delivers the entries in a consistent order, this will give the best user experience in syncing as it will consistently use the first entry for the sync comparison.	2017-07-06 14:07:26 +01:00
Nick Craig-Wood	27b157580e	Move make_test_files.go into bin	2017-07-06 11:54:57 +01:00
Nick Craig-Wood	f0512d1a52	Fix missing fs.Dir -> fs.Directory	2017-07-06 11:31:36 +01:00
Nick Craig-Wood	575e779b55	Warn about duplicate files when syncing - fixes #1506 ... Error about unsorted directories and test thoroughly	2017-06-30 21:24:13 +01:00
Nick Craig-Wood	e7d04fc103	Create fs.Directory interface and use it everywhere	2017-06-30 14:26:59 +01:00
Nick Craig-Wood	e2d7d413ef	fs: rename BasicInfo to DirEntry	2017-06-30 14:26:58 +01:00
Nick Craig-Wood	e7e9aa0dfa	fs: Remove unused ListFser interface	2017-06-30 14:26:58 +01:00
Nick Craig-Wood	f88300a153	Don't Mkdir at the start of sync - fixes #1131 ... This is possible now that the bucket based remotes will create the buckets on demand (9c1e703777).	2017-06-29 12:31:53 +01:00
Nick Craig-Wood	e54087ece1	Fix config tests to save configData which fixes subsequent tests	2017-06-29 12:31:53 +01:00
Nick Craig-Wood	54561fd2bc	s3: work around eventual consistency in bucket creation ... Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.	2017-06-29 12:31:52 +01:00
Nick Craig-Wood	f3c7e1a9dd	Debug directory creation and removal - fixes #1192	2017-06-27 22:19:35 +01:00
Nick Craig-Wood	aa20486485	Add --stats-log-level so can see --stats without -v - fixes #1180 ... The most common use for this flag is likely to be showing the stats without using -v by using `--stats-log-level NOTICE`.	2017-06-26 22:50:37 +01:00
Stefan Breunig	52b042971a	keep file permissions and try to keep user/group on supported systems (fixes #1467)	2017-06-25 09:05:24 +02:00
Stefan Breunig	2d2778eabf	don't delete remote if name does not change while renaming (fixes #1495)	2017-06-25 08:55:54 +02:00
Vasiliy Tolstov	6b89e6c381	add new http remote filesystem ... Signed-off-by: Vasiliy Tolstov <v.tolstov@selfip.ru>	2017-06-20 21:27:14 +01:00
Nick Craig-Wood	b76cd4abd2	Fix Range header option	2017-06-20 21:27:14 +01:00
Nick Craig-Wood	4fd9570332	fs: Use an in place filter in ListDirSorted	2017-06-20 21:27:14 +01:00
Nick Craig-Wood	5250fcdf08	core: fix data race in walk ... This was detected by the race detector when the client of Walk() sorted entries.	2017-06-15 16:46:32 +01:00
Nick Craig-Wood	64662bef8d	Deprecate --old-sync-method it is replaced with --fast-list ... Remove old sync method code.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	0b8d9084fc	test_all: print command line so it can be cut and pasted into bash	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	8a6a8b9623	Change List interface and add ListR optional interface ... This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	6fc88ff32e	Use --fast-list flag for sync/copy/move - fixes #1277 ... Redo test framework to take a -fast-list flag and test remotes with that flag.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	50928a5027	Implement --fast-list flag. ... This is supported remotes which can do a recursive listing. It will use more memory. This is related to #1277 but doesn't fix that issue yet.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	53c3e5f0ab	Add placeholder support for ListR interface. ... The ListR interface will be implemented by remotes that can do a recursive directory listing more efficiently than just recursing through the directories. These include the bucket based remotes.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	0edb025257	Fixup tests with dirs vs bucket based fs	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	7e20e16cff	core: Implement Walk directory listing and use in place of Lister ... This is in preparation for removing the Lister code and replacing the fundamental operation in the Fs with listing a single directory.	2017-06-14 16:49:40 +01:00
Nick Craig-Wood	68333d34a1	dropbox: make setting mod time on existing files work properly ... This is a fix left over from the v2 conversion. Dropbox ignores the client modification on an incoming file if it was identical to the existing file. This change deletes the existing file first before re-uploading the new one.	2017-06-13 13:58:39 +01:00
Nick Craig-Wood	740b3f6ae2	Fix problems found with ineffassign	2017-06-13 11:52:36 +01:00
Nick Craig-Wood	9a11d3efd9	Revert "Start Cat tests from 2 as onedrive doesn't support ranging from 1" ... Now that https://github.com/OneDrive/onedrive-api-docs/issues/543 is fixed, this can be reverted. This reverts commit 320c53eab0.	2017-06-10 13:48:00 +01:00
Nick Craig-Wood	ee14efd3c2	config: fix menu selection when no remotes	2017-06-10 11:39:40 +01:00
Nick Craig-Wood	52e1bfae2a	oauth: Allow auth_url and token_url to be set in the config file ... If set in the config file, these override the ones configured into the remote. This enables alternative oauth servers to be used for all oauth remotes. This can only be altered by editing the config file for the moment.	2017-06-08 20:35:32 +01:00
Nick Craig-Wood	b49821956a	Fix copyto/moveto test error (see #1261)	2017-06-07 14:08:46 +01:00
Nick Craig-Wood	a61ba1e7c4	moveto, copyto: report transfers and checks as per move and copy	2017-06-07 13:02:21 +01:00
Nick Craig-Wood	d30cc1e119	Factor RemoteSplit into fs	2017-06-07 12:27:33 +01:00
Nick Craig-Wood	74a3dfc4e1	Fix TestHashSums	2017-06-06 23:21:47 +01:00
Nick Craig-Wood	b047402294	config: Fix save of temp file under Windows - fixes #1458	2017-06-01 16:38:19 +01:00
Stefan	9cede6b372	fully write new config file before moving to target location (fixes #1287) ... * fully write new config file before moving to target location (fixes #1287) * do not fail if there is no previous config; print temporary config path on failure	2017-06-01 08:57:10 +01:00
Stefan Breunig	decd960867	make moveto/copyto no-ops when source and destination are the same (fixes #1261)	2017-05-30 23:01:19 +01:00
Nick Craig-Wood	20da3e6352	Add options to Put, PutUnchecked and Update, add HashOption and speed up local ... * Add options to Put, PutUnchecked and Update for all Fses * Use these to create HashOption * Implement this in local * Pass the option in fs.Copy This has the effect that we only calculate hashes we need to in the local Fs which speeds up transfers significantly.	2017-05-29 12:04:52 +01:00
Nick Craig-Wood	6381959850	dropbox: support Dropbox content hashing scheme - fixes #1302 ... * add support to hashing module * add dbhashsum to list the hashes * add support to dropbox module This means objects up and downloaded to/from Dropbox will have their hashes checked. Note after this change local objects are calculating MD5, SHA1 and DBHASH which is excessive and needs to be fixed.	2017-05-29 12:04:44 +01:00
Stefan Breunig	a2e3af0523	poll for Google Drive changes when mounted	2017-05-28 17:54:52 +01:00
Nick Craig-Wood	88ef475629	config: allow keys to be deleted from the config file	2017-05-25 10:15:22 +01:00
Nick Craig-Wood	ae9f8304fa	Attempt to make async buffer test more reliable	2017-05-24 16:24:06 +01:00
Nick Craig-Wood	e0ba1a2cd2	ftp: fix bitrot	2017-05-18 20:49:36 +01:00
Antonio Messina	c72fca2711	Add ftp backend - fixes #540	2017-05-18 20:49:36 +01:00
Nick Craig-Wood	50e79bc087	fs: Implement fs.Trace for tracing entry and exit of functions	2017-05-15 16:56:45 +01:00
Nick Craig-Wood	17633f5460	Require go1.6 for building rclone ... This is required because google.golang.org/grpc needs it.	2017-05-11 17:07:49 +01:00
Yoni Jah	cdeeff988e	Added RepetableReader to fs. used in OneDrive with io.LimitedReader to display accurate speed	2017-05-02 22:31:05 +01:00
Hraban Luyat	66e8c1600e	Print password prompts to stderr ... This makes rclone with encrypted config better suited for use in pipelines. E.g.: $ rclone lsl mydrive:Some/Dir | sort -k 4 If the password prompt ("Enter configuration password") is printed to stdout, it will be swallowed by sort. By printing it to stderr, you still see the prompt, without sacrificing compatibility with the unix pipeline.	2017-04-22 09:38:39 +01:00
Nick Craig-Wood	bc25190fc7	Fix misleading log message with --dry-run - fixes #1309	2017-04-10 16:07:22 +01:00
marvwatson	4810aa65a4	Update references from HTTP to HTTPS where possible	2017-03-29 05:38:34 -07:00
Nick Craig-Wood	503cd84919	Start v1.36-DEV development	2017-03-18 11:30:59 +00:00
Nick Craig-Wood	118e26f8e2	Version v1.36	2017-03-18 11:16:43 +00:00
Nick Craig-Wood	9b07d32c02	onedrive, drive, amazonclouddrive: make sure we find the root ... This fixes copyto copying things to the wrong place - fixes #1231	2017-03-16 09:42:49 +00:00
Nick Craig-Wood	99f7fe736a	onedrive: implement Move #197	2017-03-15 14:01:08 +00:00
Nick Craig-Wood	320c53eab0	Start Cat tests from 2 as onedrive doesn't support ranging from 1 ... This has been reported here: https://github.com/OneDrive/onedrive-api-docs/issues/543	2017-03-12 14:24:33 +00:00
Nick Craig-Wood	4d5b73df85	Fix TestListDirSorted eventual consistency listing problems	2017-03-12 14:00:22 +00:00
Nick Craig-Wood	b2a4ea9304	fs/buffer: Fix panic on concurrent Read/Close - fixes #1213	2017-03-06 19:22:17 +00:00
Nick Craig-Wood	10e532bce9	Fix --files-from with an empty file copying everything - fixes #1196	2017-03-04 10:12:54 +00:00
Nick Craig-Wood	4ab7e05e02	Fix MimeType propagation ... In fs.Copy, don't wrap objects if possible, and if not, then add a MimeType method into the wrapped object.	2017-03-04 10:10:55 +00:00
Nick Craig-Wood	79e3c67bbd	local, yandex, dropbox: fix NewObject suceeding on a directory #1079 ... Add tests to make it consistent across all remotes	2017-02-25 11:09:57 +00:00
Nick Craig-Wood	e2f0feef3c	Add debugging to print hash values on failed hash comparison	2017-02-23 11:23:19 +00:00
Nick Craig-Wood	e59dc81658	Stop --track-renames deleting case folded source files - fixes #1094 ... What was happening is that when Move was implemented as Copy + Delete, MoveFile was seeing the files didn't need transferring (because they were identical) then deleted the source. The fix uses Move instead and patches onedrive to disallow a case folded identical copy (which errors with 500 error)	2017-02-22 19:28:22 +00:00
Nick Craig-Wood	6b0f2ef4bd	Fix --delete-before deleting files on copy - fixes #1166	2017-02-22 13:17:38 +00:00
Hisham Zarka	b3a526814e	fix --ignore-checksum	2017-02-18 13:13:53 +04:00
Nick Craig-Wood	033d1eb7af	Refactor Account interface	2017-02-17 09:15:24 +00:00
Nick Craig-Wood	ac62ef430d	Prevent double closes on async buffer	2017-02-17 08:55:24 +00:00
Nick Craig-Wood	928be0f1fd	mount: fix seek with buffering to use correct interface ... Stop pre-cache before seeking which stops lots of excess data transfer	2017-02-17 08:55:24 +00:00
Nick Craig-Wood	6f75290678	Make async buffering start slowly to improve seek performance	2017-02-17 08:26:14 +00:00
Nick Craig-Wood	f3c5745468	Add srcRemote and dstRemote parameters to DirMove #954	2017-02-16 17:42:37 +00:00
Nick Craig-Wood	dac4bb22d3	mount: Make include and exclude filters apply to mount - fixes #1060	2017-02-15 23:28:53 +00:00
Nick Craig-Wood	b52c80e85c	sync: don't update mod times if --dry-run set - fixes #1100	2017-02-15 23:09:44 +00:00