TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions
8334 commits 181 branches 114 tags 335 MiB
Commit graph

188 commits

Author SHA1 Message Date
wjielai
22937e8982
docs: add Tencent COS to s3 provider list - fixes #4468 
* add Tencent COS to s3 provider list.

Co-authored-by: wjielai <wjielai@tencent.com>
 2020-09-08 16:34:25 +01:00
Nick Craig-Wood
510ac341e1 Version v1.53.0 2020-09-02 17:00:18 +01:00
Nick Craig-Wood
d5f4c74697 s3: implement cleanup and backend command to list & remove multipart uploads 
This implements `rclone cleanup` to remove multipart uploads over 24
hours old. It also implements the backend command
`list-multipart-uploads` to see which ones are available and `cleanup`
to delete them with a configurable expiry interval.

See #4302
 2020-07-28 11:37:46 +01:00
Vincent Feltz
f4d7e41f24 s3: add Scaleway provider - fixes #4338 2020-06-13 11:55:37 +01:00
Nick Craig-Wood
844b903595 docs: promote the use of -i/--interactive and "rclone sync -i" everywhere #1574 2020-06-10 12:33:53 +01:00
Nick Craig-Wood
e62c032184 docs: remove manually set dates and use git dates instead 2020-06-01 13:07:46 +01:00
Nick Craig-Wood
6882aeff97 Version v1.52.0 2020-05-27 17:31:10 +01:00
Nick Craig-Wood
41e12114a8 docs: fix validation errors 2020-05-27 17:31:09 +01:00
Nick Craig-Wood
74b8cbfb84 docs: set unsafe HTML parsing to false and fix raw HTML insertion 
This means that markdown files can't contain <thing> any more.
 2020-05-27 17:31:09 +01:00
Nick Craig-Wood
80dab10ec9 docs: Fix generated docs 
In distributed docs
- Make img tags absolute
- Add donate link in docs
- Fix provider table
 2020-05-27 17:31:08 +01:00
Martin Michlmayr
fb169a8b54
doc: fix typos throughout docs 2020-05-19 12:02:44 +01:00
Nick Craig-Wood
43daecd89b Version v1.51.0 2020-02-01 10:40:01 +00:00
Dave Koston
9f99c20232 s3: Add StackPath Object Storage Support 2020-01-31 16:05:44 +00:00
Tennix
15d19131bd s3: use aws web identity role provider 2020-01-05 19:49:31 +00:00
Thomas Kriechbaumer
584e705c0c s3: introduce list_chunk option for bucket listing 
The S3 ListObject API returns paginated bucket listings, with
"MaxKeys" items for each GET call.

The default value is 1000 entries, but for buckets with millions of
objects it might make sense to request more elements per request, if
the backend supports it. This commit adds a "list_chunk" option for
the user to specify a lower or higher value.

This commit does not add safe guards around this value - if a user
decides to request a too large list, it might result in connection
timeouts (on the server or client).

In AWS S3, there is a fixed limit of 1000, some other services might
have one too.  In Ceph, this can be configured in RadosGW.
 2020-01-02 12:15:01 +00:00
Outvi V
db1c7f9ca8 s3: Add new region Asia Patific (Hong Kong) 2020-01-02 11:10:48 +00:00
Marco Molteni
0c9a8cf776 doc: add Scaleway to the S3 table of contents 
Hello, documentation for Scaleway was already there, but the TOC was missing it.
 2019-11-14 21:49:43 +00:00
Aleksandar Jankovic
6b55b8b133 s3: add option for multipart failiure behaviour 
This is needed for resuming uploads across different sessions.
 2019-10-02 16:49:16 +01:00
Fabian Möller
33f129fbbc s3: use lib/encoder 
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
 2019-09-30 22:00:24 +01:00
Nick Craig-Wood
73e010aff9 docs: make the config walkthroughs consistent for each backend 2019-08-26 10:47:17 +01:00
Nick Craig-Wood
93810a739d docs: update fontawesome free to 5.10.2 and fixup broken images 2019-08-25 11:09:19 +01:00
Nick Craig-Wood
57d5de6fba build: fix up package paths after repo move 
git grep -l github.com/ncw/rclone | xargs -d'\n' perl -i~ -lpe 's|github.com/ncw/rclone|github.com/rclone/rclone|g'
goimports -w `find . -name \*.go`
 2019-07-28 18:47:38 +01:00
Russell Davis
6a3c3d9b89 Update docs on S3 policy to include ListAllMyBuckets permission 
This permission is required for `rclone lsd`.
 2019-07-02 07:56:54 +01:00
Matti Niemenmaa
a6dca4c13f s3: Add INTELLIGENT_TIERING storage class 
For Intelligent-Tiering:
https://aws.amazon.com/s3/storage-classes/#Unknown_or_changing_access
 2019-07-01 18:17:48 +01:00
Nick Craig-Wood
245fed513a Version v1.48.0 2019-06-15 13:55:41 +01:00
Nick Craig-Wood
d533de0f5c docs: clarify that s3 can't access Glacier Vault 2019-06-15 09:56:17 +01:00
Philip Harvey
1a2fb52266 s3: make SetModTime work for GLACIER while syncing - Fixes #3224 
Before this change rclone would fail with

    Failed to set modification time: InvalidObjectState: Operation is not valid for the source object's storage class

when attempting to set the modification time of an object in GLACIER.

After this change rclone will re-upload the object as part of a sync if it needs to change the modification time.

See: https://forum.rclone.org/t/suspected-bug-in-s3-or-compatible-sync-logic-to-glacier/10187
 2019-06-03 15:28:19 +01:00
Animosity022
b88e50cc36 docs: Typo fixes with "a existing" 
Fixed a typo with "a existing" to "an existing"
 2019-05-11 16:49:48 +01:00
Nick Craig-Wood
a0d4fdb2fa Version v1.47.0 2019-04-13 11:01:58 +01:00
Manu
6e86526c9d s3: add support for "Glacier Deep Archive" storage class - fixes #3088 2019-04-11 10:21:41 +01:00
Ben Boeckel
3d424c6e08 docs: fix various typos 2019-03-28 15:12:51 +00:00
Nick Craig-Wood
27714e29c3 s3: note incompatibility with CEPH Jewel - fixes #3015 2019-03-06 11:50:37 +00:00
Nick Craig-Wood
eb85ecc9c4 Version v1.46 2019-02-09 10:42:57 +00:00
Rémy Léone
1e6cbaa355 s3: Add Scaleway to s3 documentation 2019-02-01 17:09:57 +00:00
Nick Craig-Wood
1318c6aec8 s3: Add Alibaba OSS to integration tests and fix storage classes 2019-01-12 20:41:47 +00:00
Nick Craig-Wood
ff0b8e10af s3: Support Alibaba Cloud (Aliyun) OSS 
The existing s3 backend passed all integration tests with OSS provided
`force_path_style = false`.

This makes sure that is so and adds documentation and configuration
for OSS.

Thanks to @luolibin for their work on the OSS backend which we ended
up not needing.

Fixes #1641
Fixes #1237
 2019-01-12 17:28:04 +00:00
William Cocker
dda23baf42 s3 : update doc for Glacier storage class 
s3 : update doc for Glacier storage class : related to #923
 2018-12-06 21:53:38 +00:00
Nick Craig-Wood
d99ffde7c0 s3: change --s3-upload-concurrency default to 4 to increase perfomance #2772 
Increasing the --s3-upload-concurrency to 4 (from 2) gives an
additional 45% throughput at the cost of 10MB extra memory per transfer.

After testing the upload perfoc
 2018-12-02 17:58:34 +00:00
Nick Craig-Wood
198c34ce21 s3: implement --s3-upload-cutoff for single part uploads below this - fixes #2772 
Before this change rclone would use multipart uploads for any size of
file.  However multipart uploads are less efficient for smaller files
and don't have MD5 checksums so it is advantageous to use single part
uploads if possible.

This implements single part uploads for all files smaller than the
upload_cutoff size.  Streamed files must be uploaded as multipart
files though.
 2018-12-02 17:58:34 +00:00
Nick Craig-Wood
d35bd15762 Version v1.45 2018-11-24 13:44:25 +00:00
Nick Craig-Wood
0f02c9540c s3: make --s3-v2-auth flag 
This is an alternative to setting the region to "other-v2-signature"
which is inconvenient for multi-region providers.
 2018-10-14 00:10:29 +01:00
Nick Craig-Wood
78b9bd77f5 docs: auto generate backend options documentation 
This inserts the output of "rclone help backend xxx" into the help
pages for each backend.
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood
a9273c5da5 docs: move documentation for options from docs/content into backends 
In the following commit, the documentation will be autogenerated.
 2018-10-06 11:47:46 +01:00
Nick Craig-Wood
01af8e2905 s3: docs for how to configure Aliyun OSS / Netease NOS - thanks @xiaolei0125 2018-07-20 15:49:07 +01:00
Nick Craig-Wood
f06ba393b8 s3: Add --s3-force-path-style - fixes #2401 2018-07-20 15:41:40 +01:00
themylogin
7d2861ead6 Adjust S3 upload concurrency with --s3-upload-concurrency 2018-06-14 16:15:17 +01:00
Nick Craig-Wood
311a962011 s3: Look in S3 named profile files for credentials - fixes #2243 2018-04-21 09:00:20 +01:00
Nick Craig-Wood
dc247d21ff s3: add in config for all the supported S3 providers #2140 
These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other.

This configures endpoints where known and makes sure config doesn't
appear where it isn't valid where possible.
 2018-04-13 16:33:26 +01:00
Giri Badanahatti
acd5d4377e config,s3: hierarchical configuration support #2140 
This introduces a method of making provider specific configuration
within a remote.  This is useful particularly in s3.

This commit does the basic configuration in S3 for IBM COS.
 2018-04-13 16:05:35 +01:00
Craig Rachel
0e6faa2313 s3: add One Zone Infrequent Access storage class - fixes #2240 2018-04-13 13:36:25 +01:00
Peter Baumgartner
1db68571fd s3,swift: Add --use-server-modtime 
`--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
 2018-04-13 13:32:17 +01:00
Nick Craig-Wood
92c5aa3786 s3: add --s3-chunk-size option - fixes #2203 2018-04-05 15:40:08 +01:00
gbadanahatti
e5aa5fe7d8 s3: docs: Minor format and URL changes to IBM COS Documentation content 2018-03-26 20:49:53 +01:00
Nick Craig-Wood
e62fe06763 s3: document --ignore-checksum workaround for KMS #1824 2018-03-17 10:51:45 +00:00
Giri Badanahatti
aba43cd3a4 Documention for IBM COS (S3) configuration. 2018-03-15 20:20:43 +00:00
Nick Craig-Wood
89748feaa5 s3: update docs to discourage use of v2 auth - fixes #2120 
From testing it appears that CEPH no longer works properly with v2
auth and neither does Dreamhost, so update the docs anc configuration
to recommend v4 auth.
 2018-03-13 20:47:29 +00:00
Chris Redekop
4dd1e507f4 s3: set/get the hash for multipart files - #523 2018-01-06 14:30:10 +00:00
David Minor
4c116af1d0 s3: add support for ECS task IAM roles 
ECS container IAM metadata is in a different place than EC2 IAM metadata.
Use defaults' RemoteCredProvider function to query the standard locations
for the credentials.

Give the ECS role precedence over the role available from the underlying
EC2 instance.
 2017-11-23 08:56:56 +00:00
Corban Raun
a8dfc5ce3b Fix spelling in some documentation 2017-11-05 21:38:59 +00:00
Andrew Starr-Bochicchio
e6fb96cfd4 Initial docs for usage with DigitalOcean Spaces. 2017-10-26 21:34:42 +01:00
Nick Craig-Wood
3e9861eecf docs: improve links to cloud providers 2017-09-30 09:28:49 +01:00
Nick Craig-Wood
1d994f7330 s3: add Wasabi instructions 2017-09-30 09:00:56 +01:00
Girish Ramakrishnan
2f0551074c s3: set session token when using STS 2017-09-12 22:59:29 +01:00
Jan Varho
5a1351f141 s3: Document glacier transitions and behavior 2017-09-12 09:27:32 +01:00
Harshavardhana
8d77e48190 Minio supports ETags and metadata. 
Current doc mentioned lack of ETag and metadata
support which since has been long fixed in many
upstream Minio releases.

Also cleanup the doc to show new startup banner etc.
 2017-06-20 08:21:02 +01:00
Edward Q. Bridges
9876ba53f8 Updated permissions 
As it happens, after testing the `GetObject` permission is also required to do `HEAD` requests on a given object.
 2017-06-14 17:29:21 +01:00
Nick Craig-Wood
50928a5027 Implement --fast-list flag. 
This is supported remotes which can do a recursive listing.  It will
use more memory.

This is related to #1277 but doesn't fix that issue yet.
 2017-06-14 16:49:40 +01:00
Edward Q. Bridges
6ee4c62cae Add section on required IAM permissions. 
cf.: https://github.com/ncw/rclone/issues/1455
 2017-06-04 22:37:17 +01:00
marvwatson
4810aa65a4 Update references from HTTP to HTTPS where possible 2017-03-29 05:38:34 -07:00
Nick Craig-Wood
4652db34a4 Update config docs - fixes #1174 2017-03-05 10:14:57 +00:00
emyarod
3c87a0d0dc Update remote docs to show correct setup process 2017-01-10 13:09:52 +00:00
Radek Šenfeld
bc414b698d Command line argument for setting/overriding Amazon S3 ACL 2016-10-13 17:45:11 +01:00
Asko Tamm
61665ddd10 s3: add support for setting storage class in config and command line 2016-09-13 19:49:44 +01:00
Radek Šenfeld
2003ba356b User-configurable Amazon S3 ACL 
fixes #413
 2016-08-23 12:25:08 +01:00
Nick Craig-Wood
56adb52a21 Rename Amazon Cloud Drive to Amazon Drive - fixes #532 2016-07-11 12:42:44 +01:00
Nick Craig-Wood
8c2fc6daf8 s3: Add instructions on how to use rclone with minio 2016-07-11 12:12:28 +01:00
Justin R. Wilson
6e35a3b3ce Add AES256 server-side encryption for s3 - Fixes #491 
Add a configuration key and support for AES256 server-side encryption.
 2016-06-18 14:28:38 +01:00
Nick Craig-Wood
377986d599 Update config walk throughs with new style choice menu 2016-02-21 13:40:16 +00:00
Nick Craig-Wood
a0b9bd527e Add both forms of env var to the docs 2016-02-09 17:19:13 +00:00
Brian Stengaard
ce05ef7110 Add IAM role and Env credentials 
This will make the s3 provider authentaction logic

  - Configured credentials if both key and secret available
  - Anonymous if key and secret missing and env_auth not set
  - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool)
    - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables
    - IAM role credentials as fallback
 2016-02-09 16:32:36 +00:00
Nick Craig-Wood
0ab3f020ab Fix Amazon icon in the docs 2015-11-06 15:26:58 +00:00
Nick Craig-Wood
9f1daabb2c s3: allow anonymous access to public repositories - fixes #154 
When setting up the remote, leave both the access key and secret key
blank.
 2015-10-03 16:04:21 +01:00
Nick Craig-Wood
f8b7eaec93 s3: Document cross region bucket limitations - fixes #105 2015-08-25 20:15:50 +01:00
Nick Craig-Wood
8140869767 s3: Update documentation after transition to new SDK - also fixes #47 2015-08-16 15:51:38 +01:00
Nick Craig-Wood
bf4879f57f Expand docs and remove duplicated information 2015-06-06 14:17:14 +01:00
Nick Craig-Wood
07f9a1a9f0 Update docs for Dropbox and Google Cloud Storage 2014-07-18 10:10:08 +01:00
Nick Craig-Wood
580fa3a5a7 Documentation updates 2014-04-26 17:43:41 +01:00
Nick Craig-Wood
40f755df20 Rename docs directory 2014-04-17 22:32:39 +01:00
Renamed from rclone.org/content/s3.md (Browse further)