2014-03-28 22:34:13 +00:00
|
|
|
---
|
|
|
|
title: "Swift"
|
|
|
|
description: "Swift"
|
2014-04-26 16:43:41 +00:00
|
|
|
date: "2014-04-26"
|
2014-03-28 22:34:13 +00:00
|
|
|
---
|
|
|
|
|
2014-07-17 19:03:11 +00:00
|
|
|
<i class="fa fa-space-shuttle"></i>Swift
|
|
|
|
----------------------------------------
|
|
|
|
|
2017-03-29 12:38:34 +00:00
|
|
|
Swift refers to [Openstack Object Storage](https://www.openstack.org/software/openstack-storage/).
|
2014-03-28 22:34:13 +00:00
|
|
|
Commercial implementations of that being:
|
|
|
|
|
2017-03-29 12:38:34 +00:00
|
|
|
* [Rackspace Cloud Files](https://www.rackspace.com/cloud/files/)
|
|
|
|
* [Memset Memstore](https://www.memset.com/cloud/storage/)
|
2014-03-28 22:34:13 +00:00
|
|
|
|
2014-07-17 19:03:11 +00:00
|
|
|
Paths are specified as `remote:container` (or `remote:` for the `lsd`
|
|
|
|
command.) You may put subdirectories in too, eg `remote:container/path/to/dir`.
|
2014-03-28 22:34:13 +00:00
|
|
|
|
|
|
|
Here is an example of making a swift configuration. First run
|
|
|
|
|
|
|
|
rclone config
|
|
|
|
|
|
|
|
This will guide you through an interactive setup process.
|
|
|
|
|
|
|
|
```
|
|
|
|
No remotes found - make a new one
|
|
|
|
n) New remote
|
2016-02-21 13:39:04 +00:00
|
|
|
s) Set configuration password
|
|
|
|
n/s> n
|
2014-03-28 22:34:13 +00:00
|
|
|
name> remote
|
2016-02-21 13:39:04 +00:00
|
|
|
Type of storage to configure.
|
|
|
|
Choose a number from below, or type in your own value
|
2016-07-11 11:42:44 +00:00
|
|
|
1 / Amazon Drive
|
2016-02-21 13:39:04 +00:00
|
|
|
\ "amazon cloud drive"
|
2017-01-09 05:09:19 +00:00
|
|
|
2 / Amazon S3 (also Dreamhost, Ceph, Minio)
|
2016-02-21 13:39:04 +00:00
|
|
|
\ "s3"
|
|
|
|
3 / Backblaze B2
|
|
|
|
\ "b2"
|
|
|
|
4 / Dropbox
|
|
|
|
\ "dropbox"
|
2017-01-09 05:09:19 +00:00
|
|
|
5 / Encrypt/Decrypt a remote
|
|
|
|
\ "crypt"
|
|
|
|
6 / Google Cloud Storage (this is not Google Drive)
|
2016-02-21 13:39:04 +00:00
|
|
|
\ "google cloud storage"
|
2017-01-09 05:09:19 +00:00
|
|
|
7 / Google Drive
|
2016-02-21 13:39:04 +00:00
|
|
|
\ "drive"
|
2017-01-09 05:09:19 +00:00
|
|
|
8 / Hubic
|
2016-02-21 13:39:04 +00:00
|
|
|
\ "hubic"
|
2017-01-09 05:09:19 +00:00
|
|
|
9 / Local Disk
|
2016-02-21 13:39:04 +00:00
|
|
|
\ "local"
|
2017-01-09 05:09:19 +00:00
|
|
|
10 / Microsoft OneDrive
|
2016-02-21 13:39:04 +00:00
|
|
|
\ "onedrive"
|
2017-01-09 05:09:19 +00:00
|
|
|
11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
|
2016-02-21 13:39:04 +00:00
|
|
|
\ "swift"
|
2017-03-05 10:14:57 +00:00
|
|
|
12 / SSH/SFTP Connection
|
|
|
|
\ "sftp"
|
|
|
|
13 / Yandex Disk
|
2016-02-21 13:39:04 +00:00
|
|
|
\ "yandex"
|
2017-01-09 05:09:19 +00:00
|
|
|
Storage> 11
|
2014-03-28 22:34:13 +00:00
|
|
|
User name to log in.
|
|
|
|
user> user_name
|
|
|
|
API key or password.
|
|
|
|
key> password_or_api_key
|
|
|
|
Authentication URL for server.
|
|
|
|
Choose a number from below, or type in your own value
|
2016-02-21 13:39:04 +00:00
|
|
|
1 / Rackspace US
|
|
|
|
\ "https://auth.api.rackspacecloud.com/v1.0"
|
|
|
|
2 / Rackspace UK
|
|
|
|
\ "https://lon.auth.api.rackspacecloud.com/v1.0"
|
|
|
|
3 / Rackspace v2
|
|
|
|
\ "https://identity.api.rackspacecloud.com/v2.0"
|
|
|
|
4 / Memset Memstore UK
|
|
|
|
\ "https://auth.storage.memset.com/v1.0"
|
|
|
|
5 / Memset Memstore UK v2
|
|
|
|
\ "https://auth.storage.memset.com/v2.0"
|
|
|
|
6 / OVH
|
|
|
|
\ "https://auth.cloud.ovh.net/v2.0"
|
2014-03-28 22:34:13 +00:00
|
|
|
auth> 1
|
2016-04-20 23:00:49 +00:00
|
|
|
User domain - optional (v3 auth)
|
|
|
|
domain> Default
|
2017-01-09 05:09:19 +00:00
|
|
|
Tenant name - optional for v1 auth, required otherwise
|
|
|
|
tenant> tenant_name
|
2016-07-05 16:34:22 +00:00
|
|
|
Tenant domain - optional (v3 auth)
|
|
|
|
tenant_domain>
|
2016-02-21 13:39:04 +00:00
|
|
|
Region name - optional
|
2017-01-09 05:09:19 +00:00
|
|
|
region>
|
2016-02-02 19:45:05 +00:00
|
|
|
Storage URL - optional
|
2017-01-09 05:09:19 +00:00
|
|
|
storage_url>
|
2016-06-03 16:52:02 +00:00
|
|
|
AuthVersion - optional - set to (1,2,3) if your auth URL has no version
|
2017-01-09 05:09:19 +00:00
|
|
|
auth_version>
|
|
|
|
Remote config
|
2014-03-28 22:34:13 +00:00
|
|
|
--------------------
|
|
|
|
[remote]
|
|
|
|
user = user_name
|
|
|
|
key = password_or_api_key
|
|
|
|
auth = https://auth.api.rackspacecloud.com/v1.0
|
2017-01-09 05:09:19 +00:00
|
|
|
domain = Default
|
|
|
|
tenant =
|
|
|
|
tenant_domain =
|
|
|
|
region =
|
|
|
|
storage_url =
|
|
|
|
auth_version =
|
2014-03-28 22:34:13 +00:00
|
|
|
--------------------
|
|
|
|
y) Yes this is OK
|
|
|
|
e) Edit this remote
|
|
|
|
d) Delete this remote
|
|
|
|
y/e/d> y
|
|
|
|
```
|
|
|
|
|
|
|
|
This remote is called `remote` and can now be used like this
|
|
|
|
|
|
|
|
See all containers
|
|
|
|
|
|
|
|
rclone lsd remote:
|
|
|
|
|
|
|
|
Make a new container
|
|
|
|
|
|
|
|
rclone mkdir remote:container
|
|
|
|
|
|
|
|
List the contents of a container
|
|
|
|
|
|
|
|
rclone ls remote:container
|
|
|
|
|
|
|
|
Sync `/home/local/directory` to the remote container, deleting any
|
|
|
|
excess files in the container.
|
|
|
|
|
|
|
|
rclone sync /home/local/directory remote:container
|
|
|
|
|
2016-09-19 15:36:36 +00:00
|
|
|
### Configuration from an Openstack credentials file ###
|
|
|
|
|
|
|
|
An Opentstack credentials file typically looks something something
|
|
|
|
like this (without the comments)
|
|
|
|
|
|
|
|
```
|
|
|
|
export OS_AUTH_URL=https://a.provider.net/v2.0
|
|
|
|
export OS_TENANT_ID=ffffffffffffffffffffffffffffffff
|
|
|
|
export OS_TENANT_NAME="1234567890123456"
|
|
|
|
export OS_USERNAME="123abc567xy"
|
|
|
|
echo "Please enter your OpenStack Password: "
|
|
|
|
read -sr OS_PASSWORD_INPUT
|
|
|
|
export OS_PASSWORD=$OS_PASSWORD_INPUT
|
|
|
|
export OS_REGION_NAME="SBG1"
|
|
|
|
if [ -z "$OS_REGION_NAME" ]; then unset OS_REGION_NAME; fi
|
|
|
|
```
|
|
|
|
|
|
|
|
The config file needs to look something like this where `$OS_USERNAME`
|
|
|
|
represents the value of the `OS_USERNAME` variable - `123abc567xy` in
|
|
|
|
the example above.
|
|
|
|
|
|
|
|
```
|
|
|
|
[remote]
|
|
|
|
type = swift
|
|
|
|
user = $OS_USERNAME
|
|
|
|
key = $OS_PASSWORD
|
|
|
|
auth = $OS_AUTH_URL
|
|
|
|
tenant = $OS_TENANT_NAME
|
|
|
|
```
|
|
|
|
|
|
|
|
Note that you may (or may not) need to set `region` too - try without first.
|
|
|
|
|
2017-06-06 15:40:00 +00:00
|
|
|
### --fast-list ###
|
|
|
|
|
|
|
|
This remote supports `--fast-list` which allows you to use fewer
|
|
|
|
transactions in exchange for more memory. See the [rclone
|
|
|
|
docs](/docs/#fast-list) for more details.
|
|
|
|
|
2015-11-14 11:38:35 +00:00
|
|
|
### Specific options ###
|
|
|
|
|
|
|
|
Here are the command line options specific to this cloud storage
|
|
|
|
system.
|
|
|
|
|
|
|
|
#### --swift-chunk-size=SIZE ####
|
|
|
|
|
|
|
|
Above this size files will be chunked into a _segments container. The
|
|
|
|
default for this is 5GB which is its maximum value.
|
2017-01-09 05:09:19 +00:00
|
|
|
|
2015-06-06 09:05:21 +00:00
|
|
|
### Modified time ###
|
2014-03-28 22:34:13 +00:00
|
|
|
|
|
|
|
The modified time is stored as metadata on the object as
|
|
|
|
`X-Object-Meta-Mtime` as floating point since the epoch accurate to 1
|
|
|
|
ns.
|
|
|
|
|
|
|
|
This is a defacto standard (used in the official python-swiftclient
|
|
|
|
amongst others) for storing the modification time for an object.
|
2016-03-16 17:39:44 +00:00
|
|
|
|
|
|
|
### Limitations ###
|
|
|
|
|
|
|
|
The Swift API doesn't return a correct MD5SUM for segmented files
|
|
|
|
(Dynamic or Static Large Objects) so rclone won't check or use the
|
|
|
|
MD5SUM for these.
|
2016-05-02 11:34:15 +00:00
|
|
|
|
2016-05-06 13:09:12 +00:00
|
|
|
### Troubleshooting ###
|
2016-05-02 11:34:15 +00:00
|
|
|
|
|
|
|
#### Rclone gives Failed to create file system for "remote:": Bad Request ####
|
|
|
|
|
|
|
|
Due to an oddity of the underlying swift library, it gives a "Bad
|
|
|
|
Request" error rather than a more sensible error when the
|
|
|
|
authentication fails for Swift.
|
|
|
|
|
|
|
|
So this most likely means your username / password is wrong. You can
|
|
|
|
investigate further with the `--dump-bodies` flag.
|
|
|
|
|
2016-09-19 15:36:36 +00:00
|
|
|
This may also be caused by specifying the region when you shouldn't
|
|
|
|
have (eg OVH).
|
|
|
|
|
2016-05-02 11:34:15 +00:00
|
|
|
#### Rclone gives Failed to create file system: Response didn't have storage storage url and auth token ####
|
|
|
|
|
|
|
|
This is most likely caused by forgetting to specify your tenant when
|
|
|
|
setting up a swift remote.
|