swift: Allow authentication with storage url and auth key

Adding the option to load the storage url and the auth key
from the environment when you have an alternate authorization,
external to rclone, and you need to use it (e.g. because
it's not yet supported by the swift go library)

Allowing to get alternate authentication from config file,
and using proper way (c.Authenticated()) to know if it's authenticated.

Updated docs as well
This commit is contained in:
Giovanni Pizzi 2017-12-07 22:30:58 +01:00 committed by Nick Craig-Wood
parent 25b073c767
commit f12512dd13
2 changed files with 92 additions and 62 deletions

View file

@ -42,33 +42,39 @@ Choose a number from below, or type in your own value
\ "b2" \ "b2"
4 / Box 4 / Box
\ "box" \ "box"
5 / Dropbox 5 / Cache a remote
\ "cache"
6 / Dropbox
\ "dropbox" \ "dropbox"
6 / Encrypt/Decrypt a remote 7 / Encrypt/Decrypt a remote
\ "crypt" \ "crypt"
7 / FTP Connection 8 / FTP Connection
\ "ftp" \ "ftp"
8 / Google Cloud Storage (this is not Google Drive) 9 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage" \ "google cloud storage"
9 / Google Drive 10 / Google Drive
\ "drive" \ "drive"
10 / Hubic 11 / Hubic
\ "hubic" \ "hubic"
11 / Local Disk 12 / Local Disk
\ "local" \ "local"
12 / Microsoft Azure Blob Storage 13 / Microsoft Azure Blob Storage
\ "azureblob" \ "azureblob"
13 / Microsoft OneDrive 14 / Microsoft OneDrive
\ "onedrive" \ "onedrive"
14 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH) 15 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift" \ "swift"
15 / QingClound Object Storage 16 / Pcloud
\ "pcloud"
17 / QingClound Object Storage
\ "qingstor" \ "qingstor"
16 / SSH/SFTP Connection 18 / SSH/SFTP Connection
\ "sftp" \ "sftp"
17 / Yandex Disk 19 / Webdav
\ "webdav"
20 / Yandex Disk
\ "yandex" \ "yandex"
18 / http Connection 21 / http Connection
\ "http" \ "http"
Storage> swift Storage> swift
Get swift credentials from environment variables in standard OpenStack form. Get swift credentials from environment variables in standard OpenStack form.
@ -77,12 +83,12 @@ Choose a number from below, or type in your own value
\ "false" \ "false"
2 / Get swift credentials from environment vars. Leave other fields blank if using this. 2 / Get swift credentials from environment vars. Leave other fields blank if using this.
\ "true" \ "true"
env_auth> 1 env_auth> true
User name to log in. User name to log in (OS_USERNAME).
user> user_name user>
API key or password. API key or password (OS_PASSWORD).
key> password_or_api_key key>
Authentication URL for server. Authentication URL for server (OS_AUTH_URL).
Choose a number from below, or type in your own value Choose a number from below, or type in your own value
1 / Rackspace US 1 / Rackspace US
\ "https://auth.api.rackspacecloud.com/v1.0" \ "https://auth.api.rackspacecloud.com/v1.0"
@ -96,24 +102,26 @@ Choose a number from below, or type in your own value
\ "https://auth.storage.memset.com/v2.0" \ "https://auth.storage.memset.com/v2.0"
6 / OVH 6 / OVH
\ "https://auth.cloud.ovh.net/v2.0" \ "https://auth.cloud.ovh.net/v2.0"
auth> 1 auth>
User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID). User ID to log in - optional - most swift systems use user and leave this blank (v3 auth) (OS_USER_ID).
user_id> user_id user_id>
User domain - optional (v3 auth) User domain - optional (v3 auth) (OS_USER_DOMAIN_NAME)
domain> Default domain>
Tenant name - optional for v1 auth, this or tenant_id required otherwise Tenant name - optional for v1 auth, this or tenant_id required otherwise (OS_TENANT_NAME or OS_PROJECT_NAME)
tenant> tenant_name tenant>
Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID) Tenant ID - optional for v1 auth, this or tenant required otherwise (OS_TENANT_ID)
tenant_id> tenant_id>
Tenant domain - optional (v3 auth) Tenant domain - optional (v3 auth) (OS_PROJECT_DOMAIN_NAME)
tenant_domain> tenant_domain>
Region name - optional Region name - optional (OS_REGION_NAME)
region> region>
Storage URL - optional Storage URL - optional (OS_STORAGE_URL)
storage_url> storage_url>
AuthVersion - optional - set to (1,2,3) if your auth URL has no version Auth Token from alternate authentication - optional (OS_AUTH_TOKEN)
auth_token>
AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)
auth_version> auth_version>
Endpoint type to choose from the service catalogue Endpoint type to choose from the service catalogue (OS_ENDPOINT_TYPE)
Choose a number from below, or type in your own value Choose a number from below, or type in your own value
1 / Public (default, choose this if not sure) 1 / Public (default, choose this if not sure)
\ "public" \ "public"
@ -124,16 +132,19 @@ Choose a number from below, or type in your own value
endpoint_type> endpoint_type>
Remote config Remote config
-------------------- --------------------
[remote] [test]
env_auth = false env_auth = true
user = user_name user =
key = password_or_api_key key =
auth = https://auth.api.rackspacecloud.com/v1.0 auth =
domain = Default user_id =
domain =
tenant = tenant =
tenant_id =
tenant_domain = tenant_domain =
region = region =
storage_url = storage_url =
auth_token =
auth_version = auth_version =
endpoint_type = endpoint_type =
-------------------- --------------------
@ -208,6 +219,17 @@ the
variables](https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment) variables](https://godoc.org/github.com/ncw/swift#Connection.ApplyEnvironment)
in the docs for the swift library. in the docs for the swift library.
### Using an alternate authentication method ###
If your OpenStack installation uses a non-standard authentication method
that might not be yet supported by rclone or the underlying swift library,
you can authenticate externally (e.g. calling manually the `openstack`
commands to get a token). Then, you just need to pass the two
configuration variables ``auth_token`` and ``storage_url``.
If they are both provided, the other variables are ignored. rclone will
not try to authenticate but instead assume it is already authenticated
and use these two variables to access the OpenStack installation.
#### Using rclone without a config file #### #### Using rclone without a config file ####
You can use rclone with swift without a config file, if desired, like You can use rclone with swift without a config file, if desired, like

View file

@ -96,6 +96,9 @@ func init() {
}, { }, {
Name: "storage_url", Name: "storage_url",
Help: "Storage URL - optional (OS_STORAGE_URL)", Help: "Storage URL - optional (OS_STORAGE_URL)",
}, {
Name: "auth_token",
Help: "Auth Token from alternate authentication - optional (OS_AUTH_TOKEN)",
}, { }, {
Name: "auth_version", Name: "auth_version",
Help: "AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)", Help: "AuthVersion - optional - set to (1,2,3) if your auth URL has no version (ST_AUTH_VERSION)",
@ -197,7 +200,10 @@ func swiftConnection(name string) (*swift.Connection, error) {
TenantId: fs.ConfigFileGet(name, "tenant_id"), TenantId: fs.ConfigFileGet(name, "tenant_id"),
TenantDomain: fs.ConfigFileGet(name, "tenant_domain"), TenantDomain: fs.ConfigFileGet(name, "tenant_domain"),
Region: fs.ConfigFileGet(name, "region"), Region: fs.ConfigFileGet(name, "region"),
// StorageUrl is set below // I get the StorageUrl already here, in case the user wants to set it manually
// (e.g. when using alternate authentication)
StorageUrl: fs.ConfigFileGet(name, "storage_url"),
AuthToken: fs.ConfigFileGet(name, "auth_token"),
AuthVersion: fs.ConfigFileGetInt(name, "auth_version", 0), AuthVersion: fs.ConfigFileGetInt(name, "auth_version", 0),
EndpointType: swift.EndpointType(fs.ConfigFileGet(name, "endpoint_type", "public")), EndpointType: swift.EndpointType(fs.ConfigFileGet(name, "endpoint_type", "public")),
ConnectTimeout: 10 * fs.Config.ConnectTimeout, // Use the timeouts in the transport ConnectTimeout: 10 * fs.Config.ConnectTimeout, // Use the timeouts in the transport
@ -210,8 +216,9 @@ func swiftConnection(name string) (*swift.Connection, error) {
return nil, errors.Wrap(err, "failed to read environment variables") return nil, errors.Wrap(err, "failed to read environment variables")
} }
} }
if !c.Authenticated() {
if c.UserName == "" && c.UserId == "" { if c.UserName == "" && c.UserId == "" {
return nil, errors.New("user name or user id not found") return nil, errors.New("user name or user id not found for authentication (and no storage_url+auth_token is provided)")
} }
if c.ApiKey == "" { if c.ApiKey == "" {
return nil, errors.New("key not found") return nil, errors.New("key not found")
@ -223,6 +230,7 @@ func swiftConnection(name string) (*swift.Connection, error) {
if err != nil { if err != nil {
return nil, err return nil, err
} }
}
return c, nil return c, nil
} }