571 lines
16 KiB
Markdown
571 lines
16 KiB
Markdown
---
|
|
title: "B2"
|
|
description: "Backblaze B2"
|
|
---
|
|
|
|
# {{< icon "fa fa-fire" >}} Backblaze B2
|
|
|
|
B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/).
|
|
|
|
Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
|
|
command.) You may put subdirectories in too, e.g. `remote:bucket/path/to/dir`.
|
|
|
|
## Configuration
|
|
|
|
Here is an example of making a b2 configuration. First run
|
|
|
|
rclone config
|
|
|
|
This will guide you through an interactive setup process. To authenticate
|
|
you will either need your Account ID (a short hex number) and Master
|
|
Application Key (a long hex number) OR an Application Key, which is the
|
|
recommended method. See below for further details on generating and using
|
|
an Application Key.
|
|
|
|
```
|
|
No remotes found, make a new one?
|
|
n) New remote
|
|
q) Quit config
|
|
n/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
[snip]
|
|
XX / Backblaze B2
|
|
\ "b2"
|
|
[snip]
|
|
Storage> b2
|
|
Account ID or Application Key ID
|
|
account> 123456789abc
|
|
Application Key
|
|
key> 0123456789abcdef0123456789abcdef0123456789
|
|
Endpoint for the service - leave blank normally.
|
|
endpoint>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
account = 123456789abc
|
|
key = 0123456789abcdef0123456789abcdef0123456789
|
|
endpoint =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
```
|
|
|
|
This remote is called `remote` and can now be used like this
|
|
|
|
See all buckets
|
|
|
|
rclone lsd remote:
|
|
|
|
Create a new bucket
|
|
|
|
rclone mkdir remote:bucket
|
|
|
|
List the contents of a bucket
|
|
|
|
rclone ls remote:bucket
|
|
|
|
Sync `/home/local/directory` to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
|
|
rclone sync -i /home/local/directory remote:bucket
|
|
|
|
### Application Keys
|
|
|
|
B2 supports multiple [Application Keys for different access permission
|
|
to B2 Buckets](https://www.backblaze.com/b2/docs/application_keys.html).
|
|
|
|
You can use these with rclone too; you will need to use rclone version 1.43
|
|
or later.
|
|
|
|
Follow Backblaze's docs to create an Application Key with the required
|
|
permission and add the `applicationKeyId` as the `account` and the
|
|
`Application Key` itself as the `key`.
|
|
|
|
Note that you must put the _applicationKeyId_ as the `account` – you
|
|
can't use the master Account ID. If you try then B2 will return 401
|
|
errors.
|
|
|
|
### --fast-list
|
|
|
|
This remote supports `--fast-list` which allows you to use fewer
|
|
transactions in exchange for more memory. See the [rclone
|
|
docs](/docs/#fast-list) for more details.
|
|
|
|
### Modified time
|
|
|
|
The modified time is stored as metadata on the object as
|
|
`X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01
|
|
in the Backblaze standard. Other tools should be able to use this as
|
|
a modified time.
|
|
|
|
Modified times are used in syncing and are fully supported. Note that
|
|
if a modification time needs to be updated on an object then it will
|
|
create a new version of the object.
|
|
|
|
### Restricted filename characters
|
|
|
|
In addition to the [default restricted characters set](/overview/#restricted-characters)
|
|
the following characters are also replaced:
|
|
|
|
| Character | Value | Replacement |
|
|
| --------- |:-----:|:-----------:|
|
|
| \ | 0x5C | ＼ |
|
|
|
|
Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8),
|
|
as they can't be used in JSON strings.
|
|
|
|
Note that in 2020-05 Backblaze started allowing \ characters in file
|
|
names. Rclone hasn't changed its encoding as this could cause syncs to
|
|
re-transfer files. If you want rclone not to replace \ then see the
|
|
`--b2-encoding` flag below and remove the `BackSlash` from the
|
|
string. This can be set in the config.
|
|
|
|
### SHA1 checksums
|
|
|
|
The SHA1 checksums of the files are checked on upload and download and
|
|
will be used in the syncing process.
|
|
|
|
Large files (bigger than the limit in `--b2-upload-cutoff`) which are
|
|
uploaded in chunks will store their SHA1 on the object as
|
|
`X-Bz-Info-large_file_sha1` as recommended by Backblaze.
|
|
|
|
For a large file to be uploaded with an SHA1 checksum, the source
|
|
needs to support SHA1 checksums. The local disk supports SHA1
|
|
checksums so large file transfers from local disk will have an SHA1.
|
|
See [the overview](/overview/#features) for exactly which remotes
|
|
support SHA1.
|
|
|
|
Sources which don't support SHA1, in particular `crypt` will upload
|
|
large files without SHA1 checksums. This may be fixed in the future
|
|
(see [#1767](https://github.com/rclone/rclone/issues/1767)).
|
|
|
|
Files sizes below `--b2-upload-cutoff` will always have an SHA1
|
|
regardless of the source.
|
|
|
|
### Transfers
|
|
|
|
Backblaze recommends that you do lots of transfers simultaneously for
|
|
maximum speed. In tests from my SSD equipped laptop the optimum
|
|
setting is about `--transfers 32` though higher numbers may be used
|
|
for a slight speed improvement. The optimum number for you may vary
|
|
depending on your hardware, how big the files are, how much you want
|
|
to load your computer, etc. The default of `--transfers 4` is
|
|
definitely too low for Backblaze B2 though.
|
|
|
|
Note that uploading big files (bigger than 200 MiB by default) will use
|
|
a 96 MiB RAM buffer by default. There can be at most `--transfers` of
|
|
these in use at any moment, so this sets the upper limit on the memory
|
|
used.
|
|
|
|
### Versions
|
|
|
|
When rclone uploads a new version of a file it creates a [new version
|
|
of it](https://www.backblaze.com/b2/docs/file_versions.html).
|
|
Likewise when you delete a file, the old version will be marked hidden
|
|
and still be available. Conversely, you may opt in to a "hard delete"
|
|
of files with the `--b2-hard-delete` flag which would permanently remove
|
|
the file instead of hiding it.
|
|
|
|
Old versions of files, where available, are visible using the
|
|
`--b2-versions` flag.
|
|
|
|
It is also possible to view a bucket as it was at a certain point in time,
|
|
using the `--b2-version-at` flag. This will show the file versions as they
|
|
were at that time, showing files that have been deleted afterwards, and
|
|
hiding files that were created since.
|
|
|
|
If you wish to remove all the old versions then you can use the
|
|
`rclone cleanup remote:bucket` command which will delete all the old
|
|
versions of files, leaving the current ones intact. You can also
|
|
supply a path and only old versions under that path will be deleted,
|
|
e.g. `rclone cleanup remote:bucket/path/to/stuff`.
|
|
|
|
Note that `cleanup` will remove partially uploaded files from the bucket
|
|
if they are more than a day old.
|
|
|
|
When you `purge` a bucket, the current and the old versions will be
|
|
deleted then the bucket will be deleted.
|
|
|
|
However `delete` will cause the current versions of the files to
|
|
become hidden old versions.
|
|
|
|
Here is a session showing the listing and retrieval of an old
|
|
version followed by a `cleanup` of the old versions.
|
|
|
|
Show current version and all the versions with `--b2-versions` flag.
|
|
|
|
```
|
|
$ rclone -q ls b2:cleanup-test
|
|
9 one.txt
|
|
|
|
$ rclone -q --b2-versions ls b2:cleanup-test
|
|
9 one.txt
|
|
8 one-v2016-07-04-141032-000.txt
|
|
16 one-v2016-07-04-141003-000.txt
|
|
15 one-v2016-07-02-155621-000.txt
|
|
```
|
|
|
|
Retrieve an old version
|
|
|
|
```
|
|
$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
|
|
|
|
$ ls -l /tmp/one-v2016-07-04-141003-000.txt
|
|
-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
|
|
```
|
|
|
|
Clean up all the old versions and show that they've gone.
|
|
|
|
```
|
|
$ rclone -q cleanup b2:cleanup-test
|
|
|
|
$ rclone -q ls b2:cleanup-test
|
|
9 one.txt
|
|
|
|
$ rclone -q --b2-versions ls b2:cleanup-test
|
|
9 one.txt
|
|
```
|
|
|
|
### Data usage
|
|
|
|
It is useful to know how many requests are sent to the server in different scenarios.
|
|
|
|
All copy commands send the following 4 requests:
|
|
|
|
```
|
|
/b2api/v1/b2_authorize_account
|
|
/b2api/v1/b2_create_bucket
|
|
/b2api/v1/b2_list_buckets
|
|
/b2api/v1/b2_list_file_names
|
|
```
|
|
|
|
The `b2_list_file_names` request will be sent once for every 1k files
|
|
in the remote path, providing the checksum and modification time of
|
|
the listed files. As of version 1.33 issue
|
|
[#818](https://github.com/rclone/rclone/issues/818) causes extra requests
|
|
to be sent when using B2 with Crypt. When a copy operation does not
|
|
require any files to be uploaded, no more requests will be sent.
|
|
|
|
Uploading files that do not require chunking, will send 2 requests per
|
|
file upload:
|
|
|
|
```
|
|
/b2api/v1/b2_get_upload_url
|
|
/b2api/v1/b2_upload_file/
|
|
```
|
|
|
|
Uploading files requiring chunking, will send 2 requests (one each to
|
|
start and finish the upload) and another 2 requests for each chunk:
|
|
|
|
```
|
|
/b2api/v1/b2_start_large_file
|
|
/b2api/v1/b2_get_upload_part_url
|
|
/b2api/v1/b2_upload_part/
|
|
/b2api/v1/b2_finish_large_file
|
|
```
|
|
|
|
#### Versions
|
|
|
|
Versions can be viewed with the `--b2-versions` flag. When it is set
|
|
rclone will show and act on older versions of files. For example
|
|
|
|
Listing without `--b2-versions`
|
|
|
|
```
|
|
$ rclone -q ls b2:cleanup-test
|
|
9 one.txt
|
|
```
|
|
|
|
And with
|
|
|
|
```
|
|
$ rclone -q --b2-versions ls b2:cleanup-test
|
|
9 one.txt
|
|
8 one-v2016-07-04-141032-000.txt
|
|
16 one-v2016-07-04-141003-000.txt
|
|
15 one-v2016-07-02-155621-000.txt
|
|
```
|
|
|
|
Showing that the current version is unchanged but older versions can
|
|
be seen. These have the UTC date that they were uploaded to the
|
|
server to the nearest millisecond appended to them.
|
|
|
|
Note that when using `--b2-versions` no file write operations are
|
|
permitted, so you can't upload files or delete them.
|
|
|
|
### B2 and rclone link
|
|
|
|
Rclone supports generating file share links for private B2 buckets.
|
|
They can either be for a file for example:
|
|
|
|
```
|
|
./rclone link B2:bucket/path/to/file.txt
|
|
https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx
|
|
|
|
```
|
|
|
|
or if run on a directory you will get:
|
|
|
|
```
|
|
./rclone link B2:bucket/path
|
|
https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx
|
|
```
|
|
|
|
you can then use the authorization token (the part of the url from the
|
|
`?Authorization=` on) on any file path under that directory. For example:
|
|
|
|
```
|
|
https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx
|
|
https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx
|
|
https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx
|
|
|
|
```
|
|
|
|
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/b2/b2.go then run make backenddocs" >}}
|
|
### Standard options
|
|
|
|
Here are the standard options specific to b2 (Backblaze B2).
|
|
|
|
#### --b2-account
|
|
|
|
Account ID or Application Key ID.
|
|
|
|
Properties:
|
|
|
|
- Config: account
|
|
- Env Var: RCLONE_B2_ACCOUNT
|
|
- Type: string
|
|
- Required: true
|
|
|
|
#### --b2-key
|
|
|
|
Application Key.
|
|
|
|
Properties:
|
|
|
|
- Config: key
|
|
- Env Var: RCLONE_B2_KEY
|
|
- Type: string
|
|
- Required: true
|
|
|
|
#### --b2-hard-delete
|
|
|
|
Permanently delete files on remote removal, otherwise hide files.
|
|
|
|
Properties:
|
|
|
|
- Config: hard_delete
|
|
- Env Var: RCLONE_B2_HARD_DELETE
|
|
- Type: bool
|
|
- Default: false
|
|
|
|
### Advanced options
|
|
|
|
Here are the advanced options specific to b2 (Backblaze B2).
|
|
|
|
#### --b2-endpoint
|
|
|
|
Endpoint for the service.
|
|
|
|
Leave blank normally.
|
|
|
|
Properties:
|
|
|
|
- Config: endpoint
|
|
- Env Var: RCLONE_B2_ENDPOINT
|
|
- Type: string
|
|
- Required: false
|
|
|
|
#### --b2-test-mode
|
|
|
|
A flag string for X-Bz-Test-Mode header for debugging.
|
|
|
|
This is for debugging purposes only. Setting it to one of the strings
|
|
below will cause b2 to return specific errors:
|
|
|
|
* "fail_some_uploads"
|
|
* "expire_some_account_authorization_tokens"
|
|
* "force_cap_exceeded"
|
|
|
|
These will be set in the "X-Bz-Test-Mode" header which is documented
|
|
in the [b2 integrations checklist](https://www.backblaze.com/b2/docs/integration_checklist.html).
|
|
|
|
Properties:
|
|
|
|
- Config: test_mode
|
|
- Env Var: RCLONE_B2_TEST_MODE
|
|
- Type: string
|
|
- Required: false
|
|
|
|
#### --b2-versions
|
|
|
|
Include old versions in directory listings.
|
|
|
|
Note that when using this no file write operations are permitted,
|
|
so you can't upload files or delete them.
|
|
|
|
Properties:
|
|
|
|
- Config: versions
|
|
- Env Var: RCLONE_B2_VERSIONS
|
|
- Type: bool
|
|
- Default: false
|
|
|
|
#### --b2-upload-cutoff
|
|
|
|
Cutoff for switching to chunked upload.
|
|
|
|
Files above this size will be uploaded in chunks of "--b2-chunk-size".
|
|
|
|
This value should be set no larger than 4.657 GiB (== 5 GB).
|
|
|
|
Properties:
|
|
|
|
- Config: upload_cutoff
|
|
- Env Var: RCLONE_B2_UPLOAD_CUTOFF
|
|
- Type: SizeSuffix
|
|
- Default: 200Mi
|
|
|
|
#### --b2-copy-cutoff
|
|
|
|
Cutoff for switching to multipart copy.
|
|
|
|
Any files larger than this that need to be server-side copied will be
|
|
copied in chunks of this size.
|
|
|
|
The minimum is 0 and the maximum is 4.6 GiB.
|
|
|
|
Properties:
|
|
|
|
- Config: copy_cutoff
|
|
- Env Var: RCLONE_B2_COPY_CUTOFF
|
|
- Type: SizeSuffix
|
|
- Default: 4Gi
|
|
|
|
#### --b2-chunk-size
|
|
|
|
Upload chunk size.
|
|
|
|
When uploading large files, chunk the file into this size.
|
|
|
|
Must fit in memory. These chunks are buffered in memory and there
|
|
might a maximum of "--transfers" chunks in progress at once.
|
|
|
|
5,000,000 Bytes is the minimum size.
|
|
|
|
Properties:
|
|
|
|
- Config: chunk_size
|
|
- Env Var: RCLONE_B2_CHUNK_SIZE
|
|
- Type: SizeSuffix
|
|
- Default: 96Mi
|
|
|
|
#### --b2-disable-checksum
|
|
|
|
Disable checksums for large (> upload cutoff) files.
|
|
|
|
Normally rclone will calculate the SHA1 checksum of the input before
|
|
uploading it so it can add it to metadata on the object. This is great
|
|
for data integrity checking but can cause long delays for large files
|
|
to start uploading.
|
|
|
|
Properties:
|
|
|
|
- Config: disable_checksum
|
|
- Env Var: RCLONE_B2_DISABLE_CHECKSUM
|
|
- Type: bool
|
|
- Default: false
|
|
|
|
#### --b2-download-url
|
|
|
|
Custom endpoint for downloads.
|
|
|
|
This is usually set to a Cloudflare CDN URL as Backblaze offers
|
|
free egress for data downloaded through the Cloudflare network.
|
|
Rclone works with private buckets by sending an "Authorization" header.
|
|
If the custom endpoint rewrites the requests for authentication,
|
|
e.g., in Cloudflare Workers, this header needs to be handled properly.
|
|
Leave blank if you want to use the endpoint provided by Backblaze.
|
|
|
|
The URL provided here SHOULD have the protocol and SHOULD NOT have
|
|
a trailing slash or specify the /file/bucket subpath as rclone will
|
|
request files with "{download_url}/file/{bucket_name}/{path}".
|
|
|
|
Example:
|
|
> https://mysubdomain.mydomain.tld
|
|
(No trailing "/", "file" or "bucket")
|
|
|
|
Properties:
|
|
|
|
- Config: download_url
|
|
- Env Var: RCLONE_B2_DOWNLOAD_URL
|
|
- Type: string
|
|
- Required: false
|
|
|
|
#### --b2-download-auth-duration
|
|
|
|
Time before the authorization token will expire in s or suffix ms|s|m|h|d.
|
|
|
|
The duration before the download authorization token will expire.
|
|
The minimum value is 1 second. The maximum value is one week.
|
|
|
|
Properties:
|
|
|
|
- Config: download_auth_duration
|
|
- Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION
|
|
- Type: Duration
|
|
- Default: 1w
|
|
|
|
#### --b2-memory-pool-flush-time
|
|
|
|
How often internal memory buffer pools will be flushed.
|
|
Uploads which requires additional buffers (f.e multipart) will use memory pool for allocations.
|
|
This option controls how often unused buffers will be removed from the pool.
|
|
|
|
Properties:
|
|
|
|
- Config: memory_pool_flush_time
|
|
- Env Var: RCLONE_B2_MEMORY_POOL_FLUSH_TIME
|
|
- Type: Duration
|
|
- Default: 1m0s
|
|
|
|
#### --b2-memory-pool-use-mmap
|
|
|
|
Whether to use mmap buffers in internal memory pool.
|
|
|
|
Properties:
|
|
|
|
- Config: memory_pool_use_mmap
|
|
- Env Var: RCLONE_B2_MEMORY_POOL_USE_MMAP
|
|
- Type: bool
|
|
- Default: false
|
|
|
|
#### --b2-encoding
|
|
|
|
The encoding for the backend.
|
|
|
|
See the [encoding section in the overview](/overview/#encoding) for more info.
|
|
|
|
Properties:
|
|
|
|
- Config: encoding
|
|
- Env Var: RCLONE_B2_ENCODING
|
|
- Type: MultiEncoder
|
|
- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
|
|
|
|
{{< rem autogenerated options stop >}}
|
|
|
|
## Limitations
|
|
|
|
`rclone about` is not supported by the B2 backend. Backends without
|
|
this capability cannot determine free space for an rclone mount or
|
|
use policy `mfs` (most free space) as a member of an rclone union
|
|
remote.
|
|
|
|
See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
|
|
|
|
|