297 lines
8.3 KiB
Markdown
297 lines
8.3 KiB
Markdown
---
|
|
title: "B2"
|
|
description: "Backblaze B2"
|
|
date: "2016-10-25"
|
|
---
|
|
|
|
<i class="fa fa-fire"></i>Backblaze B2
|
|
----------------------------------------
|
|
|
|
B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/).
|
|
|
|
Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
|
|
command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`.
|
|
|
|
Here is an example of making a b2 configuration. First run
|
|
|
|
rclone config
|
|
|
|
This will guide you through an interactive setup process. You will
|
|
need your account number (a short hex number) and key (a long hex
|
|
number) which you can get from the b2 control panel.
|
|
|
|
```
|
|
No remotes found - make a new one
|
|
n) New remote
|
|
q) Quit config
|
|
n/q> n
|
|
name> remote
|
|
Type of storage to configure.
|
|
Choose a number from below, or type in your own value
|
|
1 / Amazon Drive
|
|
\ "amazon cloud drive"
|
|
2 / Amazon S3 (also Dreamhost, Ceph, Minio)
|
|
\ "s3"
|
|
3 / Backblaze B2
|
|
\ "b2"
|
|
4 / Dropbox
|
|
\ "dropbox"
|
|
5 / Encrypt/Decrypt a remote
|
|
\ "crypt"
|
|
6 / Google Cloud Storage (this is not Google Drive)
|
|
\ "google cloud storage"
|
|
7 / Google Drive
|
|
\ "drive"
|
|
8 / Hubic
|
|
\ "hubic"
|
|
9 / Local Disk
|
|
\ "local"
|
|
10 / Microsoft OneDrive
|
|
\ "onedrive"
|
|
11 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
|
|
\ "swift"
|
|
12 / Yandex Disk
|
|
\ "yandex"
|
|
Storage> 3
|
|
Account ID
|
|
account> 123456789abc
|
|
Application Key
|
|
key> 0123456789abcdef0123456789abcdef0123456789
|
|
Endpoint for the service - leave blank normally.
|
|
endpoint>
|
|
Remote config
|
|
--------------------
|
|
[remote]
|
|
account = 123456789abc
|
|
key = 0123456789abcdef0123456789abcdef0123456789
|
|
endpoint =
|
|
--------------------
|
|
y) Yes this is OK
|
|
e) Edit this remote
|
|
d) Delete this remote
|
|
y/e/d> y
|
|
```
|
|
|
|
This remote is called `remote` and can now be used like this
|
|
|
|
See all buckets
|
|
|
|
rclone lsd remote:
|
|
|
|
Make a new bucket
|
|
|
|
rclone mkdir remote:bucket
|
|
|
|
List the contents of a bucket
|
|
|
|
rclone ls remote:bucket
|
|
|
|
Sync `/home/local/directory` to the remote bucket, deleting any
|
|
excess files in the bucket.
|
|
|
|
rclone sync /home/local/directory remote:bucket
|
|
|
|
### Modified time ###
|
|
|
|
The modified time is stored as metadata on the object as
|
|
`X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01
|
|
in the Backblaze standard. Other tools should be able to use this as
|
|
a modified time.
|
|
|
|
Modified times are used in syncing and are fully supported except in
|
|
the case of updating a modification time on an existing object. In
|
|
this case the object will be uploaded again as B2 doesn't have an API
|
|
method to set the modification time independent of doing an upload.
|
|
|
|
### SHA1 checksums ###
|
|
|
|
The SHA1 checksums of the files are checked on upload and download and
|
|
will be used in the syncing process.
|
|
|
|
Large files which are uploaded in chunks will store their SHA1 on the
|
|
object as `X-Bz-Info-large_file_sha1` as recommended by Backblaze.
|
|
|
|
### Transfers ###
|
|
|
|
Backblaze recommends that you do lots of transfers simultaneously for
|
|
maximum speed. In tests from my SSD equiped laptop the optimum
|
|
setting is about `--transfers 32` though higher numbers may be used
|
|
for a slight speed improvement. The optimum number for you may vary
|
|
depending on your hardware, how big the files are, how much you want
|
|
to load your computer, etc. The default of `--transfers 4` is
|
|
definitely too low for Backblaze B2 though.
|
|
|
|
Note that uploading big files (bigger than 200 MB by default) will use
|
|
a 96 MB RAM buffer by default. There can be at most `--transfers` of
|
|
these in use at any moment, so this sets the upper limit on the memory
|
|
used.
|
|
|
|
### Versions ###
|
|
|
|
When rclone uploads a new version of a file it creates a [new version
|
|
of it](https://www.backblaze.com/b2/docs/file_versions.html).
|
|
Likewise when you delete a file, the old version will still be
|
|
available.
|
|
|
|
Old versions of files are visible using the `--b2-versions` flag.
|
|
|
|
If you wish to remove all the old versions then you can use the
|
|
`rclone cleanup remote:bucket` command which will delete all the old
|
|
versions of files, leaving the current ones intact. You can also
|
|
supply a path and only old versions under that path will be deleted,
|
|
eg `rclone cleanup remote:bucket/path/to/stuff`.
|
|
|
|
When you `purge` a bucket, the current and the old versions will be
|
|
deleted then the bucket will be deleted.
|
|
|
|
However `delete` will cause the current versions of the files to
|
|
become hidden old versions.
|
|
|
|
Here is a session showing the listing and and retreival of an old
|
|
version followed by a `cleanup` of the old versions.
|
|
|
|
Show current version and all the versions with `--b2-versions` flag.
|
|
|
|
```
|
|
$ rclone -q ls b2:cleanup-test
|
|
9 one.txt
|
|
|
|
$ rclone -q --b2-versions ls b2:cleanup-test
|
|
9 one.txt
|
|
8 one-v2016-07-04-141032-000.txt
|
|
16 one-v2016-07-04-141003-000.txt
|
|
15 one-v2016-07-02-155621-000.txt
|
|
```
|
|
|
|
Retreive an old verson
|
|
|
|
```
|
|
$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
|
|
|
|
$ ls -l /tmp/one-v2016-07-04-141003-000.txt
|
|
-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
|
|
```
|
|
|
|
Clean up all the old versions and show that they've gone.
|
|
|
|
```
|
|
$ rclone -q cleanup b2:cleanup-test
|
|
|
|
$ rclone -q ls b2:cleanup-test
|
|
9 one.txt
|
|
|
|
$ rclone -q --b2-versions ls b2:cleanup-test
|
|
9 one.txt
|
|
```
|
|
|
|
### Data usage ###
|
|
|
|
It is useful to know how many requests are sent to the server in different scenarios.
|
|
|
|
All copy commands send the following 4 requests:
|
|
|
|
```
|
|
/b2api/v1/b2_authorize_account
|
|
/b2api/v1/b2_create_bucket
|
|
/b2api/v1/b2_list_buckets
|
|
/b2api/v1/b2_list_file_names
|
|
```
|
|
|
|
The `b2_list_file_names` request will be sent once for every 1k files
|
|
in the remote path, providing the checksum and modification time of
|
|
the listed files. As of version 1.33 issue
|
|
[#818](https://github.com/ncw/rclone/issues/818) causes extra requests
|
|
to be sent when using B2 with Crypt. When a copy operation does not
|
|
require any files to be uploaded, no more requests will be sent.
|
|
|
|
Uploading files that do not require chunking, will send 2 requests per
|
|
file upload:
|
|
|
|
```
|
|
/b2api/v1/b2_get_upload_url
|
|
/b2api/v1/b2_upload_file/
|
|
```
|
|
|
|
Uploading files requiring chunking, will send 2 requests (one each to
|
|
start and finish the upload) and another 2 requests for each chunk:
|
|
|
|
```
|
|
/b2api/v1/b2_start_large_file
|
|
/b2api/v1/b2_get_upload_part_url
|
|
/b2api/v1/b2_upload_part/
|
|
/b2api/v1/b2_finish_large_file
|
|
```
|
|
|
|
### B2 with crypt ###
|
|
|
|
When using B2 with `crypt` files are encrypted into a temporary
|
|
location and streamed from there. This is required to calculate the
|
|
encrypted file's checksum before beginning the upload. On Windows the
|
|
%TMPDIR% environment variable is used as the temporary location. If
|
|
the file requires chunking, both the chunking and encryption will take
|
|
place in memory.
|
|
|
|
### Specific options ###
|
|
|
|
Here are the command line options specific to this cloud storage
|
|
system.
|
|
|
|
#### --b2-chunk-size valuee=SIZE ####
|
|
|
|
When uploading large files chunk the file into this size. Note that
|
|
these chunks are buffered in memory and there might a maximum of
|
|
`--transfers` chunks in progress at once. 100,000,000 Bytes is the
|
|
minimim size (default 96M).
|
|
|
|
#### --b2-upload-cutoff=SIZE ####
|
|
|
|
Cutoff for switching to chunked upload (default 190.735 MiB == 200
|
|
MB). Files above this size will be uploaded in chunks of
|
|
`--b2-chunk-size`.
|
|
|
|
This value should be set no larger than 4.657GiB (== 5GB) as this is
|
|
the largest file size that can be uploaded.
|
|
|
|
|
|
#### --b2-test-mode=FLAG ####
|
|
|
|
This is for debugging purposes only.
|
|
|
|
Setting FLAG to one of the strings below will cause b2 to return
|
|
specific errors for debugging purposes.
|
|
|
|
* `fail_some_uploads`
|
|
* `expire_some_account_authorization_tokens`
|
|
* `force_cap_exceeded`
|
|
|
|
These will be set in the `X-Bz-Test-Mode` header which is documented
|
|
in the [b2 integrations
|
|
checklist](https://www.backblaze.com/b2/docs/integration_checklist.html).
|
|
|
|
#### --b2-versions ####
|
|
|
|
When set rclone will show and act on older versions of files. For example
|
|
|
|
Listing without `--b2-versions`
|
|
|
|
```
|
|
$ rclone -q ls b2:cleanup-test
|
|
9 one.txt
|
|
```
|
|
|
|
And with
|
|
|
|
```
|
|
$ rclone -q --b2-versions ls b2:cleanup-test
|
|
9 one.txt
|
|
8 one-v2016-07-04-141032-000.txt
|
|
16 one-v2016-07-04-141003-000.txt
|
|
15 one-v2016-07-02-155621-000.txt
|
|
```
|
|
|
|
Showing that the current version is unchanged but older versions can
|
|
be seen. These have the UTC date that they were uploaded to the
|
|
server to the nearest millisecond appended to them.
|
|
|
|
Note that when using `--b2-versions` no file write operations are
|
|
permitted, so you can't upload files or delete them.
|