rclone/cmd/serve/nbd/nbd.md

Run a Network Block Device server using remote:path to store the object.

You can use a unix socket by setting the url to `unix:/path/to/socket`
or just by using an absolute path name.

`rclone serve nbd` will run on any OS, but the examples for using it
are Linux specific. There do exist Windows and macOS NBD clients but
these haven't been tested yet.

To see the packets on the wire use `--dump headers` or `--dump bodies`.

**NB** this has no authentication. It may in the future allow SSL
certificates. If you need access control then you will have to provide
it on the network layer, or use unix sockets.

### remote:path pointing to a file

If the `remote:path` points to a file then rclone will serve the file
directly as a network block device.

Using this with `--read-only` is recommended. You can use any
`--vfs-cache-mode` and only parts of the file that are read will be
cached locally if using `--vfs-cache-mode full`.

If you don't use `--read-only` then `--vfs-cache-mode full` is
required and the entire file will be cached locally and won't be
uploaded until the client has disconnected (`nbd-client -d`).

### remote:path pointing to a directory

If the `remote:path` points to a directory then rclone will treat the
directory as a place to store chunks of the exported network block device.

It will store an `info.json` file in the top level and store the
individual chunks in a hierarchical directory scheme with no more than
256 chunks or directories in any directory.

The first time you use this, you should use the `--create` flag
indicating how big you want the network block device to appear. Rclone
only allocates space you use so you can make this large. For example
`--create 1T`. You can also pass the `--chunk-size` flag at this
point. If you don't you will get the default of 64k chunks.

Rclone will then chunk the network block device into `--chunk-size`
chunks. Rclone has to download the entire chunk in order to change
only part of it and it will cache the chunk on disk so bear that in
mind when choosing `--chunk-size`.

If you wish to change the size of the network block device you can use
the `--resize` flag. This won't remove any data, it just changes the
size advertised. So if you have made a file system on the block device
you will need to resize it too.

If you are using `--read-only` then you can use any
`--vfs-cache-mode`.

If you are not using `--read-only` then you will need
`--vfs-cache-mode writes` or `--vfs-cache-mode full`.

Note that rclone will be acting as a writeback cache with
`--vfs-cache-mode writes` or `--vfs-cache-mode full`. Note that rclone
will only write `--transfers` files at once so the cache can get a
backlog of uploads. You can reduce the writeback caching slightly
setting `--vfs-write-back 0`, however due to the way the kernel works,
this will only reduce it slightly.

If using `--vfs-cache-mode writes` or `--vfs-cache-mode full` it is
recommended to set limits on the cache size using some or all of these
flags as the VFS can use a lot of disk space very quickly.

    --vfs-cache-max-age duration           Max time since last access of objects in the cache (default 1h0m0s)
    --vfs-cache-max-size SizeSuffix        Max total size of objects in the cache (default off)
    --vfs-cache-min-free-space SizeSuffix  Target minimum free space on the disk containing the cache (default off)

You might also need to set this smaller as the cache will only be
examined at this interval.

    --vfs-cache-poll-interval duration     Interval to poll the cache for stale objects (default 1m0s)

### Linux Examples

Install

    sudo apt install nbd-client

Start server on localhost:10809 by default.

    rclone -v --vfs-cache-mode full serve ndb remote:path

List devices

    sudo modprobe nbd
    sudo nbd-client --list localhost

Format the partition and mount read write

    sudo nbd-client -g localhost 10809 /dev/nbd0
    sudo mkfs.ext4 /dev/nbd0
    sudo mkdir -p /mnt/tmp
    sudo mount -t ext4 /dev/nbd0 /mnt/tmp

Mount read only

    rclone -v --vfs-cache-mode full --read-only serve ndb remote:path
    sudo nbd-client --readonly -g localhost 10809 /dev/nbd0
    sudo mount -t ext4 -o ro /dev/nbd0 /mnt/tmp

Disconnect

    sudo umount /mnt/tmp
    sudo nbd-client -d /dev/nbd0

### TODO

Experiment with `-connections` option. This is supported by the code.
Does it improve performance?

       -connections num

       -C     Use  num connections to the server, to allow speeding up request
              handling, at the cost of higher resource usage  on  the  server.
              Use  of this option requires kernel support available first with
              Linux 4.9.

Experiment with `-persist` option - is that a good idea?

       -persist

       -p     When  this  option is specified, nbd-client will immediately try
              to reconnect an nbd device if the connection  ever  drops  unex‐
              pectedly due to a lost server or something similar.

Need to implement Trim and see if Trim is being called.

Need to delete zero files before upload (do in VFS layer?)

FIXME need better back pressure from VFS cache to writers.

FIXME need Sync to actually work!