rclone/backend/swift
Paul Collins bd5799c079 swift: add workarounds for bad listings in Ceph RGW
Ceph's Swift API emulation does not fully confirm to the API spec.
As a result, it sometimes returns fewer items in a container than
the requested limit, which according to the spec should means
that there are no more objects left in the container.  (Note that
python-swiftclient always fetches unless the current page is empty.)

This commit adds a pair of new Swift backend settings to handle this.

Set `fetch_until_empty_page` to true to always fetch another
page of the container listing unless there are no items left.

Alternatively, set `partial_page_fetch_threshold` to an integer
percentage.  In this case rclone will fetch a new page only when
the current page is within this percentage of the limit.

Swift API reference: https://docs.openstack.org/swift/latest/api/pagination.html

PR against ncw/swift with research and discussion: https://github.com/ncw/swift/pull/167

Fixes #7924
2024-06-28 11:14:26 +01:00
..
auth.go swift: update github.com/ncw/swift to v2.0.0 2021-02-03 20:23:37 +00:00
swift.go swift: add workarounds for bad listings in Ceph RGW 2024-06-28 11:14:26 +01:00
swift_internal_test.go all: fix spelling across the project 2022-08-30 11:16:26 +02:00
swift_test.go swift: implement --swift-use-segments-container to allow >5G files on Blomp 2024-04-25 11:14:14 +01:00