c665201b85
This patch provides the support of synchronous cache space recovery to allow read threads to recover from ENOSPC errors when cache space can be recovered from cache items that are not in use or safe to be reset/emptied . The patch complements the existing cache cleaning process in two ways. Firstly, the existing cache cleaning process is time-driven that runs periodically. The cache space can run out while the cache cleaner thread is still waiting for its next scheduled run. The io threads encountering ENOSPC return an internal error to the applications in this case even when cache space can be recovered to avoid this error. This patch addresses this problem by having the read threads kick the cache cleaner thread in this condition to recover cache space preventing unnecessary ENOSPC errors from being seen by the applications. Secondly, this patch enhances the cache cleaner to support cache item reset. Currently the cache purge process removes cache items that are not in use. This may not be sufficient when the total size of the working set exceeds the cache directory's capacity. Like in the current code, this patch starts the purge process by removing cache files that are not in use. Cache items whose access times are older than vfs-cache-max-age are removed first. After that, other not-in-use items are removed in LRU order until vfs-cache-max-size is reached. If the vfs-cache-max-size (the quota) is still not reached at this time, this patch adds a cache reset step to reset/empty cache files that are still in use but not dirtied. This enables application processes to continue without seeing an error even when the working set depletes the cache space as long as there is not a large write working set hoarding the entire cache space. By design this patch does not add ENOSPC error recovery for write IOs. Rclone does not empty a write cache item until the file data is written back to the backend upon close. Allowing more cache space to be consumed by dirty cache items when the cache space is already running low would increase the risk of exhausting the cache space in a way that the vfs mount becomes unreadable.
74 lines
3.2 KiB
Modula-2
74 lines
3.2 KiB
Modula-2
module github.com/rclone/rclone
|
|
|
|
go 1.14
|
|
|
|
require (
|
|
bazil.org/fuse v0.0.0-20200407214033-5883e5a4b512
|
|
cloud.google.com/go v0.59.0 // indirect
|
|
github.com/Azure/azure-pipeline-go v0.2.2
|
|
github.com/Azure/azure-storage-blob-go v0.10.0
|
|
github.com/Unknwon/goconfig v0.0.0-20191126170842-860a72fb44fd
|
|
github.com/a8m/tree v0.0.0-20181222104329-6a0b80129de4
|
|
github.com/aalpar/deheap v0.0.0-20200318053559-9a0c2883bd56
|
|
github.com/abbot/go-http-auth v0.4.0
|
|
github.com/anacrolix/dms v1.1.0
|
|
github.com/atotto/clipboard v0.1.2
|
|
github.com/aws/aws-sdk-go v1.32.11
|
|
github.com/billziss-gh/cgofuse v1.4.0
|
|
github.com/btcsuite/btcutil v1.0.2 // indirect
|
|
github.com/calebcase/tmpfile v1.0.2 // indirect
|
|
github.com/coreos/go-semver v0.3.0
|
|
github.com/dropbox/dropbox-sdk-go-unofficial v5.6.0+incompatible
|
|
github.com/gogo/protobuf v1.3.1 // indirect
|
|
github.com/google/go-querystring v1.0.0 // indirect
|
|
github.com/hanwen/go-fuse/v2 v2.0.3
|
|
github.com/jlaffaye/ftp v0.0.0-20200720194710-13949d38913e
|
|
github.com/jzelinskie/whirlpool v0.0.0-20170603002051-c19460b8caa6
|
|
github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0 // indirect
|
|
github.com/klauspost/compress v1.10.11
|
|
github.com/koofr/go-httpclient v0.0.0-20200420163713-93aa7c75b348
|
|
github.com/koofr/go-koofrclient v0.0.0-20190724113126-8e5366da203a
|
|
github.com/mattn/go-colorable v0.1.7
|
|
github.com/mattn/go-ieproxy v0.0.1 // indirect
|
|
github.com/mattn/go-runewidth v0.0.9
|
|
github.com/mitchellh/go-homedir v1.1.0
|
|
github.com/ncw/go-acd v0.0.0-20171120105400-887eb06ab6a2
|
|
github.com/ncw/swift v1.0.52
|
|
github.com/nsf/termbox-go v0.0.0-20200418040025-38ba6e5628f1
|
|
github.com/okzk/sdnotify v0.0.0-20180710141335-d9becc38acbd
|
|
github.com/patrickmn/go-cache v2.1.0+incompatible
|
|
github.com/pkg/errors v0.9.1
|
|
github.com/pkg/sftp v1.11.0
|
|
github.com/prometheus/client_golang v1.7.1
|
|
github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8
|
|
github.com/rfjakob/eme v1.1.1
|
|
github.com/sevlyar/go-daemon v0.1.5
|
|
github.com/sirupsen/logrus v1.6.0
|
|
github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966
|
|
github.com/spf13/cobra v1.0.0
|
|
github.com/spf13/pflag v1.0.5
|
|
github.com/stretchr/testify v1.6.1
|
|
github.com/t3rm1n4l/go-mega v0.0.0-20200416171014-ffad7fcb44b8
|
|
github.com/xanzy/ssh-agent v0.2.1
|
|
github.com/youmark/pkcs8 v0.0.0-20200520070018-fad002e585ce
|
|
github.com/yunify/qingstor-sdk-go/v3 v3.2.0
|
|
go.etcd.io/bbolt v1.3.5
|
|
go.opencensus.io v0.22.4 // indirect
|
|
go.uber.org/zap v1.15.0 // indirect
|
|
goftp.io/server v0.4.0
|
|
golang.org/x/crypto v0.0.0-20200709230013-948cd5f35899
|
|
golang.org/x/net v0.0.0-20200707034311-ab3426394381
|
|
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
|
|
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208
|
|
golang.org/x/sys v0.0.0-20200720211630-cb9d2d5c5666
|
|
golang.org/x/text v0.3.3
|
|
golang.org/x/time v0.0.0-20200416051211-89c76fbcd5d1
|
|
golang.org/x/tools v0.0.0-20200820180210-c8f393745106 // indirect
|
|
google.golang.org/api v0.28.0
|
|
google.golang.org/genproto v0.0.0-20200626011028-ee7919e894b5 // indirect
|
|
google.golang.org/grpc v1.30.0 // indirect
|
|
google.golang.org/protobuf v1.25.0 // indirect
|
|
gopkg.in/yaml.v2 v2.3.0
|
|
gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776 // indirect
|
|
storj.io/uplink v1.2.0
|
|
)
|