From 5b0f9dc4e3358b6c8af6d997fdbd5f3fb4551e08 Mon Sep 17 00:00:00 2001 From: Nick Craig-Wood Date: Sat, 14 Oct 2023 12:39:48 +0100 Subject: [PATCH] local: fix copying from Windows Volume Shadows For some files the Windows Volume Shadow Service (VSS) advertises the file size as X in the directory listing but returns a different number Y on stat-ing the file. If the file is opened and read there are Y bytes available for reading. Existing copy tools copy Y bytes rather than X so for consistency rclone should do the same. This fixes the problem by stat-ing the file immediately before opening it. This will also reduce the unnecessary occurrence of "can't copy - source file is being updated" errors; if the file has finished changing by the time we come to copy it then we now can copy it successfully. See: https://forum.rclone.org/t/consistently-getting-corrupted-on-transfer-sizes-differ-syncing-to-an-smb-share/42218/ --- backend/local/local.go | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/backend/local/local.go b/backend/local/local.go index d1a613f5f..e10980a3c 100644 --- a/backend/local/local.go +++ b/backend/local/local.go @@ -146,6 +146,11 @@ time we: - Only checksum the size that stat gave - Don't update the stat info for the file +**NB** do not use this flag on a Windows Volume Shadow (VSS). For some +unknown reason, files in a VSS sometimes show different sizes from the +directory listing (where the initial stat value comes from on Windows) +and when stat is called on them directly. Other copy tools always use +the direct stat value and setting this flag will disable that. `, Default: false, Advanced: true, @@ -1123,6 +1128,12 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read } } + // Update the file info before we start reading + err = o.lstat() + if err != nil { + return nil, err + } + // If not checking updated then limit to current size. This means if // file is being extended, readers will read a o.Size() bytes rather // than the new size making for a consistent upload.