local: fix copying from Windows Volume Shadows

For some files the Windows Volume Shadow Service (VSS) advertises the
file size as X in the directory listing but returns a different number
Y on stat-ing the file. If the file is opened and read there are Y
bytes available for reading.

Existing copy tools copy Y bytes rather than X so for consistency
rclone should do the same.

This fixes the problem by stat-ing the file immediately before opening
it. This will also reduce the unnecessary occurrence of "can't copy -
source file is being updated" errors; if the file has finished
changing by the time we come to copy it then we now can copy it
successfully.

See: https://forum.rclone.org/t/consistently-getting-corrupted-on-transfer-sizes-differ-syncing-to-an-smb-share/42218/
This commit is contained in:
Nick Craig-Wood 2023-10-14 12:39:48 +01:00
parent b0a87d7cf1
commit 5b0f9dc4e3

View file

@ -146,6 +146,11 @@ time we:
- Only checksum the size that stat gave - Only checksum the size that stat gave
- Don't update the stat info for the file - Don't update the stat info for the file
**NB** do not use this flag on a Windows Volume Shadow (VSS). For some
unknown reason, files in a VSS sometimes show different sizes from the
directory listing (where the initial stat value comes from on Windows)
and when stat is called on them directly. Other copy tools always use
the direct stat value and setting this flag will disable that.
`, `,
Default: false, Default: false,
Advanced: true, Advanced: true,
@ -1123,6 +1128,12 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
} }
} }
// Update the file info before we start reading
err = o.lstat()
if err != nil {
return nil, err
}
// If not checking updated then limit to current size. This means if // If not checking updated then limit to current size. This means if
// file is being extended, readers will read a o.Size() bytes rather // file is being extended, readers will read a o.Size() bytes rather
// than the new size making for a consistent upload. // than the new size making for a consistent upload.