rclone/backend/local/clone_darwin.go

94 lines
2.5 KiB
Go
Raw Normal View History

local: add server-side copy with xattrs on macOS (part-fix #1710) Before this change, macOS-specific metadata was not preserved by rclone, even for local-to-local transfers (it does not use the "user." prefix, nor is Mac metadata limited to xattrs.) Additionally, rclone did not take advantage of APFS's native "cloning" functionality for fast and deduplicated transfers. After this change, local (on macOS only) supports "server-side copy" similarly to other remotes, and achieves this by using (when possible) macOS's native APFS "cloning", which is the same underlying mechanism deployed when a user duplicates a file via the Finder UI. This has several advantages over the previous behavior: - It is extremely fast (even large files can be cloned instantly) - It is very efficient in terms of storage, as it automatically deduplicates when possible (i.e. so that having two identical files does not consume more storage than having just one.) (The concept is similar to a "hard link", but subsequent modifications will not affect the original file.) - It preserves Mac-specific metadata to the maximum degree, including not only xattrs but also metadata not easily settable by other methods, including Finder and Spotlight params. When server-side "clone" is not available (for example, on non-APFS volumes), it falls back to server-side "copy" (still preserving metadata but using more disk storage.) It is only used when both remotes are local (and not wrapped by other remotes, such as crypt.) The behavior of local on non-mac systems is unchanged.
2023-12-28 17:30:47 +00:00
//go:build darwin && cgo
// Package local provides a filesystem interface
package local
import (
"context"
"fmt"
"path/filepath"
local: add server-side copy with xattrs on macOS (part-fix #1710) Before this change, macOS-specific metadata was not preserved by rclone, even for local-to-local transfers (it does not use the "user." prefix, nor is Mac metadata limited to xattrs.) Additionally, rclone did not take advantage of APFS's native "cloning" functionality for fast and deduplicated transfers. After this change, local (on macOS only) supports "server-side copy" similarly to other remotes, and achieves this by using (when possible) macOS's native APFS "cloning", which is the same underlying mechanism deployed when a user duplicates a file via the Finder UI. This has several advantages over the previous behavior: - It is extremely fast (even large files can be cloned instantly) - It is very efficient in terms of storage, as it automatically deduplicates when possible (i.e. so that having two identical files does not consume more storage than having just one.) (The concept is similar to a "hard link", but subsequent modifications will not affect the original file.) - It preserves Mac-specific metadata to the maximum degree, including not only xattrs but also metadata not easily settable by other methods, including Finder and Spotlight params. When server-side "clone" is not available (for example, on non-APFS volumes), it falls back to server-side "copy" (still preserving metadata but using more disk storage.) It is only used when both remotes are local (and not wrapped by other remotes, such as crypt.) The behavior of local on non-mac systems is unchanged.
2023-12-28 17:30:47 +00:00
"runtime"
"github.com/go-darwin/apfs"
"github.com/rclone/rclone/fs"
)
// Copy src to this remote using server-side copy operations.
//
// # This is stored with the remote path given
//
// # It returns the destination Object and a possible error
//
// Will only be called if src.Fs().Name() == f.Name()
//
// If it isn't possible then return fs.ErrorCantCopy
func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
if runtime.GOOS != "darwin" || f.opt.NoClone {
local: add server-side copy with xattrs on macOS (part-fix #1710) Before this change, macOS-specific metadata was not preserved by rclone, even for local-to-local transfers (it does not use the "user." prefix, nor is Mac metadata limited to xattrs.) Additionally, rclone did not take advantage of APFS's native "cloning" functionality for fast and deduplicated transfers. After this change, local (on macOS only) supports "server-side copy" similarly to other remotes, and achieves this by using (when possible) macOS's native APFS "cloning", which is the same underlying mechanism deployed when a user duplicates a file via the Finder UI. This has several advantages over the previous behavior: - It is extremely fast (even large files can be cloned instantly) - It is very efficient in terms of storage, as it automatically deduplicates when possible (i.e. so that having two identical files does not consume more storage than having just one.) (The concept is similar to a "hard link", but subsequent modifications will not affect the original file.) - It preserves Mac-specific metadata to the maximum degree, including not only xattrs but also metadata not easily settable by other methods, including Finder and Spotlight params. When server-side "clone" is not available (for example, on non-APFS volumes), it falls back to server-side "copy" (still preserving metadata but using more disk storage.) It is only used when both remotes are local (and not wrapped by other remotes, such as crypt.) The behavior of local on non-mac systems is unchanged.
2023-12-28 17:30:47 +00:00
return nil, fs.ErrorCantCopy
}
srcObj, ok := src.(*Object)
if !ok {
fs.Debugf(src, "Can't clone - not same remote type")
return nil, fs.ErrorCantCopy
}
if f.opt.TranslateSymlinks && srcObj.translatedLink { // in --links mode, use cloning only for regular files
return nil, fs.ErrorCantCopy
}
local: add server-side copy with xattrs on macOS (part-fix #1710) Before this change, macOS-specific metadata was not preserved by rclone, even for local-to-local transfers (it does not use the "user." prefix, nor is Mac metadata limited to xattrs.) Additionally, rclone did not take advantage of APFS's native "cloning" functionality for fast and deduplicated transfers. After this change, local (on macOS only) supports "server-side copy" similarly to other remotes, and achieves this by using (when possible) macOS's native APFS "cloning", which is the same underlying mechanism deployed when a user duplicates a file via the Finder UI. This has several advantages over the previous behavior: - It is extremely fast (even large files can be cloned instantly) - It is very efficient in terms of storage, as it automatically deduplicates when possible (i.e. so that having two identical files does not consume more storage than having just one.) (The concept is similar to a "hard link", but subsequent modifications will not affect the original file.) - It preserves Mac-specific metadata to the maximum degree, including not only xattrs but also metadata not easily settable by other methods, including Finder and Spotlight params. When server-side "clone" is not available (for example, on non-APFS volumes), it falls back to server-side "copy" (still preserving metadata but using more disk storage.) It is only used when both remotes are local (and not wrapped by other remotes, such as crypt.) The behavior of local on non-mac systems is unchanged.
2023-12-28 17:30:47 +00:00
// Fetch metadata if --metadata is in use
meta, err := fs.GetMetadataOptions(ctx, f, src, fs.MetadataAsOpenOptions(ctx))
if err != nil {
return nil, fmt.Errorf("copy: failed to read metadata: %w", err)
}
local: add server-side copy with xattrs on macOS (part-fix #1710) Before this change, macOS-specific metadata was not preserved by rclone, even for local-to-local transfers (it does not use the "user." prefix, nor is Mac metadata limited to xattrs.) Additionally, rclone did not take advantage of APFS's native "cloning" functionality for fast and deduplicated transfers. After this change, local (on macOS only) supports "server-side copy" similarly to other remotes, and achieves this by using (when possible) macOS's native APFS "cloning", which is the same underlying mechanism deployed when a user duplicates a file via the Finder UI. This has several advantages over the previous behavior: - It is extremely fast (even large files can be cloned instantly) - It is very efficient in terms of storage, as it automatically deduplicates when possible (i.e. so that having two identical files does not consume more storage than having just one.) (The concept is similar to a "hard link", but subsequent modifications will not affect the original file.) - It preserves Mac-specific metadata to the maximum degree, including not only xattrs but also metadata not easily settable by other methods, including Finder and Spotlight params. When server-side "clone" is not available (for example, on non-APFS volumes), it falls back to server-side "copy" (still preserving metadata but using more disk storage.) It is only used when both remotes are local (and not wrapped by other remotes, such as crypt.) The behavior of local on non-mac systems is unchanged.
2023-12-28 17:30:47 +00:00
// Create destination
dstObj := f.newObject(remote)
err = dstObj.mkdirAll()
local: add server-side copy with xattrs on macOS (part-fix #1710) Before this change, macOS-specific metadata was not preserved by rclone, even for local-to-local transfers (it does not use the "user." prefix, nor is Mac metadata limited to xattrs.) Additionally, rclone did not take advantage of APFS's native "cloning" functionality for fast and deduplicated transfers. After this change, local (on macOS only) supports "server-side copy" similarly to other remotes, and achieves this by using (when possible) macOS's native APFS "cloning", which is the same underlying mechanism deployed when a user duplicates a file via the Finder UI. This has several advantages over the previous behavior: - It is extremely fast (even large files can be cloned instantly) - It is very efficient in terms of storage, as it automatically deduplicates when possible (i.e. so that having two identical files does not consume more storage than having just one.) (The concept is similar to a "hard link", but subsequent modifications will not affect the original file.) - It preserves Mac-specific metadata to the maximum degree, including not only xattrs but also metadata not easily settable by other methods, including Finder and Spotlight params. When server-side "clone" is not available (for example, on non-APFS volumes), it falls back to server-side "copy" (still preserving metadata but using more disk storage.) It is only used when both remotes are local (and not wrapped by other remotes, such as crypt.) The behavior of local on non-mac systems is unchanged.
2023-12-28 17:30:47 +00:00
if err != nil {
return nil, err
}
srcPath := srcObj.path
if f.opt.FollowSymlinks { // in --copy-links mode, find the real file being pointed to and pass that in instead
srcPath, err = filepath.EvalSymlinks(srcPath)
if err != nil {
return nil, err
}
}
err = Clone(srcPath, f.localPath(remote))
local: add server-side copy with xattrs on macOS (part-fix #1710) Before this change, macOS-specific metadata was not preserved by rclone, even for local-to-local transfers (it does not use the "user." prefix, nor is Mac metadata limited to xattrs.) Additionally, rclone did not take advantage of APFS's native "cloning" functionality for fast and deduplicated transfers. After this change, local (on macOS only) supports "server-side copy" similarly to other remotes, and achieves this by using (when possible) macOS's native APFS "cloning", which is the same underlying mechanism deployed when a user duplicates a file via the Finder UI. This has several advantages over the previous behavior: - It is extremely fast (even large files can be cloned instantly) - It is very efficient in terms of storage, as it automatically deduplicates when possible (i.e. so that having two identical files does not consume more storage than having just one.) (The concept is similar to a "hard link", but subsequent modifications will not affect the original file.) - It preserves Mac-specific metadata to the maximum degree, including not only xattrs but also metadata not easily settable by other methods, including Finder and Spotlight params. When server-side "clone" is not available (for example, on non-APFS volumes), it falls back to server-side "copy" (still preserving metadata but using more disk storage.) It is only used when both remotes are local (and not wrapped by other remotes, such as crypt.) The behavior of local on non-mac systems is unchanged.
2023-12-28 17:30:47 +00:00
if err != nil {
return nil, err
}
// Set metadata if --metadata is in use
if meta != nil {
err = dstObj.writeMetadata(meta)
if err != nil {
return nil, fmt.Errorf("copy: failed to set metadata: %w", err)
}
}
local: add server-side copy with xattrs on macOS (part-fix #1710) Before this change, macOS-specific metadata was not preserved by rclone, even for local-to-local transfers (it does not use the "user." prefix, nor is Mac metadata limited to xattrs.) Additionally, rclone did not take advantage of APFS's native "cloning" functionality for fast and deduplicated transfers. After this change, local (on macOS only) supports "server-side copy" similarly to other remotes, and achieves this by using (when possible) macOS's native APFS "cloning", which is the same underlying mechanism deployed when a user duplicates a file via the Finder UI. This has several advantages over the previous behavior: - It is extremely fast (even large files can be cloned instantly) - It is very efficient in terms of storage, as it automatically deduplicates when possible (i.e. so that having two identical files does not consume more storage than having just one.) (The concept is similar to a "hard link", but subsequent modifications will not affect the original file.) - It preserves Mac-specific metadata to the maximum degree, including not only xattrs but also metadata not easily settable by other methods, including Finder and Spotlight params. When server-side "clone" is not available (for example, on non-APFS volumes), it falls back to server-side "copy" (still preserving metadata but using more disk storage.) It is only used when both remotes are local (and not wrapped by other remotes, such as crypt.) The behavior of local on non-mac systems is unchanged.
2023-12-28 17:30:47 +00:00
return f.NewObject(ctx, remote)
}
// Clone uses APFS cloning if possible, otherwise falls back to copying (with full metadata preservation)
// note that this is closely related to unix.Clonefile(src, dst, unix.CLONE_NOFOLLOW) but not 100% identical
// https://opensource.apple.com/source/copyfile/copyfile-173.40.2/copyfile.c.auto.html
func Clone(src, dst string) error {
state := apfs.CopyFileStateAlloc()
defer func() {
if err := apfs.CopyFileStateFree(state); err != nil {
fs.Errorf(dst, "free state error: %v", err)
}
}()
cloned, err := apfs.CopyFile(src, dst, state, apfs.COPYFILE_CLONE)
fs.Debugf(dst, "isCloned: %v, error: %v", cloned, err)
return err
}
// Check the interfaces are satisfied
var (
_ fs.Copier = &Fs{}
)