TrueCloudLab/rclone
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions 3
rclone/backend/onedrive/onedrive_internal_test.go

520 lines
19 KiB
Go
Raw Normal View History

onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
package onedrive
import (
"context"
"encoding/json"
"fmt"
build: replace "golang.org/x/exp/slices" with "slices" now go1.21 is required
2024-09-25 15:02:36 +00:00
"slices"
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
"testing"
"time"
_ "github.com/rclone/rclone/backend/local"
"github.com/rclone/rclone/backend/onedrive/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/random"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// go test -timeout 30m -run ^TestIntegration/FsMkdir/FsPutFiles/Internal$ github.com/rclone/rclone/backend/onedrive -remote TestOneDrive:meta -v
// go test -timeout 30m -run ^TestIntegration/FsMkdir/FsPutFiles/Internal$ github.com/rclone/rclone/backend/onedrive -remote TestOneDriveBusiness:meta -v
// go run ./fstest/test_all -remotes TestOneDriveBusiness:meta,TestOneDrive:meta -verbose -maxtries 1
var (
t1 = fstest.Time("2023-08-26T23:13:06.499999999Z")
t2 = fstest.Time("2020-02-29T12:34:56.789Z")
t3 = time.Date(1994, time.December, 24, 9+12, 0, 0, 525600, time.FixedZone("Eastern Standard Time", -5))
ctx = context.Background()
content = "hello"
)
const (
testUserID = "ryan@contoso.com" // demo user from doc examples (can't share files with yourself)
// https://learn.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_invite?view=odsp-graph-online#http-request-1
)
// TestMain drives the tests
func TestMain(m *testing.M) {
fstest.TestMain(m)
}
// TestWritePermissions tests reading and writing permissions
func (f *Fs) TestWritePermissions(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
_ = f.opt.MetadataPermissions.Set("read,write")
file1 := r.WriteFile(randomFilename(), content, t2)
// add a permission with "read" role
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
permissions := defaultPermissions(f.driveType)
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
permissions[0].Roles[0] = api.ReadRole
expectedMeta, actualMeta := f.putWithMeta(ctx, t, &file1, permissions)
f.compareMeta(t, expectedMeta, actualMeta, false)
expectedP, actualP := unmarshalPerms(t, expectedMeta["permissions"]), unmarshalPerms(t, actualMeta["permissions"])
found, num := false, 0
foundCount := 0
for i, p := range actualP {
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
for _, identity := range p.GetGrantedToIdentities(f.driveType) {
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
if identity.User.DisplayName == testUserID {
// note: expected will always be element 0 here, but actual may be variable based on org settings
assert.Equal(t, expectedP[0].Roles, p.Roles)
found, num = true, i
foundCount++
}
}
if f.driveType == driveTypePersonal {
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
if p.GetGrantedTo(f.driveType) != nil && p.GetGrantedTo(f.driveType).User != (api.Identity{}) && p.GetGrantedTo(f.driveType).User.ID == testUserID { // shows up in a different place on biz vs. personal
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
assert.Equal(t, expectedP[0].Roles, p.Roles)
found, num = true, i
foundCount++
}
}
}
assert.True(t, found, fmt.Sprintf("no permission found with expected role (want: \n\n%v \n\ngot: \n\n%v\n\n)", indent(t, expectedMeta["permissions"]), indent(t, actualMeta["permissions"])))
assert.Equal(t, 1, foundCount, "expected to find exactly 1 match")
// update it to "write"
permissions = actualP
permissions[num].Roles[0] = api.WriteRole
expectedMeta, actualMeta = f.putWithMeta(ctx, t, &file1, permissions)
f.compareMeta(t, expectedMeta, actualMeta, false)
if f.driveType != driveTypePersonal {
// zero out some things we expect to be different
expectedP, actualP = unmarshalPerms(t, expectedMeta["permissions"]), unmarshalPerms(t, actualMeta["permissions"])
normalize(expectedP)
normalize(actualP)
expectedMeta.Set("permissions", marshalPerms(t, expectedP))
actualMeta.Set("permissions", marshalPerms(t, actualP))
}
assert.JSONEq(t, expectedMeta["permissions"], actualMeta["permissions"])
// remove it
permissions[num] = nil
_, actualMeta = f.putWithMeta(ctx, t, &file1, permissions)
if f.driveType == driveTypePersonal {
perms, ok := actualMeta["permissions"]
assert.False(t, ok, fmt.Sprintf("permissions metadata key was unexpectedly found: %v", perms))
return
}
_, actualP = unmarshalPerms(t, expectedMeta["permissions"]), unmarshalPerms(t, actualMeta["permissions"])
found = false
var foundP *api.PermissionsType
for _, p := range actualP {
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
if p.GetGrantedTo(f.driveType) == nil || p.GetGrantedTo(f.driveType).User == (api.Identity{}) || p.GetGrantedTo(f.driveType).User.ID != testUserID {
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
continue
}
found = true
foundP = p
}
assert.False(t, found, fmt.Sprintf("permission was found but expected to be removed: %v", foundP))
}
// TestUploadSinglePart tests reading/writing permissions using uploadSinglepart()
// This is only used when file size is exactly 0.
func (f *Fs) TestUploadSinglePart(t *testing.T, r *fstest.Run) {
content = ""
f.TestWritePermissions(t, r)
content = "hello"
}
// TestReadPermissions tests that no permissions are written when --onedrive-metadata-permissions has "read" but not "write"
func (f *Fs) TestReadPermissions(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
file1 := r.WriteFile(randomFilename(), "hello", t2)
// try adding a permission without --onedrive-metadata-permissions -- should fail
// test that what we got before vs. after is the same
_ = f.opt.MetadataPermissions.Set("read")
_, expectedMeta := f.putWithMeta(ctx, t, &file1, []*api.PermissionsType{}) // return var intentionally switched here
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
permissions := defaultPermissions(f.driveType)
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
_, actualMeta := f.putWithMeta(ctx, t, &file1, permissions)
if f.driveType == driveTypePersonal {
perms, ok := actualMeta["permissions"]
assert.False(t, ok, fmt.Sprintf("permissions metadata key was unexpectedly found: %v", perms))
return
}
assert.JSONEq(t, expectedMeta["permissions"], actualMeta["permissions"])
}
// TestReadMetadata tests that all the read-only system properties are present and non-blank
func (f *Fs) TestReadMetadata(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
file1 := r.WriteFile(randomFilename(), "hello", t2)
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
permissions := defaultPermissions(f.driveType)
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
_ = f.opt.MetadataPermissions.Set("read,write")
_, actualMeta := f.putWithMeta(ctx, t, &file1, permissions)
optionals := []string{"package-type", "shared-by-id", "shared-scope", "shared-time", "shared-owner-id"} // not always present
for k := range systemMetadataInfo {
if slices.Contains(optionals, k) {
continue
}
if k == "description" && f.driveType != driveTypePersonal {
continue // not supported
}
gotV, ok := actualMeta[k]
assert.True(t, ok, fmt.Sprintf("property is missing: %v", k))
assert.NotEmpty(t, gotV, fmt.Sprintf("property is blank: %v", k))
}
}
// TestDirectoryMetadata tests reading and writing modtime and other metadata and permissions for directories
func (f *Fs) TestDirectoryMetadata(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
_ = f.opt.MetadataPermissions.Set("read,write")
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
permissions := defaultPermissions(f.driveType)
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
permissions[0].Roles[0] = api.ReadRole
expectedMeta := fs.Metadata{
"mtime": t1.Format(timeFormatOut),
"btime": t2.Format(timeFormatOut),
"content-type": dirMimeType,
"description": "that is so meta!",
}
b, err := json.MarshalIndent(permissions, "", "\t")
assert.NoError(t, err)
expectedMeta.Set("permissions", string(b))
compareDirMeta := func(expectedMeta, actualMeta fs.Metadata, ignoreID bool) {
f.compareMeta(t, expectedMeta, actualMeta, ignoreID)
// check that all required system properties are present
optionals := []string{"package-type", "shared-by-id", "shared-scope", "shared-time", "shared-owner-id"} // not always present
for k := range systemMetadataInfo {
if slices.Contains(optionals, k) {
continue
}
if k == "description" && f.driveType != driveTypePersonal {
continue // not supported
}
gotV, ok := actualMeta[k]
assert.True(t, ok, fmt.Sprintf("property is missing: %v", k))
assert.NotEmpty(t, gotV, fmt.Sprintf("property is blank: %v", k))
}
}
newDst, err := operations.MkdirMetadata(ctx, f, "subdir", expectedMeta)
assert.NoError(t, err)
require.NotNil(t, newDst)
assert.Equal(t, "subdir", newDst.Remote())
actualMeta, err := fs.GetMetadata(ctx, newDst)
assert.NoError(t, err)
assert.NotNil(t, actualMeta)
compareDirMeta(expectedMeta, actualMeta, false)
// modtime
onedrive: fix integration tests after precision change We changed the precision of the onedrive personal backend in c053429b9c0bbca5 from 1mS to 1S. However the tests did not get updated. This changes the time tests to use `fstest.AssertTimeEqualWithPrecision` which compares with precision so hopefully won't break again.
2024-11-12 13:05:57 +00:00
fstest.AssertTimeEqualWithPrecision(t, newDst.Remote(), t1, newDst.ModTime(ctx), f.Precision())
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
// try changing it and re-check it
newDst, err = operations.SetDirModTime(ctx, f, newDst, "", t2)
assert.NoError(t, err)
onedrive: fix integration tests after precision change We changed the precision of the onedrive personal backend in c053429b9c0bbca5 from 1mS to 1S. However the tests did not get updated. This changes the time tests to use `fstest.AssertTimeEqualWithPrecision` which compares with precision so hopefully won't break again.
2024-11-12 13:05:57 +00:00
fstest.AssertTimeEqualWithPrecision(t, newDst.Remote(), t2, newDst.ModTime(ctx), f.Precision())
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
// ensure that f.DirSetModTime also works
err = f.DirSetModTime(ctx, "subdir", t3)
assert.NoError(t, err)
entries, err := f.List(ctx, "")
assert.NoError(t, err)
entries.ForDir(func(dir fs.Directory) {
if dir.Remote() == "subdir" {
onedrive: fix integration tests after precision change We changed the precision of the onedrive personal backend in c053429b9c0bbca5 from 1mS to 1S. However the tests did not get updated. This changes the time tests to use `fstest.AssertTimeEqualWithPrecision` which compares with precision so hopefully won't break again.
2024-11-12 13:05:57 +00:00
fstest.AssertTimeEqualWithPrecision(t, dir.Remote(), t3, dir.ModTime(ctx), f.Precision())
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
}
})
// test updating metadata on existing dir
actualMeta, err = fs.GetMetadata(ctx, newDst) // get fresh info as we've been changing modtimes
assert.NoError(t, err)
expectedMeta = actualMeta
expectedMeta.Set("description", "metadata is fun!")
expectedMeta.Set("btime", t3.Format(timeFormatOut))
expectedMeta.Set("mtime", t1.Format(timeFormatOut))
expectedMeta.Set("content-type", dirMimeType)
perms := unmarshalPerms(t, expectedMeta["permissions"])
perms[0].Roles[0] = api.WriteRole
b, err = json.MarshalIndent(perms, "", "\t")
assert.NoError(t, err)
expectedMeta.Set("permissions", string(b))
newDst, err = operations.MkdirMetadata(ctx, f, "subdir", expectedMeta)
assert.NoError(t, err)
require.NotNil(t, newDst)
assert.Equal(t, "subdir", newDst.Remote())
actualMeta, err = fs.GetMetadata(ctx, newDst)
assert.NoError(t, err)
assert.NotNil(t, actualMeta)
compareDirMeta(expectedMeta, actualMeta, false)
// test copying metadata from one dir to another
copiedDir, err := operations.CopyDirMetadata(ctx, f, nil, "subdir2", newDst)
assert.NoError(t, err)
require.NotNil(t, copiedDir)
assert.Equal(t, "subdir2", copiedDir.Remote())
actualMeta, err = fs.GetMetadata(ctx, copiedDir)
assert.NoError(t, err)
assert.NotNil(t, actualMeta)
compareDirMeta(expectedMeta, actualMeta, true)
// test DirModTimeUpdatesOnWrite
expectedTime := copiedDir.ModTime(ctx)
assert.True(t, !expectedTime.IsZero())
r.WriteObject(ctx, copiedDir.Remote()+"/"+randomFilename(), "hi there", t3)
entries, err = f.List(ctx, "")
assert.NoError(t, err)
entries.ForDir(func(dir fs.Directory) {
if dir.Remote() == copiedDir.Remote() {
assert.True(t, expectedTime.Equal(dir.ModTime(ctx)), fmt.Sprintf("want %v got %v", expectedTime, dir.ModTime(ctx)))
}
})
}
// TestServerSideCopyMove tests server-side Copy and Move
func (f *Fs) TestServerSideCopyMove(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
_ = f.opt.MetadataPermissions.Set("read,write")
file1 := r.WriteFile(randomFilename(), content, t2)
// add a permission with "read" role
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
permissions := defaultPermissions(f.driveType)
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
permissions[0].Roles[0] = api.ReadRole
expectedMeta, actualMeta := f.putWithMeta(ctx, t, &file1, permissions)
f.compareMeta(t, expectedMeta, actualMeta, false)
comparePerms := func(expectedMeta, actualMeta fs.Metadata) (newExpectedMeta, newActualMeta fs.Metadata) {
expectedP, actualP := unmarshalPerms(t, expectedMeta["permissions"]), unmarshalPerms(t, actualMeta["permissions"])
normalize(expectedP)
normalize(actualP)
expectedMeta.Set("permissions", marshalPerms(t, expectedP))
actualMeta.Set("permissions", marshalPerms(t, actualP))
assert.JSONEq(t, expectedMeta["permissions"], actualMeta["permissions"])
return expectedMeta, actualMeta
}
// Copy
obj1, err := f.NewObject(ctx, file1.Path)
assert.NoError(t, err)
originalMeta := actualMeta
obj2, err := f.Copy(ctx, obj1, randomFilename())
assert.NoError(t, err)
actualMeta, err = fs.GetMetadata(ctx, obj2)
assert.NoError(t, err)
expectedMeta, actualMeta = comparePerms(originalMeta, actualMeta)
f.compareMeta(t, expectedMeta, actualMeta, true)
// Move
obj3, err := f.Move(ctx, obj1, randomFilename())
assert.NoError(t, err)
actualMeta, err = fs.GetMetadata(ctx, obj3)
assert.NoError(t, err)
expectedMeta, actualMeta = comparePerms(originalMeta, actualMeta)
f.compareMeta(t, expectedMeta, actualMeta, true)
}
onedrive: fix --metadata-mapper called twice if writing permissions Before this change, the --metadata-mapper was called twice if an object was uploaded via multipart upload with --metadata and --onedrive-metadata-permissions "write" or "read,write". This change fixes the issue.
2024-03-28 17:00:43 +00:00
// TestMetadataMapper tests adding permissions with the --metadata-mapper
func (f *Fs) TestMetadataMapper(t *testing.T, r *fstest.Run) {
// setup
ctx, ci := fs.AddConfig(ctx)
ci.Metadata = true
_ = f.opt.MetadataPermissions.Set("read,write")
file1 := r.WriteFile(randomFilename(), content, t2)
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
blob := `{"Metadata":{"permissions":"[{\"grantedToIdentities\":[{\"user\":{\"id\":\"ryan@contoso.com\"}}],\"roles\":[\"read\"]}]"}}`
if f.driveType != driveTypePersonal {
blob = `{"Metadata":{"permissions":"[{\"grantedToIdentitiesV2\":[{\"user\":{\"id\":\"ryan@contoso.com\"}}],\"roles\":[\"read\"]}]"}}`
}
onedrive: fix --metadata-mapper called twice if writing permissions Before this change, the --metadata-mapper was called twice if an object was uploaded via multipart upload with --metadata and --onedrive-metadata-permissions "write" or "read,write". This change fixes the issue.
2024-03-28 17:00:43 +00:00
// Copy
ci.MetadataMapper = []string{"echo", blob}
require.NoError(t, ci.Dump.Set("mapper"))
obj1, err := r.Flocal.NewObject(ctx, file1.Path)
assert.NoError(t, err)
obj2, err := operations.Copy(ctx, f, nil, randomFilename(), obj1)
assert.NoError(t, err)
actualMeta, err := fs.GetMetadata(ctx, obj2)
assert.NoError(t, err)
actualP := unmarshalPerms(t, actualMeta["permissions"])
found := false
foundCount := 0
for _, p := range actualP {
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
for _, identity := range p.GetGrantedToIdentities(f.driveType) {
onedrive: fix --metadata-mapper called twice if writing permissions Before this change, the --metadata-mapper was called twice if an object was uploaded via multipart upload with --metadata and --onedrive-metadata-permissions "write" or "read,write". This change fixes the issue.
2024-03-28 17:00:43 +00:00
if identity.User.DisplayName == testUserID {
assert.Equal(t, []api.Role{api.ReadRole}, p.Roles)
found = true
foundCount++
}
}
if f.driveType == driveTypePersonal {
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
if p.GetGrantedTo(f.driveType) != nil && p.GetGrantedTo(f.driveType).User != (api.Identity{}) && p.GetGrantedTo(f.driveType).User.ID == testUserID { // shows up in a different place on biz vs. personal
onedrive: fix --metadata-mapper called twice if writing permissions Before this change, the --metadata-mapper was called twice if an object was uploaded via multipart upload with --metadata and --onedrive-metadata-permissions "write" or "read,write". This change fixes the issue.
2024-03-28 17:00:43 +00:00
assert.Equal(t, []api.Role{api.ReadRole}, p.Roles)
found = true
foundCount++
}
}
}
assert.True(t, found, fmt.Sprintf("no permission found with expected role (want: \n\n%v \n\ngot: \n\n%v\n\n)", blob, actualMeta))
assert.Equal(t, 1, foundCount, "expected to find exactly 1 match")
}
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
// helper function to put an object with metadata and permissions
func (f *Fs) putWithMeta(ctx context.Context, t *testing.T, file *fstest.Item, perms []*api.PermissionsType) (expectedMeta, actualMeta fs.Metadata) {
t.Helper()
expectedMeta = fs.Metadata{
"mtime": t1.Format(timeFormatOut),
"btime": t2.Format(timeFormatOut),
"description": "that is so meta!",
}
expectedMeta.Set("permissions", marshalPerms(t, perms))
b2: Include custom upload headers in large file info - fixes #7744
2024-04-09 21:13:33 +00:00
obj := fstests.PutTestContentsMetadata(ctx, t, f, file, false, content, true, "plain/text", expectedMeta)
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
do, ok := obj.(fs.Metadataer)
require.True(t, ok)
actualMeta, err := do.Metadata(ctx)
require.NoError(t, err)
return expectedMeta, actualMeta
}
func randomFilename() string {
return "some file-" + random.String(8) + ".txt"
}
func (f *Fs) compareMeta(t *testing.T, expectedMeta, actualMeta fs.Metadata, ignoreID bool) {
t.Helper()
for k, v := range expectedMeta {
gotV, ok := actualMeta[k]
switch k {
case "shared-owner-id", "shared-time", "shared-by-id", "shared-scope":
continue
case "permissions":
continue
case "utime":
assert.True(t, ok, fmt.Sprintf("expected metadata key is missing: %v", k))
if f.driveType == driveTypePersonal {
compareTimeStrings(t, k, v, gotV, time.Minute) // read-only upload time, so slight difference expected -- use larger precision
continue
}
compareTimeStrings(t, k, expectedMeta["btime"], gotV, time.Minute) // another bizarre difference between personal and business...
continue
case "id":
if ignoreID {
continue // different id is expected when copying meta from one item to another
}
case "mtime", "btime":
assert.True(t, ok, fmt.Sprintf("expected metadata key is missing: %v", k))
compareTimeStrings(t, k, v, gotV, time.Second)
continue
case "description":
if f.driveType != driveTypePersonal {
continue // not supported
}
}
assert.True(t, ok, fmt.Sprintf("expected metadata key is missing: %v", k))
assert.Equal(t, v, gotV, actualMeta)
}
}
func compareTimeStrings(t *testing.T, remote, want, got string, precision time.Duration) {
wantT, err := time.Parse(timeFormatIn, want)
assert.NoError(t, err)
gotT, err := time.Parse(timeFormatIn, got)
assert.NoError(t, err)
fstest.AssertTimeEqualWithPrecision(t, remote, wantT, gotT, precision)
}
func marshalPerms(t *testing.T, p []*api.PermissionsType) string {
b, err := json.MarshalIndent(p, "", "\t")
assert.NoError(t, err)
return string(b)
}
func unmarshalPerms(t *testing.T, perms string) (p []*api.PermissionsType) {
t.Helper()
err := json.Unmarshal([]byte(perms), &p)
assert.NoError(t, err)
return p
}
func indent(t *testing.T, s string) string {
p := unmarshalPerms(t, s)
return marshalPerms(t, p)
}
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
func defaultPermissions(driveType string) []*api.PermissionsType {
if driveType == driveTypePersonal {
return []*api.PermissionsType{{
GrantedTo: &api.IdentitySet{User: api.Identity{}},
GrantedToIdentities: []*api.IdentitySet{{User: api.Identity{ID: testUserID}}},
Roles: []api.Role{api.WriteRole},
}}
}
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
return []*api.PermissionsType{{
onedrive: fix references to deprecated permissions properties Before this change, metadata permissions used the `grantedTo` and `grantedToIdentities` properties, which are deprecated on OneDrive Business in favor of `grantedToV2` and `grantedToIdentitiesV2`. After this change, OneDrive Business uses the new V2 versions, while OneDrive Personal still uses the originals, as the V2 versions are not available for OneDrive Personal. (see https://learn.microsoft.com/en-us/answers/questions/1079737/inconsistency-between-grantedtov2-and-grantedto-re)
2024-04-22 01:39:15 +00:00
GrantedToV2: &api.IdentitySet{User: api.Identity{}},
GrantedToIdentitiesV2: []*api.IdentitySet{{User: api.Identity{ID: testUserID}}},
Roles: []api.Role{api.WriteRole},
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
}}
}
// zeroes out some things we expect to be different when copying/moving between objects
func normalize(Ps []*api.PermissionsType) {
for _, ep := range Ps {
ep.ID = ""
ep.Link = nil
ep.ShareID = ""
}
}
func (f *Fs) resetTestDefaults(r *fstest.Run) {
ci := fs.GetConfig(ctx)
ci.Metadata = false
_ = f.opt.MetadataPermissions.Set("off")
r.Finalise()
}
// InternalTest dispatches all internal tests
func (f *Fs) InternalTest(t *testing.T) {
newTestF := func() (*Fs, *fstest.Run) {
r := fstest.NewRunIndividual(t)
testF, ok := r.Fremote.(*Fs)
if !ok {
t.FailNow()
}
return testF, r
}
testF, r := newTestF()
t.Run("TestWritePermissions", func(t *testing.T) { testF.TestWritePermissions(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestUploadSinglePart", func(t *testing.T) { testF.TestUploadSinglePart(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestReadPermissions", func(t *testing.T) { testF.TestReadPermissions(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestReadMetadata", func(t *testing.T) { testF.TestReadMetadata(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestDirectoryMetadata", func(t *testing.T) { testF.TestDirectoryMetadata(t, r) })
testF.resetTestDefaults(r)
testF, r = newTestF()
t.Run("TestServerSideCopyMove", func(t *testing.T) { testF.TestServerSideCopyMove(t, r) })
testF.resetTestDefaults(r)
onedrive: fix --metadata-mapper called twice if writing permissions Before this change, the --metadata-mapper was called twice if an object was uploaded via multipart upload with --metadata and --onedrive-metadata-permissions "write" or "read,write". This change fixes the issue.
2024-03-28 17:00:43 +00:00
t.Run("TestMetadataMapper", func(t *testing.T) { testF.TestMetadataMapper(t, r) })
testF.resetTestDefaults(r)
onedrive: add metadata support This change adds support for metadata on OneDrive. Metadata (including permissions) is supported for both files and directories. OneDrive supports System Metadata (not User Metadata, as of this writing.) Much of the metadata is read-only, and there are some differences between OneDrive Personal and Business (see table in OneDrive backend docs for details). Permissions are also supported, if --onedrive-metadata-permissions is set. The accepted values for --onedrive-metadata-permissions are read, write, read,write, and off (the default). write supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use read,write instead of write if you wish to update/remove permissions. Permissions are read/written in JSON format using the same schema as the OneDrive API, which differs slightly between OneDrive Personal and Business. (See OneDrive backend docs for examples.) To write permissions, pass in a "permissions" metadata key using this same format. The --metadata-mapper tool can be very helpful for this. When adding permissions, an email address can be provided in the User.ID or DisplayName properties of grantedTo or grantedToIdentities. Alternatively, an ObjectID can be provided in User.ID. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if Link.Scope is set to "anonymous". Note that adding a permission can fail if a conflicting permission already exists for the file/folder. To update an existing permission, include both the Permission ID and the new roles to be assigned. roles is the only property that can be changed. To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that both reading and writing permissions requires extra API calls, so if you don't need to read or write permissions it is recommended to omit --onedrive- metadata-permissions. Metadata and permissions are supported for Folders (directories) as well as Files. Note that setting the mtime or btime on a Folder requires one extra API call on OneDrive Business only. OneDrive does not currently support User Metadata. When writing metadata, only writeable system properties will be written -- any read-only or unrecognized keys passed in will be ignored. TIP: to see the metadata and permissions for any file or folder, run: rclone lsjson remote:path --stat -M --onedrive-metadata-permissions read See the OneDrive backend docs for a table of all the supported metadata properties.
2024-02-22 14:17:14 +00:00
}
var _ fstests.InternalTester = (*Fs)(nil)
Reference in a new issue Copy permalink