rclone/backend/chunker/chunker_test.go

63 lines
1.8 KiB
Go
Raw Normal View History

chunker: implementation + required fstest patch Note: chunker implements many irrelevant methods (UserInfo, Disconnect etc), but they are required by TestIntegration/FsCheckWrap and cannot be removed. Dropped API methods: MergeDirs DirCacheFlush PublicLink UserInfo Disconnect OpenWriterAt Meta formats: - renamed old simplejson format to wdmrcompat. - new simplejson format supports hash sums and verification of chunk size/count. Change list: - split-chunking overlay for mailru - add to all - fix linter errors - fix integration tests - support chunks without meta object - fix package paths - propagate context - fix formatting - implement new required wrapper interfaces - also test large file uploads - simplify options - user friendly name pattern - set default chunk size 2G - fix building with golang 1.9 - fix ci/cd on a separate branch - fix updated object name (SyncUTFNorm failed) - fix panic in Box overlay - workaround: Box rename failed if name taken - enhance comments in unit test - fix formatting - embed wrapped remote rather than inherit - require wrapped remote to support move (or copy) - implement 3 (keep fstest) - drop irrelevant file system interfaces - factor out Object.mainChunk - refactor TestLargeUpload as InternalTest - add unit test for chunk name formats - new improved simplejson meta format - tricky case in test FsIsFile (fix+ignore) - remove debugging print - hide temporary objects from listings - fix bugs in chunking reader: - return EOF immediately when all data is sent - handle case when wrapped remote puts by hash (bug detected by TestRcat) - chunked file hashing (feature) - server-side copy across configs (feature) - robust cleanup of temporary chunks in Put - linear download strategy (no read-ahead, feature) - fix unexpected EOF in the box multipart uploader - throw error if destination ignores data
2019-06-09 17:41:48 +00:00
// Test the Chunker filesystem interface
package chunker_test
import (
"flag"
"os"
"path/filepath"
"testing"
_ "github.com/rclone/rclone/backend/all" // for integration tests
"github.com/rclone/rclone/backend/chunker"
"github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests"
)
// Command line flags
var (
// Invalid characters are not supported by some remotes, e.g. Mailru.
chunker: implementation + required fstest patch Note: chunker implements many irrelevant methods (UserInfo, Disconnect etc), but they are required by TestIntegration/FsCheckWrap and cannot be removed. Dropped API methods: MergeDirs DirCacheFlush PublicLink UserInfo Disconnect OpenWriterAt Meta formats: - renamed old simplejson format to wdmrcompat. - new simplejson format supports hash sums and verification of chunk size/count. Change list: - split-chunking overlay for mailru - add to all - fix linter errors - fix integration tests - support chunks without meta object - fix package paths - propagate context - fix formatting - implement new required wrapper interfaces - also test large file uploads - simplify options - user friendly name pattern - set default chunk size 2G - fix building with golang 1.9 - fix ci/cd on a separate branch - fix updated object name (SyncUTFNorm failed) - fix panic in Box overlay - workaround: Box rename failed if name taken - enhance comments in unit test - fix formatting - embed wrapped remote rather than inherit - require wrapped remote to support move (or copy) - implement 3 (keep fstest) - drop irrelevant file system interfaces - factor out Object.mainChunk - refactor TestLargeUpload as InternalTest - add unit test for chunk name formats - new improved simplejson meta format - tricky case in test FsIsFile (fix+ignore) - remove debugging print - hide temporary objects from listings - fix bugs in chunking reader: - return EOF immediately when all data is sent - handle case when wrapped remote puts by hash (bug detected by TestRcat) - chunked file hashing (feature) - server-side copy across configs (feature) - robust cleanup of temporary chunks in Put - linear download strategy (no read-ahead, feature) - fix unexpected EOF in the box multipart uploader - throw error if destination ignores data
2019-06-09 17:41:48 +00:00
// We enable testing with invalid characters when -remote is not set, so
// chunker overlays a local directory, but invalid characters are disabled
// by default when -remote is set, e.g. when test_all runs backend tests.
chunker: implementation + required fstest patch Note: chunker implements many irrelevant methods (UserInfo, Disconnect etc), but they are required by TestIntegration/FsCheckWrap and cannot be removed. Dropped API methods: MergeDirs DirCacheFlush PublicLink UserInfo Disconnect OpenWriterAt Meta formats: - renamed old simplejson format to wdmrcompat. - new simplejson format supports hash sums and verification of chunk size/count. Change list: - split-chunking overlay for mailru - add to all - fix linter errors - fix integration tests - support chunks without meta object - fix package paths - propagate context - fix formatting - implement new required wrapper interfaces - also test large file uploads - simplify options - user friendly name pattern - set default chunk size 2G - fix building with golang 1.9 - fix ci/cd on a separate branch - fix updated object name (SyncUTFNorm failed) - fix panic in Box overlay - workaround: Box rename failed if name taken - enhance comments in unit test - fix formatting - embed wrapped remote rather than inherit - require wrapped remote to support move (or copy) - implement 3 (keep fstest) - drop irrelevant file system interfaces - factor out Object.mainChunk - refactor TestLargeUpload as InternalTest - add unit test for chunk name formats - new improved simplejson meta format - tricky case in test FsIsFile (fix+ignore) - remove debugging print - hide temporary objects from listings - fix bugs in chunking reader: - return EOF immediately when all data is sent - handle case when wrapped remote puts by hash (bug detected by TestRcat) - chunked file hashing (feature) - server-side copy across configs (feature) - robust cleanup of temporary chunks in Put - linear download strategy (no read-ahead, feature) - fix unexpected EOF in the box multipart uploader - throw error if destination ignores data
2019-06-09 17:41:48 +00:00
// You can still test with invalid characters using the below flag.
UseBadChars = flag.Bool("bad-chars", false, "Set to test bad characters in file names when -remote is set")
)
// TestIntegration runs integration tests against a concrete remote
// set by the -remote flag. If the flag is not set, it creates a
// dynamic chunker overlay wrapping a local temporary directory.
func TestIntegration(t *testing.T) {
opt := fstests.Opt{
RemoteName: *fstest.RemoteName,
NilObject: (*chunker.Object)(nil),
SkipBadWindowsCharacters: !*UseBadChars,
UnimplementableObjectMethods: []string{
"MimeType",
"GetTier",
"SetTier",
"Metadata",
"SetMetadata",
},
chunker: implementation + required fstest patch Note: chunker implements many irrelevant methods (UserInfo, Disconnect etc), but they are required by TestIntegration/FsCheckWrap and cannot be removed. Dropped API methods: MergeDirs DirCacheFlush PublicLink UserInfo Disconnect OpenWriterAt Meta formats: - renamed old simplejson format to wdmrcompat. - new simplejson format supports hash sums and verification of chunk size/count. Change list: - split-chunking overlay for mailru - add to all - fix linter errors - fix integration tests - support chunks without meta object - fix package paths - propagate context - fix formatting - implement new required wrapper interfaces - also test large file uploads - simplify options - user friendly name pattern - set default chunk size 2G - fix building with golang 1.9 - fix ci/cd on a separate branch - fix updated object name (SyncUTFNorm failed) - fix panic in Box overlay - workaround: Box rename failed if name taken - enhance comments in unit test - fix formatting - embed wrapped remote rather than inherit - require wrapped remote to support move (or copy) - implement 3 (keep fstest) - drop irrelevant file system interfaces - factor out Object.mainChunk - refactor TestLargeUpload as InternalTest - add unit test for chunk name formats - new improved simplejson meta format - tricky case in test FsIsFile (fix+ignore) - remove debugging print - hide temporary objects from listings - fix bugs in chunking reader: - return EOF immediately when all data is sent - handle case when wrapped remote puts by hash (bug detected by TestRcat) - chunked file hashing (feature) - server-side copy across configs (feature) - robust cleanup of temporary chunks in Put - linear download strategy (no read-ahead, feature) - fix unexpected EOF in the box multipart uploader - throw error if destination ignores data
2019-06-09 17:41:48 +00:00
UnimplementableFsMethods: []string{
"PublicLink",
"OpenWriterAt",
"OpenChunkWriter",
chunker: implementation + required fstest patch Note: chunker implements many irrelevant methods (UserInfo, Disconnect etc), but they are required by TestIntegration/FsCheckWrap and cannot be removed. Dropped API methods: MergeDirs DirCacheFlush PublicLink UserInfo Disconnect OpenWriterAt Meta formats: - renamed old simplejson format to wdmrcompat. - new simplejson format supports hash sums and verification of chunk size/count. Change list: - split-chunking overlay for mailru - add to all - fix linter errors - fix integration tests - support chunks without meta object - fix package paths - propagate context - fix formatting - implement new required wrapper interfaces - also test large file uploads - simplify options - user friendly name pattern - set default chunk size 2G - fix building with golang 1.9 - fix ci/cd on a separate branch - fix updated object name (SyncUTFNorm failed) - fix panic in Box overlay - workaround: Box rename failed if name taken - enhance comments in unit test - fix formatting - embed wrapped remote rather than inherit - require wrapped remote to support move (or copy) - implement 3 (keep fstest) - drop irrelevant file system interfaces - factor out Object.mainChunk - refactor TestLargeUpload as InternalTest - add unit test for chunk name formats - new improved simplejson meta format - tricky case in test FsIsFile (fix+ignore) - remove debugging print - hide temporary objects from listings - fix bugs in chunking reader: - return EOF immediately when all data is sent - handle case when wrapped remote puts by hash (bug detected by TestRcat) - chunked file hashing (feature) - server-side copy across configs (feature) - robust cleanup of temporary chunks in Put - linear download strategy (no read-ahead, feature) - fix unexpected EOF in the box multipart uploader - throw error if destination ignores data
2019-06-09 17:41:48 +00:00
"MergeDirs",
"DirCacheFlush",
"UserInfo",
"Disconnect",
},
}
if *fstest.RemoteName == "" {
name := "TestChunker"
opt.RemoteName = name + ":"
tempDir := filepath.Join(os.TempDir(), "rclone-chunker-test-standard")
opt.ExtraConfig = []fstests.ExtraConfigItem{
{Name: name, Key: "type", Value: "chunker"},
{Name: name, Key: "remote", Value: tempDir},
}
opt.QuickTestOK = true
chunker: implementation + required fstest patch Note: chunker implements many irrelevant methods (UserInfo, Disconnect etc), but they are required by TestIntegration/FsCheckWrap and cannot be removed. Dropped API methods: MergeDirs DirCacheFlush PublicLink UserInfo Disconnect OpenWriterAt Meta formats: - renamed old simplejson format to wdmrcompat. - new simplejson format supports hash sums and verification of chunk size/count. Change list: - split-chunking overlay for mailru - add to all - fix linter errors - fix integration tests - support chunks without meta object - fix package paths - propagate context - fix formatting - implement new required wrapper interfaces - also test large file uploads - simplify options - user friendly name pattern - set default chunk size 2G - fix building with golang 1.9 - fix ci/cd on a separate branch - fix updated object name (SyncUTFNorm failed) - fix panic in Box overlay - workaround: Box rename failed if name taken - enhance comments in unit test - fix formatting - embed wrapped remote rather than inherit - require wrapped remote to support move (or copy) - implement 3 (keep fstest) - drop irrelevant file system interfaces - factor out Object.mainChunk - refactor TestLargeUpload as InternalTest - add unit test for chunk name formats - new improved simplejson meta format - tricky case in test FsIsFile (fix+ignore) - remove debugging print - hide temporary objects from listings - fix bugs in chunking reader: - return EOF immediately when all data is sent - handle case when wrapped remote puts by hash (bug detected by TestRcat) - chunked file hashing (feature) - server-side copy across configs (feature) - robust cleanup of temporary chunks in Put - linear download strategy (no read-ahead, feature) - fix unexpected EOF in the box multipart uploader - throw error if destination ignores data
2019-06-09 17:41:48 +00:00
}
fstests.Run(t, &opt)
}