TrueCloudLab/rclone
16
0
Fork 2
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
rclone/backend/s3/s3.go

1476 lines
42 KiB
Go
Raw Normal View History

Fix golint warnings
2015-09-22 17:47:16 +00:00
// Package s3 provides an interface to Amazon S3 oject storage
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
package s3
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
Implement fs.ListDir / lsd command
2013-01-23 22:43:20 +00:00
// FIXME need to prevent anything but ListDir working for s3://
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
/*
Progress of port to aws-sdk
* Don't really need o.meta at all?
What happens if you CTRL-C a multipart upload
* get an incomplete upload
* disappears when you delete the bucket
*/
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
import (
s3: set/get the hash for multipart files - #523
2018-01-06 14:30:10 +00:00
"encoding/base64"
"encoding/hex"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
"fmt"
"io"
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
"net/http"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
"path"
"regexp"
"strings"
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
"sync"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
"time"
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114
2015-08-28 07:47:41 +00:00
"github.com/aws/aws-sdk-go/aws/corehandlers"
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
"github.com/aws/aws-sdk-go/aws/credentials"
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
s3: add support for ECS task IAM roles ECS container IAM metadata is in a different place than EC2 IAM metadata. Use defaults' RemoteCredProvider function to query the standard locations for the credentials. Give the ECS role precedence over the role available from the underlying EC2 instance.
2017-11-22 21:21:36 +00:00
"github.com/aws/aws-sdk-go/aws/defaults"
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
"github.com/aws/aws-sdk-go/aws/ec2metadata"
s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114
2015-08-28 07:47:41 +00:00
"github.com/aws/aws-sdk-go/aws/request"
Update to AWS SDK 0.10.0 Tested with S3 and Dreamhost Here is a link to the release notes: http://aws.amazon.com/releasenotes/5476699172355228
2015-10-30 11:50:45 +00:00
"github.com/aws/aws-sdk-go/aws/session"
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
"github.com/ncw/rclone/fs"
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
"github.com/ncw/rclone/fs/config/configmap"
"github.com/ncw/rclone/fs/config/configstruct"
Break the fs package up into smaller parts. The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.
2018-01-12 16:30:54 +00:00
"github.com/ncw/rclone/fs/fshttp"
"github.com/ncw/rclone/fs/hash"
"github.com/ncw/rclone/fs/walk"
Move dircache, oauthutil, rest and pacer modules into lib
2018-01-11 16:29:20 +00:00
"github.com/ncw/rclone/lib/rest"
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
"github.com/ncw/swift"
Convert to using github.com/pkg/errors everywhere
2016-06-12 14:06:02 +00:00
"github.com/pkg/errors"
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
)
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
// Register with Fs
func init() {
Create separate interface for object information. Take out read-only information about a Fs in a separate struct to limit access. See discussion at #282.
2016-02-18 11:35:25 +00:00
fs.Register(&fs.RegInfo{
Add help to remote chooser in rclone config - fixes #43
2016-02-15 18:11:53 +00:00
Name: "s3",
config,s3: hierarchical configuration support #2140 This introduces a method of making provider specific configuration within a remote. This is useful particularly in s3. This commit does the basic configuration in S3 for IBM COS.
2018-04-12 16:05:53 +00:00
Description: "Amazon S3 Compliant Storage Providers (AWS, Ceph, Dreamhost, IBM COS, Minio)",
Add help to remote chooser in rclone config - fixes #43
2016-02-15 18:11:53 +00:00
NewFs: NewFs,
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
Options: []fs.Option{{
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
Name: fs.ConfigProvider,
Help: "Choose your S3 provider.",
Examples: []fs.OptionExample{{
Value: "AWS",
Help: "Amazon Web Services (AWS) S3",
}, {
Value: "Ceph",
Help: "Ceph Object Storage",
}, {
Value: "DigitalOcean",
Help: "Digital Ocean Spaces",
}, {
Value: "Dreamhost",
Help: "Dreamhost DreamObjects",
}, {
Value: "IBMCOS",
Help: "IBM COS S3",
}, {
Value: "Minio",
Help: "Minio Object Storage",
}, {
Value: "Wasabi",
Help: "Wasabi Object Storage",
}, {
Value: "Other",
Help: "Any other S3 compatible provider",
}},
}, {
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
Name: "env_auth",
Help: "Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).\nOnly applies if access_key_id and secret_access_key is blank.",
Default: false,
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
Examples: []fs.OptionExample{{
Value: "false",
Help: "Enter AWS credentials in the next step",
}, {
Value: "true",
Help: "Get AWS credentials from the environment (env vars or IAM)",
}},
}, {
Name: "access_key_id",
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
Help: "AWS Access Key ID.\nLeave blank for anonymous access or runtime credentials.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
}, {
Name: "secret_access_key",
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
Help: "AWS Secret Access Key (password)\nLeave blank for anonymous access or runtime credentials.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
}, {
Name: "region",
Help: "Region to connect to.",
Provider: "AWS",
Examples: []fs.OptionExample{{
Value: "us-east-1",
Help: "The default endpoint - a good choice if you are unsure.\nUS Region, Northern Virginia or Pacific Northwest.\nLeave location constraint empty.",
}, {
Value: "us-east-2",
Help: "US East (Ohio) Region\nNeeds location constraint us-east-2.",
}, {
Value: "us-west-2",
Help: "US West (Oregon) Region\nNeeds location constraint us-west-2.",
}, {
Value: "us-west-1",
Help: "US West (Northern California) Region\nNeeds location constraint us-west-1.",
}, {
Value: "ca-central-1",
Help: "Canada (Central) Region\nNeeds location constraint ca-central-1.",
}, {
Value: "eu-west-1",
Help: "EU (Ireland) Region\nNeeds location constraint EU or eu-west-1.",
}, {
Value: "eu-west-2",
Help: "EU (London) Region\nNeeds location constraint eu-west-2.",
}, {
Value: "eu-central-1",
Help: "EU (Frankfurt) Region\nNeeds location constraint eu-central-1.",
}, {
Value: "ap-southeast-1",
Help: "Asia Pacific (Singapore) Region\nNeeds location constraint ap-southeast-1.",
}, {
Value: "ap-southeast-2",
Help: "Asia Pacific (Sydney) Region\nNeeds location constraint ap-southeast-2.",
}, {
Value: "ap-northeast-1",
Help: "Asia Pacific (Tokyo) Region\nNeeds location constraint ap-northeast-1.",
}, {
Value: "ap-northeast-2",
Help: "Asia Pacific (Seoul)\nNeeds location constraint ap-northeast-2.",
}, {
Value: "ap-south-1",
Help: "Asia Pacific (Mumbai)\nNeeds location constraint ap-south-1.",
}, {
Value: "sa-east-1",
Help: "South America (Sao Paulo) Region\nNeeds location constraint sa-east-1.",
}},
}, {
Name: "region",
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
Help: "Region to connect to.\nLeave blank if you are using an S3 clone and you don't have a region.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
Provider: "!AWS",
Examples: []fs.OptionExample{{
Value: "",
Help: "Use this if unsure. Will use v4 signatures and an empty region.",
}, {
Value: "other-v2-signature",
Help: "Use this only if v4 signatures don't work, eg pre Jewel/v10 CEPH.",
}},
}, {
Name: "endpoint",
Help: "Endpoint for S3 API.\nLeave blank if using AWS to use the default endpoint for the region.",
Provider: "AWS",
}, {
Name: "endpoint",
Help: "Endpoint for IBM COS S3 API.\nSpecify if using an IBM COS On Premise.",
Provider: "IBMCOS",
Examples: []fs.OptionExample{{
Value: "s3-api.us-geo.objectstorage.softlayer.net",
Help: "US Cross Region Endpoint",
}, {
Value: "s3-api.dal.us-geo.objectstorage.softlayer.net",
Help: "US Cross Region Dallas Endpoint",
}, {
Value: "s3-api.wdc-us-geo.objectstorage.softlayer.net",
Help: "US Cross Region Washington DC Endpoint",
}, {
Value: "s3-api.sjc-us-geo.objectstorage.softlayer.net",
Help: "US Cross Region San Jose Endpoint",
}, {
Value: "s3-api.us-geo.objectstorage.service.networklayer.com",
Help: "US Cross Region Private Endpoint",
}, {
Value: "s3-api.dal-us-geo.objectstorage.service.networklayer.com",
Help: "US Cross Region Dallas Private Endpoint",
}, {
Value: "s3-api.wdc-us-geo.objectstorage.service.networklayer.com",
Help: "US Cross Region Washington DC Private Endpoint",
}, {
Value: "s3-api.sjc-us-geo.objectstorage.service.networklayer.com",
Help: "US Cross Region San Jose Private Endpoint",
}, {
Value: "s3.us-east.objectstorage.softlayer.net",
Help: "US Region East Endpoint",
}, {
Value: "s3.us-east.objectstorage.service.networklayer.com",
Help: "US Region East Private Endpoint",
}, {
Value: "s3.us-south.objectstorage.softlayer.net",
Help: "US Region South Endpoint",
}, {
Value: "s3.us-south.objectstorage.service.networklayer.com",
Help: "US Region South Private Endpoint",
}, {
Value: "s3.eu-geo.objectstorage.softlayer.net",
Help: "EU Cross Region Endpoint",
}, {
Value: "s3.fra-eu-geo.objectstorage.softlayer.net",
Help: "EU Cross Region Frankfurt Endpoint",
}, {
Value: "s3.mil-eu-geo.objectstorage.softlayer.net",
Help: "EU Cross Region Milan Endpoint",
}, {
Value: "s3.ams-eu-geo.objectstorage.softlayer.net",
Help: "EU Cross Region Amsterdam Endpoint",
}, {
Value: "s3.eu-geo.objectstorage.service.networklayer.com",
Help: "EU Cross Region Private Endpoint",
}, {
Value: "s3.fra-eu-geo.objectstorage.service.networklayer.com",
Help: "EU Cross Region Frankfurt Private Endpoint",
}, {
Value: "s3.mil-eu-geo.objectstorage.service.networklayer.com",
Help: "EU Cross Region Milan Private Endpoint",
}, {
Value: "s3.ams-eu-geo.objectstorage.service.networklayer.com",
Help: "EU Cross Region Amsterdam Private Endpoint",
}, {
Value: "s3.eu-gb.objectstorage.softlayer.net",
Help: "Great Britan Endpoint",
}, {
Value: "s3.eu-gb.objectstorage.service.networklayer.com",
Help: "Great Britan Private Endpoint",
}, {
Value: "s3.ap-geo.objectstorage.softlayer.net",
Help: "APAC Cross Regional Endpoint",
}, {
Value: "s3.tok-ap-geo.objectstorage.softlayer.net",
Help: "APAC Cross Regional Tokyo Endpoint",
}, {
Value: "s3.hkg-ap-geo.objectstorage.softlayer.net",
Help: "APAC Cross Regional HongKong Endpoint",
}, {
Value: "s3.seo-ap-geo.objectstorage.softlayer.net",
Help: "APAC Cross Regional Seoul Endpoint",
}, {
Value: "s3.ap-geo.objectstorage.service.networklayer.com",
Help: "APAC Cross Regional Private Endpoint",
}, {
Value: "s3.tok-ap-geo.objectstorage.service.networklayer.com",
Help: "APAC Cross Regional Tokyo Private Endpoint",
}, {
Value: "s3.hkg-ap-geo.objectstorage.service.networklayer.com",
Help: "APAC Cross Regional HongKong Private Endpoint",
}, {
Value: "s3.seo-ap-geo.objectstorage.service.networklayer.com",
Help: "APAC Cross Regional Seoul Private Endpoint",
}, {
Value: "s3.mel01.objectstorage.softlayer.net",
Help: "Melbourne Single Site Endpoint",
}, {
Value: "s3.mel01.objectstorage.service.networklayer.com",
Help: "Melbourne Single Site Private Endpoint",
}, {
Value: "s3.tor01.objectstorage.softlayer.net",
Help: "Toronto Single Site Endpoint",
}, {
Value: "s3.tor01.objectstorage.service.networklayer.com",
Help: "Toronto Single Site Private Endpoint",
}},
}, {
Name: "endpoint",
Help: "Endpoint for S3 API.\nRequired when using an S3 clone.",
Provider: "!AWS,IBMCOS",
Examples: []fs.OptionExample{{
Value: "objects-us-west-1.dream.io",
Help: "Dream Objects endpoint",
Provider: "Dreamhost",
}, {
Value: "nyc3.digitaloceanspaces.com",
Help: "Digital Ocean Spaces New York 3",
Provider: "DigitalOcean",
}, {
Value: "ams3.digitaloceanspaces.com",
Help: "Digital Ocean Spaces Amsterdam 3",
Provider: "DigitalOcean",
}, {
Value: "sgp1.digitaloceanspaces.com",
Help: "Digital Ocean Spaces Singapore 1",
Provider: "DigitalOcean",
}, {
Value: "s3.wasabisys.com",
Help: "Wasabi Object Storage",
Provider: "Wasabi",
}},
}, {
Name: "location_constraint",
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
Help: "Location constraint - must be set to match the Region.\nUsed when creating buckets only.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
Provider: "AWS",
Examples: []fs.OptionExample{{
Value: "",
Help: "Empty for US Region, Northern Virginia or Pacific Northwest.",
}, {
Value: "us-east-2",
Help: "US East (Ohio) Region.",
}, {
Value: "us-west-2",
Help: "US West (Oregon) Region.",
}, {
Value: "us-west-1",
Help: "US West (Northern California) Region.",
}, {
Value: "ca-central-1",
Help: "Canada (Central) Region.",
}, {
Value: "eu-west-1",
Help: "EU (Ireland) Region.",
}, {
Value: "eu-west-2",
Help: "EU (London) Region.",
}, {
Value: "EU",
Help: "EU Region.",
}, {
Value: "ap-southeast-1",
Help: "Asia Pacific (Singapore) Region.",
}, {
Value: "ap-southeast-2",
Help: "Asia Pacific (Sydney) Region.",
}, {
Value: "ap-northeast-1",
Help: "Asia Pacific (Tokyo) Region.",
}, {
Value: "ap-northeast-2",
Help: "Asia Pacific (Seoul)",
}, {
Value: "ap-south-1",
Help: "Asia Pacific (Mumbai)",
}, {
Value: "sa-east-1",
Help: "South America (Sao Paulo) Region.",
}},
}, {
Name: "location_constraint",
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
Help: "Location constraint - must match endpoint when using IBM Cloud Public.\nFor on-prem COS, do not make a selection from this list, hit enter",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
Provider: "IBMCOS",
Examples: []fs.OptionExample{{
Value: "us-standard",
Help: "US Cross Region Standard",
}, {
Value: "us-vault",
Help: "US Cross Region Vault",
}, {
Value: "us-cold",
Help: "US Cross Region Cold",
}, {
Value: "us-flex",
Help: "US Cross Region Flex",
}, {
Value: "us-east-standard",
Help: "US East Region Standard",
}, {
Value: "us-east-vault",
Help: "US East Region Vault",
}, {
Value: "us-east-cold",
Help: "US East Region Cold",
}, {
Value: "us-east-flex",
Help: "US East Region Flex",
}, {
Value: "us-south-standard",
Help: "US Sout hRegion Standard",
}, {
Value: "us-south-vault",
Help: "US South Region Vault",
}, {
Value: "us-south-cold",
Help: "US South Region Cold",
}, {
Value: "us-south-flex",
Help: "US South Region Flex",
}, {
Value: "eu-standard",
Help: "EU Cross Region Standard",
}, {
Value: "eu-vault",
Help: "EU Cross Region Vault",
}, {
Value: "eu-cold",
Help: "EU Cross Region Cold",
}, {
Value: "eu-flex",
Help: "EU Cross Region Flex",
}, {
Value: "eu-gb-standard",
Help: "Great Britan Standard",
}, {
Value: "eu-gb-vault",
Help: "Great Britan Vault",
}, {
Value: "eu-gb-cold",
Help: "Great Britan Cold",
}, {
Value: "eu-gb-flex",
Help: "Great Britan Flex",
}, {
Value: "ap-standard",
Help: "APAC Standard",
}, {
Value: "ap-vault",
Help: "APAC Vault",
}, {
Value: "ap-cold",
Help: "APAC Cold",
}, {
Value: "ap-flex",
Help: "APAC Flex",
}, {
Value: "mel01-standard",
Help: "Melbourne Standard",
}, {
Value: "mel01-vault",
Help: "Melbourne Vault",
}, {
Value: "mel01-cold",
Help: "Melbourne Cold",
}, {
Value: "mel01-flex",
Help: "Melbourne Flex",
}, {
Value: "tor01-standard",
Help: "Toronto Standard",
}, {
Value: "tor01-vault",
Help: "Toronto Vault",
}, {
Value: "tor01-cold",
Help: "Toronto Cold",
}, {
Value: "tor01-flex",
Help: "Toronto Flex",
}},
}, {
Name: "location_constraint",
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
Help: "Location constraint - must be set to match the Region.\nLeave blank if not sure. Used when creating buckets only.",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
Provider: "!AWS,IBMCOS",
}, {
Name: "acl",
Help: "Canned ACL used when creating buckets and/or storing objects in S3.\nFor more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl",
Examples: []fs.OptionExample{{
Value: "private",
Help: "Owner gets FULL_CONTROL. No one else has access rights (default).",
Provider: "!IBMCOS",
}, {
Value: "public-read",
Help: "Owner gets FULL_CONTROL. The AllUsers group gets READ access.",
Provider: "!IBMCOS",
}, {
Value: "public-read-write",
Help: "Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.\nGranting this on a bucket is generally not recommended.",
Provider: "!IBMCOS",
}, {
Value: "authenticated-read",
Help: "Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.",
Provider: "!IBMCOS",
}, {
Value: "bucket-owner-read",
Help: "Object owner gets FULL_CONTROL. Bucket owner gets READ access.\nIf you specify this canned ACL when creating a bucket, Amazon S3 ignores it.",
Provider: "!IBMCOS",
}, {
Value: "bucket-owner-full-control",
Help: "Both the object owner and the bucket owner get FULL_CONTROL over the object.\nIf you specify this canned ACL when creating a bucket, Amazon S3 ignores it.",
Provider: "!IBMCOS",
}, {
Value: "private",
Help: "Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS",
config,s3: hierarchical configuration support #2140 This introduces a method of making provider specific configuration within a remote. This is useful particularly in s3. This commit does the basic configuration in S3 for IBM COS.
2018-04-12 16:05:53 +00:00
Provider: "IBMCOS",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
}, {
Value: "public-read",
Help: "Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS",
Provider: "IBMCOS",
}, {
Value: "public-read-write",
Help: "Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS",
config,s3: hierarchical configuration support #2140 This introduces a method of making provider specific configuration within a remote. This is useful particularly in s3. This commit does the basic configuration in S3 for IBM COS.
2018-04-12 16:05:53 +00:00
Provider: "IBMCOS",
s3: add in config for all the supported S3 providers #2140 These are AWS, Ceph, Dreamhost, IBM COS S3, Minio, Wasabi and Other. This configures endpoints where known and makes sure config doesn't appear where it isn't valid where possible.
2018-04-13 15:08:00 +00:00
}, {
Value: "authenticated-read",
Help: "Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS",
Provider: "IBMCOS",
}},
}, {
Name: "server_side_encryption",
Help: "The server-side encryption algorithm used when storing this object in S3.",
Provider: "AWS",
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
}, {
Value: "AES256",
Help: "AES256",
}},
}, {
Name: "storage_class",
Help: "The storage class to use when storing objects in S3.",
Provider: "AWS",
Examples: []fs.OptionExample{{
Value: "",
Help: "Default",
}, {
Value: "STANDARD",
Help: "Standard storage class",
}, {
Value: "REDUCED_REDUNDANCY",
Help: "Reduced redundancy storage class",
}, {
Value: "STANDARD_IA",
Help: "Standard Infrequent Access storage class",
}, {
Value: "ONEZONE_IA",
Help: "One Zone Infrequent Access storage class",
}},
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
}, {
Name: "chunk_size",
Help: "Chunk size to use for uploading",
Default: fs.SizeSuffix(s3manager.MinUploadPartSize),
Advanced: true,
}, {
Name: "disable_checksum",
Help: "Don't store MD5 checksum with object metadata",
Default: false,
Advanced: true,
}, {
Name: "session_token",
Help: "An AWS session token",
Hide: fs.OptionHideBoth,
Advanced: true,
}, {
Name: "upload_concurrency",
Help: "Concurrency for multipart uploads.",
Default: 2,
Advanced: true,
s3: Add --s3-force-path-style - fixes #2401
2018-07-18 15:40:59 +00:00
}, {
Name: "force_path_style",
Help: "If true use path style access if false use virtual hosted style.\nSome providers (eg Aliyun OSS or Netease COS) require this.",
Default: true,
Advanced: true,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
}},
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
})
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
// Constants
const (
s3: allow streaming upload of files with unknown file size (see #1614)
2017-09-15 18:20:32 +00:00
metaMtime = "Mtime" // the meta key to store mtime in - eg X-Amz-Meta-Mtime
s3: set/get the hash for multipart files - #523
2018-01-06 14:30:10 +00:00
metaMD5Hash = "Md5chksum" // the meta key to store md5hash in
s3: allow streaming upload of files with unknown file size (see #1614)
2017-09-15 18:20:32 +00:00
listChunkSize = 1000 // number of items to read at once
maxRetries = 10 // number of retries to make of operations
maxSizeForCopy = 5 * 1024 * 1024 * 1024 // The maximum size of object we can COPY
maxFileSize = 5 * 1024 * 1024 * 1024 * 1024 // largest possible upload file size
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
)
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
// Options defines the configuration for this backend
type Options struct {
Provider string `config:"provider"`
EnvAuth bool `config:"env_auth"`
AccessKeyID string `config:"access_key_id"`
SecretAccessKey string `config:"secret_access_key"`
Region string `config:"region"`
Endpoint string `config:"endpoint"`
LocationConstraint string `config:"location_constraint"`
ACL string `config:"acl"`
ServerSideEncryption string `config:"server_side_encryption"`
StorageClass string `config:"storage_class"`
ChunkSize fs.SizeSuffix `config:"chunk_size"`
DisableChecksum bool `config:"disable_checksum"`
SessionToken string `config:"session_token"`
UploadConcurrency int `config:"upload_concurrency"`
s3: Add --s3-force-path-style - fixes #2401
2018-07-18 15:40:59 +00:00
ForcePathStyle bool `config:"force_path_style"`
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
}
s3: add support for setting storage class in config and command line
2016-09-01 21:27:50 +00:00
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
// Fs represents a remote s3 server
type Fs struct {
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
name string // the name of the remote
root string // root of the bucket - ignore all objects above this
opt Options // parsed options
features *fs.Features // optional features
c *s3.S3 // the connection to the s3 server
ses *session.Session // the s3 session
bucket string // the bucket we are working on
bucketOKMu sync.Mutex // mutex to protect bucket OK
bucketOK bool // true if we have created the bucket
bucketDeleted bool // true if we have deleted the bucket
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
// Object describes a s3 object
type Object struct {
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
// Will definitely have everything but meta which may be nil
//
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 21:13:24 +00:00
// List will read everything but meta & mimeType - to fill
// that in you need to call readMetaData
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
fs *Fs // what this object is part of
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
remote string // The remote path
etag string // md5sum of the object
bytes int64 // size of the object
lastModified time.Time // Last modified
meta map[string]*string // The object metadata if known - may be nil
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 21:13:24 +00:00
mimeType string // MimeType of object - may be ""
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
// ------------------------------------------------------------
Fix golint warnings
2015-09-22 17:47:16 +00:00
// Name of the remote (as passed into NewFs)
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (f *Fs) Name() string {
Add Name() to Fs interface to return name as passed to NewFs
2015-08-22 15:53:11 +00:00
return f.name
}
Fix golint warnings
2015-09-22 17:47:16 +00:00
// Root of the remote (as passed into NewFs)
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (f *Fs) Root() string {
Add Root() to Fs interface
2015-09-01 19:45:27 +00:00
if f.root == "" {
return f.bucket
}
return f.bucket + "/" + f.root
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
// String converts this Fs to a string
func (f *Fs) String() string {
s3: support sub-bucket paths
2014-05-05 17:25:32 +00:00
if f.root == "" {
return fmt.Sprintf("S3 bucket %s", f.bucket)
}
return fmt.Sprintf("S3 bucket %s path %s", f.bucket, f.root)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Define a new Features() method for Fs Optional interfaces are becoming more important in rclone, --track-renames and --backup-dir both rely on them. Up to this point rclone has used interface upgrades to define optional behaviour on Fs objects. However when one Fs object wraps another it is very difficult for this scheme to work accurately. rclone has relied on specific error messages being returned when the interface isn't supported - this is unsatisfactory because it means you have to call the interface to see whether it is supported. This change enables accurate detection of optional interfaces by use of a Features struct as returned by an obligatory Fs.Features() method. The Features struct contains flags and function pointers which can be tested against nil to see whether they can be used. As a result crypt and hubic can accurately reflect the capabilities of the underlying Fs they are wrapping.
2017-01-13 17:21:47 +00:00
// Features returns the optional features of this Fs
func (f *Fs) Features() *fs.Features {
return f.features
}
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
// Pattern to match a s3 path
azureblob,b2,qingstor,s3,swift: remove leading / from paths - fixes #2484
2018-08-26 22:19:28 +00:00
var matcher = regexp.MustCompile(`^/*([^/]*)(.*)$`)
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
// parseParse parses a s3 'url'
func s3ParsePath(path string) (bucket, directory string, err error) {
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
parts := matcher.FindStringSubmatch(path)
if parts == nil {
Convert to using github.com/pkg/errors everywhere
2016-06-12 14:06:02 +00:00
err = errors.Errorf("couldn't parse bucket out of s3 path %q", path)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
} else {
Remove filesystem flags and put in config file with editor
2014-03-15 16:06:11 +00:00
bucket, directory = parts[1], parts[2]
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
directory = strings.Trim(directory, "/")
}
return
}
// s3Connection makes a connection to s3
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
func s3Connection(opt *Options) (*s3.S3, *session.Session, error) {
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
// Make the auth
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
v := credentials.Value{
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
AccessKeyID: opt.AccessKeyID,
SecretAccessKey: opt.SecretAccessKey,
SessionToken: opt.SessionToken,
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
}
s3: add support for ECS task IAM roles ECS container IAM metadata is in a different place than EC2 IAM metadata. Use defaults' RemoteCredProvider function to query the standard locations for the credentials. Give the ECS role precedence over the role available from the underlying EC2 instance.
2017-11-22 21:21:36 +00:00
lowTimeoutClient := &http.Client{Timeout: 1 * time.Second} // low timeout to ec2 metadata service
def := defaults.Get()
def.Config.HTTPClient = lowTimeoutClient
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
// first provider to supply a credential set "wins"
providers := []credentials.Provider{
// use static credentials if they're present (checked by provider)
&credentials.StaticProvider{Value: v},
// * Access Key ID: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY
// * Secret Access Key: AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY
&credentials.EnvProvider{},
s3: Look in S3 named profile files for credentials - fixes #2243
2018-04-16 11:14:35 +00:00
// A SharedCredentialsProvider retrieves credentials
// from the current user's home directory. It checks
// AWS_SHARED_CREDENTIALS_FILE and AWS_PROFILE too.
&credentials.SharedCredentialsProvider{},
s3: add support for ECS task IAM roles ECS container IAM metadata is in a different place than EC2 IAM metadata. Use defaults' RemoteCredProvider function to query the standard locations for the credentials. Give the ECS role precedence over the role available from the underlying EC2 instance.
2017-11-22 21:21:36 +00:00
// Pick up IAM role if we're in an ECS task
defaults.RemoteCredProvider(*def.Config, def.Handlers),
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
// Pick up IAM role in case we're on EC2
&ec2rolecreds.EC2RoleProvider{
Client: ec2metadata.New(session.New(), &aws.Config{
s3: add support for ECS task IAM roles ECS container IAM metadata is in a different place than EC2 IAM metadata. Use defaults' RemoteCredProvider function to query the standard locations for the credentials. Give the ECS role precedence over the role available from the underlying EC2 instance.
2017-11-22 21:21:36 +00:00
HTTPClient: lowTimeoutClient,
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
}),
ExpiryWindow: 3,
},
}
cred := credentials.NewChainCredentials(providers)
s3: allow anonymous access to public repositories - fixes #154 When setting up the remote, leave both the access key and secret key blank.
2015-09-29 08:58:03 +00:00
switch {
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
case opt.EnvAuth:
s3: Fix empty checks in auth
2016-02-09 17:19:33 +00:00
// No need for empty checks if "env_auth" is true
case v.AccessKeyID == "" && v.SecretAccessKey == "":
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
// if no access key/secret and iam is explicitly disabled then fall back to anon interaction
cred = credentials.AnonymousCredentials
case v.AccessKeyID == "":
Update to AWS SDK 0.10.0 Tested with S3 and Dreamhost Here is a link to the release notes: http://aws.amazon.com/releasenotes/5476699172355228
2015-10-30 11:50:45 +00:00
return nil, nil, errors.New("access_key_id not found")
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
case v.SecretAccessKey == "":
Update to AWS SDK 0.10.0 Tested with S3 and Dreamhost Here is a link to the release notes: http://aws.amazon.com/releasenotes/5476699172355228
2015-10-30 11:50:45 +00:00
return nil, nil, errors.New("secret_access_key not found")
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
if opt.Region == "" && opt.Endpoint == "" {
opt.Endpoint = "https://s3.amazonaws.com/"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
if opt.Region == "" {
opt.Region = "us-east-1"
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
awsConfig := aws.NewConfig().
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
WithRegion(opt.Region).
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
WithMaxRetries(maxRetries).
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
WithCredentials(cred).
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
WithEndpoint(opt.Endpoint).
Break the fs package up into smaller parts. The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.
2018-01-12 16:30:54 +00:00
WithHTTPClient(fshttp.NewClient(fs.Config)).
s3: Add --s3-force-path-style - fixes #2401
2018-07-18 15:40:59 +00:00
WithS3ForcePathStyle(opt.ForcePathStyle)
s3: make v2 signatures work for ceph
2015-08-10 10:02:34 +00:00
// awsConfig.WithLogLevel(aws.LogDebugWithSigning)
Update to AWS SDK 0.10.0 Tested with S3 and Dreamhost Here is a link to the release notes: http://aws.amazon.com/releasenotes/5476699172355228
2015-10-30 11:50:45 +00:00
ses := session.New()
c := s3.New(ses, awsConfig)
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
if opt.Region == "other-v2-signature" {
fs.Debugf(nil, "Using v2 auth")
s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114
2015-08-28 07:47:41 +00:00
signer := func(req *request.Request) {
s3: make v2 signatures work for ceph
2015-08-10 10:02:34 +00:00
// Ignore AnonymousCredentials object
Update to AWS SDK 0.10.0 Tested with S3 and Dreamhost Here is a link to the release notes: http://aws.amazon.com/releasenotes/5476699172355228
2015-10-30 11:50:45 +00:00
if req.Config.Credentials == credentials.AnonymousCredentials {
s3: make v2 signatures work for ceph
2015-08-10 10:02:34 +00:00
return
}
Add IAM role and Env credentials This will make the s3 provider authentaction logic - Configured credentials if both key and secret available - Anonymous if key and secret missing and env_auth not set - if env_auth is set to truthy (https://golang.org/pkg/strconv/#ParseBool) - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables - IAM role credentials as fallback
2016-02-01 13:11:27 +00:00
sign(v.AccessKeyID, v.SecretAccessKey, req.HTTPRequest)
s3: make v2 signatures work for ceph
2015-08-10 10:02:34 +00:00
}
c.Handlers.Sign.Clear()
s3: Fix after upstream API changes in aws-sdk-go/aws - fixes #114
2015-08-28 07:47:41 +00:00
c.Handlers.Sign.PushBackNamed(corehandlers.BuildContentLengthHandler)
s3: make v2 signatures work for ceph
2015-08-10 10:02:34 +00:00
c.Handlers.Sign.PushBack(signer)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Update to AWS SDK 0.10.0 Tested with S3 and Dreamhost Here is a link to the release notes: http://aws.amazon.com/releasenotes/5476699172355228
2015-10-30 11:50:45 +00:00
return c, ses, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
s3: add support for setting storage class in config and command line
2016-09-01 21:27:50 +00:00
// NewFs constructs an Fs from the path, bucket:path
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
// Parse config into Options struct
opt := new(Options)
err := configstruct.Set(m, opt)
if err != nil {
return nil, err
}
if opt.ChunkSize < fs.SizeSuffix(s3manager.MinUploadPartSize) {
return nil, errors.Errorf("s3 chunk size (%v) must be >= %v", opt.ChunkSize, fs.SizeSuffix(s3manager.MinUploadPartSize))
}
Implement single file operations for all file systems
2014-05-05 18:52:52 +00:00
bucket, directory, err := s3ParsePath(root)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
if err != nil {
return nil, err
}
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
c, ses, err := s3Connection(opt)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
if err != nil {
return nil, err
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
f := &Fs{
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
name: name,
root: directory,
opt: *opt,
c: c,
bucket: bucket,
ses: ses,
s3: add support for setting storage class in config and command line
2016-09-01 21:27:50 +00:00
}
fs: Add CanHaveEmptyDirectories and BucketBased feature flags to all remotes
2017-08-09 14:27:43 +00:00
f.features = (&fs.Features{
ReadMimeType: true,
WriteMimeType: true,
BucketBased: true,
}).Fill(f)
Implement single file operations for all file systems
2014-05-05 18:52:52 +00:00
if f.root != "" {
f.root += "/"
// Check to see if the object exists
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
req := s3.HeadObjectInput{
Bucket: &f.bucket,
Key: &directory,
}
_, err = f.c.HeadObject(&req)
Implement single file operations for all file systems
2014-05-05 18:52:52 +00:00
if err == nil {
f.root = path.Dir(directory)
if f.root == "." {
f.root = ""
} else {
f.root += "/"
}
Get rid of LimitedFs - FIXME needs docs on copying single files If remote:path points to a file make NewFs return a sentinel error fs.ErrorIsFile and an Fs which points to the parent. Use this to remove the LimitedFs and just add this file to the --files-from list. This means that server side operations can be used also. Fixes #518 Fixes #545
2016-06-21 17:01:53 +00:00
// return an error with an fs which points to the parent
return f, fs.ErrorIsFile
Implement single file operations for all file systems
2014-05-05 18:52:52 +00:00
}
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
// f.listMultipartUploads()
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
return f, nil
}
Rename NewFsObject to NewObject
2016-06-25 20:58:34 +00:00
// Return an Object from a path
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
//
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 20:23:20 +00:00
//If it can't be found it returns the error ErrorObjectNotFound.
func (f *Fs) newObjectWithInfo(remote string, info *s3.Object) (fs.Object, error) {
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
o := &Object{
fs: f,
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
remote: remote,
}
if info != nil {
// Set info but not meta
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
if info.LastModified == nil {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Logf(o, "Failed to read last modified")
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
o.lastModified = time.Now()
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
} else {
o.lastModified = *info.LastModified
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
o.etag = aws.StringValue(info.ETag)
o.bytes = aws.Int64Value(info.Size)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
} else {
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
err := o.readMetaData() // reads info and meta, returning an error
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
if err != nil {
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 20:23:20 +00:00
return nil, err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
}
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 20:23:20 +00:00
return o, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 20:23:20 +00:00
// NewObject finds the Object at remote. If it can't be found
// it returns the error fs.ErrorObjectNotFound.
func (f *Fs) NewObject(remote string) (fs.Object, error) {
Rename NewFsObject to NewObject
2016-06-25 20:58:34 +00:00
return f.newObjectWithInfo(remote, nil)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
// listFn is called from list to handle an object.
type listFn func(remote string, object *s3.Object, isDirectory bool) error
s3: support sub-bucket paths
2014-05-05 17:25:32 +00:00
// list the objects into the function supplied
//
Add a directory parameter to Fs.List()
2016-04-23 20:46:52 +00:00
// dir is the starting directory, "" for root
//
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
// Set recurse to read sub directories
func (f *Fs) list(dir string, recurse bool, fn listFn) error {
Add a directory parameter to Fs.List()
2016-04-23 20:46:52 +00:00
root := f.root
if dir != "" {
root += dir + "/"
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
maxKeys := int64(listChunkSize)
s3: support sub-bucket paths
2014-05-05 17:25:32 +00:00
delimiter := ""
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
if !recurse {
s3: support sub-bucket paths
2014-05-05 17:25:32 +00:00
delimiter = "/"
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
var marker *string
s3: list an unlimited number of items - fixes #22
2015-02-10 17:58:29 +00:00
for {
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
// FIXME need to implement ALL loop
req := s3.ListObjectsInput{
Bucket: &f.bucket,
Delimiter: &delimiter,
Add a directory parameter to Fs.List()
2016-04-23 20:46:52 +00:00
Prefix: &root,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
MaxKeys: &maxKeys,
Marker: marker,
}
resp, err := f.c.ListObjects(&req)
s3: list an unlimited number of items - fixes #22
2015-02-10 17:58:29 +00:00
if err != nil {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
if awsErr, ok := err.(awserr.RequestFailure); ok {
if awsErr.StatusCode() == http.StatusNotFound {
err = fs.ErrorDirNotFound
}
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
return err
}
rootLength := len(f.root)
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
if !recurse {
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
for _, commonPrefix := range resp.CommonPrefixes {
if commonPrefix.Prefix == nil {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Logf(f, "Nil common prefix received")
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
continue
s3: list an unlimited number of items - fixes #22
2015-02-10 17:58:29 +00:00
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
remote := *commonPrefix.Prefix
if !strings.HasPrefix(remote, f.root) {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Logf(f, "Odd name received %q", remote)
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
continue
}
remote = remote[rootLength:]
if strings.HasSuffix(remote, "/") {
remote = remote[:len(remote)-1]
}
err = fn(remote, &s3.Object{Key: &remote}, true)
if err != nil {
return err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
}
for _, object := range resp.Contents {
key := aws.StringValue(object.Key)
if !strings.HasPrefix(key, f.root) {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Logf(f, "Odd name received %q", key)
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
continue
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
remote := key[rootLength:]
s3: ignore zero length directory markers - fixes #1621
2018-03-19 17:41:46 +00:00
// is this a directory marker?
s3: ignore zero length directory markers at the root too
2018-03-21 20:09:37 +00:00
if (strings.HasSuffix(remote, "/") || remote == "") && *object.Size == 0 {
s3: fix index out of range error with --fast-list fixes #2388
2018-07-09 16:00:05 +00:00
if recurse && remote != "" {
s3: ignore zero length directory markers - fixes #1621
2018-03-19 17:41:46 +00:00
// add a directory in if --fast-list since will have no prefixes
remote = remote[:len(remote)-1]
err = fn(remote, &s3.Object{Key: &remote}, true)
if err != nil {
return err
}
}
continue // skip directory marker
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
err = fn(remote, object, false)
if err != nil {
return err
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
}
s3: list an unlimited number of items - fixes #22
2015-02-10 17:58:29 +00:00
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
if !aws.BoolValue(resp.IsTruncated) {
break
}
// Use NextMarker if set, otherwise use last Key
if resp.NextMarker == nil || *resp.NextMarker == "" {
s3: fix crash if a bad listing is received - fixes #1927 Caringo Swarm is returning a listing with IsTruncated set but no NextMarker and no Keys. Rclone doesn't know how to continue the listing at this point, so it returns an error rather than truncating the listing or risking a loop.
2017-12-20 16:40:41 +00:00
if len(resp.Contents) == 0 {
return errors.New("s3 protocol error: received listing with IsTruncated set, no NextMarker and no Contents")
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
marker = resp.Contents[len(resp.Contents)-1].Key
} else {
marker = resp.NextMarker
}
s3: support sub-bucket paths
2014-05-05 17:25:32 +00:00
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
return nil
s3: support sub-bucket paths
2014-05-05 17:25:32 +00:00
}
fs: rename BasicInfo to DirEntry
2017-06-30 09:54:14 +00:00
// Convert a list item into a DirEntry
func (f *Fs) itemToDirEntry(remote string, object *s3.Object, isDirectory bool) (fs.DirEntry, error) {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
if isDirectory {
size := int64(0)
if object.Size != nil {
size = *object.Size
}
Create fs.Directory interface and use it everywhere
2017-06-30 12:37:29 +00:00
d := fs.NewDir(remote, time.Time{}).SetSize(size)
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
return d, nil
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
}
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
o, err := f.newObjectWithInfo(remote, object)
if err != nil {
return nil, err
}
return o, nil
}
azureblob,b2,gcs,qingstor,s3,swift: Don't check for bucket/container presense if listing was OK In a typical rclone copy to a bucket/container based remote, before this change we were doing a list, followed by a HEAD of the bucket to check it existed before doing the copy. The fact the list succeeded means the bucket exists so mark it OK at that point. Issue #1421
2018-03-01 12:11:34 +00:00
// mark the bucket as being OK
func (f *Fs) markBucketOK() {
if f.bucket != "" {
f.bucketOKMu.Lock()
f.bucketOK = true
f.bucketDeleted = false
f.bucketOKMu.Unlock()
}
}
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
// listDir lists files and directories to out
func (f *Fs) listDir(dir string) (entries fs.DirEntries, err error) {
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
// List the objects and directories
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
err = f.list(dir, false, func(remote string, object *s3.Object, isDirectory bool) error {
entry, err := f.itemToDirEntry(remote, object, isDirectory)
if err != nil {
return err
}
if entry != nil {
entries = append(entries, entry)
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
}
return nil
})
if err != nil {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
return nil, err
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
}
azureblob,b2,gcs,qingstor,s3,swift: Don't check for bucket/container presense if listing was OK In a typical rclone copy to a bucket/container based remote, before this change we were doing a list, followed by a HEAD of the bucket to check it existed before doing the copy. The fact the list succeeded means the bucket exists so mark it OK at that point. Issue #1421
2018-03-01 12:11:34 +00:00
// bucket must be present if listing succeeded
f.markBucketOK()
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
return entries, nil
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
}
// listBuckets lists the buckets to out
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
func (f *Fs) listBuckets(dir string) (entries fs.DirEntries, err error) {
Add a directory parameter to Fs.List()
2016-04-23 20:46:52 +00:00
if dir != "" {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
return nil, fs.ErrorListBucketRequired
Add a directory parameter to Fs.List()
2016-04-23 20:46:52 +00:00
}
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
req := s3.ListBucketsInput{}
resp, err := f.c.ListBuckets(&req)
if err != nil {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
return nil, err
Refactor the List and ListDir interface Gives more accurate error propagation, control of depth of recursion and short circuit recursion where possible. Most of the the heavy lifting is done in the "fs" package, making file system implementations a bit simpler. This commit contains some code originally by Klaus Post. Fixes #316
2016-04-21 19:06:21 +00:00
}
for _, bucket := range resp.Buckets {
Create fs.Directory interface and use it everywhere
2017-06-30 12:37:29 +00:00
d := fs.NewDir(aws.StringValue(bucket.Name), aws.TimeValue(bucket.CreationDate))
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
entries = append(entries, d)
s3: support sub-bucket paths
2014-05-05 17:25:32 +00:00
}
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
return entries, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
// List the objects and directories in dir into entries. The
// entries can be returned in any order but should be for a
// complete directory.
//
// dir should be "" to list the root, and should not have
// trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(dir string) (entries fs.DirEntries, err error) {
s3: support sub-bucket paths
2014-05-05 17:25:32 +00:00
if f.bucket == "" {
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
return f.listBuckets(dir)
s3: support sub-bucket paths
2014-05-05 17:25:32 +00:00
}
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
return f.listDir(dir)
Implement fs.ListDir / lsd command
2013-01-23 22:43:20 +00:00
}
Add placeholder support for ListR interface. The ListR interface will be implemented by remotes that can do a recursive directory listing more efficiently than just recursing through the directories. These include the bucket based remotes.
2017-06-05 15:14:24 +00:00
// ListR lists the objects and directories of the Fs starting
// from dir recursively into out.
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
//
// dir should be "" to start from the root, and should not
// have trailing slashes.
//
// This should return ErrDirNotFound if the directory isn't
// found.
//
// It should call callback for each tranche of entries read.
// These need not be returned in any particular order. If
// callback returns an error then the listing will stop
// immediately.
//
// Don't implement this unless you have a more efficient way
// of listing recursively that doing a directory traversal.
func (f *Fs) ListR(dir string, callback fs.ListRCallback) (err error) {
if f.bucket == "" {
return fs.ErrorListBucketRequired
}
Break the fs package up into smaller parts. The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.
2018-01-12 16:30:54 +00:00
list := walk.NewListRHelper(callback)
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
err = f.list(dir, true, func(remote string, object *s3.Object, isDirectory bool) error {
entry, err := f.itemToDirEntry(remote, object, isDirectory)
if err != nil {
return err
}
return list.Add(entry)
})
if err != nil {
return err
}
azureblob,b2,gcs,qingstor,s3,swift: Don't check for bucket/container presense if listing was OK In a typical rclone copy to a bucket/container based remote, before this change we were doing a list, followed by a HEAD of the bucket to check it existed before doing the copy. The fact the list succeeded means the bucket exists so mark it OK at that point. Issue #1421
2018-03-01 12:11:34 +00:00
// bucket must be present if listing succeeded
f.markBucketOK()
Change List interface and add ListR optional interface This simplifies the implementation of remotes. The only required interface is now `List` which is a simple one level directory list. Optionally remotes may implement `ListR` if they have an efficient way of doing a recursive list.
2017-06-11 21:43:31 +00:00
return list.Flush()
Add placeholder support for ListR interface. The ListR interface will be implemented by remotes that can do a recursive directory listing more efficiently than just recursing through the directories. These include the bucket based remotes.
2017-06-05 15:14:24 +00:00
}
Rename NewFsObject to NewObject
2016-06-25 20:58:34 +00:00
// Put the Object into the bucket
Add options to Put, PutUnchecked and Update, add HashOption and speed up local * Add options to Put, PutUnchecked and Update for all Fses * Use these to create HashOption * Implement this in local * Pass the option in fs.Copy This has the effect that we only calculate hashes we need to in the local Fs which speeds up transfers significantly.
2017-05-28 11:44:22 +00:00
func (f *Fs) Put(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
// Temporary Object under construction
fs := &Object{
fs: f,
Create separate interface for object information. Take out read-only information about a Fs in a separate struct to limit access. See discussion at #282.
2016-02-18 11:35:25 +00:00
remote: src.Remote(),
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
}
Add options to Put, PutUnchecked and Update, add HashOption and speed up local * Add options to Put, PutUnchecked and Update for all Fses * Use these to create HashOption * Implement this in local * Pass the option in fs.Copy This has the effect that we only calculate hashes we need to in the local Fs which speeds up transfers significantly.
2017-05-28 11:44:22 +00:00
return fs, fs.Update(in, src, options...)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
s3: allow streaming upload of files with unknown file size (see #1614)
2017-09-15 18:20:32 +00:00
// PutStream uploads to the remote path with the modTime given of indeterminate size
func (f *Fs) PutStream(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
return f.Put(in, src, options...)
}
s3: Check if directory exists during Mkdir If you dont have privs to create a bucket in S3 but it exists, don't fail with an auth error, but detect that the mkdir was not needed and return successfully.
2016-02-24 01:58:55 +00:00
// Check if the bucket exists
s3: work around eventual consistency in bucket creation Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.
2017-06-29 11:26:14 +00:00
//
// NB this can return incorrect results if called immediately after bucket deletion
s3: Check if directory exists during Mkdir If you dont have privs to create a bucket in S3 but it exists, don't fail with an auth error, but detect that the mkdir was not needed and return successfully.
2016-02-24 01:58:55 +00:00
func (f *Fs) dirExists() (bool, error) {
req := s3.HeadBucketInput{
Bucket: &f.bucket,
}
_, err := f.c.HeadBucket(&req)
if err == nil {
return true, nil
}
if err, ok := err.(awserr.RequestFailure); ok {
if err.StatusCode() == http.StatusNotFound {
return false, nil
}
}
return false, err
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
// Mkdir creates the bucket if it doesn't exist
Add directory parameter to Rmdir and Mkdir #100 #831 This will enable rclone to manage directories properly in the future.
2016-11-25 21:52:43 +00:00
func (f *Fs) Mkdir(dir string) error {
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
f.bucketOKMu.Lock()
defer f.bucketOKMu.Unlock()
if f.bucketOK {
Add directory parameter to Rmdir and Mkdir #100 #831 This will enable rclone to manage directories properly in the future.
2016-11-25 21:52:43 +00:00
return nil
}
s3: work around eventual consistency in bucket creation Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.
2017-06-29 11:26:14 +00:00
if !f.bucketDeleted {
exists, err := f.dirExists()
if err == nil {
f.bucketOK = exists
}
if err != nil || exists {
return err
}
s3: Check if directory exists during Mkdir If you dont have privs to create a bucket in S3 but it exists, don't fail with an auth error, but detect that the mkdir was not needed and return successfully.
2016-02-24 01:58:55 +00:00
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
req := s3.CreateBucketInput{
Bucket: &f.bucket,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
ACL: &f.opt.ACL,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
}
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
if f.opt.LocationConstraint != "" {
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
req.CreateBucketConfiguration = &s3.CreateBucketConfiguration{
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
LocationConstraint: &f.opt.LocationConstraint,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
}
}
s3: work around eventual consistency in bucket creation Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.
2017-06-29 11:26:14 +00:00
_, err := f.c.CreateBucket(&req)
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
if err, ok := err.(awserr.Error); ok {
if err.Code() == "BucketAlreadyOwnedByYou" {
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
err = nil
S3: Fix mkdir not to error if bucket exists and implement SetModTime
2013-01-08 22:31:16 +00:00
}
}
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
if err == nil {
f.bucketOK = true
s3: work around eventual consistency in bucket creation Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.
2017-06-29 11:26:14 +00:00
f.bucketDeleted = false
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
}
S3: Fix mkdir not to error if bucket exists and implement SetModTime
2013-01-08 22:31:16 +00:00
return err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
swift, s3, googlecloudstorage: Don't delete the container/bucket if fs wasn't at root - fixes #172
2015-11-07 15:31:04 +00:00
// Rmdir deletes the bucket if the fs is at the root
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
//
// Returns an error if it isn't empty
Add directory parameter to Rmdir and Mkdir #100 #831 This will enable rclone to manage directories properly in the future.
2016-11-25 21:52:43 +00:00
func (f *Fs) Rmdir(dir string) error {
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
f.bucketOKMu.Lock()
defer f.bucketOKMu.Unlock()
Add directory parameter to Rmdir and Mkdir #100 #831 This will enable rclone to manage directories properly in the future.
2016-11-25 21:52:43 +00:00
if f.root != "" || dir != "" {
swift, s3, googlecloudstorage: Don't delete the container/bucket if fs wasn't at root - fixes #172
2015-11-07 15:31:04 +00:00
return nil
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
req := s3.DeleteBucketInput{
Bucket: &f.bucket,
}
_, err := f.c.DeleteBucket(&req)
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
if err == nil {
f.bucketOK = false
s3: work around eventual consistency in bucket creation Deleting a bucket then testing its existence can give the wrong result. Work around by keeping a flag as to whether we have deleted the bucket.
2017-06-29 11:26:14 +00:00
f.bucketDeleted = true
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
return err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Fix golint warnings
2015-09-22 17:47:16 +00:00
// Precision of the remote
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (f *Fs) Precision() time.Duration {
Implement Precision interface in all filesystems * drive: add mime type on upload * drive: correct info on upload to fix crash * local: measure time precision by making a local file and using Chtimes * swift: move snet parameter here * core: add -modify-window and compute the optimum one * core: use modify window when checking times
2013-01-18 23:21:02 +00:00
return time.Nanosecond
}
s3: fix server side copy and set modtime on files with + in - fixes #2001 This was broken in 64ea94c1a4e9d7a104b393a17d2e883593e20809 when putting a work-around for Digital Ocean. PathEscape has now been adjusted so it works with both providers.
2018-01-23 10:50:50 +00:00
// pathEscape escapes s as for a URL path. It uses rest.URLPathEscape
// but also escapes '+' for S3 and Digital Ocean spaces compatibility
func pathEscape(s string) string {
return strings.Replace(rest.URLPathEscape(s), "+", "%2B", -1)
}
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 18:48:08 +00:00
// Copy src to this remote using server side copy operations.
//
// This is stored with the remote path given
//
// It returns the destination Object and a possible error
//
// Will only be called if src.Fs().Name() == f.Name()
//
// If it isn't possible then return fs.ErrorCantCopy
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
swift, s3, gcs: create container if necessary on server side copy
2017-06-28 20:14:53 +00:00
err := f.Mkdir("")
if err != nil {
return nil, err
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
srcObj, ok := src.(*Object)
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 18:48:08 +00:00
if !ok {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Debugf(src, "Can't copy - not same remote type")
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 18:48:08 +00:00
return nil, fs.ErrorCantCopy
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
srcFs := srcObj.fs
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 18:48:08 +00:00
key := f.root + remote
s3: fix server side copy and set modtime on files with + in - fixes #2001 This was broken in 64ea94c1a4e9d7a104b393a17d2e883593e20809 when putting a work-around for Digital Ocean. PathEscape has now been adjusted so it works with both providers.
2018-01-23 10:50:50 +00:00
source := pathEscape(srcFs.bucket + "/" + srcFs.root + srcObj.remote)
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 18:48:08 +00:00
req := s3.CopyObjectInput{
Bucket: &f.bucket,
Key: &key,
CopySource: &source,
MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
}
swift, s3, gcs: create container if necessary on server side copy
2017-06-28 20:14:53 +00:00
_, err = f.c.CopyObject(&req)
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 18:48:08 +00:00
if err != nil {
return nil, err
}
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 20:23:20 +00:00
return f.NewObject(remote)
Implement server side copies if possible - fixes #99 Add optional fs.Copier interface Implemented for * swift * s3 * drive * dropbox * google cloud storage
2015-02-14 18:48:08 +00:00
}
Add support for multiple hash types. Add support for multiple hash types with negotiation of common hash types for comparison. Manually rebased version of #277 (see discussion there)
2016-01-11 12:39:33 +00:00
// Hashes returns the supported hash sets.
Break the fs package up into smaller parts. The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.
2018-01-12 16:30:54 +00:00
func (f *Fs) Hashes() hash.Set {
config/hash: rename more symbols after factoring into own package
2018-01-18 20:27:52 +00:00
return hash.Set(hash.MD5)
Add support for multiple hash types. Add support for multiple hash types with negotiation of common hash types for comparison. Manually rebased version of #277 (see discussion there)
2016-01-11 12:39:33 +00:00
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
// ------------------------------------------------------------
Fix golint warnings
2015-09-22 17:47:16 +00:00
// Fs returns the parent Fs
Create separate interface for object information. Take out read-only information about a Fs in a separate struct to limit access. See discussion at #282.
2016-02-18 11:35:25 +00:00
func (o *Object) Fs() fs.Info {
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
return o.fs
Factor the generic code into fs and add some more intefaces
2014-03-28 17:56:04 +00:00
}
// Return a string version
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (o *Object) String() string {
Factor the generic code into fs and add some more intefaces
2014-03-28 17:56:04 +00:00
if o == nil {
return "<nil>"
}
return o.remote
}
Fix golint warnings
2015-09-22 17:47:16 +00:00
// Remote returns the remote path
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (o *Object) Remote() string {
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
return o.remote
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
s3: ignore etags from multipart uploads which aren't md5sums - fixes #56
2015-05-09 09:37:43 +00:00
var matchMd5 = regexp.MustCompile(`^[0-9a-f]{32}$`)
Add support for multiple hash types. Add support for multiple hash types with negotiation of common hash types for comparison. Manually rebased version of #277 (see discussion there)
2016-01-11 12:39:33 +00:00
// Hash returns the Md5sum of an object returning a lowercase hex string
Break the fs package up into smaller parts. The purpose of this is to make it easier to maintain and eventually to allow the rclone backends to be re-used in other projects without having to use the rclone configuration system. The new code layout is documented in CONTRIBUTING.
2018-01-12 16:30:54 +00:00
func (o *Object) Hash(t hash.Type) (string, error) {
config/hash: rename more symbols after factoring into own package
2018-01-18 20:27:52 +00:00
if t != hash.MD5 {
return "", hash.ErrUnsupported
Add support for multiple hash types. Add support for multiple hash types with negotiation of common hash types for comparison. Manually rebased version of #277 (see discussion there)
2016-01-11 12:39:33 +00:00
}
s3: set/get the hash for multipart files - #523
2018-01-06 14:30:10 +00:00
hash := strings.Trim(strings.ToLower(o.etag), `"`)
s3: ignore etags from multipart uploads which aren't md5sums - fixes #56
2015-05-09 09:37:43 +00:00
// Check the etag is a valid md5sum
s3: set/get the hash for multipart files - #523
2018-01-06 14:30:10 +00:00
if !matchMd5.MatchString(hash) {
err := o.readMetaData()
if err != nil {
return "", err
}
if md5sum, ok := o.meta[metaMD5Hash]; ok {
md5sumBytes, err := base64.StdEncoding.DecodeString(*md5sum)
if err != nil {
return "", err
}
hash = hex.EncodeToString(md5sumBytes)
} else {
hash = ""
}
s3: ignore etags from multipart uploads which aren't md5sums - fixes #56
2015-05-09 09:37:43 +00:00
}
s3: set/get the hash for multipart files - #523
2018-01-06 14:30:10 +00:00
return hash, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
// Size returns the size of an object in bytes
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (o *Object) Size() int64 {
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
return o.bytes
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (o *Object) readMetaData() (err error) {
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
if o.meta != nil {
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
return nil
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
key := o.fs.root + o.remote
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
req := s3.HeadObjectInput{
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
Bucket: &o.fs.bucket,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
Key: &key,
s3: make reading metadata more reliable to work around eventual consistency problems
2014-07-28 21:32:15 +00:00
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
resp, err := o.fs.c.HeadObject(&req)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
if err != nil {
Make NewObject return an error * make it return an error * make a canonical error fs.ErrorNotFound * make a test for it * remove logs/debugs of error
2016-06-25 20:23:20 +00:00
if awsErr, ok := err.(awserr.RequestFailure); ok {
if awsErr.StatusCode() == http.StatusNotFound {
return fs.ErrorObjectNotFound
}
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
return err
}
s3: Treat missing Content-Length as 0 for some ceph installations
2014-05-16 15:27:53 +00:00
var size int64
// Ignore missing Content-Length assuming it is 0
// Some versions of ceph do this due their apache proxies
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
if resp.ContentLength != nil {
size = *resp.ContentLength
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
o.etag = aws.StringValue(resp.ETag)
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
o.bytes = size
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
o.meta = resp.Metadata
if resp.LastModified == nil {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Logf(o, "Failed to read last modified from HEAD: %v", err)
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
o.lastModified = time.Now()
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
} else {
o.lastModified = *resp.LastModified
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 21:13:24 +00:00
o.mimeType = aws.StringValue(resp.ContentType)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
return nil
}
// ModTime returns the modification time of the object
//
// It attempts to read the objects mtime and if that isn't present the
// LastModified returned in the http headers
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (o *Object) ModTime() time.Time {
s3,swift: Add --use-server-modtime `--use-server-modtime` stops s3 and swift retrieving the modtime from metadata which enables a fast sync mode with the `--update` flag.
2018-04-13 12:32:17 +00:00
if fs.Config.UseServerModTime {
return o.lastModified
}
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
err := o.readMetaData()
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
if err != nil {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Logf(o, "Failed to read metadata: %v", err)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
return time.Now()
}
// read mtime out of metadata if available
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
d, ok := o.meta[metaMtime]
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
if !ok || d == nil {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
// fs.Debugf(o, "No metadata")
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
return o.lastModified
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
modTime, err := swift.FloatStringToTime(*d)
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
if err != nil {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Logf(o, "Failed to read mtime from object: %v", err)
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
return o.lastModified
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
return modTime
}
Fix golint warnings
2015-09-22 17:47:16 +00:00
// SetModTime sets the modification time of the local fs object
Change the interface of SetModTime to return an error - #348
2016-03-22 15:07:10 +00:00
func (o *Object) SetModTime(modTime time.Time) error {
Refactor into sub filesystems
2013-06-27 19:13:07 +00:00
err := o.readMetaData()
S3: Fix mkdir not to error if bucket exists and implement SetModTime
2013-01-08 22:31:16 +00:00
if err != nil {
Change the interface of SetModTime to return an error - #348
2016-03-22 15:07:10 +00:00
return err
S3: Fix mkdir not to error if bucket exists and implement SetModTime
2013-01-08 22:31:16 +00:00
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
o.meta[metaMtime] = aws.String(swift.TimeToFloatString(modTime))
s3: skip SetModTime for objects > 5GB - fixes #534
2016-06-19 16:26:44 +00:00
if o.bytes >= maxSizeForCopy {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Debugf(o, "SetModTime is unsupported for objects bigger than %v bytes", fs.SizeSuffix(maxSizeForCopy))
s3: skip SetModTime for objects > 5GB - fixes #534
2016-06-19 16:26:44 +00:00
return nil
}
s3: Fix corrupting Content-Type on mod time update This fixes an issue where updating the modification time resets the content-type to the S3 default of binary/octet-stream which breaks static websites that expect an html file to have a content-type of text/html.
2016-01-02 08:58:48 +00:00
// Guess the content type
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 21:13:24 +00:00
mimeType := fs.MimeType(o)
s3: Fix corrupting Content-Type on mod time update This fixes an issue where updating the modification time resets the content-type to the S3 default of binary/octet-stream which breaks static websites that expect an html file to have a content-type of text/html.
2016-01-02 08:58:48 +00:00
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
// Copy the object to itself to update the metadata
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
key := o.fs.root + o.remote
sourceKey := o.fs.bucket + "/" + key
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
directive := s3.MetadataDirectiveReplace // replace metadata with that passed in
req := s3.CopyObjectInput{
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
Bucket: &o.fs.bucket,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
ACL: &o.fs.opt.ACL,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
Key: &key,
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 21:13:24 +00:00
ContentType: &mimeType,
s3: fix server side copy and set modtime on files with + in - fixes #2001 This was broken in 64ea94c1a4e9d7a104b393a17d2e883593e20809 when putting a work-around for Digital Ocean. PathEscape has now been adjusted so it works with both providers.
2018-01-23 10:50:50 +00:00
CopySource: aws.String(pathEscape(sourceKey)),
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
Metadata: o.meta,
MetadataDirective: &directive,
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
_, err = o.fs.c.CopyObject(&req)
Change the interface of SetModTime to return an error - #348
2016-03-22 15:07:10 +00:00
return err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Fix golint warnings
2015-09-22 17:47:16 +00:00
// Storable raturns a boolean indicating if this object is storable
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (o *Object) Storable() bool {
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
return true
}
// Open an object for read
Add options for Open and implement Range for all remotes
2016-09-10 10:29:57 +00:00
func (o *Object) Open(options ...fs.OpenOption) (in io.ReadCloser, err error) {
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
key := o.fs.root + o.remote
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
req := s3.GetObjectInput{
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
Bucket: &o.fs.bucket,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
Key: &key,
}
Add options for Open and implement Range for all remotes
2016-09-10 10:29:57 +00:00
for _, option := range options {
switch option.(type) {
case *fs.RangeOption, *fs.SeekOption:
_, value := option.Header()
req.Range = &value
default:
if option.Mandatory() {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Logf(o, "Unsupported mandatory option: %v", option)
Add options for Open and implement Range for all remotes
2016-09-10 10:29:57 +00:00
}
}
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
resp, err := o.fs.c.GetObject(&req)
s3: Error message for objects in glacier
2017-09-09 12:02:26 +00:00
if err, ok := err.(awserr.RequestFailure); ok {
if err.Code() == "InvalidObjectState" {
return nil, errors.Errorf("Object in GLACIER, restore first: %v", key)
}
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
if err != nil {
return nil, err
}
return resp.Body, nil
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Factor Fs.Put into Object.Update and call Update rather than Put if possible
2014-04-18 16:04:21 +00:00
// Update the Object from in with modTime and size
Add options to Put, PutUnchecked and Update, add HashOption and speed up local * Add options to Put, PutUnchecked and Update for all Fses * Use these to create HashOption * Implement this in local * Pass the option in fs.Copy This has the effect that we only calculate hashes we need to in the local Fs which speeds up transfers significantly.
2017-05-28 11:44:22 +00:00
func (o *Object) Update(in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
err := o.fs.Mkdir("")
if err != nil {
return err
}
Create separate interface for object information. Take out read-only information about a Fs in a separate struct to limit access. See discussion at #282.
2016-02-18 11:35:25 +00:00
modTime := src.ModTime()
s3: set/get the hash for multipart files - #523
2018-01-06 14:30:10 +00:00
size := src.Size()
Create separate interface for object information. Take out read-only information about a Fs in a separate struct to limit access. See discussion at #282.
2016-02-18 11:35:25 +00:00
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
uploader := s3manager.NewUploader(o.fs.ses, func(u *s3manager.Uploader) {
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
u.Concurrency = o.fs.opt.UploadConcurrency
Update to AWS SDK 0.10.0 Tested with S3 and Dreamhost Here is a link to the release notes: http://aws.amazon.com/releasenotes/5476699172355228
2015-10-30 11:50:45 +00:00
u.LeavePartsOnError = false
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
u.S3 = o.fs.c
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
u.PartSize = int64(o.fs.opt.ChunkSize)
s3: Fix uploading files bigger than 50GB - fixes #386
2016-03-10 16:48:55 +00:00
s3: allow streaming upload of files with unknown file size (see #1614)
2017-09-15 18:20:32 +00:00
if size == -1 {
// Make parts as small as possible while still being able to upload to the
// S3 file size limit. Rounded up to nearest MB.
u.PartSize = (((maxFileSize / s3manager.MaxUploadParts) >> 20) + 1) << 20
return
}
s3: Fix uploading files bigger than 50GB - fixes #386
2016-03-10 16:48:55 +00:00
// Adjust PartSize until the number of parts is small enough.
if size/u.PartSize >= s3manager.MaxUploadParts {
// Calculate partition size rounded up to the nearest MB
u.PartSize = (((size / s3manager.MaxUploadParts) >> 20) + 1) << 20
}
Update to AWS SDK 0.10.0 Tested with S3 and Dreamhost Here is a link to the release notes: http://aws.amazon.com/releasenotes/5476699172355228
2015-10-30 11:50:45 +00:00
})
Factor Fs.Put into Object.Update and call Update rather than Put if possible
2014-04-18 16:04:21 +00:00
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
// Set the mtime in the meta data
metadata := map[string]*string{
metaMtime: aws.String(swift.TimeToFloatString(modTime)),
}
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
if !o.fs.opt.DisableChecksum && size > uploader.PartSize {
config/hash: rename more symbols after factoring into own package
2018-01-18 20:27:52 +00:00
hash, err := src.Hash(hash.MD5)
s3: set/get the hash for multipart files - #523
2018-01-06 14:30:10 +00:00
if err == nil && matchMd5.MatchString(hash) {
hashBytes, err := hex.DecodeString(hash)
if err == nil {
metadata[metaMD5Hash] = aws.String(base64.StdEncoding.EncodeToString(hashBytes))
}
}
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
// Guess the content type
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 21:13:24 +00:00
mimeType := fs.MimeType(src)
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
key := o.fs.root + o.remote
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
req := s3manager.UploadInput{
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
Bucket: &o.fs.bucket,
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
ACL: &o.fs.opt.ACL,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
Key: &key,
Body: in,
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 21:13:24 +00:00
ContentType: &mimeType,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
Metadata: metadata,
//ContentLength: &size,
}
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
if o.fs.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
Add AES256 server-side encryption for s3 - Fixes #491 Add a configuration key and support for AES256 server-side encryption.
2016-06-14 20:22:54 +00:00
}
Implement new backend config system This unifies the 3 methods of reading config * command line * environment variable * config file And allows them all to be configured in all places. This is done by making the []fs.Option in the backend registration be the master source of what the backend options are. The backend changes are: * Use the new configmap.Mapper parameter * Use configstruct to parse it into an Options struct * Add all config to []fs.Option including defaults and help * Remove all uses of pflag * Remove all uses of config.FileGet
2018-05-14 17:06:57 +00:00
if o.fs.opt.StorageClass != "" {
req.StorageClass = &o.fs.opt.StorageClass
s3: add support for setting storage class in config and command line
2016-09-01 21:27:50 +00:00
}
swift, b2, gcs, s3: Fix moveto and copyto We now make sure the container/bucket is created before creating any objects.
2017-06-07 13:16:50 +00:00
_, err = uploader.Upload(&req)
s3: Read metadata after Update or Put
2014-07-19 11:37:11 +00:00
if err != nil {
return err
}
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
s3: Read metadata after Update or Put
2014-07-19 11:37:11 +00:00
// Read the metadata from the newly created object
swift, s3, dropbox: fix metadata read on Update() This was causing changed files to be marked as corrupted on upload
2014-07-20 10:23:05 +00:00
o.meta = nil // wipe old metadata
s3: Read metadata after Update or Put
2014-07-19 11:37:11 +00:00
err = o.readMetaData()
Factor Fs.Put into Object.Update and call Update rather than Put if possible
2014-04-18 16:04:21 +00:00
return err
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
// Remove an object
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
func (o *Object) Remove() error {
key := o.fs.root + o.remote
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
req := s3.DeleteObjectInput{
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
Bucket: &o.fs.bucket,
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
Key: &key,
}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
_, err := o.fs.c.DeleteObject(&req)
s3: use official github.com/aws/aws-sdk-go including multipart upload - fixes #101
2014-12-23 12:09:02 +00:00
return err
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
}
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 21:13:24 +00:00
// MimeType of an Object if known, "" otherwise
func (o *Object) MimeType() string {
err := o.readMetaData()
if err != nil {
Rename log functions and factor into own file
2017-02-09 11:01:20 +00:00
fs.Logf(o, "Failed to read metadata: %v", err)
Make ContentType be preserved for cloud -> cloud copies - fixes #733
2016-09-21 21:13:24 +00:00
return ""
}
return o.mimeType
}
Add initial S3 support * Requires a patched goamz to enable metadata reading and writing * Still todo: updating metadata * EU region not working yet
2013-01-08 18:53:35 +00:00
// Check the interfaces are satisfied
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
var (
s3: allow streaming upload of files with unknown file size (see #1614)
2017-09-15 18:20:32 +00:00
_ fs.Fs = &Fs{}
_ fs.Copier = &Fs{}
_ fs.PutStreamer = &Fs{}
_ fs.ListRer = &Fs{}
_ fs.Object = &Object{}
_ fs.MimeTyper = &Object{}
Be more constistent with naming in remotes * External objects are called Fs and Object * Object.fs always points to the Fs
2015-11-07 11:14:46 +00:00
)
Reference in a new issue Copy permalink