TrueCloudLab/rclone
16
0
Fork 4
Code Issues 1 Pull requests Projects Releases 4 Packages Wiki Activity Actions

vendor: update all dependencies

Browse source
This commit is contained in:
Nick Craig-Wood 2017-07-23 08:51:42 +01:00
parent 0b6fba34a3
commit eb87cf6f12
2008 changed files with 352633 additions and 1004750 deletions
Download patch file Download diff file Expand all files Collapse all files

124
vendor/github.com/aws/aws-sdk-go/example/service/s3/presignURL/README.md generated vendored Normal file
View file

@ -0,0 +1,124 @@
# Presigned Amazon S3 API Operation Example
This example demonstrates how you can build a client application to retrieve and
upload object data from Amazon S3 without needing to know anything about Amazon
S3 or have access to any AWS credentials. Only the service would have knowledge
of how and where the objects are stored in Amazon S3.
The example is split into two parts `server.go` and `client.go`. These two parts
simulate the client/server architecture. In this example the client will represent
a third part user that will request resource URLs from the service. The service
will generate presigned S3 URLs which the client can use to download and
upload S3 object content.
The service supports generating presigned URLs for two S3 APIs; `GetObject` and
`PutObject`. The client will request a presigned URL from the service with an
object Key. In this example the value is the S3 object's `key`. Alternatively,
you could use your own pattern with no visible relation to the S3 object's key.
The server would then perform a cross reference with client provided value to
one that maps to the S3 object's key.
Before using the client to upload and download S3 objects you'll need to start the
service. The service will use the SDK's default credential chain to source your
AWS credentials. See the [`Configuring Credentials`](http://docs.aws.amazon.com/sdk-for-go/api/)
section of the SDK's API Reference guide on how the SDK loads your AWS credentials.
The server requires the S3 `-b bucket` the presigned URLs will be generated for. A
`-r region` is only needed if the bucket is in AWS China or AWS Gov Cloud. For
buckets in AWS the server will use the [`s3manager.GetBucketRegion`](http://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#GetBucketRegion) utility to lookup the bucket's region.
You should run the service in the background or in a separate terminal tab before
moving onto the client.
```sh
go run -tags example server/server.go -b mybucket
> Starting Server On: 127.0.0.1:8080
```
Use the `--help` flag to see a list of additional configuration flags, and their
defaults.
## Downloading an Amazon S3 Object
Use the client application to request a presigned URL from the server and use
that presigned URL to download the object from S3. Calling the client with the
`-get key` flag will do this. An optional `-f filename` flag can be provided as
well to write the object to. If no flag is provided the object will be written
to `stdout`
```sh
go run -tags example client/client.go -get "my-object/key" -f outputfilename
```
Use the `--help` flag to see a list of additional configuration flags, and their
defaults.
The following curl request demonstrates the request the client makes to the server
for the presigned URL for the `my-object/key` S3 object. The `method` query
parameter lets the server know that we are requesting the `GetObject`'s presigned
URL. The `method` value can be `GET` or `PUT` for the `GetObject` or `PutObject` APIs.
```sh
curl -v "http://127.0.0.1:8080/presign/my-object/key?method=GET"
```
The server will respond with a JSON value. The value contains three pieces of
information that the client will need to correctly make the request. First is
the presigned URL. This is the URL the client will make the request to. Second
is the HTTP method the request should be sent as. This is included to simplify
the client's request building. Finally the response will include a list of
additional headers that the client should include that the presigned request
was signed with.
```json
{
"URL": "https://mybucket.s3-us-west-2.amazonaws.com/my-object/key?<signature>",
"Method": "GET",
"Header": {
"x-amz-content-sha256":["UNSIGNED-PAYLOAD"]
}
}
```
With this URL our client will build a HTTP request for the S3 object's data. The
`client.go` will then write the object's data to the `filename` if one is provided,
or to `stdout` of a filename is not set in the command line arguments.
## Uploading a File to Amazon S3
Just like the download, uploading a file to S3 will use a presigned URL requested
from the server. The resigned URL will be built into an HTTP request using the
URL, Method, and Headers. The `-put key` flag will upload the content of `-f filename`
or stdin if no filename is provided to S3 using a presigned URL provided by the
service
```sh
go run -tags example client/client.go -put "my-object/key" -f filename
```
Like the download case this will make a HTTP request to the server for the
presigned URL. The Server will respond with a presigned URL for S3's `PutObject`
API operation. In addition the `method` query parameter the client will also
include a `contentLength` this value instructs the server to generate the presigned
PutObject request with a `Content-Length` header value included in the signature.
This is done so the content that is uploaded by the client can only be the size
the presigned request was generated for.
```sh
curl -v "http://127.0.0.1:8080/presign/my-object/key?method=PUT&contentLength=1024"
```
## Expanding the Example
This example provides a spring board you can use to vend presigned URLs to your
clients instead of streaming the object's content through your service. This
client and server example can be expanded and customized. Adding new functionality
such as additional constraints the server puts on the presigned URLs like
`Content-Type`.
In addition to adding constraints to the presigned URLs the service could be
updated to obfuscate S3 object's key. Instead of the client knowing the object's
key, a lookup system could be used instead. This could be substitution based,
or lookup into an external data store such as DynamoDB.

266
vendor/github.com/aws/aws-sdk-go/example/service/s3/presignURL/client/client.go generated vendored Normal file
View file

@ -0,0 +1,266 @@
// +build example
package main
import (
"bytes"
"encoding/json"
"flag"
"fmt"
"io"
"io/ioutil"
"net/http"
"os"
"strconv"
"strings"
)
// client.go is an example of a client that will request URLs from a service that
// the client will use to upload and download content with.
//
// The server must be started before the client is run.
//
// Use "--help" command line argument flag to see all options and defaults. If
// filename is not provided the client will read from stdin for uploads and
// write to stdout for downloads.
//
// Usage:
// go run -tags example client.go -get myObjectKey -f filename
func main() {
method, filename, key, serverURL := loadConfig()
var err error
switch method {
case GetMethod:
// Requests the URL from the server that the client will use to download
// the content from. The content will be written to the file pointed to
// by filename. Creating it if the file does not exist. If filename is
// not set the contents will be written to stdout.
err = downloadFile(serverURL, key, filename)
case PutMethod:
// Requests the URL from the service that the client will use to upload
// content to. The content will be read from the file pointed to by the
// filename. If the filename is not set, content will be read from stdin.
err = uploadFile(serverURL, key, filename)
}
if err != nil {
exitError(err)
}
}
// loadConfig configures the client based on the command line arguments used.
func loadConfig() (method Method, serverURL, key, filename string) {
var getKey, putKey string
flag.StringVar(&getKey, "get", "",
"Downloads the object from S3 by the `key`. Writes the object to a file the filename is provided, otherwise writes to stdout.")
flag.StringVar(&putKey, "put", "",
"Uploads data to S3 at the `key` provided. Uploads the file if filename is provided, otherwise reads from stdin.")
flag.StringVar(&serverURL, "s", "http://127.0.0.1:8080", "Required `URL` the client will request presigned S3 operation from.")
flag.StringVar(&filename, "f", "", "The `filename` of the file to upload and get from S3.")
flag.Parse()
var errs Errors
if len(serverURL) == 0 {
errs = append(errs, fmt.Errorf("server URL required"))
}
if !((len(getKey) != 0) != (len(putKey) != 0)) {
errs = append(errs, fmt.Errorf("either `get` or `put` can be provided, and one of the two is required."))
}
if len(getKey) > 0 {
method = GetMethod
key = getKey
} else {
method = PutMethod
key = putKey
}
if len(errs) > 0 {
fmt.Fprintf(os.Stderr, "Failed to load configuration:%v\n", errs)
flag.PrintDefaults()
os.Exit(1)
}
return method, filename, key, serverURL
}
// downloadFile will request a URL from the server that the client can download
// the content pointed to by "key". The content will be written to the file
// pointed to by filename, creating the file if it doesn't exist. If filename
// is not set the content will be written to stdout.
func downloadFile(serverURL, key, filename string) error {
var w *os.File
if len(filename) > 0 {
f, err := os.Create(filename)
if err != nil {
return fmt.Errorf("failed to create download file %s, %v", filename, err)
}
w = f
} else {
w = os.Stdout
}
defer w.Close()
// Get the presigned URL from the remote service.
req, err := getPresignedRequest(serverURL, "GET", key, 0)
if err != nil {
return fmt.Errorf("failed to get get presigned request, %v", err)
}
// Gets the file contents with the URL provided by the service.
resp, err := http.DefaultClient.Do(req)
if err != nil {
return fmt.Errorf("failed to do GET request, %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("failed to get S3 object, %d:%s",
resp.StatusCode, resp.Status)
}
if _, err = io.Copy(w, resp.Body); err != nil {
return fmt.Errorf("failed to write S3 object, %v", err)
}
return nil
}
// uploadFile will request a URL from the service that the client can use to
// upload content to. The content will be read from the file pointed to by filename.
// If filename is not set the content will be read from stdin.
func uploadFile(serverURL, key, filename string) error {
var r io.ReadCloser
var size int64
if len(filename) > 0 {
f, err := os.Open(filename)
if err != nil {
return fmt.Errorf("failed to open upload file %s, %v", filename, err)
}
// Get the size of the file so that the constraint of Content-Length
// can be included with the presigned URL. This can be used by the
// server or client to ensure the content uploaded is of a certain size.
//
// These constraints can further be expanded to include things like
// Content-Type. Additionally constraints such as X-Amz-Content-Sha256
// header set restricting the content of the file to only the content
// the client initially made the request with. This prevents the object
// from being overwritten or used to upload other unintended content.
stat, err := f.Stat()
if err != nil {
return fmt.Errorf("failed to stat file, %s, %v", filename, err)
}
size = stat.Size()
r = f
} else {
buf := &bytes.Buffer{}
io.Copy(buf, os.Stdin)
size = int64(buf.Len())
r = ioutil.NopCloser(buf)
}
defer r.Close()
// Get the Presigned URL from the remote service. Pass in the file's
// size if it is known so that the presigned URL returned will be required
// to be used with the size of content requested.
req, err := getPresignedRequest(serverURL, "PUT", key, size)
if err != nil {
return fmt.Errorf("failed to get get presigned request, %v", err)
}
req.Body = r
// Upload the file contents to S3.
resp, err := http.DefaultClient.Do(req)
if err != nil {
return fmt.Errorf("failed to do GET request, %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("failed to put S3 object, %d:%s",
resp.StatusCode, resp.Status)
}
return nil
}
// getPresignRequest will request a URL from the service for the content specified
// by the key and method. Returns a constructed Request that can be used to
// upload or download content with based on the method used.
//
// If the PUT method is used the request's Body will need to be set on the returned
// request value.
func getPresignedRequest(serverURL, method, key string, contentLen int64) (*http.Request, error) {
u := fmt.Sprintf("%s/presign/%s?method=%s&contentLength=%d",
serverURL, key, method, contentLen,
)
resp, err := http.Get(u)
if err != nil {
return nil, fmt.Errorf("failed to make request for presigned URL, %v", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("failed to get valid presign response, %s", resp.Status)
}
p := PresignResp{}
if err := json.NewDecoder(resp.Body).Decode(&p); err != nil {
return nil, fmt.Errorf("failed to decode response body, %v", err)
}
req, err := http.NewRequest(p.Method, p.URL, nil)
if err != nil {
return nil, fmt.Errorf("failed to build presigned request, %v", err)
}
for k, vs := range p.Header {
for _, v := range vs {
req.Header.Add(k, v)
}
}
// Need to ensure that the content length member is set of the HTTP Request
// or the request will not be transmitted correctly with a content length
// value across the wire.
if contLen := req.Header.Get("Content-Length"); len(contLen) > 0 {
req.ContentLength, _ = strconv.ParseInt(contLen, 10, 64)
}
return req, nil
}
type Method int
const (
PutMethod Method = iota
GetMethod
)
type Errors []error
func (es Errors) Error() string {
out := make([]string, len(es))
for _, e := range es {
out = append(out, e.Error())
}
return strings.Join(out, "\n")
}
type PresignResp struct {
Method, URL string
Header http.Header
}
func exitError(err error) {
fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}

186
vendor/github.com/aws/aws-sdk-go/example/service/s3/presignURL/server/server.go generated vendored Normal file
View file

@ -0,0 +1,186 @@
// +build example
package main
import (
"encoding/json"
"flag"
"fmt"
"net"
"net/http"
"os"
"strconv"
"strings"
"time"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/endpoints"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3iface"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
)
// server.go is an example of a service that vends lists for requests for presigned
// URLs for S3 objects. The service supports two S3 operations, "GetObject" and
// "PutObject".
//
// Example GetObject request to the service for the object with the key "MyObjectKey":
//
// curl -v "http://127.0.0.1:8080/presign/my-object/key?method=GET"
//
// Example PutObject request to the service for the object with the key "MyObjectKey":
//
// curl -v "http://127.0.0.1:8080/presign/my-object/key?method=PUT&contentLength=1024"
//
// Use "--help" command line argument flag to see all options and defaults.
//
// Usage:
// go run -tags example service.go -b myBucket
func main() {
addr, bucket, region := loadConfig()
// Create a AWS SDK for Go Session that will load credentials using the SDK's
// default credential change.
sess := session.Must(session.NewSession())
// Use the GetBucketRegion utility to lookup the bucket's region automatically.
// The service.go will only do this correctly for AWS regions. For AWS China
// and AWS Gov Cloud the region needs to be specified to let the service know
// to look in those partitions instead of AWS.
if len(region) == 0 {
var err error
region, err = s3manager.GetBucketRegion(aws.BackgroundContext(), sess, bucket, endpoints.UsWest2RegionID)
if err != nil {
exitError(fmt.Errorf("failed to get bucket region, %v", err))
}
}
// Create a new S3 service client that will be use by the service to generate
// presigned URLs with. Not actual API requests will be made with this client.
// The credentials loaded when the Session was created above will be used
// to sign the requests with.
s3Svc := s3.New(sess, &aws.Config{
Region: aws.String(region),
})
// Start the server listening and serve presigned URLs for GetObject and
// PutObject requests.
if err := listenAndServe(addr, bucket, s3Svc); err != nil {
exitError(err)
}
}
func loadConfig() (addr, bucket, region string) {
flag.StringVar(&bucket, "b", "", "S3 `bucket` object should be uploaded to.")
flag.StringVar(&region, "r", "", "AWS `region` the bucket exists in, If not set region will be looked up, only valid for AWS Regions, not AWS China or Gov Cloud.")
flag.StringVar(&addr, "a", "127.0.0.1:8080", "The TCP `address` the server will be started on.")
flag.Parse()
if len(bucket) == 0 {
fmt.Fprintln(os.Stderr, "bucket is required")
flag.PrintDefaults()
os.Exit(1)
}
return addr, bucket, region
}
func listenAndServe(addr, bucket string, svc s3iface.S3API) error {
l, err := net.Listen("tcp", addr)
if err != nil {
return fmt.Errorf("failed to start service listener, %v", err)
}
const presignPath = "/presign/"
// Create the HTTP handler for the "/presign/" path prefix. This will handle
// all requests on this path, extracting the object's key from the path.
http.HandleFunc(presignPath, func(w http.ResponseWriter, r *http.Request) {
var u string
var err error
var signedHeaders http.Header
query := r.URL.Query()
var contentLen int64
// Optionally the Content-Length header can be included with the signature
// of the request. This is helpful to ensure the content uploaded is the
// size that is expected. Constraints like these can be further expanded
// with headers such as `Content-Type`. These can be enforced by the service
// requiring the client to satisfying those constraints when uploading
//
// In addition the client could provide the service with a SHA256 of the
// content to be uploaded. This prevents any other third party from uploading
// anything else with the presigned URL
if contLenStr := query.Get("contentLength"); len(contLenStr) > 0 {
contentLen, err = strconv.ParseInt(contLenStr, 10, 64)
if err != nil {
fmt.Fprintf(os.Stderr, "unable to parse request content length, %v", err)
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
}
// Extract the object key from the path
key := strings.Replace(r.URL.Path, presignPath, "", 1)
method := query.Get("method")
switch method {
case "PUT":
// For creating PutObject presigned URLs
fmt.Println("Received request to presign PutObject for,", key)
sdkReq, _ := svc.PutObjectRequest(&s3.PutObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
// If ContentLength is 0 the header will not be included in the signature.
ContentLength: aws.Int64(contentLen),
})
u, signedHeaders, err = sdkReq.PresignRequest(15 * time.Minute)
case "GET":
// For creating GetObject presigned URLs
fmt.Println("Received request to presign GetObject for,", key)
sdkReq, _ := svc.GetObjectRequest(&s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
})
u, signedHeaders, err = sdkReq.PresignRequest(15 * time.Minute)
default:
fmt.Fprintf(os.Stderr, "invalid method provided, %s, %v\n", method, err)
err = fmt.Errorf("invalid request")
}
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
// Create the response back to the client with the information on the
// presigned request and additional headers to include.
if err := json.NewEncoder(w).Encode(PresignResp{
Method: method,
URL: u,
Header: signedHeaders,
}); err != nil {
fmt.Fprintf(os.Stderr, "failed to encode presign response, %v", err)
}
})
fmt.Println("Starting Server On:", "http://"+l.Addr().String())
s := &http.Server{}
return s.Serve(l)
}
// PresignResp provides the Go representation of the JSON value that will be
// sent to the client.
type PresignResp struct {
Method, URL string
Header http.Header
}
func exitError(err error) {
fmt.Fprintln(os.Stderr, err.Error())
os.Exit(1)
}