Specify and implement Docker-Upload-UUID

This changeset adds support for a header to identify docker upload uuids. This
id can be used as a key to manage local state for resumable uploads. The goal
is remove the necessity for a client to parse the url to get an upload uuid.
The restrictions for clients to use the location header are still strongly in
place.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
This commit is contained in:
Stephen J Day 2015-02-26 16:43:47 -08:00
parent 47a8ad7a61
commit 996235dc59
5 changed files with 72 additions and 11 deletions

View file

@ -347,6 +347,7 @@ with the upload URL in the `Location` header:
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: bytes=0-<offset> Range: bytes=0-<offset>
Content-Length: 0 Content-Length: 0
Docker-Upload-UUID: <uuid>
``` ```
The rest of the upload process can be carried out with the returned url, The rest of the upload process can be carried out with the returned url,
@ -358,6 +359,10 @@ try to assemble the it. While the `uuid` parameter may be an actual UUID, this
proposal imposes no constraints on the format and clients should never impose proposal imposes no constraints on the format and clients should never impose
any. any.
If clients need to correlate local upload state with remote upload state, the
contents of the `Docker-Upload-UUID` header should be used. Such an id can be
used to key the last used location header when implementing resumable uploads.
##### Upload Progress ##### Upload Progress
The progress and chunk coordination of the upload process will be coordinated The progress and chunk coordination of the upload process will be coordinated
@ -384,6 +389,7 @@ The response will be similar to the above, except will return 204 status:
204 No Content 204 No Content
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: bytes=0-<offset> Range: bytes=0-<offset>
Docker-Upload-UUID: <uuid>
``` ```
Note that the HTTP `Range` header byte ranges are inclusive and that will be Note that the HTTP `Range` header byte ranges are inclusive and that will be
@ -453,6 +459,7 @@ current status:
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: 0-<last valid range> Range: 0-<last valid range>
Content-Length: 0 Content-Length: 0
Docker-Upload-UUID: <uuid>
``` ```
If this response is received, the client should resume from the "last valid If this response is received, the client should resume from the "last valid
@ -471,6 +478,7 @@ be returned, including a `Range` header with the current upload status:
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: bytes=0-<offset> Range: bytes=0-<offset>
Content-Length: 0 Content-Length: 0
Docker-Upload-UUID: <uuid>
``` ```
##### Completed Upload ##### Completed Upload
@ -1787,6 +1795,7 @@ The following parameters should be specified on the request:
201 Created 201 Created
Location: <blob location> Location: <blob location>
Content-Length: 0 Content-Length: 0
Docker-Upload-UUID: <uuid>
``` ```
The blob has been created in the registry and is available at the provided location. The blob has been created in the registry and is available at the provided location.
@ -1797,6 +1806,7 @@ The following headers will be returned with the response:
|----|-----------| |----|-----------|
|`Location`|| |`Location`||
|`Content-Length`|The `Content-Length` header must be zero and the body must be empty.| |`Content-Length`|The `Content-Length` header must be zero and the body must be empty.|
|`Docker-Upload-UUID`|Identifies the docker upload uuid for the current request.|
@ -1890,6 +1900,7 @@ The following parameters should be specified on the request:
Content-Length: 0 Content-Length: 0
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: 0-0 Range: 0-0
Docker-Upload-UUID: <uuid>
``` ```
The upload has been created. The `Location` header must be used to complete the upload. The response should be identical to a `GET` request on the contents of the returned `Location` header. The upload has been created. The `Location` header must be used to complete the upload. The response should be identical to a `GET` request on the contents of the returned `Location` header.
@ -1901,6 +1912,7 @@ The following headers will be returned with the response:
|`Content-Length`|The `Content-Length` header must be zero and the body must be empty.| |`Content-Length`|The `Content-Length` header must be zero and the body must be empty.|
|`Location`|The location of the created upload. Clients should use the contents verbatim to complete the upload, adding parameters where required.| |`Location`|The location of the created upload. Clients should use the contents verbatim to complete the upload, adding parameters where required.|
|`Range`|Range header indicating the progress of the upload. When starting an upload, it will return an empty range, since no content has been received.| |`Range`|Range header indicating the progress of the upload. When starting an upload, it will return an empty range, since no content has been received.|
|`Docker-Upload-UUID`|Identifies the docker upload uuid for the current request.|
@ -2004,6 +2016,7 @@ The following parameters should be specified on the request:
204 No Content 204 No Content
Range: 0-<offset> Range: 0-<offset>
Content-Length: 0 Content-Length: 0
Docker-Upload-UUID: <uuid>
``` ```
The upload is known and in progress. The last received offset is available in the `Range` header. The upload is known and in progress. The last received offset is available in the `Range` header.
@ -2014,6 +2027,7 @@ The following headers will be returned with the response:
|----|-----------| |----|-----------|
|`Range`|Range indicating the current progress of the upload.| |`Range`|Range indicating the current progress of the upload.|
|`Content-Length`|The `Content-Length` header must be zero and the body must be empty.| |`Content-Length`|The `Content-Length` header must be zero and the body must be empty.|
|`Docker-Upload-UUID`|Identifies the docker upload uuid for the current request.|
@ -2161,6 +2175,7 @@ The following parameters should be specified on the request:
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: 0-<offset> Range: 0-<offset>
Content-Length: 0 Content-Length: 0
Docker-Upload-UUID: <uuid>
``` ```
The chunk of data has been accepted and the current progress is available in the range header. The updated upload location is available in the `Location` header. The chunk of data has been accepted and the current progress is available in the range header. The updated upload location is available in the `Location` header.
@ -2172,6 +2187,7 @@ The following headers will be returned with the response:
|`Location`|The location of the upload. Clients should assume this changes after each request. Clients should use the contents verbatim to complete the upload, adding parameters where required.| |`Location`|The location of the upload. Clients should assume this changes after each request. Clients should use the contents verbatim to complete the upload, adding parameters where required.|
|`Range`|Range indicating the current progress of the upload.| |`Range`|Range indicating the current progress of the upload.|
|`Content-Length`|The `Content-Length` header must be zero and the body must be empty.| |`Content-Length`|The `Content-Length` header must be zero and the body must be empty.|
|`Docker-Upload-UUID`|Identifies the docker upload uuid for the current request.|

View file

@ -347,6 +347,7 @@ with the upload URL in the `Location` header:
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: bytes=0-<offset> Range: bytes=0-<offset>
Content-Length: 0 Content-Length: 0
Docker-Upload-UUID: <uuid>
``` ```
The rest of the upload process can be carried out with the returned url, The rest of the upload process can be carried out with the returned url,
@ -358,6 +359,10 @@ try to assemble the it. While the `uuid` parameter may be an actual UUID, this
proposal imposes no constraints on the format and clients should never impose proposal imposes no constraints on the format and clients should never impose
any. any.
If clients need to correlate local upload state with remote upload state, the
contents of the `Docker-Upload-UUID` header should be used. Such an id can be
used to key the last used location header when implementing resumable uploads.
##### Upload Progress ##### Upload Progress
The progress and chunk coordination of the upload process will be coordinated The progress and chunk coordination of the upload process will be coordinated
@ -384,6 +389,7 @@ The response will be similar to the above, except will return 204 status:
204 No Content 204 No Content
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: bytes=0-<offset> Range: bytes=0-<offset>
Docker-Upload-UUID: <uuid>
``` ```
Note that the HTTP `Range` header byte ranges are inclusive and that will be Note that the HTTP `Range` header byte ranges are inclusive and that will be
@ -453,6 +459,7 @@ current status:
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: 0-<last valid range> Range: 0-<last valid range>
Content-Length: 0 Content-Length: 0
Docker-Upload-UUID: <uuid>
``` ```
If this response is received, the client should resume from the "last valid If this response is received, the client should resume from the "last valid
@ -471,6 +478,7 @@ be returned, including a `Range` header with the current upload status:
Location: /v2/<name>/blobs/uploads/<uuid> Location: /v2/<name>/blobs/uploads/<uuid>
Range: bytes=0-<offset> Range: bytes=0-<offset>
Content-Length: 0 Content-Length: 0
Docker-Upload-UUID: <uuid>
``` ```
##### Completed Upload ##### Completed Upload

View file

@ -72,6 +72,13 @@ var (
Format: "0", Format: "0",
} }
dockerUploadUUIDHeader = ParameterDescriptor{
Name: "Docker-Upload-UUID",
Description: "Identifies the docker upload uuid for the current request.",
Type: "uuid",
Format: "<uuid>",
}
unauthorizedResponse = ResponseDescriptor{ unauthorizedResponse = ResponseDescriptor{
Description: "The client does not have access to the repository.", Description: "The client does not have access to the repository.",
StatusCode: http.StatusUnauthorized, StatusCode: http.StatusUnauthorized,
@ -898,6 +905,7 @@ var routeDescriptors = []RouteDescriptor{
Format: "<blob location>", Format: "<blob location>",
}, },
contentLengthZeroHeader, contentLengthZeroHeader,
dockerUploadUUIDHeader,
}, },
}, },
}, },
@ -941,6 +949,7 @@ var routeDescriptors = []RouteDescriptor{
Format: "0-0", Format: "0-0",
Description: "Range header indicating the progress of the upload. When starting an upload, it will return an empty range, since no content has been received.", Description: "Range header indicating the progress of the upload. When starting an upload, it will return an empty range, since no content has been received.",
}, },
dockerUploadUUIDHeader,
}, },
}, },
}, },
@ -994,6 +1003,7 @@ var routeDescriptors = []RouteDescriptor{
Description: "Range indicating the current progress of the upload.", Description: "Range indicating the current progress of the upload.",
}, },
contentLengthZeroHeader, contentLengthZeroHeader,
dockerUploadUUIDHeader,
}, },
}, },
}, },
@ -1077,6 +1087,7 @@ var routeDescriptors = []RouteDescriptor{
Description: "Range indicating the current progress of the upload.", Description: "Range indicating the current progress of the upload.",
}, },
contentLengthZeroHeader, contentLengthZeroHeader,
dockerUploadUUIDHeader,
}, },
}, },
}, },

View file

@ -11,6 +11,7 @@ import (
"net/http/httputil" "net/http/httputil"
"net/url" "net/url"
"os" "os"
"path"
"reflect" "reflect"
"testing" "testing"
@ -97,8 +98,20 @@ func TestLayerAPI(t *testing.T) {
checkResponse(t, "checking head on non-existent layer", resp, http.StatusNotFound) checkResponse(t, "checking head on non-existent layer", resp, http.StatusNotFound)
// ------------------------------------------ // ------------------------------------------
// Start an upload and cancel // Start an upload, check the status then cancel
uploadURLBase := startPushLayer(t, env.builder, imageName) uploadURLBase, uploadUUID := startPushLayer(t, env.builder, imageName)
// A status check should work
resp, err = http.Get(uploadURLBase)
if err != nil {
t.Fatalf("unexpected error getting upload status: %v", err)
}
checkResponse(t, "status of deleted upload", resp, http.StatusNoContent)
checkHeaders(t, resp, http.Header{
"Location": []string{"*"},
"Range": []string{"0-0"},
"Docker-Upload-UUID": []string{uploadUUID},
})
req, err := http.NewRequest("DELETE", uploadURLBase, nil) req, err := http.NewRequest("DELETE", uploadURLBase, nil)
if err != nil { if err != nil {
@ -121,7 +134,7 @@ func TestLayerAPI(t *testing.T) {
// ----------------------------------------- // -----------------------------------------
// Do layer push with an empty body and different digest // Do layer push with an empty body and different digest
uploadURLBase = startPushLayer(t, env.builder, imageName) uploadURLBase, uploadUUID = startPushLayer(t, env.builder, imageName)
resp, err = doPushLayer(t, env.builder, imageName, layerDigest, uploadURLBase, bytes.NewReader([]byte{})) resp, err = doPushLayer(t, env.builder, imageName, layerDigest, uploadURLBase, bytes.NewReader([]byte{}))
if err != nil { if err != nil {
t.Fatalf("unexpected error doing bad layer push: %v", err) t.Fatalf("unexpected error doing bad layer push: %v", err)
@ -137,7 +150,7 @@ func TestLayerAPI(t *testing.T) {
t.Fatalf("unexpected error digesting empty buffer: %v", err) t.Fatalf("unexpected error digesting empty buffer: %v", err)
} }
uploadURLBase = startPushLayer(t, env.builder, imageName) uploadURLBase, uploadUUID = startPushLayer(t, env.builder, imageName)
pushLayer(t, env.builder, imageName, zeroDigest, uploadURLBase, bytes.NewReader([]byte{})) pushLayer(t, env.builder, imageName, zeroDigest, uploadURLBase, bytes.NewReader([]byte{}))
// ----------------------------------------- // -----------------------------------------
@ -150,7 +163,7 @@ func TestLayerAPI(t *testing.T) {
t.Fatalf("unexpected error digesting empty tar: %v", err) t.Fatalf("unexpected error digesting empty tar: %v", err)
} }
uploadURLBase = startPushLayer(t, env.builder, imageName) uploadURLBase, uploadUUID = startPushLayer(t, env.builder, imageName)
pushLayer(t, env.builder, imageName, emptyDigest, uploadURLBase, bytes.NewReader(emptyTar)) pushLayer(t, env.builder, imageName, emptyDigest, uploadURLBase, bytes.NewReader(emptyTar))
// ------------------------------------------ // ------------------------------------------
@ -158,7 +171,7 @@ func TestLayerAPI(t *testing.T) {
layerLength, _ := layerFile.Seek(0, os.SEEK_END) layerLength, _ := layerFile.Seek(0, os.SEEK_END)
layerFile.Seek(0, os.SEEK_SET) layerFile.Seek(0, os.SEEK_SET)
uploadURLBase = startPushLayer(t, env.builder, imageName) uploadURLBase, uploadUUID = startPushLayer(t, env.builder, imageName)
pushLayer(t, env.builder, imageName, layerDigest, uploadURLBase, layerFile) pushLayer(t, env.builder, imageName, layerDigest, uploadURLBase, layerFile)
// ------------------------ // ------------------------
@ -284,7 +297,7 @@ func TestManifestAPI(t *testing.T) {
expectedLayers[dgst] = rs expectedLayers[dgst] = rs
unsignedManifest.FSLayers[i].BlobSum = dgst unsignedManifest.FSLayers[i].BlobSum = dgst
uploadURLBase := startPushLayer(t, env.builder, imageName) uploadURLBase, _ := startPushLayer(t, env.builder, imageName)
pushLayer(t, env.builder, imageName, dgst, uploadURLBase, rs) pushLayer(t, env.builder, imageName, dgst, uploadURLBase, rs)
} }
@ -411,7 +424,7 @@ func putManifest(t *testing.T, msg, url string, v interface{}) *http.Response {
return resp return resp
} }
func startPushLayer(t *testing.T, ub *v2.URLBuilder, name string) string { func startPushLayer(t *testing.T, ub *v2.URLBuilder, name string) (location string, uuid string) {
layerUploadURL, err := ub.BuildBlobUploadURL(name) layerUploadURL, err := ub.BuildBlobUploadURL(name)
if err != nil { if err != nil {
t.Fatalf("unexpected error building layer upload url: %v", err) t.Fatalf("unexpected error building layer upload url: %v", err)
@ -424,12 +437,20 @@ func startPushLayer(t *testing.T, ub *v2.URLBuilder, name string) string {
defer resp.Body.Close() defer resp.Body.Close()
checkResponse(t, fmt.Sprintf("pushing starting layer push %v", name), resp, http.StatusAccepted) checkResponse(t, fmt.Sprintf("pushing starting layer push %v", name), resp, http.StatusAccepted)
u, err := url.Parse(resp.Header.Get("Location"))
if err != nil {
t.Fatalf("error parsing location header: %v", err)
}
uuid = path.Base(u.Path)
checkHeaders(t, resp, http.Header{ checkHeaders(t, resp, http.Header{
"Location": []string{"*"}, "Location": []string{"*"},
"Content-Length": []string{"0"}, "Content-Length": []string{"0"},
"Docker-Upload-UUID": []string{uuid},
}) })
return resp.Header.Get("Location") return resp.Header.Get("Location"), uuid
} }
// doPushLayer pushes the layer content returning the url on success returning // doPushLayer pushes the layer content returning the url on success returning

View file

@ -138,6 +138,8 @@ func (luh *layerUploadHandler) StartLayerUpload(w http.ResponseWriter, r *http.R
luh.Errors.Push(v2.ErrorCodeUnknown, err) luh.Errors.Push(v2.ErrorCodeUnknown, err)
return return
} }
w.Header().Set("Docker-Upload-UUID", luh.Upload.UUID())
w.WriteHeader(http.StatusAccepted) w.WriteHeader(http.StatusAccepted)
} }
@ -155,6 +157,7 @@ func (luh *layerUploadHandler) GetUploadStatus(w http.ResponseWriter, r *http.Re
return return
} }
w.Header().Set("Docker-Upload-UUID", luh.UUID)
w.WriteHeader(http.StatusNoContent) w.WriteHeader(http.StatusNoContent)
} }
@ -235,6 +238,7 @@ func (luh *layerUploadHandler) CancelLayerUpload(w http.ResponseWriter, r *http.
return return
} }
w.Header().Set("Docker-Upload-UUID", luh.UUID)
if err := luh.Upload.Cancel(); err != nil { if err := luh.Upload.Cancel(); err != nil {
ctxu.GetLogger(luh).Errorf("error encountered canceling upload: %v", err) ctxu.GetLogger(luh).Errorf("error encountered canceling upload: %v", err)
w.WriteHeader(http.StatusInternalServerError) w.WriteHeader(http.StatusInternalServerError)
@ -277,6 +281,7 @@ func (luh *layerUploadHandler) layerUploadResponse(w http.ResponseWriter, r *htt
return err return err
} }
w.Header().Set("Docker-Upload-UUID", luh.UUID)
w.Header().Set("Location", uploadURL) w.Header().Set("Location", uploadURL)
w.Header().Set("Content-Length", "0") w.Header().Set("Content-Length", "0")
w.Header().Set("Range", fmt.Sprintf("0-%d", luh.State.Offset)) w.Header().Set("Range", fmt.Sprintf("0-%d", luh.State.Offset))