TrueCloudLab/distribution
13
0
Fork 3
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions
distribution/registry/handlers/api_test.go

838 lines
24 KiB
Go
Raw Normal View History

Move registry package into handler package The goal is to free up the distribution/registry package to include common registry types. This moves the webapp definitions out of the way to allow for this change in the future. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 01:25:40 +00:00
package handlers
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
import (
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
"bytes"
"encoding/json"
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
"fmt"
"io"
Implement V2 API base endpoint This implements a base endpoint that will respond with a 200 OK and an empty json response. Such an endpoint can be used as to ping the v2 service or as an endpoint to check authorization status.
2014-12-11 06:33:36 +00:00
"io/ioutil"
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
"net/http"
"net/http/httptest"
"net/http/httputil"
"net/url"
"os"
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
"path"
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
"reflect"
Path prefix support for running registry somewhere other than root of server Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-02-24 22:59:01 +00:00
"strings"
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
"testing"
Move from docker-registry to distribution
2014-12-24 00:01:38 +00:00
"github.com/docker/distribution/configuration"
"github.com/docker/distribution/digest"
Move manifest to discrete package Because manifests and their signatures are a discrete component of the registry, we are moving the definitions into a separate package. This causes us to lose some test coverage, but we can fill this in shortly. No changes have been made to the external interfaces, but they are likely to come. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-02 21:21:29 +00:00
"github.com/docker/distribution/manifest"
Move storagedriver package to registry/storage/driver This change is slightly more complex than previous package maves in that the package name changed. To address this, we simply always reference the package driver as storagedriver to avoid compatbility issues with existing code. While unfortunate, this can be cleaned up over time. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 02:14:23 +00:00
"github.com/docker/distribution/registry/api/v2"
_ "github.com/docker/distribution/registry/storage/driver/inmemory"
Move testutil package to top-level Since the common package no longer exists, the testutil package is being moved up to the root. Ideally, we don't have large omnibus packages, like testutil, but we can fix that in another refactoring round. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-06 00:53:13 +00:00
"github.com/docker/distribution/testutil"
Lock down HTTP API error codes This commit locks down the set of http error codes that will be part of the inital V2 specification, proposed in docker/docker#9015. The naming order has been slightly changed and there are few tweaks to ensure all conditions are captured but this will be set the docker core will be impleemnted against. To support this, the errors have been moved into an api/errors package. A new type, ErrorDescriptor, has been defined to centralize the code, message and definitions used with each type. The information therein can be used to generate documentation and response code mappings (yet to come...). In addition to the refactoring that came along with this change, several tests have been added to ensure serialization round trips are reliable. This allows better support for using these error types on the client and server side. This is coupled with some tweaks in the client code to fix issues with error reporting. Other fixes in the client include moving client-specific errors out of the base package and ensuring that we have correct parameters for finishing uploads.
2014-12-10 05:25:54 +00:00
"github.com/docker/libtrust"
"github.com/gorilla/handlers"
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
"golang.org/x/net/context"
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
)
Implement V2 API base endpoint This implements a base endpoint that will respond with a 200 OK and an empty json response. Such an endpoint can be used as to ping the v2 service or as an endpoint to check authorization status.
2014-12-11 06:33:36 +00:00
// TestCheckAPI hits the base endpoint (/v2/) ensures we return the specified
// 200 OK response.
func TestCheckAPI(t *testing.T) {
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
env := newTestEnv(t)
Implement V2 API base endpoint This implements a base endpoint that will respond with a 200 OK and an empty json response. Such an endpoint can be used as to ping the v2 service or as an endpoint to check authorization status.
2014-12-11 06:33:36 +00:00
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
baseURL, err := env.builder.BuildBaseURL()
Implement V2 API base endpoint This implements a base endpoint that will respond with a 200 OK and an empty json response. Such an endpoint can be used as to ping the v2 service or as an endpoint to check authorization status.
2014-12-11 06:33:36 +00:00
if err != nil {
t.Fatalf("unexpected error building base url: %v", err)
}
resp, err := http.Get(baseURL)
if err != nil {
t.Fatalf("unexpected error issuing request: %v", err)
}
defer resp.Body.Close()
checkResponse(t, "issuing api base check", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{
Use full json content type with charset parameter
2014-12-31 04:09:45 +00:00
"Content-Type": []string{"application/json; charset=utf-8"},
Implement V2 API base endpoint This implements a base endpoint that will respond with a 200 OK and an empty json response. Such an endpoint can be used as to ping the v2 service or as an endpoint to check authorization status.
2014-12-11 06:33:36 +00:00
"Content-Length": []string{"2"},
})
p, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Fatalf("unexpected error reading response body: %v", err)
}
if string(p) != "{}" {
t.Fatalf("unexpected response body: %v", string(p))
}
}
Path prefix support for running registry somewhere other than root of server Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-02-24 22:59:01 +00:00
func TestURLPrefix(t *testing.T) {
config := configuration.Configuration{
Storage: configuration.Storage{
"inmemory": configuration.Parameters{},
},
}
config.HTTP.Prefix = "/test/"
env := newTestEnvWithConfig(t, &config)
baseURL, err := env.builder.BuildBaseURL()
if err != nil {
t.Fatalf("unexpected error building base url: %v", err)
}
parsed, _ := url.Parse(baseURL)
if !strings.HasPrefix(parsed.Path, config.HTTP.Prefix) {
t.Fatalf("Prefix %v not included in test url %v", config.HTTP.Prefix, baseURL)
}
resp, err := http.Get(baseURL)
if err != nil {
t.Fatalf("unexpected error issuing request: %v", err)
}
defer resp.Body.Close()
checkResponse(t, "issuing api base check", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{
"Content-Type": []string{"application/json; charset=utf-8"},
"Content-Length": []string{"2"},
})
}
Refactor Blob Service API This PR refactors the blob service API to be oriented around blob descriptors. Identified by digests, blobs become an abstract entity that can be read and written using a descriptor as a handle. This allows blobs to take many forms, such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented operations to better integrate with blob agnostic APIs (such as the `io` package). The error definitions are now better organized to reflect conditions that can only be seen when interacting with the blob API. The main benefit of this is to separate the much smaller metadata from large file storage. Many benefits also follow from this. Reading and writing has been separated into discrete services. Backend implementation is also simplified, by reducing the amount of metadata that needs to be picked up to simply serve a read. This also improves cacheability. "Opening" a blob simply consists of an access check (Stat) and a path calculation. Caching is greatly simplified and we've made the mapping of provisional to canonical hashes a first-class concept. BlobDescriptorService and BlobProvider can be combined in different ways to achieve varying effects. Recommend Review Approach ------------------------- This is a very large patch. While apologies are in order, we are getting a considerable amount of refactoring. Most changes follow from the changes to the root package (distribution), so start there. From there, the main changes are in storage. Looking at (*repository).Blobs will help to understand the how the linkedBlobStore is wired. One can explore the internals within and also branch out into understanding the changes to the caching layer. Following the descriptions below will also help to guide you. To reduce the chances for regressions, it was critical that major changes to unit tests were avoided. Where possible, they are left untouched and where not, the spirit is hopefully captured. Pay particular attention to where behavior may have changed. Storage ------- The primary changes to the `storage` package, other than the interface updates, were to merge the layerstore and blobstore. Blob access is now layered even further. The first layer, blobStore, exposes a global `BlobStatter` and `BlobProvider`. Operations here provide a fast path for most read operations that don't take access control into account. The `linkedBlobStore` layers on top of the `blobStore`, providing repository- scoped blob link management in the backend. The `linkedBlobStore` implements the full `BlobStore` suite, providing access-controlled, repository-local blob writers. The abstraction between the two is slightly broken in that `linkedBlobStore` is the only channel under which one can write into the global blob store. The `linkedBlobStore` also provides flexibility in that it can act over different link sets depending on configuration. This allows us to use the same code for signature links, manifest links and blob links. Eventually, we will fully consolidate this storage. The improved cache flow comes from the `linkedBlobStatter` component of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to provide a simple cache hierarchy that should streamline access checks on read and write operations, or at least provide a single path to optimize. The metrics have been changed in a slightly incompatible way since the former operations, Fetch and Exists, are no longer relevant. The fileWriter and fileReader have been slightly modified to support the rest of the changes. The most interesting is the removal of the `Stat` call from `newFileReader`. This was the source of unnecessary round trips that were only present to look up the size of the resulting reader. Now, one must simply pass in the size, requiring the caller to decide whether or not the `Stat` call is appropriate. In several cases, it turned out the caller already had the size already. The `WriterAt` implementation has been removed from `fileWriter`, since it is no longer required for `BlobWriter`, reducing the number of paths which writes may take. Cache ----- Unfortunately, the `cache` package required a near full rewrite. It was pretty mechanical in that the cache is oriented around the `BlobDescriptorService` slightly modified to include the ability to set the values for individual digests. While the implementation is oriented towards caching, it can act as a primary store. Provisions are in place to have repository local metadata, in addition to global metadata. Fallback is implemented as a part of the storage package to maintain this flexibility. One unfortunate side-effect is that caching is now repository-scoped, rather than global. This should have little effect on performance but may increase memory usage. Handlers -------- The `handlers` package has been updated to leverage the new API. For the most part, the changes are superficial or mechanical based on the API changes. This did expose a bug in the handling of provisional vs canonical digests that was fixed in the unit tests. Configuration ------------- One user-facing change has been made to the configuration and is updated in the associated documentation. The `layerinfo` cache parameter has been deprecated by the `blobdescriptor` cache parameter. Both are equivalent and configuration files should be backward compatible. Notifications ------------- Changes the `notification` package are simply to support the interface changes. Context ------- A small change has been made to the tracing log-level. Traces have been moved from "info" to "debug" level to reduce output when not needed. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-12 07:10:29 +00:00
// TestBlobAPI conducts a full test of the of the blob api.
func TestBlobAPI(t *testing.T) {
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
// TODO(stevvooe): This test code is complete junk but it should cover the
// complete flow. This must be broken down and checked against the
// specification *before* we submit the final to docker core.
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
env := newTestEnv(t)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
imageName := "foo/bar"
// "build" our layer file
layerFile, tarSumStr, err := testutil.CreateRandomTarFile()
if err != nil {
t.Fatalf("error creating random layer file: %v", err)
}
layerDigest := digest.Digest(tarSumStr)
// -----------------------------------
// Test fetch for non-existent content
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
layerURL, err := env.builder.BuildBlobURL(imageName, layerDigest)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
if err != nil {
t.Fatalf("error building url: %v", err)
}
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
resp, err := http.Get(layerURL)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
if err != nil {
t.Fatalf("unexpected error fetching non-existent layer: %v", err)
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkResponse(t, "fetching non-existent content", resp, http.StatusNotFound)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
// ------------------------------------------
// Test head request for non-existent content
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
resp, err = http.Head(layerURL)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
if err != nil {
t.Fatalf("unexpected error checking head on non-existent layer: %v", err)
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkResponse(t, "checking head on non-existent layer", resp, http.StatusNotFound)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
// ------------------------------------------
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
// Start an upload, check the status then cancel
uploadURLBase, uploadUUID := startPushLayer(t, env.builder, imageName)
// A status check should work
resp, err = http.Get(uploadURLBase)
if err != nil {
t.Fatalf("unexpected error getting upload status: %v", err)
}
checkResponse(t, "status of deleted upload", resp, http.StatusNoContent)
checkHeaders(t, resp, http.Header{
"Location": []string{"*"},
"Range": []string{"0-0"},
"Docker-Upload-UUID": []string{uploadUUID},
})
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
req, err := http.NewRequest("DELETE", uploadURLBase, nil)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
if err != nil {
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
t.Fatalf("unexpected error creating delete request: %v", err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
resp, err = http.DefaultClient.Do(req)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
if err != nil {
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
t.Fatalf("unexpected error sending delete request: %v", err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
checkResponse(t, "deleting upload", resp, http.StatusNoContent)
// A status check should result in 404
resp, err = http.Get(uploadURLBase)
if err != nil {
t.Fatalf("unexpected error getting upload status: %v", err)
}
checkResponse(t, "status of deleted upload", resp, http.StatusNotFound)
// -----------------------------------------
Handle empty blob files more appropriately Several API tests were added to ensure correct acceptance of zero-size and empty tar files. This led to several changes in the storage backend around the guarantees of remote file reading, which backs the layer and layer upload type. In support of these changes, zero-length and empty checks have been added to the digest package. These provide a sanity check against upstream tarsum changes. The fileReader has been modified to be more robust when reading and seeking on zero-length or non-existent files. The file no longer needs to exist for the reader to be created. Seeks can now move beyond the end of the file, causing reads to issue an io.EOF. This eliminates errors during certain race conditions for reading files which should be detected by stat calls. As a part of this, a few error types were factored out and the read buffer size was increased to something more reasonable. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 21:01:49 +00:00
// Do layer push with an empty body and different digest
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
uploadURLBase, uploadUUID = startPushLayer(t, env.builder, imageName)
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
resp, err = doPushLayer(t, env.builder, imageName, layerDigest, uploadURLBase, bytes.NewReader([]byte{}))
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
if err != nil {
t.Fatalf("unexpected error doing bad layer push: %v", err)
}
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
checkResponse(t, "bad layer push", resp, http.StatusBadRequest)
Handle empty blob files more appropriately Several API tests were added to ensure correct acceptance of zero-size and empty tar files. This led to several changes in the storage backend around the guarantees of remote file reading, which backs the layer and layer upload type. In support of these changes, zero-length and empty checks have been added to the digest package. These provide a sanity check against upstream tarsum changes. The fileReader has been modified to be more robust when reading and seeking on zero-length or non-existent files. The file no longer needs to exist for the reader to be created. Seeks can now move beyond the end of the file, causing reads to issue an io.EOF. This eliminates errors during certain race conditions for reading files which should be detected by stat calls. As a part of this, a few error types were factored out and the read buffer size was increased to something more reasonable. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 21:01:49 +00:00
checkBodyHasErrorCodes(t, "bad layer push", resp, v2.ErrorCodeDigestInvalid)
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
// -----------------------------------------
Handle empty blob files more appropriately Several API tests were added to ensure correct acceptance of zero-size and empty tar files. This led to several changes in the storage backend around the guarantees of remote file reading, which backs the layer and layer upload type. In support of these changes, zero-length and empty checks have been added to the digest package. These provide a sanity check against upstream tarsum changes. The fileReader has been modified to be more robust when reading and seeking on zero-length or non-existent files. The file no longer needs to exist for the reader to be created. Seeks can now move beyond the end of the file, causing reads to issue an io.EOF. This eliminates errors during certain race conditions for reading files which should be detected by stat calls. As a part of this, a few error types were factored out and the read buffer size was increased to something more reasonable. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 21:01:49 +00:00
// Do layer push with an empty body and correct digest
zeroDigest, err := digest.FromTarArchive(bytes.NewReader([]byte{}))
if err != nil {
t.Fatalf("unexpected error digesting empty buffer: %v", err)
}
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
uploadURLBase, uploadUUID = startPushLayer(t, env.builder, imageName)
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
pushLayer(t, env.builder, imageName, zeroDigest, uploadURLBase, bytes.NewReader([]byte{}))
Handle empty blob files more appropriately Several API tests were added to ensure correct acceptance of zero-size and empty tar files. This led to several changes in the storage backend around the guarantees of remote file reading, which backs the layer and layer upload type. In support of these changes, zero-length and empty checks have been added to the digest package. These provide a sanity check against upstream tarsum changes. The fileReader has been modified to be more robust when reading and seeking on zero-length or non-existent files. The file no longer needs to exist for the reader to be created. Seeks can now move beyond the end of the file, causing reads to issue an io.EOF. This eliminates errors during certain race conditions for reading files which should be detected by stat calls. As a part of this, a few error types were factored out and the read buffer size was increased to something more reasonable. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 21:01:49 +00:00
// -----------------------------------------
// Do layer push with an empty body and correct digest
// This is a valid but empty tarfile!
emptyTar := bytes.Repeat([]byte("\x00"), 1024)
emptyDigest, err := digest.FromTarArchive(bytes.NewReader(emptyTar))
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
if err != nil {
Handle empty blob files more appropriately Several API tests were added to ensure correct acceptance of zero-size and empty tar files. This led to several changes in the storage backend around the guarantees of remote file reading, which backs the layer and layer upload type. In support of these changes, zero-length and empty checks have been added to the digest package. These provide a sanity check against upstream tarsum changes. The fileReader has been modified to be more robust when reading and seeking on zero-length or non-existent files. The file no longer needs to exist for the reader to be created. Seeks can now move beyond the end of the file, causing reads to issue an io.EOF. This eliminates errors during certain race conditions for reading files which should be detected by stat calls. As a part of this, a few error types were factored out and the read buffer size was increased to something more reasonable. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 21:01:49 +00:00
t.Fatalf("unexpected error digesting empty tar: %v", err)
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
}
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
uploadURLBase, uploadUUID = startPushLayer(t, env.builder, imageName)
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
pushLayer(t, env.builder, imageName, emptyDigest, uploadURLBase, bytes.NewReader(emptyTar))
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
// ------------------------------------------
// Now, actually do successful upload.
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
layerLength, _ := layerFile.Seek(0, os.SEEK_END)
layerFile.Seek(0, os.SEEK_SET)
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
uploadURLBase, uploadUUID = startPushLayer(t, env.builder, imageName)
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
pushLayer(t, env.builder, imageName, layerDigest, uploadURLBase, layerFile)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
Modify blob upload API - Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it. - Add PATCH support for streaming data on upload. - Add messaging to specification that PATCH with content range is currently not supported. - Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported. closes #470 Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-04 15:56:37 +00:00
// ------------------------------------------
// Now, push just a chunk
layerFile.Seek(0, 0)
Refactor Blob Service API This PR refactors the blob service API to be oriented around blob descriptors. Identified by digests, blobs become an abstract entity that can be read and written using a descriptor as a handle. This allows blobs to take many forms, such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented operations to better integrate with blob agnostic APIs (such as the `io` package). The error definitions are now better organized to reflect conditions that can only be seen when interacting with the blob API. The main benefit of this is to separate the much smaller metadata from large file storage. Many benefits also follow from this. Reading and writing has been separated into discrete services. Backend implementation is also simplified, by reducing the amount of metadata that needs to be picked up to simply serve a read. This also improves cacheability. "Opening" a blob simply consists of an access check (Stat) and a path calculation. Caching is greatly simplified and we've made the mapping of provisional to canonical hashes a first-class concept. BlobDescriptorService and BlobProvider can be combined in different ways to achieve varying effects. Recommend Review Approach ------------------------- This is a very large patch. While apologies are in order, we are getting a considerable amount of refactoring. Most changes follow from the changes to the root package (distribution), so start there. From there, the main changes are in storage. Looking at (*repository).Blobs will help to understand the how the linkedBlobStore is wired. One can explore the internals within and also branch out into understanding the changes to the caching layer. Following the descriptions below will also help to guide you. To reduce the chances for regressions, it was critical that major changes to unit tests were avoided. Where possible, they are left untouched and where not, the spirit is hopefully captured. Pay particular attention to where behavior may have changed. Storage ------- The primary changes to the `storage` package, other than the interface updates, were to merge the layerstore and blobstore. Blob access is now layered even further. The first layer, blobStore, exposes a global `BlobStatter` and `BlobProvider`. Operations here provide a fast path for most read operations that don't take access control into account. The `linkedBlobStore` layers on top of the `blobStore`, providing repository- scoped blob link management in the backend. The `linkedBlobStore` implements the full `BlobStore` suite, providing access-controlled, repository-local blob writers. The abstraction between the two is slightly broken in that `linkedBlobStore` is the only channel under which one can write into the global blob store. The `linkedBlobStore` also provides flexibility in that it can act over different link sets depending on configuration. This allows us to use the same code for signature links, manifest links and blob links. Eventually, we will fully consolidate this storage. The improved cache flow comes from the `linkedBlobStatter` component of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to provide a simple cache hierarchy that should streamline access checks on read and write operations, or at least provide a single path to optimize. The metrics have been changed in a slightly incompatible way since the former operations, Fetch and Exists, are no longer relevant. The fileWriter and fileReader have been slightly modified to support the rest of the changes. The most interesting is the removal of the `Stat` call from `newFileReader`. This was the source of unnecessary round trips that were only present to look up the size of the resulting reader. Now, one must simply pass in the size, requiring the caller to decide whether or not the `Stat` call is appropriate. In several cases, it turned out the caller already had the size already. The `WriterAt` implementation has been removed from `fileWriter`, since it is no longer required for `BlobWriter`, reducing the number of paths which writes may take. Cache ----- Unfortunately, the `cache` package required a near full rewrite. It was pretty mechanical in that the cache is oriented around the `BlobDescriptorService` slightly modified to include the ability to set the values for individual digests. While the implementation is oriented towards caching, it can act as a primary store. Provisions are in place to have repository local metadata, in addition to global metadata. Fallback is implemented as a part of the storage package to maintain this flexibility. One unfortunate side-effect is that caching is now repository-scoped, rather than global. This should have little effect on performance but may increase memory usage. Handlers -------- The `handlers` package has been updated to leverage the new API. For the most part, the changes are superficial or mechanical based on the API changes. This did expose a bug in the handling of provisional vs canonical digests that was fixed in the unit tests. Configuration ------------- One user-facing change has been made to the configuration and is updated in the associated documentation. The `layerinfo` cache parameter has been deprecated by the `blobdescriptor` cache parameter. Both are equivalent and configuration files should be backward compatible. Notifications ------------- Changes the `notification` package are simply to support the interface changes. Context ------- A small change has been made to the tracing log-level. Traces have been moved from "info" to "debug" level to reduce output when not needed. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-12 07:10:29 +00:00
canonicalDigester := digest.NewCanonicalDigester()
if _, err := io.Copy(canonicalDigester, layerFile); err != nil {
t.Fatalf("error copying to digest: %v", err)
}
canonicalDigest := canonicalDigester.Digest()
layerFile.Seek(0, 0)
Modify blob upload API - Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it. - Add PATCH support for streaming data on upload. - Add messaging to specification that PATCH with content range is currently not supported. - Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported. closes #470 Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-04 15:56:37 +00:00
uploadURLBase, uploadUUID = startPushLayer(t, env.builder, imageName)
uploadURLBase, dgst := pushChunk(t, env.builder, imageName, uploadURLBase, layerFile, layerLength)
finishUpload(t, env.builder, imageName, uploadURLBase, dgst)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
// ------------------------
// Use a head request to see if the layer exists.
resp, err = http.Head(layerURL)
if err != nil {
t.Fatalf("unexpected error checking head on existing layer: %v", err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkResponse(t, "checking head on existing layer", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
"Content-Length": []string{fmt.Sprint(layerLength)},
Refactor Blob Service API This PR refactors the blob service API to be oriented around blob descriptors. Identified by digests, blobs become an abstract entity that can be read and written using a descriptor as a handle. This allows blobs to take many forms, such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented operations to better integrate with blob agnostic APIs (such as the `io` package). The error definitions are now better organized to reflect conditions that can only be seen when interacting with the blob API. The main benefit of this is to separate the much smaller metadata from large file storage. Many benefits also follow from this. Reading and writing has been separated into discrete services. Backend implementation is also simplified, by reducing the amount of metadata that needs to be picked up to simply serve a read. This also improves cacheability. "Opening" a blob simply consists of an access check (Stat) and a path calculation. Caching is greatly simplified and we've made the mapping of provisional to canonical hashes a first-class concept. BlobDescriptorService and BlobProvider can be combined in different ways to achieve varying effects. Recommend Review Approach ------------------------- This is a very large patch. While apologies are in order, we are getting a considerable amount of refactoring. Most changes follow from the changes to the root package (distribution), so start there. From there, the main changes are in storage. Looking at (*repository).Blobs will help to understand the how the linkedBlobStore is wired. One can explore the internals within and also branch out into understanding the changes to the caching layer. Following the descriptions below will also help to guide you. To reduce the chances for regressions, it was critical that major changes to unit tests were avoided. Where possible, they are left untouched and where not, the spirit is hopefully captured. Pay particular attention to where behavior may have changed. Storage ------- The primary changes to the `storage` package, other than the interface updates, were to merge the layerstore and blobstore. Blob access is now layered even further. The first layer, blobStore, exposes a global `BlobStatter` and `BlobProvider`. Operations here provide a fast path for most read operations that don't take access control into account. The `linkedBlobStore` layers on top of the `blobStore`, providing repository- scoped blob link management in the backend. The `linkedBlobStore` implements the full `BlobStore` suite, providing access-controlled, repository-local blob writers. The abstraction between the two is slightly broken in that `linkedBlobStore` is the only channel under which one can write into the global blob store. The `linkedBlobStore` also provides flexibility in that it can act over different link sets depending on configuration. This allows us to use the same code for signature links, manifest links and blob links. Eventually, we will fully consolidate this storage. The improved cache flow comes from the `linkedBlobStatter` component of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to provide a simple cache hierarchy that should streamline access checks on read and write operations, or at least provide a single path to optimize. The metrics have been changed in a slightly incompatible way since the former operations, Fetch and Exists, are no longer relevant. The fileWriter and fileReader have been slightly modified to support the rest of the changes. The most interesting is the removal of the `Stat` call from `newFileReader`. This was the source of unnecessary round trips that were only present to look up the size of the resulting reader. Now, one must simply pass in the size, requiring the caller to decide whether or not the `Stat` call is appropriate. In several cases, it turned out the caller already had the size already. The `WriterAt` implementation has been removed from `fileWriter`, since it is no longer required for `BlobWriter`, reducing the number of paths which writes may take. Cache ----- Unfortunately, the `cache` package required a near full rewrite. It was pretty mechanical in that the cache is oriented around the `BlobDescriptorService` slightly modified to include the ability to set the values for individual digests. While the implementation is oriented towards caching, it can act as a primary store. Provisions are in place to have repository local metadata, in addition to global metadata. Fallback is implemented as a part of the storage package to maintain this flexibility. One unfortunate side-effect is that caching is now repository-scoped, rather than global. This should have little effect on performance but may increase memory usage. Handlers -------- The `handlers` package has been updated to leverage the new API. For the most part, the changes are superficial or mechanical based on the API changes. This did expose a bug in the handling of provisional vs canonical digests that was fixed in the unit tests. Configuration ------------- One user-facing change has been made to the configuration and is updated in the associated documentation. The `layerinfo` cache parameter has been deprecated by the `blobdescriptor` cache parameter. Both are equivalent and configuration files should be backward compatible. Notifications ------------- Changes the `notification` package are simply to support the interface changes. Context ------- A small change has been made to the tracing log-level. Traces have been moved from "info" to "debug" level to reduce output when not needed. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-12 07:10:29 +00:00
"Docker-Content-Digest": []string{canonicalDigest.String()},
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
})
// ----------------
// Fetch the layer!
resp, err = http.Get(layerURL)
if err != nil {
t.Fatalf("unexpected error fetching layer: %v", err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkResponse(t, "fetching layer", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
"Content-Length": []string{fmt.Sprint(layerLength)},
Refactor Blob Service API This PR refactors the blob service API to be oriented around blob descriptors. Identified by digests, blobs become an abstract entity that can be read and written using a descriptor as a handle. This allows blobs to take many forms, such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented operations to better integrate with blob agnostic APIs (such as the `io` package). The error definitions are now better organized to reflect conditions that can only be seen when interacting with the blob API. The main benefit of this is to separate the much smaller metadata from large file storage. Many benefits also follow from this. Reading and writing has been separated into discrete services. Backend implementation is also simplified, by reducing the amount of metadata that needs to be picked up to simply serve a read. This also improves cacheability. "Opening" a blob simply consists of an access check (Stat) and a path calculation. Caching is greatly simplified and we've made the mapping of provisional to canonical hashes a first-class concept. BlobDescriptorService and BlobProvider can be combined in different ways to achieve varying effects. Recommend Review Approach ------------------------- This is a very large patch. While apologies are in order, we are getting a considerable amount of refactoring. Most changes follow from the changes to the root package (distribution), so start there. From there, the main changes are in storage. Looking at (*repository).Blobs will help to understand the how the linkedBlobStore is wired. One can explore the internals within and also branch out into understanding the changes to the caching layer. Following the descriptions below will also help to guide you. To reduce the chances for regressions, it was critical that major changes to unit tests were avoided. Where possible, they are left untouched and where not, the spirit is hopefully captured. Pay particular attention to where behavior may have changed. Storage ------- The primary changes to the `storage` package, other than the interface updates, were to merge the layerstore and blobstore. Blob access is now layered even further. The first layer, blobStore, exposes a global `BlobStatter` and `BlobProvider`. Operations here provide a fast path for most read operations that don't take access control into account. The `linkedBlobStore` layers on top of the `blobStore`, providing repository- scoped blob link management in the backend. The `linkedBlobStore` implements the full `BlobStore` suite, providing access-controlled, repository-local blob writers. The abstraction between the two is slightly broken in that `linkedBlobStore` is the only channel under which one can write into the global blob store. The `linkedBlobStore` also provides flexibility in that it can act over different link sets depending on configuration. This allows us to use the same code for signature links, manifest links and blob links. Eventually, we will fully consolidate this storage. The improved cache flow comes from the `linkedBlobStatter` component of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to provide a simple cache hierarchy that should streamline access checks on read and write operations, or at least provide a single path to optimize. The metrics have been changed in a slightly incompatible way since the former operations, Fetch and Exists, are no longer relevant. The fileWriter and fileReader have been slightly modified to support the rest of the changes. The most interesting is the removal of the `Stat` call from `newFileReader`. This was the source of unnecessary round trips that were only present to look up the size of the resulting reader. Now, one must simply pass in the size, requiring the caller to decide whether or not the `Stat` call is appropriate. In several cases, it turned out the caller already had the size already. The `WriterAt` implementation has been removed from `fileWriter`, since it is no longer required for `BlobWriter`, reducing the number of paths which writes may take. Cache ----- Unfortunately, the `cache` package required a near full rewrite. It was pretty mechanical in that the cache is oriented around the `BlobDescriptorService` slightly modified to include the ability to set the values for individual digests. While the implementation is oriented towards caching, it can act as a primary store. Provisions are in place to have repository local metadata, in addition to global metadata. Fallback is implemented as a part of the storage package to maintain this flexibility. One unfortunate side-effect is that caching is now repository-scoped, rather than global. This should have little effect on performance but may increase memory usage. Handlers -------- The `handlers` package has been updated to leverage the new API. For the most part, the changes are superficial or mechanical based on the API changes. This did expose a bug in the handling of provisional vs canonical digests that was fixed in the unit tests. Configuration ------------- One user-facing change has been made to the configuration and is updated in the associated documentation. The `layerinfo` cache parameter has been deprecated by the `blobdescriptor` cache parameter. Both are equivalent and configuration files should be backward compatible. Notifications ------------- Changes the `notification` package are simply to support the interface changes. Context ------- A small change has been made to the tracing log-level. Traces have been moved from "info" to "debug" level to reduce output when not needed. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-12 07:10:29 +00:00
"Docker-Content-Digest": []string{canonicalDigest.String()},
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
})
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
// Verify the body
digest: Minor refactoring Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-03-10 21:40:58 +00:00
verifier, err := digest.NewDigestVerifier(layerDigest)
if err != nil {
t.Fatalf("unexpected error getting digest verifier: %s", err)
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
io.Copy(verifier, resp.Body)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
if !verifier.Verified() {
t.Fatalf("response body did not pass verification")
}
Lock down HTTP API error codes This commit locks down the set of http error codes that will be part of the inital V2 specification, proposed in docker/docker#9015. The naming order has been slightly changed and there are few tweaks to ensure all conditions are captured but this will be set the docker core will be impleemnted against. To support this, the errors have been moved into an api/errors package. A new type, ErrorDescriptor, has been defined to centralize the code, message and definitions used with each type. The information therein can be used to generate documentation and response code mappings (yet to come...). In addition to the refactoring that came along with this change, several tests have been added to ensure serialization round trips are reliable. This allows better support for using these error types on the client and server side. This is coupled with some tweaks in the client code to fix issues with error reporting. Other fixes in the client include moving client-specific errors out of the base package and ensuring that we have correct parameters for finishing uploads.
2014-12-10 05:25:54 +00:00
Ensure the instrumentedResponseWriter correctly sets the http status in the context. Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-05 21:21:33 +00:00
// ----------------
// Fetch the layer with an invalid digest
badURL := strings.Replace(layerURL, "tarsum", "trsum", 1)
resp, err = http.Get(badURL)
if err != nil {
t.Fatalf("unexpected error fetching layer: %v", err)
}
checkResponse(t, "fetching layer bad digest", resp, http.StatusBadRequest)
Set cache headers for layers. - Set an Etag header - Check If-None-Match and respond appropriately - Set a Cache-Control header with a default of 1 week Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-13 00:49:18 +00:00
// Cache headers
resp, err = http.Get(layerURL)
if err != nil {
t.Fatalf("unexpected error fetching layer: %v", err)
}
checkResponse(t, "fetching layer", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{
"Content-Length": []string{fmt.Sprint(layerLength)},
Refactor Blob Service API This PR refactors the blob service API to be oriented around blob descriptors. Identified by digests, blobs become an abstract entity that can be read and written using a descriptor as a handle. This allows blobs to take many forms, such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented operations to better integrate with blob agnostic APIs (such as the `io` package). The error definitions are now better organized to reflect conditions that can only be seen when interacting with the blob API. The main benefit of this is to separate the much smaller metadata from large file storage. Many benefits also follow from this. Reading and writing has been separated into discrete services. Backend implementation is also simplified, by reducing the amount of metadata that needs to be picked up to simply serve a read. This also improves cacheability. "Opening" a blob simply consists of an access check (Stat) and a path calculation. Caching is greatly simplified and we've made the mapping of provisional to canonical hashes a first-class concept. BlobDescriptorService and BlobProvider can be combined in different ways to achieve varying effects. Recommend Review Approach ------------------------- This is a very large patch. While apologies are in order, we are getting a considerable amount of refactoring. Most changes follow from the changes to the root package (distribution), so start there. From there, the main changes are in storage. Looking at (*repository).Blobs will help to understand the how the linkedBlobStore is wired. One can explore the internals within and also branch out into understanding the changes to the caching layer. Following the descriptions below will also help to guide you. To reduce the chances for regressions, it was critical that major changes to unit tests were avoided. Where possible, they are left untouched and where not, the spirit is hopefully captured. Pay particular attention to where behavior may have changed. Storage ------- The primary changes to the `storage` package, other than the interface updates, were to merge the layerstore and blobstore. Blob access is now layered even further. The first layer, blobStore, exposes a global `BlobStatter` and `BlobProvider`. Operations here provide a fast path for most read operations that don't take access control into account. The `linkedBlobStore` layers on top of the `blobStore`, providing repository- scoped blob link management in the backend. The `linkedBlobStore` implements the full `BlobStore` suite, providing access-controlled, repository-local blob writers. The abstraction between the two is slightly broken in that `linkedBlobStore` is the only channel under which one can write into the global blob store. The `linkedBlobStore` also provides flexibility in that it can act over different link sets depending on configuration. This allows us to use the same code for signature links, manifest links and blob links. Eventually, we will fully consolidate this storage. The improved cache flow comes from the `linkedBlobStatter` component of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to provide a simple cache hierarchy that should streamline access checks on read and write operations, or at least provide a single path to optimize. The metrics have been changed in a slightly incompatible way since the former operations, Fetch and Exists, are no longer relevant. The fileWriter and fileReader have been slightly modified to support the rest of the changes. The most interesting is the removal of the `Stat` call from `newFileReader`. This was the source of unnecessary round trips that were only present to look up the size of the resulting reader. Now, one must simply pass in the size, requiring the caller to decide whether or not the `Stat` call is appropriate. In several cases, it turned out the caller already had the size already. The `WriterAt` implementation has been removed from `fileWriter`, since it is no longer required for `BlobWriter`, reducing the number of paths which writes may take. Cache ----- Unfortunately, the `cache` package required a near full rewrite. It was pretty mechanical in that the cache is oriented around the `BlobDescriptorService` slightly modified to include the ability to set the values for individual digests. While the implementation is oriented towards caching, it can act as a primary store. Provisions are in place to have repository local metadata, in addition to global metadata. Fallback is implemented as a part of the storage package to maintain this flexibility. One unfortunate side-effect is that caching is now repository-scoped, rather than global. This should have little effect on performance but may increase memory usage. Handlers -------- The `handlers` package has been updated to leverage the new API. For the most part, the changes are superficial or mechanical based on the API changes. This did expose a bug in the handling of provisional vs canonical digests that was fixed in the unit tests. Configuration ------------- One user-facing change has been made to the configuration and is updated in the associated documentation. The `layerinfo` cache parameter has been deprecated by the `blobdescriptor` cache parameter. Both are equivalent and configuration files should be backward compatible. Notifications ------------- Changes the `notification` package are simply to support the interface changes. Context ------- A small change has been made to the tracing log-level. Traces have been moved from "info" to "debug" level to reduce output when not needed. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-05-12 07:10:29 +00:00
"Docker-Content-Digest": []string{canonicalDigest.String()},
"ETag": []string{canonicalDigest.String()},
"Cache-Control": []string{"max-age=31536000"},
Set cache headers for layers. - Set an Etag header - Check If-None-Match and respond appropriately - Set a Cache-Control header with a default of 1 week Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-05-13 00:49:18 +00:00
})
// Matching etag, gives 304
etag := resp.Header.Get("Etag")
req, err = http.NewRequest("GET", layerURL, nil)
if err != nil {
t.Fatalf("Error constructing request: %s", err)
}
req.Header.Set("If-None-Match", etag)
resp, err = http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("Error constructing request: %s", err)
}
checkResponse(t, "fetching layer with etag", resp, http.StatusNotModified)
// Non-matching etag, gives 200
req, err = http.NewRequest("GET", layerURL, nil)
if err != nil {
t.Fatalf("Error constructing request: %s", err)
}
req.Header.Set("If-None-Match", "")
resp, err = http.DefaultClient.Do(req)
checkResponse(t, "fetching layer with invalid etag", resp, http.StatusOK)
Lock down HTTP API error codes This commit locks down the set of http error codes that will be part of the inital V2 specification, proposed in docker/docker#9015. The naming order has been slightly changed and there are few tweaks to ensure all conditions are captured but this will be set the docker core will be impleemnted against. To support this, the errors have been moved into an api/errors package. A new type, ErrorDescriptor, has been defined to centralize the code, message and definitions used with each type. The information therein can be used to generate documentation and response code mappings (yet to come...). In addition to the refactoring that came along with this change, several tests have been added to ensure serialization round trips are reliable. This allows better support for using these error types on the client and server side. This is coupled with some tweaks in the client code to fix issues with error reporting. Other fixes in the client include moving client-specific errors out of the base package and ensuring that we have correct parameters for finishing uploads.
2014-12-10 05:25:54 +00:00
// Missing tests:
// - Upload the same tarsum file under and different repository and
// ensure the content remains uncorrupted.
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
}
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
func TestManifestAPI(t *testing.T) {
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
env := newTestEnv(t)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
imageName := "foo/bar"
tag := "thetag"
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
manifestURL, err := env.builder.BuildManifestURL(imageName, tag)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
if err != nil {
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
t.Fatalf("unexpected error getting manifest url: %v", err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
// -----------------------------
// Attempt to fetch the manifest
resp, err := http.Get(manifestURL)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
if err != nil {
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
t.Fatalf("unexpected error getting manifest: %v", err)
}
defer resp.Body.Close()
checkResponse(t, "getting non-existent manifest", resp, http.StatusNotFound)
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
checkBodyHasErrorCodes(t, "getting non-existent manifest", resp, v2.ErrorCodeManifestUnknown)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
tagsURL, err := env.builder.BuildTagsURL(imageName)
Test Tags HTTP API methods during manifest upload
2014-12-09 23:36:26 +00:00
if err != nil {
t.Fatalf("unexpected error building tags url: %v", err)
}
resp, err = http.Get(tagsURL)
if err != nil {
t.Fatalf("unexpected error getting unknown tags: %v", err)
}
defer resp.Body.Close()
// Check that we get an unknown repository error when asking for tags
checkResponse(t, "getting unknown manifest tags", resp, http.StatusNotFound)
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
checkBodyHasErrorCodes(t, "getting unknown manifest tags", resp, v2.ErrorCodeNameUnknown)
Test Tags HTTP API methods during manifest upload
2014-12-09 23:36:26 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
// --------------------------------
// Attempt to push unsigned manifest with missing layers
Move manifest to discrete package Because manifests and their signatures are a discrete component of the registry, we are moving the definitions into a separate package. This causes us to lose some test coverage, but we can fill this in shortly. No changes have been made to the external interfaces, but they are likely to come. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-02 21:21:29 +00:00
unsignedManifest := &manifest.Manifest{
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
Versioned: manifest.Versioned{
SchemaVersion: 1,
},
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
Name: imageName,
Tag: tag,
Move manifest to discrete package Because manifests and their signatures are a discrete component of the registry, we are moving the definitions into a separate package. This causes us to lose some test coverage, but we can fill this in shortly. No changes have been made to the external interfaces, but they are likely to come. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-02 21:21:29 +00:00
FSLayers: []manifest.FSLayer{
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
{
BlobSum: "asdf",
},
{
BlobSum: "qwer",
},
},
}
resp = putManifest(t, "putting unsigned manifest", manifestURL, unsignedManifest)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
defer resp.Body.Close()
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkResponse(t, "posting unsigned manifest", resp, http.StatusBadRequest)
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
_, p, counts := checkBodyHasErrorCodes(t, "getting unknown manifest tags", resp,
v2.ErrorCodeManifestUnverified, v2.ErrorCodeBlobUnknown, v2.ErrorCodeDigestInvalid)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
expectedCounts := map[v2.ErrorCode]int{
v2.ErrorCodeManifestUnverified: 1,
v2.ErrorCodeBlobUnknown: 2,
v2.ErrorCodeDigestInvalid: 2,
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
}
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
if !reflect.DeepEqual(counts, expectedCounts) {
t.Fatalf("unexpected number of error codes encountered: %v\n!=\n%v\n---\n%s", counts, expectedCounts, string(p))
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
}
Add TODO about manifest tampering test
2014-12-02 01:40:14 +00:00
// TODO(stevvooe): Add a test case where we take a mostly valid registry,
// tamper with the content and ensure that we get a unverified manifest
// error.
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
// Push 2 random layers
expectedLayers := make(map[digest.Digest]io.ReadSeeker)
for i := range unsignedManifest.FSLayers {
rs, dgstStr, err := testutil.CreateRandomTarFile()
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
if err != nil {
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
t.Fatalf("error creating random layer %d: %v", i, err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
dgst := digest.Digest(dgstStr)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
expectedLayers[dgst] = rs
unsignedManifest.FSLayers[i].BlobSum = dgst
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
uploadURLBase, _ := startPushLayer(t, env.builder, imageName)
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
pushLayer(t, env.builder, imageName, dgst, uploadURLBase, rs)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
// -------------------
// Push the signed manifest with all layers pushed.
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
signedManifest, err := manifest.Sign(unsignedManifest, env.pk)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
if err != nil {
t.Fatalf("unexpected error signing manifest: %v", err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
payload, err := signedManifest.Payload()
checkErr(t, err, "getting manifest payload")
dgst, err := digest.FromBytes(payload)
checkErr(t, err, "digesting manifest")
manifestDigestURL, err := env.builder.BuildManifestURL(imageName, dgst.String())
checkErr(t, err, "building manifest url")
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
resp = putManifest(t, "putting signed manifest", manifestURL, signedManifest)
Manifest PUT should return 202 Accepted status Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-04 02:27:40 +00:00
checkResponse(t, "putting signed manifest", resp, http.StatusAccepted)
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
checkHeaders(t, resp, http.Header{
"Location": []string{manifestDigestURL},
"Docker-Content-Digest": []string{dgst.String()},
})
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
// --------------------
// Push by digest -- should get same result
resp = putManifest(t, "putting signed manifest", manifestDigestURL, signedManifest)
checkResponse(t, "putting signed manifest", resp, http.StatusAccepted)
checkHeaders(t, resp, http.Header{
"Location": []string{manifestDigestURL},
"Docker-Content-Digest": []string{dgst.String()},
})
// ------------------
// Fetch by tag name
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
resp, err = http.Get(manifestURL)
if err != nil {
t.Fatalf("unexpected error fetching manifest: %v", err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
defer resp.Body.Close()
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkResponse(t, "fetching uploaded manifest", resp, http.StatusOK)
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
checkHeaders(t, resp, http.Header{
"Docker-Content-Digest": []string{dgst.String()},
})
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Move manifest to discrete package Because manifests and their signatures are a discrete component of the registry, we are moving the definitions into a separate package. This causes us to lose some test coverage, but we can fill this in shortly. No changes have been made to the external interfaces, but they are likely to come. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-02 21:21:29 +00:00
var fetchedManifest manifest.SignedManifest
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
dec := json.NewDecoder(resp.Body)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
if err := dec.Decode(&fetchedManifest); err != nil {
t.Fatalf("error decoding fetched manifest: %v", err)
}
if !bytes.Equal(fetchedManifest.Raw, signedManifest.Raw) {
t.Fatalf("manifests do not match")
}
Test Tags HTTP API methods during manifest upload
2014-12-09 23:36:26 +00:00
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
// ---------------
// Fetch by digest
resp, err = http.Get(manifestDigestURL)
checkErr(t, err, "fetching manifest by digest")
defer resp.Body.Close()
checkResponse(t, "fetching uploaded manifest", resp, http.StatusOK)
checkHeaders(t, resp, http.Header{
"Docker-Content-Digest": []string{dgst.String()},
})
var fetchedManifestByDigest manifest.SignedManifest
dec = json.NewDecoder(resp.Body)
if err := dec.Decode(&fetchedManifestByDigest); err != nil {
t.Fatalf("error decoding fetched manifest: %v", err)
}
if !bytes.Equal(fetchedManifestByDigest.Raw, signedManifest.Raw) {
t.Fatalf("manifests do not match")
}
Test Tags HTTP API methods during manifest upload
2014-12-09 23:36:26 +00:00
// Ensure that the tag is listed.
resp, err = http.Get(tagsURL)
if err != nil {
t.Fatalf("unexpected error getting unknown tags: %v", err)
}
defer resp.Body.Close()
// Check that we get an unknown repository error when asking for tags
checkResponse(t, "getting unknown manifest tags", resp, http.StatusOK)
dec = json.NewDecoder(resp.Body)
var tagsResponse tagsAPIResponse
if err := dec.Decode(&tagsResponse); err != nil {
t.Fatalf("unexpected error decoding error response: %v", err)
}
if tagsResponse.Name != imageName {
t.Fatalf("tags name should match image name: %v != %v", tagsResponse.Name, imageName)
}
if len(tagsResponse.Tags) != 1 {
t.Fatalf("expected some tags in response: %v", tagsResponse.Tags)
}
if tagsResponse.Tags[0] != tag {
t.Fatalf("tag not as expected: %q != %q", tagsResponse.Tags[0], tag)
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
}
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
type testEnv struct {
pk libtrust.PrivateKey
ctx context.Context
config configuration.Configuration
app *App
server *httptest.Server
builder *v2.URLBuilder
}
func newTestEnv(t *testing.T) *testEnv {
config := configuration.Configuration{
Storage: configuration.Storage{
"inmemory": configuration.Parameters{},
},
}
Path prefix support for running registry somewhere other than root of server Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-02-24 22:59:01 +00:00
return newTestEnvWithConfig(t, &config)
}
func newTestEnvWithConfig(t *testing.T, config *configuration.Configuration) *testEnv {
ctx := context.Background()
app := NewApp(ctx, *config)
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
server := httptest.NewServer(handlers.CombinedLoggingHandler(os.Stderr, app))
Path prefix support for running registry somewhere other than root of server Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-02-24 22:59:01 +00:00
builder, err := v2.NewURLBuilderFromString(server.URL + config.HTTP.Prefix)
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
if err != nil {
t.Fatalf("error creating url builder: %v", err)
}
pk, err := libtrust.GenerateECP256PrivateKey()
if err != nil {
t.Fatalf("unexpected error generating private key: %v", err)
}
return &testEnv{
pk: pk,
ctx: ctx,
Path prefix support for running registry somewhere other than root of server Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-02-24 22:59:01 +00:00
config: *config,
Integrate contextual logging with regsitry app This changeset integrates contextual logging into the registry web application. Idiomatic context use is attempted within the current webapp layout. The functionality is centered around making lifecycle objects (application and request context) into contexts themselves. Relevant data has been moved into the context where appropriate. We still have some work to do to factor out the registry.Context object and the dispatching functionality to remove some awkward portions. The api tests were slightly refactored to use a test environment to eliminate common code. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-07 00:19:19 +00:00
app: app,
server: server,
builder: builder,
}
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
func putManifest(t *testing.T, msg, url string, v interface{}) *http.Response {
Avoid manifest verification errors by using Raw Because json.Marshal does compaction on returned results, applications must directly use SignedManifest.Raw when the marshaled value is required. Otherwise, the returned manifest will fail signature checks.
2014-12-02 01:10:33 +00:00
var body []byte
Move manifest to discrete package Because manifests and their signatures are a discrete component of the registry, we are moving the definitions into a separate package. This causes us to lose some test coverage, but we can fill this in shortly. No changes have been made to the external interfaces, but they are likely to come. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-02 21:21:29 +00:00
if sm, ok := v.(*manifest.SignedManifest); ok {
Avoid manifest verification errors by using Raw Because json.Marshal does compaction on returned results, applications must directly use SignedManifest.Raw when the marshaled value is required. Otherwise, the returned manifest will fail signature checks.
2014-12-02 01:10:33 +00:00
body = sm.Raw
} else {
var err error
body, err = json.MarshalIndent(v, "", " ")
if err != nil {
t.Fatalf("unexpected error marshaling %v: %v", v, err)
}
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
req, err := http.NewRequest("PUT", url, bytes.NewReader(body))
if err != nil {
t.Fatalf("error creating request for %s: %v", msg, err)
}
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("error doing put request while %s: %v", msg, err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
return resp
}
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
func startPushLayer(t *testing.T, ub *v2.URLBuilder, name string) (location string, uuid string) {
Migrate webapp to use exported urls.URLBuilder package
2014-12-12 05:59:59 +00:00
layerUploadURL, err := ub.BuildBlobUploadURL(name)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
if err != nil {
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
t.Fatalf("unexpected error building layer upload url: %v", err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
resp, err := http.Post(layerUploadURL, "", nil)
if err != nil {
t.Fatalf("unexpected error starting layer push: %v", err)
}
defer resp.Body.Close()
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkResponse(t, fmt.Sprintf("pushing starting layer push %v", name), resp, http.StatusAccepted)
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
u, err := url.Parse(resp.Header.Get("Location"))
if err != nil {
t.Fatalf("error parsing location header: %v", err)
}
uuid = path.Base(u.Path)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkHeaders(t, resp, http.Header{
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
"Location": []string{"*"},
"Content-Length": []string{"0"},
"Docker-Upload-UUID": []string{uuid},
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
})
Specify and implement Docker-Upload-UUID This changeset adds support for a header to identify docker upload uuids. This id can be used as a key to manage local state for resumable uploads. The goal is remove the necessity for a client to parse the url to get an upload uuid. The restrictions for clients to use the location header are still strongly in place. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-27 00:43:47 +00:00
return resp.Header.Get("Location"), uuid
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
}
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
// doPushLayer pushes the layer content returning the url on success returning
// the response. If you're only expecting a successful response, use pushLayer.
func doPushLayer(t *testing.T, ub *v2.URLBuilder, name string, dgst digest.Digest, uploadURLBase string, body io.Reader) (*http.Response, error) {
Migrate webapp to use exported urls.URLBuilder package
2014-12-12 05:59:59 +00:00
u, err := url.Parse(uploadURLBase)
if err != nil {
t.Fatalf("unexpected error parsing pushLayer url: %v", err)
}
u.RawQuery = url.Values{
Serializes upload state to an HMAC token for subsequent requests To support clustered registry, upload UUIDs must be recognizable by registries that did not issue the UUID. By creating an HMAC verifiable upload state token, registries can validate upload requests that other instances authorized. The tokenProvider interface could also use a redis store or other system for token handling in the future.
2015-01-05 07:59:29 +00:00
"_state": u.Query()["_state"],
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
"digest": []string{dgst.String()},
Migrate webapp to use exported urls.URLBuilder package
2014-12-12 05:59:59 +00:00
}.Encode()
uploadURL := u.String()
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
// Just do a monolithic upload
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
req, err := http.NewRequest("PUT", uploadURL, body)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
if err != nil {
t.Fatalf("unexpected error creating new request: %v", err)
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
return http.DefaultClient.Do(req)
}
// pushLayer pushes the layer content returning the url on success.
func pushLayer(t *testing.T, ub *v2.URLBuilder, name string, dgst digest.Digest, uploadURLBase string, body io.Reader) string {
fixing up tests to work with for non-tarsum future Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-05 04:26:56 +00:00
digester := digest.NewCanonicalDigester()
resp, err := doPushLayer(t, ub, name, dgst, uploadURLBase, io.TeeReader(body, &digester))
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
if err != nil {
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
t.Fatalf("unexpected error doing push layer request: %v", err)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
}
defer resp.Body.Close()
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkResponse(t, "putting monolithic chunk", resp, http.StatusCreated)
fixing up tests to work with for non-tarsum future Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-05 04:26:56 +00:00
if err != nil {
t.Fatalf("error generating sha256 digest of body")
}
sha256Dgst := digester.Digest()
expectedLayerURL, err := ub.BuildBlobURL(name, sha256Dgst)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
if err != nil {
t.Fatalf("error building expected layer url: %v", err)
}
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
checkHeaders(t, resp, http.Header{
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
"Location": []string{expectedLayerURL},
"Content-Length": []string{"0"},
fixing up tests to work with for non-tarsum future Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-05 04:26:56 +00:00
"Docker-Content-Digest": []string{sha256Dgst.String()},
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
})
return resp.Header.Get("Location")
}
Modify blob upload API - Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it. - Add PATCH support for streaming data on upload. - Add messaging to specification that PATCH with content range is currently not supported. - Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported. closes #470 Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-05-04 15:56:37 +00:00
func finishUpload(t *testing.T, ub *v2.URLBuilder, name string, uploadURLBase string, dgst digest.Digest) string {
resp, err := doPushLayer(t, ub, name, dgst, uploadURLBase, nil)
if err != nil {
t.Fatalf("unexpected error doing push layer request: %v", err)
}
defer resp.Body.Close()
checkResponse(t, "putting monolithic chunk", resp, http.StatusCreated)
expectedLayerURL, err := ub.BuildBlobURL(name, dgst)
if err != nil {
t.Fatalf("error building expected layer url: %v", err)
}
checkHeaders(t, resp, http.Header{
"Location": []string{expectedLayerURL},
"Content-Length": []string{"0"},
"Docker-Content-Digest": []string{dgst.String()},
})
return resp.Header.Get("Location")
}
func doPushChunk(t *testing.T, uploadURLBase string, body io.Reader) (*http.Response, digest.Digest, error) {
u, err := url.Parse(uploadURLBase)
if err != nil {
t.Fatalf("unexpected error parsing pushLayer url: %v", err)
}
u.RawQuery = url.Values{
"_state": u.Query()["_state"],
}.Encode()
uploadURL := u.String()
digester := digest.NewCanonicalDigester()
req, err := http.NewRequest("PATCH", uploadURL, io.TeeReader(body, digester))
if err != nil {
t.Fatalf("unexpected error creating new request: %v", err)
}
req.Header.Set("Content-Type", "application/octet-stream")
resp, err := http.DefaultClient.Do(req)
return resp, digester.Digest(), err
}
func pushChunk(t *testing.T, ub *v2.URLBuilder, name string, uploadURLBase string, body io.Reader, length int64) (string, digest.Digest) {
resp, dgst, err := doPushChunk(t, uploadURLBase, body)
if err != nil {
t.Fatalf("unexpected error doing push layer request: %v", err)
}
defer resp.Body.Close()
checkResponse(t, "putting chunk", resp, http.StatusAccepted)
if err != nil {
t.Fatalf("error generating sha256 digest of body")
}
checkHeaders(t, resp, http.Header{
"Range": []string{fmt.Sprintf("0-%d", length-1)},
"Content-Length": []string{"0"},
})
return resp.Header.Get("Location"), dgst
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
func checkResponse(t *testing.T, msg string, resp *http.Response, expectedStatus int) {
if resp.StatusCode != expectedStatus {
t.Logf("unexpected status %s: %v != %v", msg, resp.StatusCode, expectedStatus)
maybeDumpResponse(t, resp)
t.FailNow()
}
}
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
// checkBodyHasErrorCodes ensures the body is an error body and has the
// expected error codes, returning the error structure, the json slice and a
// count of the errors by code.
func checkBodyHasErrorCodes(t *testing.T, msg string, resp *http.Response, errorCodes ...v2.ErrorCode) (v2.Errors, []byte, map[v2.ErrorCode]int) {
p, err := ioutil.ReadAll(resp.Body)
if err != nil {
t.Fatalf("unexpected error reading body %s: %v", msg, err)
}
var errs v2.Errors
if err := json.Unmarshal(p, &errs); err != nil {
t.Fatalf("unexpected error decoding error response: %v", err)
}
if len(errs.Errors) == 0 {
t.Fatalf("expected errors in response")
}
// TODO(stevvooe): Shoot. The error setup is not working out. The content-
// type headers are being set after writing the status code.
// if resp.Header.Get("Content-Type") != "application/json; charset=utf-8" {
// t.Fatalf("unexpected content type: %v != 'application/json'",
// resp.Header.Get("Content-Type"))
// }
expected := map[v2.ErrorCode]struct{}{}
counts := map[v2.ErrorCode]int{}
// Initialize map with zeros for expected
for _, code := range errorCodes {
expected[code] = struct{}{}
counts[code] = 0
}
for _, err := range errs.Errors {
if _, ok := expected[err.Code]; !ok {
Handle empty blob files more appropriately Several API tests were added to ensure correct acceptance of zero-size and empty tar files. This led to several changes in the storage backend around the guarantees of remote file reading, which backs the layer and layer upload type. In support of these changes, zero-length and empty checks have been added to the digest package. These provide a sanity check against upstream tarsum changes. The fileReader has been modified to be more robust when reading and seeking on zero-length or non-existent files. The file no longer needs to exist for the reader to be created. Seeks can now move beyond the end of the file, causing reads to issue an io.EOF. This eliminates errors during certain race conditions for reading files which should be detected by stat calls. As a part of this, a few error types were factored out and the read buffer size was increased to something more reasonable. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 21:01:49 +00:00
t.Fatalf("unexpected error code %v encountered during %s: %s ", err.Code, msg, string(p))
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
}
counts[err.Code]++
}
// Ensure that counts of expected errors were all non-zero
for code := range expected {
if counts[code] == 0 {
Handle empty blob files more appropriately Several API tests were added to ensure correct acceptance of zero-size and empty tar files. This led to several changes in the storage backend around the guarantees of remote file reading, which backs the layer and layer upload type. In support of these changes, zero-length and empty checks have been added to the digest package. These provide a sanity check against upstream tarsum changes. The fileReader has been modified to be more robust when reading and seeking on zero-length or non-existent files. The file no longer needs to exist for the reader to be created. Seeks can now move beyond the end of the file, causing reads to issue an io.EOF. This eliminates errors during certain race conditions for reading files which should be detected by stat calls. As a part of this, a few error types were factored out and the read buffer size was increased to something more reasonable. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 21:01:49 +00:00
t.Fatalf("expected error code %v not encounterd during %s: %s", code, msg, string(p))
Address server errors received during layer upload This changeset addresses intermittent internal server errors encountered during pushes. The root cause has been isolated to layers that result in identical, empty filesystems but may have some path declarations (imaginge "./"), resulting in different tarsums. The main error message reported during these upload problems was a 500 error, which was not correct. Further investigation showed the errors to be rooted in digest verification when finishing uploads. Inspection of the surrounding code also identified a few issues. PutLayerChunk was slightly refactered into PutLayerUploadComplete. Helper methods were avoided to make handler less confusing. This simplification leveraged an earlier change in the spec that moved non-complete chunk uploads to the PATCH method. Simple logging was also added in the unknown error case that should help to avoid mysterious 500 errors in the future. At the same time, the glaring omission of a proper layer upload cancel method was rectified. This has been added in this change so it is not missed in the future. In the future, we may want to refactor the handler code to be more straightforward, hopefully letting us avoid these problems in the future. Added test cases that reproduce these errors and drove these changes include the following: 1. Push a layer with an empty body results in invalid blob upload. 2. Push a layer with a different tarsum (in this case, empty tar) 3. Deleting a layer upload works. 4. Getting status on a deleted layer upload returns 404. Common functionality was grouped into shared functions to remove repitition. The API tests will still require future love. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 05:26:35 +00:00
}
}
return errs, p, counts
}
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
func maybeDumpResponse(t *testing.T, resp *http.Response) {
if d, err := httputil.DumpResponse(resp, true); err != nil {
t.Logf("error dumping response: %v", err)
} else {
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
t.Logf("response:\n%s", string(d))
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
}
}
// matchHeaders checks that the response has at least the headers. If not, the
// test will fail. If a passed in header value is "*", any non-zero value will
// suffice as a match.
func checkHeaders(t *testing.T, resp *http.Response, headers http.Header) {
for k, vs := range headers {
if resp.Header.Get(k) == "" {
t.Fatalf("response missing header %q", k)
}
for _, v := range vs {
if v == "*" {
// Just ensure there is some value.
if len(resp.Header[k]) > 0 {
continue
}
}
for _, hv := range resp.Header[k] {
if hv != v {
fixing up tests to work with for non-tarsum future Signed-off-by: David Lawrence <david.lawrence@docker.com> (github: endophage)
2015-03-05 04:26:56 +00:00
t.Fatalf("%v header value not matched in response: %q != %q", k, hv, v)
Initial implementation of Manifest HTTP API Push, pull and delete of manifest files in the registry have been implemented on top of the storage services. Basic workflows, including reporting of missing manifests are tested, including various proposed response codes. Common testing functionality has been collected into shared methods. A test suite may be emerging but it might better to capture more edge cases (such as resumable upload, range requests, etc.) before we commit to a full approach. To support clearer test cases and simpler handler methods, an application aware urlBuilder has been added. We may want to export the functionality for use in the client, which could allow us to abstract away from gorilla/mux. A few error codes have been added to fill in error conditions missing from the proposal. Some use cases have identified some problems with the approach to error reporting that requires more work to reconcile. To resolve this, the mapping of Go errors into error types needs to pulled out of the handlers and into the application. We also need to move to type-based errors, with rich information, rather than value-based errors. ErrorHandlers will probably replace the http.Handlers to make this work correctly. Unrelated to the above, the "length" parameter has been migrated to "size" for completing layer uploads. This change should have gone out before but these diffs ending up being coupled with the parameter name change due to updates to the layer unit tests.
2014-11-26 20:16:58 +00:00
}
}
}
Initial implementation of Layer API The http API has its first set of endpoints to implement the core aspects of fetching and uploading layers. Uploads can be started and completed in a single chunk and the content can be fetched via tarsum. Most proposed error conditions should be represented but edge cases likely remain. In this version, note that the layers are still called layers, even though the routes are pointing to blobs. This will change with backend refactoring over the next few weeks. The unit tests are a bit of a shamble but these need to be carefully written along with the core specification process. As the the client-server interaction solidifies, we can port this into a verification suite for registry providers.
2014-11-21 03:57:01 +00:00
}
}
Implement immutable manifest reference support This changeset implements immutable manifest references via the HTTP API. Most of the changes follow from modifications to ManifestService. Once updates were made across the repo to implement these changes, the http handlers were change accordingly. The new methods on ManifestService will be broken out into a tagging service in a later PR. Unfortunately, due to complexities around managing the manifest tag index in an eventually consistent manner, direct deletes of manifests have been disabled. Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-26 23:47:04 +00:00
func checkErr(t *testing.T, err error, msg string) {
if err != nil {
t.Fatalf("unexpected error %s: %v", msg, err)
}
}
Reference in a new issue Copy permalink