TrueCloudLab/distribution
13
0
Fork 3
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions

Merge pull request #653 from pdevine/catalog-api

Browse source 
Catalog for V2 API Implementation
This commit is contained in:
Stephen Day 2015-07-22 18:54:48 -07:00
commit 76f29c2630
13 changed files with 1352 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

331
docs/spec/api.md
View file

@ -120,6 +120,16 @@ indicating what is different. Optionally, we may start marking parts of the
specification to correspond with the versions enumerated here.
<dl>
<dt>2.0.4</dt>>
<dd>
<ul>
<li>Added support for listing registry contents.</li>
<li>Added pagination to tags API.</li>
<li>Added common approach to support pagination.</li>
</ul>
</dd>
<dt>2.0.3</dt>
<dd>
<li>Allow repository name components to be one character.</li>
@ -131,7 +141,6 @@ specification to correspond with the versions enumerated here.
<li>Added section covering digest format.</li>
<li>Added more clarification that manifest cannot be deleted by tag.</li>
</dd>
<dt>2.0.1</dt>
<dd>
<ul>
@ -745,7 +754,131 @@ each unknown blob. The response format is as follows:
]
}
#### Listing Image Tags
### Listing Repositories
Images are stored in collections, known as a _repository_, which is keyed by a
`name`, as seen throughout the API specification. A registry instance may
contain several repositories. The list of available repositories is made
available through the _catalog_.
The catalog for a given registry can be retrived with the following request:
```
GET /v2/_catalog
```
The response will be in the following format:
```
200 OK
Content-Type: application/json
{
"repositories": [
<name>,
...
]
}
```
Note that the contents of the response are specific to the registry
implementation. Some registries may opt to provide a full catalog output,
limit it based on the user's access level or omit upstream results, if
providing mirroring functionality. Subsequently, the presence of a repository
in the catalog listing only means that the registry *may* provide access to
the repository at the time of the request. Conversely, a missing entry does
*not* mean that the registry does not have the repository. More succinctly,
the presence of a repository only guarantees that it is there but not that it
is _not_ there.
For registries with a large number of repositories, this response may be quite
large. If such a response is expected, one should use pagination.
#### Pagination
Paginated catalog results can be retrieved by adding an `n` parameter to the
request URL, declaring that the response should be limited to `n` results.
Starting a paginated flow begins as follows:
```
GET /v2/_catalog?n=<integer>
```
The above specifies that a catalog response should be returned, from the start of
the result set, ordered lexically, limiting the number of results to `n`. The
response to such a request would look as follows:
```
200 OK
Content-Type: application/json
Link: <<url>?n=<n from the request>&last=<last repository in response>>; rel="next"
{
"repositories": [
<name>,
...
]
}
```
The above includes the _first_ `n` entries from the result set. To get the
_next_ `n` entries, one can create a URL where the argument `last` has the
value from `repositories[len(repositories)-1]`. If there are indeed more
results, the URL for the next block is encoded in an
[RFC5988](https://tools.ietf.org/html/rfc5988) `Link` header, as a "next"
relation. The presence of the `Link` header communicates to the client that
the entire result set has not been returned and another request must be
issued. If the header is not present, the client can assume that all results
have been recieved.
> __NOTE:__ In the request template above, note that the brackets
> are required. For example, if the url is
> `http://example.com/v2/_catalog?n=20&last=b`, the value of the header would
> be `<http://example.com/v2/_catalog?n=20&last=b>; rel="next"`. Please see
> [RFC5988](https://tools.ietf.org/html/rfc5988) for details.
Compliant client implementations should always use the `Link` header
value when proceeding through results linearly. The client may construct URLs
to skip forward in the catalog.
To get the next result set, a client would issue the request as follows, using
the URL encoded in the described `Link` header:
```
GET /v2/_catalog?n=<n from the request>&last=<last repostory value from previous response>
```
The above process should then be repeated until the `Link` header is no longer
set.
The catalog result set is represented abstractly as a lexically sorted list,
where the position in that list can be specified by the query term `last`. The
entries in the response start _after_ the term specified by `last`, up to `n`
entries.
The behavior of `last` is quite simple when demonstrated with an example. Let
us say the registry has the following repositories:
```
a
b
c
d
```
If the value of `n` is 2, _a_ and _b_ will be returned on the first response.
The `Link` header returned on the response will have `n` set to 2 and last set
to _b_:
```
Link: <<url>?n=2&last=b>; rel="next"
```
The client can then issue the request with above value from the `Link` header,
receiving the values _c_ and _d_. Note that n may change on second to last
response or be omitted fully, if the server may so choose.
### Listing Image Tags
It may be necessary to list all of the tags under a given repository. The tags
for an image repository can be retrieved with the following request:
@ -766,8 +899,51 @@ The response will be in the following format:
}
For repositories with a large number of tags, this response may be quite
large, so care should be taken by the client when parsing the response to
reduce copying.
large. If such a response is expected, one should use the pagination.
#### Pagination
Paginated tag results can be retrieved by adding the appropriate parameters to
the request URL described above. The behavior of tag pagination is identical
to that specified for catalog pagination. We cover a simple flow to highlight
any differences.
Starting a paginated flow may begin as follows:
```
GET /v2/<name>/tags/list?n=<integer>
```
The above specifies that a tags response should be returned, from the start of
the result set, ordered lexically, limiting the number of results to `n`. The
response to such a request would look as follows:
```
200 OK
Content-Type: application/json
Link: <<url>?n=<n from the request>&last=<last tag value from previous response>>; rel="next"
{
"name": <name>,
"tags": [
<tag>,
...
]
}
```
To get the next result set, a client would issue the request as follows, using
the value encoded in the [RFC5988](https://tools.ietf.org/html/rfc5988) `Link`
header:
```
GET /v2/<name>/tags/list?n=<n from the request>&last=<last tag value from previous response>
```
The above process should then be repeated until the `Link` header is no longer
set in the response. The behavior of the `last` parameter, the provided
response result, lexical ordering and encoding of the `Link` header are
identical to that of catalog pagination.
### Deleting an Image
@ -817,6 +993,7 @@ A list of methods and URIs are covered in the table below:
| PATCH | `/v2/<name>/blobs/uploads/<uuid>` | Blob Upload | Upload a chunk of data for the specified upload. |
| PUT | `/v2/<name>/blobs/uploads/<uuid>` | Blob Upload | Complete the upload specified by `uuid`, optionally appending the body as the final chunk. |
| DELETE | `/v2/<name>/blobs/uploads/<uuid>` | Blob Upload | Cancel outstanding upload processes, releasing associated resources. If this is not called, the unfinished uploads will eventually timeout. |
| GET | `/v2/_catalog` | Catalog | Retrieve a sorted, json list of repositories available in the registry. |
The detail for each endpoint is covered in the following sections.
@ -886,7 +1063,6 @@ The API implements V2 protocol and is accessible.
###### On Failure: Unauthorized
```
@ -973,6 +1149,7 @@ The following parameters should be specified on the request:
```
200 OK
Content-Length: <length>
Link: <<url>?n=<last n value>&last=<last entry from response>>; rel="next"
Content-Type: application/json; charset=utf-8
{
@ -1056,6 +1233,52 @@ The error codes that may be included in the response body are enumerated below:
```
GET /v2/<name>/tags/list?n=<integer>last=<integer>
```
Return a portion of the tags for the specified repository.
The following parameters should be specified on the request:
|Name|Kind|Description|
|----|----|-----------|
|`name`|path|Name of the target repository.|
|`n`|query|Limit the number of entries in each response. It not present, all entries will be returned.|
|`last`|query|Result set will include values lexically after last.|
###### On Success: OK
```
200 OK
Content-Length: <length>
Content-Type: application/json; charset=utf-8
{
"name": <name>,
"tags": [
<tag>,
...
],
}
```
A list of tags for the named repository.
The following headers will be returned with the response:
|Name|Description|
|----|-----------|
|`Content-Length`|Length of the JSON response body.|
|`Link`|RFC5988 compliant rel='next' with URL to next result set, if available|
### Manifest
@ -1453,7 +1676,6 @@ The following parameters should be specified on the request:
###### On Failure: Invalid Name or Reference
```
@ -2907,3 +3129,100 @@ The error codes that may be included in the response body are enumerated below:
### Catalog
List a set of available repositories in the local registry cluster. Does not provide any indication of what may be available upstream. Applications can only determine if a repository is available but not if it is not available.
#### GET Catalog
Retrieve a sorted, json list of repositories available in the registry.
##### Catalog Fetch Complete
```
GET /v2/_catalog
```
Request an unabridged list of repositories available.
###### On Success: OK
```
200 OK
Content-Length: <length>
Link: <<url>?n=<last n value>&last=<last entry from response>>; rel="next"
Content-Type: application/json; charset=utf-8
{
"repositories": [
<name>,
...
]
}
```
Returns the unabridged list of repositories as a json response.
The following headers will be returned with the response:
|Name|Description|
|----|-----------|
|`Content-Length`|Length of the JSON response body.|
##### Catalog Fetch Paginated
```
GET /v2/_catalog?n=<integer>last=<integer>
```
Return the specified portion of repositories.
The following parameters should be specified on the request:
|Name|Kind|Description|
|----|----|-----------|
|`n`|query|Limit the number of entries in each response. It not present, all entries will be returned.|
|`last`|query|Result set will include values lexically after last.|
###### On Success: OK
```
200 OK
Content-Length: <length>
Content-Type: application/json; charset=utf-8
{
"repositories": [
<name>,
...
]
"next": "<url>?last=<name>&n=<last value of n>"
}
```
The following headers will be returned with the response:
|Name|Description|
|----|-----------|
|`Content-Length`|Length of the JSON response body.|
|`Link`|RFC5988 compliant rel='next' with URL to next result set, if available|

191
docs/spec/api.md.tmpl
View file

@ -120,6 +120,16 @@ indicating what is different. Optionally, we may start marking parts of the
specification to correspond with the versions enumerated here.
<dl>
<dt>2.0.4</dt>>
<dd>
<ul>
<li>Added support for listing registry contents.</li>
<li>Added pagination to tags API.</li>
<li>Added common approach to support pagination.</li>
</ul>
</dd>
<dt>2.0.3</dt>
<dd>
<li>Allow repository name components to be one character.</li>
@ -131,7 +141,6 @@ specification to correspond with the versions enumerated here.
<li>Added section covering digest format.</li>
<li>Added more clarification that manifest cannot be deleted by tag.</li>
</dd>
<dt>2.0.1</dt>
<dd>
<ul>
@ -745,7 +754,131 @@ each unknown blob. The response format is as follows:
]
}
#### Listing Image Tags
### Listing Repositories
Images are stored in collections, known as a _repository_, which is keyed by a
`name`, as seen throughout the API specification. A registry instance may
contain several repositories. The list of available repositories is made
available through the _catalog_.
The catalog for a given registry can be retrived with the following request:
```
GET /v2/_catalog
```
The response will be in the following format:
```
200 OK
Content-Type: application/json
{
"repositories": [
<name>,
...
]
}
```
Note that the contents of the response are specific to the registry
implementation. Some registries may opt to provide a full catalog output,
limit it based on the user's access level or omit upstream results, if
providing mirroring functionality. Subsequently, the presence of a repository
in the catalog listing only means that the registry *may* provide access to
the repository at the time of the request. Conversely, a missing entry does
*not* mean that the registry does not have the repository. More succinctly,
the presence of a repository only guarantees that it is there but not that it
is _not_ there.
For registries with a large number of repositories, this response may be quite
large. If such a response is expected, one should use pagination.
#### Pagination
Paginated catalog results can be retrieved by adding an `n` parameter to the
request URL, declaring that the response should be limited to `n` results.
Starting a paginated flow begins as follows:
```
GET /v2/_catalog?n=<integer>
```
The above specifies that a catalog response should be returned, from the start of
the result set, ordered lexically, limiting the number of results to `n`. The
response to such a request would look as follows:
```
200 OK
Content-Type: application/json
Link: <<url>?n=<n from the request>&last=<last repository in response>>; rel="next"
{
"repositories": [
<name>,
...
]
}
```
The above includes the _first_ `n` entries from the result set. To get the
_next_ `n` entries, one can create a URL where the argument `last` has the
value from `repositories[len(repositories)-1]`. If there are indeed more
results, the URL for the next block is encoded in an
[RFC5988](https://tools.ietf.org/html/rfc5988) `Link` header, as a "next"
relation. The presence of the `Link` header communicates to the client that
the entire result set has not been returned and another request must be
issued. If the header is not present, the client can assume that all results
have been recieved.
> __NOTE:__ In the request template above, note that the brackets
> are required. For example, if the url is
> `http://example.com/v2/_catalog?n=20&last=b`, the value of the header would
> be `<http://example.com/v2/_catalog?n=20&last=b>; rel="next"`. Please see
> [RFC5988](https://tools.ietf.org/html/rfc5988) for details.
Compliant client implementations should always use the `Link` header
value when proceeding through results linearly. The client may construct URLs
to skip forward in the catalog.
To get the next result set, a client would issue the request as follows, using
the URL encoded in the described `Link` header:
```
GET /v2/_catalog?n=<n from the request>&last=<last repostory value from previous response>
```
The above process should then be repeated until the `Link` header is no longer
set.
The catalog result set is represented abstractly as a lexically sorted list,
where the position in that list can be specified by the query term `last`. The
entries in the response start _after_ the term specified by `last`, up to `n`
entries.
The behavior of `last` is quite simple when demonstrated with an example. Let
us say the registry has the following repositories:
```
a
b
c
d
```
If the value of `n` is 2, _a_ and _b_ will be returned on the first response.
The `Link` header returned on the response will have `n` set to 2 and last set
to _b_:
```
Link: <<url>?n=2&last=b>; rel="next"
```
The client can then issue the request with above value from the `Link` header,
receiving the values _c_ and _d_. Note that n may change on second to last
response or be omitted fully, if the server may so choose.
### Listing Image Tags
It may be necessary to list all of the tags under a given repository. The tags
for an image repository can be retrieved with the following request:
@ -766,8 +899,51 @@ The response will be in the following format:
}
For repositories with a large number of tags, this response may be quite
large, so care should be taken by the client when parsing the response to
reduce copying.
large. If such a response is expected, one should use the pagination.
#### Pagination
Paginated tag results can be retrieved by adding the appropriate parameters to
the request URL described above. The behavior of tag pagination is identical
to that specified for catalog pagination. We cover a simple flow to highlight
any differences.
Starting a paginated flow may begin as follows:
```
GET /v2/<name>/tags/list?n=<integer>
```
The above specifies that a tags response should be returned, from the start of
the result set, ordered lexically, limiting the number of results to `n`. The
response to such a request would look as follows:
```
200 OK
Content-Type: application/json
Link: <<url>?n=<n from the request>&last=<last tag value from previous response>>; rel="next"
{
"name": <name>,
"tags": [
<tag>,
...
]
}
```
To get the next result set, a client would issue the request as follows, using
the value encoded in the [RFC5988](https://tools.ietf.org/html/rfc5988) `Link`
header:
```
GET /v2/<name>/tags/list?n=<n from the request>&last=<last tag value from previous response>
```
The above process should then be repeated until the `Link` header is no longer
set in the response. The behavior of the `last` parameter, the provided
response result, lexical ordering and encoding of the `Link` header are
identical to that of catalog pagination.
### Deleting an Image
@ -867,8 +1043,13 @@ Content-Type: {{.Body.ContentType}}{{end}}{{if .Body.Format}}
```
{{.Description}}
{{if .Fields}}The following fields may be returned in the response body:
{{if .Headers}}The following headers will be returned with the response:
|Name|Description|
|----|-----------|
{{range .Fields}}|`{{.Name}}`|{{.Description}}|
{{end}}{{end}}{{if .Headers}}
The following headers will be returned with the response:
|Name|Description|
|----|-----------|

6
registry.go
View file

@ -35,6 +35,12 @@ type Namespace interface {
// registry may or may not have the repository but should always return a
// reference.
Repository(ctx context.Context, name string) (Repository, error)
// Repositories fills 'repos' with a lexigraphically sorted catalog of repositories
// up to the size of 'repos' and returns the value 'n' for the number of entries
// which were filled. 'last' contains an offset in the catalog, and 'err' will be
// set to io.EOF if there are no more entries to obtain.
Repositories(ctx context.Context, repos []string, last string) (n int, err error)
}
// ManifestServiceOption is a function argument for Manifest Service methods

127
registry/api/v2/descriptors.go
View file

@ -87,6 +87,30 @@ var (
Format: "<digest>",
}
linkHeader = ParameterDescriptor{
Name: "Link",
Type: "link",
Description: "RFC5988 compliant rel='next' with URL to next result set, if available",
Format: `<<url>?n=<last n value>&last=<last entry from response>>; rel="next"`,
}
paginationParameters = []ParameterDescriptor{
{
Name: "n",
Type: "integer",
Description: "Limit the number of entries in each response. It not present, all entries will be returned.",
Format: "<integer>",
Required: false,
},
{
Name: "last",
Type: "string",
Description: "Result set will include values lexically after last.",
Format: "<integer>",
Required: false,
},
}
unauthorizedResponse = ResponseDescriptor{
Description: "The client does not have access to the repository.",
StatusCode: http.StatusUnauthorized,
@ -269,6 +293,9 @@ type ResponseDescriptor struct {
// Headers covers any headers that may be returned from the response.
Headers []ParameterDescriptor
// Fields describes any fields that may be present in the response.
Fields []ParameterDescriptor
// ErrorCodes enumerates the error codes that may be returned along with
// the response.
ErrorCodes []errcode.ErrorCode
@ -427,6 +454,36 @@ var routeDescriptors = []RouteDescriptor{
},
},
},
{
Description: "Return a portion of the tags for the specified repository.",
PathParameters: []ParameterDescriptor{nameParameterDescriptor},
QueryParameters: paginationParameters,
Successes: []ResponseDescriptor{
{
StatusCode: http.StatusOK,
Description: "A list of tags for the named repository.",
Headers: []ParameterDescriptor{
{
Name: "Content-Length",
Type: "integer",
Description: "Length of the JSON response body.",
Format: "<length>",
},
linkHeader,
},
Body: BodyDescriptor{
ContentType: "application/json; charset=utf-8",
Format: `{
"name": <name>,
"tags": [
<tag>,
...
],
}`,
},
},
},
},
},
},
},
@ -1320,6 +1377,76 @@ var routeDescriptors = []RouteDescriptor{
},
},
},
{
Name: RouteNameCatalog,
Path: "/v2/_catalog",
Entity: "Catalog",
Description: "List a set of available repositories in the local registry cluster. Does not provide any indication of what may be available upstream. Applications can only determine if a repository is available but not if it is not available.",
Methods: []MethodDescriptor{
{
Method: "GET",
Description: "Retrieve a sorted, json list of repositories available in the registry.",
Requests: []RequestDescriptor{
{
Name: "Catalog Fetch Complete",
Description: "Request an unabridged list of repositories available.",
Successes: []ResponseDescriptor{
{
Description: "Returns the unabridged list of repositories as a json response.",
StatusCode: http.StatusOK,
Headers: []ParameterDescriptor{
{
Name: "Content-Length",
Type: "integer",
Description: "Length of the JSON response body.",
Format: "<length>",
},
},
Body: BodyDescriptor{
ContentType: "application/json; charset=utf-8",
Format: `{
"repositories": [
<name>,
...
]
}`,
},
},
},
},
{
Name: "Catalog Fetch Paginated",
Description: "Return the specified portion of repositories.",
QueryParameters: paginationParameters,
Successes: []ResponseDescriptor{
{
StatusCode: http.StatusOK,
Body: BodyDescriptor{
ContentType: "application/json; charset=utf-8",
Format: `{
"repositories": [
<name>,
...
]
"next": "<url>?last=<name>&n=<last value of n>"
}`,
},
Headers: []ParameterDescriptor{
{
Name: "Content-Length",
Type: "integer",
Description: "Length of the JSON response body.",
Format: "<length>",
},
linkHeader,
},
},
},
},
},
},
},
},
}
var routeDescriptorsMap map[string]RouteDescriptor

2
registry/api/v2/routes.go
View file

@ -11,10 +11,12 @@ const (
RouteNameBlob = "blob"
RouteNameBlobUpload = "blob-upload"
RouteNameBlobUploadChunk = "blob-upload-chunk"
RouteNameCatalog = "catalog"
)
var allEndpoints = []string{
RouteNameManifest,
RouteNameCatalog,
RouteNameTags,
RouteNameBlob,
RouteNameBlobUpload,

12
registry/api/v2/urls.go
View file

@ -100,6 +100,18 @@ func (ub *URLBuilder) BuildBaseURL() (string, error) {
return baseURL.String(), nil
}
// BuildCatalogURL constructs a url get a catalog of repositories
func (ub *URLBuilder) BuildCatalogURL(values ...url.Values) (string, error) {
route := ub.cloneRoute(RouteNameCatalog)
catalogURL, err := route.URL()
if err != nil {
return "", err
}
return appendValuesURL(catalogURL, values...).String(), nil
}
// BuildTagsURL constructs a url to list the tags in the named repository.
func (ub *URLBuilder) BuildTagsURL(name string) (string, error) {
route := ub.cloneRoute(RouteNameTags)

91
registry/client/repository.go
View file

@ -21,6 +21,83 @@ import (
"github.com/docker/distribution/registry/storage/cache/memory"
)
// Registry provides an interface for calling Repositories, which returns a catalog of repositories.
type Registry interface {
Repositories(ctx context.Context, repos []string, last string) (n int, err error)
}
// NewRegistry creates a registry namespace which can be used to get a listing of repositories
func NewRegistry(ctx context.Context, baseURL string, transport http.RoundTripper) (Registry, error) {
ub, err := v2.NewURLBuilderFromString(baseURL)
if err != nil {
return nil, err
}
client := &http.Client{
Transport: transport,
Timeout: 1 * time.Minute,
}
return &registry{
client: client,
ub: ub,
context: ctx,
}, nil
}
type registry struct {
client *http.Client
ub *v2.URLBuilder
context context.Context
}
// Repositories returns a lexigraphically sorted catalog given a base URL. The 'entries' slice will be filled up to the size
// of the slice, starting at the value provided in 'last'. The number of entries will be returned along with io.EOF if there
// are no more entries
func (r *registry) Repositories(ctx context.Context, entries []string, last string) (int, error) {
var numFilled int
var returnErr error
values := buildCatalogValues(len(entries), last)
u, err := r.ub.BuildCatalogURL(values)
if err != nil {
return 0, err
}
resp, err := r.client.Get(u)
if err != nil {
return 0, err
}
defer resp.Body.Close()
switch resp.StatusCode {
case http.StatusOK:
var ctlg struct {
Repositories []string `json:"repositories"`
}
decoder := json.NewDecoder(resp.Body)
if err := decoder.Decode(&ctlg); err != nil {
return 0, err
}
for cnt := range ctlg.Repositories {
entries[cnt] = ctlg.Repositories[cnt]
}
numFilled = len(ctlg.Repositories)
link := resp.Header.Get("Link")
if link == "" {
returnErr = io.EOF
}
default:
return 0, handleErrorResponse(resp)
}
return numFilled, returnErr
}
// NewRepository creates a new Repository for the given repository name and base URL
func NewRepository(ctx context.Context, name, baseURL string, transport http.RoundTripper) (distribution.Repository, error) {
if err := v2.ValidateRepositoryName(name); err != nil {
@ -444,3 +521,17 @@ func (bs *blobStatter) Stat(ctx context.Context, dgst digest.Digest) (distributi
return distribution.Descriptor{}, handleErrorResponse(resp)
}
}
func buildCatalogValues(maxEntries int, last string) url.Values {
values := url.Values{}
if maxEntries > 0 {
values.Add("n", strconv.Itoa(maxEntries))
}
if last != "" {
values.Add("last", last)
}
return values
}

92
registry/client/repository_test.go
View file

@ -5,9 +5,11 @@ import (
"crypto/rand"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"time"
@ -77,6 +79,28 @@ func addTestFetch(repo string, dgst digest.Digest, content []byte, m *testutil.R
})
}
func addTestCatalog(route string, content []byte, link string, m *testutil.RequestResponseMap) {
headers := map[string][]string{
"Content-Length": {strconv.Itoa(len(content))},
"Content-Type": {"application/json; charset=utf-8"},
}
if link != "" {
headers["Link"] = append(headers["Link"], link)
}
*m = append(*m, testutil.RequestResponseMapping{
Request: testutil.Request{
Method: "GET",
Route: route,
},
Response: testutil.Response{
StatusCode: http.StatusOK,
Body: content,
Headers: http.Header(headers),
},
})
}
func TestBlobFetch(t *testing.T) {
d1, b1 := newRandomBlob(1024)
var m testutil.RequestResponseMap
@ -732,3 +756,71 @@ func TestManifestUnauthorized(t *testing.T) {
t.Fatalf("Unexpected message value: %q, expected %q", v2Err.Message, expected)
}
}
func TestCatalog(t *testing.T) {
var m testutil.RequestResponseMap
addTestCatalog(
"/v2/_catalog?n=5",
[]byte("{\"repositories\":[\"foo\", \"bar\", \"baz\"]}"), "", &m)
e, c := testServer(m)
defer c()
entries := make([]string, 5)
r, err := NewRegistry(context.Background(), e, nil)
if err != nil {
t.Fatal(err)
}
ctx := context.Background()
numFilled, err := r.Repositories(ctx, entries, "")
if err != io.EOF {
t.Fatal(err)
}
if numFilled != 3 {
t.Fatalf("Got wrong number of repos")
}
}
func TestCatalogInParts(t *testing.T) {
var m testutil.RequestResponseMap
addTestCatalog(
"/v2/_catalog?n=2",
[]byte("{\"repositories\":[\"bar\", \"baz\"]}"),
"</v2/_catalog?last=baz&n=2>", &m)
addTestCatalog(
"/v2/_catalog?last=baz&n=2",
[]byte("{\"repositories\":[\"foo\"]}"),
"", &m)
e, c := testServer(m)
defer c()
entries := make([]string, 2)
r, err := NewRegistry(context.Background(), e, nil)
if err != nil {
t.Fatal(err)
}
ctx := context.Background()
numFilled, err := r.Repositories(ctx, entries, "")
if err != nil {
t.Fatal(err)
}
if numFilled != 2 {
t.Fatalf("Got wrong number of repos")
}
numFilled, err = r.Repositories(ctx, entries, "baz")
if err != io.EOF {
t.Fatal(err)
}
if numFilled != 1 {
t.Fatalf("Got wrong number of repos")
}
}

205
registry/handlers/api_test.go
View file

@ -13,6 +13,8 @@ import (
"os"
"path"
"reflect"
"regexp"
"strconv"
"strings"
"testing"
@ -60,6 +62,152 @@ func TestCheckAPI(t *testing.T) {
}
}
// TestCatalogAPI tests the /v2/_catalog endpoint
func TestCatalogAPI(t *testing.T) {
chunkLen := 2
env := newTestEnv(t)
values := url.Values{
"last": []string{""},
"n": []string{strconv.Itoa(chunkLen)}}
catalogURL, err := env.builder.BuildCatalogURL(values)
if err != nil {
t.Fatalf("unexpected error building catalog url: %v", err)
}
// -----------------------------------
// try to get an empty catalog
resp, err := http.Get(catalogURL)
if err != nil {
t.Fatalf("unexpected error issuing request: %v", err)
}
defer resp.Body.Close()
checkResponse(t, "issuing catalog api check", resp, http.StatusOK)
var ctlg struct {
Repositories []string `json:"repositories"`
}
dec := json.NewDecoder(resp.Body)
if err := dec.Decode(&ctlg); err != nil {
t.Fatalf("error decoding fetched manifest: %v", err)
}
// we haven't pushed anything to the registry yet
if len(ctlg.Repositories) != 0 {
t.Fatalf("repositories has unexpected values")
}
if resp.Header.Get("Link") != "" {
t.Fatalf("repositories has more data when none expected")
}
// -----------------------------------
// push something to the registry and try again
images := []string{"foo/aaaa", "foo/bbbb", "foo/cccc"}
for _, image := range images {
createRepository(env, t, image, "sometag")
}
resp, err = http.Get(catalogURL)
if err != nil {
t.Fatalf("unexpected error issuing request: %v", err)
}
defer resp.Body.Close()
checkResponse(t, "issuing catalog api check", resp, http.StatusOK)
dec = json.NewDecoder(resp.Body)
if err = dec.Decode(&ctlg); err != nil {
t.Fatalf("error decoding fetched manifest: %v", err)
}
if len(ctlg.Repositories) != chunkLen {
t.Fatalf("repositories has unexpected values")
}
for _, image := range images[:chunkLen] {
if !contains(ctlg.Repositories, image) {
t.Fatalf("didn't find our repository '%s' in the catalog", image)
}
}
link := resp.Header.Get("Link")
if link == "" {
t.Fatalf("repositories has less data than expected")
}
newValues := checkLink(t, link, chunkLen, ctlg.Repositories[len(ctlg.Repositories)-1])
// -----------------------------------
// get the last chunk of data
catalogURL, err = env.builder.BuildCatalogURL(newValues)
if err != nil {
t.Fatalf("unexpected error building catalog url: %v", err)
}
resp, err = http.Get(catalogURL)
if err != nil {
t.Fatalf("unexpected error issuing request: %v", err)
}
defer resp.Body.Close()
checkResponse(t, "issuing catalog api check", resp, http.StatusOK)
dec = json.NewDecoder(resp.Body)
if err = dec.Decode(&ctlg); err != nil {
t.Fatalf("error decoding fetched manifest: %v", err)
}
if len(ctlg.Repositories) != 1 {
t.Fatalf("repositories has unexpected values")
}
lastImage := images[len(images)-1]
if !contains(ctlg.Repositories, lastImage) {
t.Fatalf("didn't find our repository '%s' in the catalog", lastImage)
}
link = resp.Header.Get("Link")
if link != "" {
t.Fatalf("catalog has unexpected data")
}
}
func checkLink(t *testing.T, urlStr string, numEntries int, last string) url.Values {
re := regexp.MustCompile("<(/v2/_catalog.*)>; rel=\"next\"")
matches := re.FindStringSubmatch(urlStr)
if len(matches) != 2 {
t.Fatalf("Catalog link address response was incorrect")
}
linkURL, _ := url.Parse(matches[1])
urlValues := linkURL.Query()
if urlValues.Get("n") != strconv.Itoa(numEntries) {
t.Fatalf("Catalog link entry size is incorrect")
}
if urlValues.Get("last") != last {
t.Fatal("Catalog link last entry is incorrect")
}
return urlValues
}
func contains(elems []string, e string) bool {
for _, elem := range elems {
if elem == e {
return true
}
}
return false
}
func TestURLPrefix(t *testing.T) {
config := configuration.Configuration{
Storage: configuration.Storage{
@ -869,3 +1017,60 @@ func checkErr(t *testing.T, err error, msg string) {
t.Fatalf("unexpected error %s: %v", msg, err)
}
}
func createRepository(env *testEnv, t *testing.T, imageName string, tag string) {
unsignedManifest := &manifest.Manifest{
Versioned: manifest.Versioned{
SchemaVersion: 1,
},
Name: imageName,
Tag: tag,
FSLayers: []manifest.FSLayer{
{
BlobSum: "asdf",
},
{
BlobSum: "qwer",
},
},
}
// Push 2 random layers
expectedLayers := make(map[digest.Digest]io.ReadSeeker)
for i := range unsignedManifest.FSLayers {
rs, dgstStr, err := testutil.CreateRandomTarFile()
if err != nil {
t.Fatalf("error creating random layer %d: %v", i, err)
}
dgst := digest.Digest(dgstStr)
expectedLayers[dgst] = rs
unsignedManifest.FSLayers[i].BlobSum = dgst
uploadURLBase, _ := startPushLayer(t, env.builder, imageName)
pushLayer(t, env.builder, imageName, dgst, uploadURLBase, rs)
}
signedManifest, err := manifest.Sign(unsignedManifest, env.pk)
if err != nil {
t.Fatalf("unexpected error signing manifest: %v", err)
}
payload, err := signedManifest.Payload()
checkErr(t, err, "getting manifest payload")
dgst, err := digest.FromBytes(payload)
checkErr(t, err, "digesting manifest")
manifestDigestURL, err := env.builder.BuildManifestURL(imageName, dgst.String())
checkErr(t, err, "building manifest url")
resp := putManifest(t, "putting signed manifest", manifestDigestURL, signedManifest)
checkResponse(t, "putting signed manifest", resp, http.StatusAccepted)
checkHeaders(t, resp, http.Header{
"Location": []string{manifestDigestURL},
"Docker-Content-Digest": []string{dgst.String()},
})
}

25
registry/handlers/app.go
View file

@ -69,6 +69,7 @@ func NewApp(ctx context.Context, configuration configuration.Configuration) *App
return http.HandlerFunc(apiBase)
})
app.register(v2.RouteNameManifest, imageManifestDispatcher)
app.register(v2.RouteNameCatalog, catalogDispatcher)
app.register(v2.RouteNameTags, tagsDispatcher)
app.register(v2.RouteNameBlob, blobDispatcher)
app.register(v2.RouteNameBlobUpload, blobUploadDispatcher)
@ -493,6 +494,7 @@ func (app *App) authorized(w http.ResponseWriter, r *http.Request, context *Cont
}
return fmt.Errorf("forbidden: no repository name")
}
accessRecords = appendCatalogAccessRecord(accessRecords, r)
}
ctx, err := app.accessController.Authorized(context.Context, accessRecords...)
@ -538,7 +540,8 @@ func (app *App) eventBridge(ctx *Context, r *http.Request) notifications.Listene
// nameRequired returns true if the route requires a name.
func (app *App) nameRequired(r *http.Request) bool {
route := mux.CurrentRoute(r)
return route == nil || route.GetName() != v2.RouteNameBase
routeName := route.GetName()
return route == nil || (routeName != v2.RouteNameBase && routeName != v2.RouteNameCatalog)
}
// apiBase implements a simple yes-man for doing overall checks against the
@ -588,6 +591,26 @@ func appendAccessRecords(records []auth.Access, method string, repo string) []au
return records
}
// Add the access record for the catalog if it's our current route
func appendCatalogAccessRecord(accessRecords []auth.Access, r *http.Request) []auth.Access {
route := mux.CurrentRoute(r)
routeName := route.GetName()
if routeName == v2.RouteNameCatalog {
resource := auth.Resource{
Type: "registry",
Name: "catalog",
}
accessRecords = append(accessRecords,
auth.Access{
Resource: resource,
Action: "*",
})
}
return accessRecords
}
// applyRegistryMiddleware wraps a registry instance with the configured middlewares
func applyRegistryMiddleware(registry distribution.Namespace, middlewares []configuration.Middleware) (distribution.Namespace, error) {
for _, mw := range middlewares {

95
registry/handlers/catalog.go Normal file
View file

@ -0,0 +1,95 @@
package handlers
import (
"encoding/json"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"github.com/docker/distribution/registry/api/errcode"
"github.com/gorilla/handlers"
)
const maximumReturnedEntries = 100
func catalogDispatcher(ctx *Context, r *http.Request) http.Handler {
catalogHandler := &catalogHandler{
Context: ctx,
}
return handlers.MethodHandler{
"GET": http.HandlerFunc(catalogHandler.GetCatalog),
}
}
type catalogHandler struct {
*Context
}
type catalogAPIResponse struct {
Repositories []string `json:"repositories"`
}
func (ch *catalogHandler) GetCatalog(w http.ResponseWriter, r *http.Request) {
var moreEntries = true
q := r.URL.Query()
lastEntry := q.Get("last")
maxEntries, err := strconv.Atoi(q.Get("n"))
if err != nil || maxEntries < 0 {
maxEntries = maximumReturnedEntries
}
repos := make([]string, maxEntries)
filled, err := ch.App.registry.Repositories(ch.Context, repos, lastEntry)
if err == io.EOF {
moreEntries = false
} else if err != nil {
ch.Errors = append(ch.Errors, errcode.ErrorCodeUnknown.WithDetail(err))
return
}
w.Header().Set("Content-Type", "application/json; charset=utf-8")
// Add a link header if there are more entries to retrieve
if moreEntries {
lastEntry = repos[len(repos)-1]
urlStr, err := createLinkEntry(r.URL.String(), maxEntries, lastEntry)
if err != nil {
ch.Errors = append(ch.Errors, errcode.ErrorCodeUnknown.WithDetail(err))
return
}
w.Header().Set("Link", urlStr)
}
enc := json.NewEncoder(w)
if err := enc.Encode(catalogAPIResponse{
Repositories: repos[0:filled],
}); err != nil {
ch.Errors = append(ch.Errors, errcode.ErrorCodeUnknown.WithDetail(err))
return
}
}
// Use the original URL from the request to create a new URL for
// the link header
func createLinkEntry(origURL string, maxEntries int, lastEntry string) (string, error) {
calledURL, err := url.Parse(origURL)
if err != nil {
return "", err
}
v := url.Values{}
v.Add("n", strconv.Itoa(maxEntries))
v.Add("last", lastEntry)
calledURL.RawQuery = v.Encode()
calledURL.Fragment = ""
urlStr := fmt.Sprintf("<%s>; rel=\"next\"", calledURL.String())
return urlStr, nil
}

65
registry/storage/catalog.go Normal file
View file

@ -0,0 +1,65 @@
package storage
import (
"errors"
"io"
"path"
"sort"
"strings"
"github.com/docker/distribution/context"
"github.com/docker/distribution/registry/storage/driver"
)
// Returns a list, or partial list, of repositories in the registry.
// Because it's a quite expensive operation, it should only be used when building up
// an initial set of repositories.
func (reg *registry) Repositories(ctx context.Context, repos []string, last string) (n int, err error) {
var foundRepos []string
var errVal error
if len(repos) == 0 {
return 0, errors.New("no space in slice")
}
root, err := defaultPathMapper.path(repositoriesRootPathSpec{})
if err != nil {
return 0, err
}
// Walk each of the directories in our storage. Unfortunately since there's no
// guarantee that storage will return files in lexigraphical order, we have
// to store everything another slice, sort it and then copy it back to our
// passed in slice.
Walk(ctx, reg.blobStore.driver, root, func(fileInfo driver.FileInfo) error {
filePath := fileInfo.Path()
// lop the base path off
repoPath := filePath[len(root)+1:]
_, file := path.Split(repoPath)
if file == "_layers" {
repoPath = strings.TrimSuffix(repoPath, "/_layers")
if repoPath > last {
foundRepos = append(foundRepos, repoPath)
}
return ErrSkipDir
} else if strings.HasPrefix(file, "_") {
return ErrSkipDir
}
return nil
})
sort.Strings(foundRepos)
n = copy(repos, foundRepos)
// Signal that we have no more entries by setting EOF
if len(foundRepos) <= len(repos) {
errVal = io.EOF
}
return n, errVal
}

122
registry/storage/catalog_test.go Normal file
View file

@ -0,0 +1,122 @@
package storage
import (
"io"
"testing"
"github.com/docker/distribution"
"github.com/docker/distribution/context"
"github.com/docker/distribution/registry/storage/cache/memory"
"github.com/docker/distribution/registry/storage/driver"
"github.com/docker/distribution/registry/storage/driver/inmemory"
)
type setupEnv struct {
ctx context.Context
driver driver.StorageDriver
expected []string
registry distribution.Namespace
}
func setupFS(t *testing.T) *setupEnv {
d := inmemory.New()
c := []byte("")
ctx := context.Background()
registry := NewRegistryWithDriver(ctx, d, memory.NewInMemoryBlobDescriptorCacheProvider())
rootpath, _ := defaultPathMapper.path(repositoriesRootPathSpec{})
repos := []string{
"/foo/a/_layers/1",
"/foo/b/_layers/2",
"/bar/c/_layers/3",
"/bar/d/_layers/4",
"/foo/d/in/_layers/5",
"/an/invalid/repo",
"/bar/d/_layers/ignored/dir/6",
}
for _, repo := range repos {
if err := d.PutContent(ctx, rootpath+repo, c); err != nil {
t.Fatalf("Unable to put to inmemory fs")
}
}
expected := []string{
"bar/c",
"bar/d",
"foo/a",
"foo/b",
"foo/d/in",
}
return &setupEnv{
ctx: ctx,
driver: d,
expected: expected,
registry: registry,
}
}
func TestCatalog(t *testing.T) {
env := setupFS(t)
p := make([]string, 50)
numFilled, err := env.registry.Repositories(env.ctx, p, "")
if !testEq(p, env.expected, numFilled) {
t.Errorf("Expected catalog repos err")
}
if err != io.EOF {
t.Errorf("Catalog has more values which we aren't expecting")
}
}
func TestCatalogInParts(t *testing.T) {
env := setupFS(t)
chunkLen := 2
p := make([]string, chunkLen)
numFilled, err := env.registry.Repositories(env.ctx, p, "")
if err == io.EOF || numFilled != len(p) {
t.Errorf("Expected more values in catalog")
}
if !testEq(p, env.expected[0:chunkLen], numFilled) {
t.Errorf("Expected catalog first chunk err")
}
lastRepo := p[len(p)-1]
numFilled, err = env.registry.Repositories(env.ctx, p, lastRepo)
if err == io.EOF || numFilled != len(p) {
t.Errorf("Expected more values in catalog")
}
if !testEq(p, env.expected[chunkLen:chunkLen*2], numFilled) {
t.Errorf("Expected catalog second chunk err")
}
lastRepo = p[len(p)-1]
numFilled, err = env.registry.Repositories(env.ctx, p, lastRepo)
if err != io.EOF {
t.Errorf("Catalog has more values which we aren't expecting")
}
if !testEq(p, env.expected[chunkLen*2:chunkLen*3-1], numFilled) {
t.Errorf("Expected catalog third chunk err")
}
}
func testEq(a, b []string, size int) bool {
for cnt := 0; cnt < size-1; cnt++ {
if a[cnt] != b[cnt] {
return false
}
}
return true
}