Commit graph

882 commits

Author SHA1 Message Date
Owen W. Taylor
132abc6de5 Test storing OCI image manifests and indexes with/without a media type
OCI Image manifests and indexes are supported both with and without
an embeded MediaType (the field is reserved according to the spec).
Test storing and retrieving both types from the manifest store.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2018-06-19 11:24:25 -04:00
Owen W. Taylor
60d9c5dfad Handle OCI manifests and image indexes without a media type
In the OCI image specification, the MediaType field is reserved
and otherwise undefined; assume that manifests without a media
in storage are OCI images or image indexes, and determine which
by looking at what fields are in the JSON. We do keep a check
that when unmarshalling an OCI image or image index, if it has
a MediaType field, it must match that media type of the upload.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2018-06-19 11:24:25 -04:00
Mike Brown
ad7ab0853c folow commit 9c88801a12
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
7b47fb13cf update url policy support; testing for annoations in index
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
ec2aa05cdf addressing comments from stevvooe
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
6bae7ca597 refactor adding enum for storage types
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
9e3f78b8c8 addresses minor debug comments
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
b0cef05626 removes oci. namespace feature
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
426afb3a4c address get manifest issue with oci. namespace; and comment descriptions
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
c94f28805e OCI media types; annotation support; oci index
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:48:58 -05:00
Mike Brown
6fcea22b0a add an ocischema manifest handler for the registry
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:48:58 -05:00
Mike Brown
9986e8ca7c adds support for oci manifests and manifestlists
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:48:58 -05:00
Yu Wang
62797237b9 closes #2496 and #2552
Update Azure SDK with release v16.2.1
Update Azure autorest SDK with release v10.8.1

Signed-off-by: Yu Wang <yuwa@microsoft.com>
2018-06-08 16:53:02 -07:00
Olivier Gambier
23bef416bd
Merge pull request #2538 from tianon/challanges
Fix minor "Challanges" typo
2018-03-21 09:58:14 -07:00
Derek McGowan
607ae5d128
Merge pull request #2501 from xiaonancc77/master
Added ignore event types into notifications
2018-03-14 10:37:02 -07:00
Michal Minář
492844e09c redis: delete right size attribute
Signed-off-by: Michal Minář <miminar@redhat.com>
2018-03-13 15:25:32 +01:00
elsanli(李楠)
fc1d3647c6 Added ignore event typs into notifications
Signed-off-by: elsanli(李楠) <elsanli@tencent.com>
2018-03-13 16:00:44 +08:00
Tianon Gravi
f2805894c8 Fix minor "Challanges" typo
Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
2018-03-01 11:25:05 -08:00
Stephen Day
6664ec7039
Merge pull request #2466 from tifayuki/prometheus_go_metrics
add prometheus metrics
2018-02-09 15:49:04 -08:00
tifayuki
e3c37a46e2 Add Prometheus Metrics
at the first iteration, only the following metrics are collected:

  - HTTP metrics of each API endpoint
  - cache counter for request/hit/miss
  - histogram of storage actions, including:
    GetContent, PutContent, Stat, List, Move, and Delete

Signed-off-by: tifayuki <tifayuki@gmail.com>
2018-02-09 14:27:51 -08:00
Derek McGowan
13076371a6
Merge pull request #2520 from ywk253100/ignore_path_not_found_error
Ignore path not found error when look up tags
2018-02-09 11:19:52 -08:00
Wenkai Yin
005c6e0236 ignore path not found error when look up tags
Signed-off-by: Wenkai Yin <yinw@vmware.com>
2018-02-08 17:32:25 +08:00
Stephen Day
d707ea2428
Merge pull request #2469 from felixbuenemann/letsencrypt-host-whitelist-support
Let's Encrypt Host Whitelist Support
2018-02-06 16:11:20 -08:00
Stephen J Day
1ba5b3b553
registry/storage: ignore missing tag on delete
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-02-06 15:25:51 -08:00
Stephen Day
fea8bd5114
Merge pull request #2471 from GarageDeveloper/master
fixes #2249: sanitize tenant and tenantid
2018-02-02 16:29:34 -08:00
Felix Bünemann
4ecb17cc4c registry: support whitelisting letsencrypt hosts
This adds a configuration setting `HTTP.TLS.LetsEncrypt.Hosts` which can
be set to a list of hosts that the registry will whitelist for retrieving
certificates from Let's Encrypt. HTTPS connections with SNI hostnames
that are not whitelisted will be closed with an "unknown host" error.
It is required to avoid lots of unsuccessful registrations attempts that
are triggered by malicious clients connecting with bogus SNI hostnames.

NOTE: Due to a bug in the deprecated vendored rsc.io/letsencrypt library
clearing the host list requires deleting or editing of the cachefile to
reset the hosts list to null.

Signed-off-by: Felix Buenemann <felix.buenemann@gmail.com>
2018-02-01 21:16:58 +01:00
Derek McGowan
5cb406d511
Merge pull request #2455 from sargun/make-walk-faster
s3: improve walk performance
2018-01-19 13:00:03 -08:00
Sargun Dhillon
cbcbcb02c5 Remove old walk function
This removes the old global walk function, and changes all
the code to use the per-driver walk functions.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-18 13:06:07 -08:00
Sargun Dhillon
c7b0da2622 Use the new walk method for catalog enumeration
This change is primarily to make GC faster.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-18 11:58:46 -08:00
Stephen J Day
585cdeb571
context, registry/handlers: remove net/context references
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-01-17 16:01:41 -08:00
Sargun Dhillon
35b29a609e Use the New Driver Walk method for catalog enumeration
This changes the Walk Method used for catalog enumeration. Just to show
how much an effect this has on our s3 storage:
Original:
List calls: 6839

real    3m16.636s
user    0m0.000s
sys    0m0.016s

New:
ListObjectsV2 Calls: 1805

real    0m49.970s
user    0m0.008s
sys    0m0.000s

This is because it no longer performs a list and stat per item, and instead
is able to use the metadata gained from the list as a replacement to stat.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Sargun Dhillon
32ac467992 Introduce Walk Method Per Storage Driver
Move the Walk types into registry/storage/driver, and add a Walk method to each
storage driver. Although this is yet another API to implement, there is a fall
back implementation that relies on List and Stat. For some filesystems this is
very slow.

Also, this WalkDir Method conforms better do a traditional WalkDir (a la filepath).

This change is in preparation for refactoring.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Jesse Haka
ff87ad884c add possibility to clean untagged manifests
add tests

add possibility to clean untagged manifests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add dry tests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove underscores

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fixes

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

opts struct+use camelcase

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

doublecheck manifest in paths.go

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add gofmt

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix lint

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add log print

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

move log to dryrun as well

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove counter

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove manifest tag references

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add tag to tests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

manifestsWithoutTags -> removeUntagged

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove RemoveManifestTagReferences and use removemanifests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove comment

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove pathfor

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

move removemanifest out of manifestenumerator, it does not work correctly if we delete stuff in it

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add comment

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix context -> dcontext

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix gofmt
2018-01-05 12:20:01 +02:00
Raphaël Enrici
8777e97b72 fixes #2249: sanitize tenant and tenantid
If tenant or tenantid are passed as env variables, we systematically use Sprint to make sure they are string and not integer as it would make mapstructure fail.

Signed-off-by: Raphaël Enrici <raphael@root-42.com>
2017-12-17 18:33:34 +01:00
Derek McGowan
f411848591
Merge pull request #2447 from tifayuki/cloudfront-s3-filter
add s3 region filters for cloudfront
2017-12-07 10:04:35 -08:00
tifayuki
e8ecc6dc55 add s3 region filters for cloudfront
Signed-off-by: tifayuki <tifayuki@gmail.com>
2017-12-01 15:58:58 -08:00
Kevin Lin
1bfbeca726 Properly follow relative links when listing tags
The previous code assumed that the link returned when listing tags was
always absolute. However, some registries, such as quay.io, return the
link as a relative link (e.g. the second page for the quay.io/coreos/etcd
image is /v2/coreos/etcd/tags/list?next_page=<truncated>&n=50). Because
the relative link was retrieved directly, the fetch failed (with the
error `unsupported protocol scheme ""`).

Signed-off-by: Kevin Lin <kevin@kelda.io>
2017-11-18 22:04:19 -08:00
Clayton Coleman
3c5f85abd1
Allow clients to request specific manifest media types
The current registry/client sends the registered manifest types in
random order. Allow clients to request a single specific manifest type
or a preferred order as per the HTTP spec.

Signed-off-by: Clayton Coleman <ccoleman@redhat.com>
2017-10-03 17:06:11 -04:00
Stephen J Day
4abf680c76
registry: feed the linter by removing redundant err check
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-09-29 13:27:49 -07:00
Derek McGowan
8710fa58ae Merge pull request #2377 from stevvooe/healthcheck-storage-ignore-notfound
registry/handlers: ignore notfound on storage driver healthcheck
2017-09-29 13:14:15 -07:00
Tibor Vass
13e0608bc8 gofmt -w -s registry/client/repository.go
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2017-09-05 13:14:43 -07:00
Tibor Vass
2c58ce1a7f Remove context in NewRegistry and NewRepository
The context parameter was either not used or misused.

Signed-off-by: Tibor Vass <teabee89@gmail.com>
2017-09-05 10:34:56 -07:00
Clayton Coleman
a2015272c1
Support HEAD requests without Docker-Content-Digest header
A statically hosted registry that responds correctly to GET with a
manifest will load the right digest (by looking at the manifest body and
calculating the digest). If the registry returns a HEAD without
`Docker-Content-Digest`, then the client Tags().Get() call will return
an empty digest.

This commit changes the client to fallback to loading the tag via GET if
the `Docker-Content-Digest` header is not set.

Signed-off-by: Clayton Coleman <ccoleman@redhat.com>
2017-08-25 17:18:01 -04:00
Clayton Coleman
23f8ca88e1
If the request already has the scope, don't force token fetch
AuthorizeRequest() injects the 'pull' scope if `from` is set
unconditionally. If the current token already has that scope, it will
be inserted into the scope list twice and `addedScopes` will be set to
true, resulting in a new token being fetched that has no net new scopes.

Instead, check whether `additionalScopes` are actually new.

Signed-off-by: Clayton Coleman <ccoleman@redhat.com>
2017-08-23 19:27:37 -04:00
Stephen J Day
1618b49d5b
registry/handlers: ignore notfound on storage driver healthcheck
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-21 15:04:31 -07:00
Stephen J Day
860b28c5b9
registry: remove dependency on logrus for client
To simplify the vendoring story for the client, we have now removed the
requirement for `logrus` and the forked `context` package (usually
imported as `dcontext`). We inject the logger via the metrics tracker
for the blob cache and via options on the token handler. We preserve
logs on the proxy cache for that case. Clients expecting these log
messages may need to be updated accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-15 10:28:38 -07:00
Stephen J Day
9c88801a12
context: remove definition of Context
Back in the before time, the best practices surrounding usage of Context
weren't quite worked out. We defined our own type to make usage easier.
As this packaged was used elsewhere, it make it more and more
challenging to integrate with the forked `Context` type. Now that it is
available in the standard library, we can just use that one directly.

To make usage more consistent, we now use `dcontext` when referring to
the distribution context package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-11 15:53:31 -07:00
Derek McGowan
edc3ab29cd Merge pull request #2316 from igmor/logrus_case_dep_update
Update logrus and logrus-logstash-hook libraries
2017-07-26 10:46:10 -07:00
Stephen Day
91c507a39a Merge pull request #2340 from stevvooe/limit-payload-size
registry/{storage,handlers}: limit content sizes
2017-07-20 13:57:55 -07:00
Derek McGowan
5cfdfbdce5 Merge pull request #2338 from stevvooe/api-typo
api: url typo in specification
2017-07-18 15:49:00 -07:00