Commit graph

852 commits

Author SHA1 Message Date
Sargun Dhillon
c7b0da2622 Use the new walk method for catalog enumeration
This change is primarily to make GC faster.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-18 11:58:46 -08:00
Sargun Dhillon
35b29a609e Use the New Driver Walk method for catalog enumeration
This changes the Walk Method used for catalog enumeration. Just to show
how much an effect this has on our s3 storage:
Original:
List calls: 6839

real    3m16.636s
user    0m0.000s
sys    0m0.016s

New:
ListObjectsV2 Calls: 1805

real    0m49.970s
user    0m0.008s
sys    0m0.000s

This is because it no longer performs a list and stat per item, and instead
is able to use the metadata gained from the list as a replacement to stat.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Sargun Dhillon
32ac467992 Introduce Walk Method Per Storage Driver
Move the Walk types into registry/storage/driver, and add a Walk method to each
storage driver. Although this is yet another API to implement, there is a fall
back implementation that relies on List and Stat. For some filesystems this is
very slow.

Also, this WalkDir Method conforms better do a traditional WalkDir (a la filepath).

This change is in preparation for refactoring.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Jesse Haka
ff87ad884c add possibility to clean untagged manifests
add tests

add possibility to clean untagged manifests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add dry tests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove underscores

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fixes

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

opts struct+use camelcase

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

doublecheck manifest in paths.go

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add gofmt

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix lint

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add log print

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

move log to dryrun as well

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove counter

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove manifest tag references

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add tag to tests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

manifestsWithoutTags -> removeUntagged

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove RemoveManifestTagReferences and use removemanifests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove comment

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove pathfor

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

move removemanifest out of manifestenumerator, it does not work correctly if we delete stuff in it

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add comment

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix context -> dcontext

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix gofmt
2018-01-05 12:20:01 +02:00
Derek McGowan
f411848591
Merge pull request #2447 from tifayuki/cloudfront-s3-filter
add s3 region filters for cloudfront
2017-12-07 10:04:35 -08:00
tifayuki
e8ecc6dc55 add s3 region filters for cloudfront
Signed-off-by: tifayuki <tifayuki@gmail.com>
2017-12-01 15:58:58 -08:00
Kevin Lin
1bfbeca726 Properly follow relative links when listing tags
The previous code assumed that the link returned when listing tags was
always absolute. However, some registries, such as quay.io, return the
link as a relative link (e.g. the second page for the quay.io/coreos/etcd
image is /v2/coreos/etcd/tags/list?next_page=<truncated>&n=50). Because
the relative link was retrieved directly, the fetch failed (with the
error `unsupported protocol scheme ""`).

Signed-off-by: Kevin Lin <kevin@kelda.io>
2017-11-18 22:04:19 -08:00
Clayton Coleman
3c5f85abd1
Allow clients to request specific manifest media types
The current registry/client sends the registered manifest types in
random order. Allow clients to request a single specific manifest type
or a preferred order as per the HTTP spec.

Signed-off-by: Clayton Coleman <ccoleman@redhat.com>
2017-10-03 17:06:11 -04:00
Stephen J Day
4abf680c76
registry: feed the linter by removing redundant err check
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-09-29 13:27:49 -07:00
Derek McGowan
8710fa58ae Merge pull request #2377 from stevvooe/healthcheck-storage-ignore-notfound
registry/handlers: ignore notfound on storage driver healthcheck
2017-09-29 13:14:15 -07:00
Tibor Vass
13e0608bc8 gofmt -w -s registry/client/repository.go
Signed-off-by: Tibor Vass <teabee89@gmail.com>
2017-09-05 13:14:43 -07:00
Tibor Vass
2c58ce1a7f Remove context in NewRegistry and NewRepository
The context parameter was either not used or misused.

Signed-off-by: Tibor Vass <teabee89@gmail.com>
2017-09-05 10:34:56 -07:00
Clayton Coleman
a2015272c1
Support HEAD requests without Docker-Content-Digest header
A statically hosted registry that responds correctly to GET with a
manifest will load the right digest (by looking at the manifest body and
calculating the digest). If the registry returns a HEAD without
`Docker-Content-Digest`, then the client Tags().Get() call will return
an empty digest.

This commit changes the client to fallback to loading the tag via GET if
the `Docker-Content-Digest` header is not set.

Signed-off-by: Clayton Coleman <ccoleman@redhat.com>
2017-08-25 17:18:01 -04:00
Clayton Coleman
23f8ca88e1
If the request already has the scope, don't force token fetch
AuthorizeRequest() injects the 'pull' scope if `from` is set
unconditionally. If the current token already has that scope, it will
be inserted into the scope list twice and `addedScopes` will be set to
true, resulting in a new token being fetched that has no net new scopes.

Instead, check whether `additionalScopes` are actually new.

Signed-off-by: Clayton Coleman <ccoleman@redhat.com>
2017-08-23 19:27:37 -04:00
Stephen J Day
1618b49d5b
registry/handlers: ignore notfound on storage driver healthcheck
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-21 15:04:31 -07:00
Stephen J Day
860b28c5b9
registry: remove dependency on logrus for client
To simplify the vendoring story for the client, we have now removed the
requirement for `logrus` and the forked `context` package (usually
imported as `dcontext`). We inject the logger via the metrics tracker
for the blob cache and via options on the token handler. We preserve
logs on the proxy cache for that case. Clients expecting these log
messages may need to be updated accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-15 10:28:38 -07:00
Stephen J Day
9c88801a12
context: remove definition of Context
Back in the before time, the best practices surrounding usage of Context
weren't quite worked out. We defined our own type to make usage easier.
As this packaged was used elsewhere, it make it more and more
challenging to integrate with the forked `Context` type. Now that it is
available in the standard library, we can just use that one directly.

To make usage more consistent, we now use `dcontext` when referring to
the distribution context package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-11 15:53:31 -07:00
Derek McGowan
edc3ab29cd Merge pull request #2316 from igmor/logrus_case_dep_update
Update logrus and logrus-logstash-hook libraries
2017-07-26 10:46:10 -07:00
Stephen Day
91c507a39a Merge pull request #2340 from stevvooe/limit-payload-size
registry/{storage,handlers}: limit content sizes
2017-07-20 13:57:55 -07:00
Derek McGowan
5cfdfbdce5 Merge pull request #2338 from stevvooe/api-typo
api: url typo in specification
2017-07-18 15:49:00 -07:00
Stephen J Day
5e5156afa3
api: url typo in specification
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-18 14:48:55 -07:00
fate-grand-order
a11fe173d5 fix misspelling "algorithm" for cache/redis/redis.go
Signed-off-by: Helen Chen <chenjg@harmonycloud.cn>
2017-07-18 16:02:42 +08:00
Stephen J Day
55ea440428
registry/{storage,handlers}: limit content sizes
Under certain circumstances, the use of `StorageDriver.GetContent` can
result in unbounded memory allocations. In particualr, this happens when
accessing a layer through the manifests endpoint.

This problem is mitigated by setting a 4MB limit when using to access
content that may have been accepted from a user. In practice, this means
setting the limit with the use of `BlobProvider.Get` by wrapping
`StorageDriver.GetContent` in a helper that uses `StorageDriver.Reader`
with a `limitReader` that returns an error.

When mitigating this security issue, we also noticed that the size of
manifests uploaded to the registry is also unlimited. We apply similar
logic to the request body of payloads that are full buffered.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-06 17:13:39 -07:00
Aaron Lehmann
caa175c710 Merge pull request #2299 from dmage/regulator
Fix signalling Wait in regulator.enter
2017-06-23 15:25:35 -07:00
Igor Morozov
a97d7c0c15 moved Sirupsen to sirupsen on a case sensitive system
Signed-off-by: Igor Morozov <igor@adhoc05-sjc1.prod.uber.internal>
2017-06-23 20:28:48 +00:00
Oleg Bulatov
5b1b6afae2 Update Docker-Content-Digest if manifest list is rewritten
If the client doesn't support manifest lists, the registry will
rewrite a manifest list into the old format. The Docker-Content-Digest
header should be updated in this case.

Signed-off-by: Oleg Bulatov <oleg@bulatov.me>
2017-06-16 16:20:14 +02:00
Oleg Bulatov
258345ba0d Fix signalling Wait in regulator.enter
In some conditions, regulator.exit may not send a signal to blocked
regulator.enter.

Let's assume we are in the critical section of regulator.exit and r.available
is equal to 0. And there are three more gorotines. One goroutine also executes
regulator.exit and waits for the lock. Rest run regulator.enter and wait for
the signal.

We send the signal, and after releasing the lock, there will be lock
contention:

  1. Wait from regulator.enter
  2. Lock from regulator.exit

If the winner is Lock from regulator.exit, we will not send another signal to
unlock the second Wait.

Signed-off-by: Oleg Bulatov <obulatov@redhat.com>
2017-06-02 15:41:55 +02:00
william wei
37ca688dc0 Remove unused function
Signed-off-by: william wei <1342247033@qq.com>
2017-05-16 17:47:13 +08:00
Derek McGowan
1d7824702b Merge pull request #2247 from yuwaMSFT2/revendorazuresdk
closes #2224 re-vendor the latest Azure Storage SDK for better performance
2017-04-27 10:54:25 -07:00
Troels Thomsen
1935c8d50b Return early to prevent nil pointer dereference
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2017-04-27 14:59:37 +02:00
Troels Thomsen
7f510ae9c9 Support session token
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2017-04-25 19:31:50 +02:00
Yu Wang (UC)
ac05d143d8 closes #2224: re-vendor the latest Azure Storage SDK for better performance
Signed-off-by: Yu Wang <yuwa@microsoft.com>
2017-04-14 14:20:18 -07:00
Stephen J Day
1f0a9dbca0
register/handlers: remove context manager
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-04-10 15:27:30 -07:00
Stephen Day
7b9ebdc54f Merge pull request #2215 from AlvinFeng/expires
Remove expires tag from s3 upload
2017-04-07 12:14:10 -07:00
Derek McGowan
81a47d9766 Remove support for X-Forwarded-Port
Partially reverts change adding support for X-Forwarded-Port.
Changes the logic to prefer the standard Forwarded header over
X-Forwarded headers. Prefer forwarded "host" over "for" since
"for" represents the client and not the client's request.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-20 16:13:33 -07:00
Troels Thomsen
d4c3e88426
Add test for precendence with standard port
Signed-off-by: Troels Thomsen <troels@thomsen.io>
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-03-20 16:10:44 -07:00
Alvin Feng
45bb7c9cc9 Remove expires tag from s3 upload
Signed-off-by: Alvin Feng <alvin4feng@yahoo.com>
2017-03-17 23:41:15 +00:00
Christy Perez
0810eba2ad Better error message for BuildManifestURL if not tagged or digested
Since there's no default case, if there's not a tag or digest you get
back a confusing error from the router about it not matching the
expected pattern.

Also redoing the tests for URLs a bit so that they can handle checking
for failures.

Signed-off-by: Christy Perez <christy@linux.vnet.ibm.com>
2017-03-02 11:27:23 -06:00
Derek McGowan
4f87c80073 Merge pull request #2192 from uhayate/refactor-code-style
refactor the code style in distribution/registry/storage/driver/s3-goamz/s3.go
2017-02-15 17:12:16 -08:00
Aaron Lehmann
62d8d910b5 Merge pull request #2184 from sakeven/master
Check whether must use v4 auth in specific aws region ( storage driver s3-goamz )
2017-02-13 22:29:02 -08:00
sakeven
72bdf0e320 check whether must use v4 auth in specific aws region
Signed-off-by: sakeven <jc5930@sina.cn>
2017-02-14 10:42:20 +08:00
uhayate
75c2e524a1 refactor the code style in distribution/registry/storage/driver/s3-goamz/s3.go
Signed-off-by: uhayate <uhayate.gong@daocloud.io>
2017-02-13 17:29:08 +08:00
Eric Yang
2d20471cd7 fixing acronyms case
Signed-off-by: Eric Yang <qizhao.yang@daocloud.io>
2017-01-31 01:29:30 +08:00
Stephen J Day
7f565ed65a
registry/storage: clean up zero-length blob comments
A previous inspection of the code surrounding zero-length blobs led to
some interesting question. After inspection, it was found that the hash
was indeed for the empty string (""), and not an empty tar, so the code
was correct. The variable naming and comments have been updated
accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 15:28:30 -08:00
Derek McGowan
1bcc73251d
Update handler test to use WithName
Fix test panic in handler test.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-19 15:21:40 -08:00
Derek McGowan
11cb04e994 Merge pull request #2142 from dmcgowan/reference-enforce-canonical-parsing
reference: ParseNamed updated to enforce canonical format
2017-01-18 17:06:08 -08:00
Derek McGowan
d8fcbeeb71
Update registry to use WithName for creating Named values
The registry uses partial Named values which the named parsers
no longer support. To allow the registry service to continue
to operate without canonicalization, switch to use WithName.
In the future, the registry should start using fully canonical
values on the backend and WithName should no longer support
creating partial values.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-18 13:56:56 -08:00
Stephen Day
0111f1e3cf Merge pull request #2138 from yuwaMSFT2/master
closes issue#2135 image pull returns 404 on manifest request if there is storage error
2017-01-18 12:02:55 -08:00
yuwaMSFT2
0bb696c5bf add test for manifest handler where storage driver fails
Signed-off-by: Yu Wang (UC) <yuwa@microsoft.com>
2017-01-18 11:39:45 -08:00
Michal Fojtik
9e510d67f5 Add more regions to registry S3 storage driver
Namely adding ca-central-1, ap-south-1 and eu-west-1.

Signed-off-by: Michal Fojtik <mfojtik@redhat.com>
2017-01-11 22:38:24 +01:00