Commit graph

1073 commits

Author SHA1 Message Date
Milos Gajdos
1563384b69
Merge pull request #3480 from CollinShoop/optimize-s3-walk
Optimize storagedriver/s3 Walk (up to ~500x) + small bugfix
2021-09-26 10:24:39 +01:00
Oleg Bulatov
f5709b285a updatefrequency should not be saved into duration
When updatefrequency is set and is a string, its value should be saved
into updateFrequency, and it shouldn't override duration.

Signed-off-by: Oleg Bulatov <oleg@bulatov.me>
2021-09-21 11:12:34 +02:00
Collin Shoop
cf81f67a16 storagedriver/s3: Optimized Walk implementation + bugfix
Optimized S3 Walk impl by no longer listing files recursively. Overall gives a huge performance increase both in terms of runtime and S3 calls (up to ~500x).

Fixed a bug in WalkFallback where ErrSkipDir for was not handled as documented for non-directory.

Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-16 16:07:25 -04:00
Collin Shoop
9e873f31ec storagedriver/s3: Adding back missing import.
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
e625bc7160 storagedriver/s3: Removed temporary S3 test
(cherry picked from commit ce80e98cea1d15aa2a2ab931c8b9a1161fc6e218)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
dc5b77101d storagedriver/s3: Cleaning up tests
(cherry picked from commit 483ba26165ca66bcf18a1eaadf41ebe4d3bd5f85)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
6da7217b99 storagedriver/s3: Optimize s3 Delete test cleanup.
(cherry picked from commit e4af4dc3a6da6da724e7cff18cf5b6da6ef2a3fd)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
03f9eb3a18 storagedriver/s3: Fixed a Delete noop edgecase
Delete was not working when the subpath immediately followed the given path started with an ascii lower than "/" such as dash "-" and underscore "_" and requests no files to be deleted.

(cherry picked from commit 5d8fa0ce94b68cce70237805db92cdd8d40de282)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
05a258e711 storagedriver/s3: Added Delete tests to s3_test
(cherry picked from commit 1e3b6b67a8e6d7f01307518370f0731212935d05)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 10:57:24 -04:00
Brandon Mitchell
9c7967a32d Update PUT and PATCH APIs
Signed-off-by: Brandon Mitchell <git@bmitch.net>
2021-06-29 14:16:33 -04:00
João Pereira
6d75bd043a
Merge pull request #3425 from lostsquirrel/patch-1
Fix the /v2/_catalog n parameter description
2021-06-17 11:10:14 +01:00
lostsquirrel
6d9a3aba04 fix the /v2/_catalog n parameter description
If `n` is not present only 100 entries returned

Signed-off-by: lisong <lisong@cdsunrise.net>
2021-06-17 17:33:55 +08:00
Milos Gajdos
af8ac80933
Merge pull request #3427 from joaodrp/tag-delete
Add tag delete API
2021-06-14 17:52:26 +01:00
João Pereira
033683d629
apply feedback
Signed-off-by: João Pereira <484633+joaodrp@users.noreply.github.com>
2021-06-06 19:54:55 +01:00
João Pereira
81f081f91b
Group case values
Signed-off-by: João Pereira <484633+joaodrp@users.noreply.github.com>
2021-05-28 10:33:40 +01:00
João Pereira
6ae6df7d75
Add tag delete API
Signed-off-by: João Pereira <484633+joaodrp@users.noreply.github.com>
2021-05-27 23:27:02 +01:00
João Pereira
8ef268df25
Add tests for tags list pagination
Signed-off-by: João Pereira <484633+joaodrp@users.noreply.github.com>
2021-05-22 15:31:18 +01:00
João Pereira
d80a63f1ea
Merge pull request #3143 from eyJhb/pagination
OCI: Add pagination on `/v2/<name>/tags/list`
2021-05-22 15:05:18 +01:00
Wang Yan
f65a33d3c8
Merge pull request #3227 from wy65701436/fix-conformance-416
OCI: add content range handling in patch blob
2021-05-11 23:51:17 +08:00
wang yan
d7a2b14489 add content range handling in patch blob
Fixes #3141

1, return 416 for Out-of-order blob upload
2, return 400 for content length and content size mismatch

Signed-off-by: wang yan <wangyan@vmware.com>
2021-05-11 23:35:15 +08:00
Hayley Swimelar
9329f6a62b
Merge pull request #3395 from fuweid/verify-layer-config-descriptor
OCI: verify digest and check blob presence when put manifest
2021-05-07 10:38:45 -07:00
eyjhb
4da2712b52
added pagination to v2/<name>/tags/list endpoint
Signed-off-by: eyjhb <eyjhbb@gmail.com>
2021-04-23 18:06:04 +02:00
eyjhbb@gmail.com
febc8733d2
added error codes for pagination
Signed-off-by: eyjhb <eyjhbb@gmail.com>
2021-04-23 18:06:03 +02:00
João Pereira
a27b0c4952
Merge pull request #3161 from bloodorangeio/redis-tls
Add configuration option for Redis TLS
2021-04-23 16:52:54 +01:00
Wei Fu
9e618c90c3 registry: verify digest and check blob presence when put manifest
According to OCI image spec, the descriptor's digest field is required.
For the normal config/layer blobs, the valivation should check the
presence of the blob when put manifest.

REF: https://github.com/opencontainers/image-spec/blob/v1.0.1/descriptor.md

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2021-04-16 16:11:52 +08:00
sangluo
334a7e7ff6 close the io.ReadCloser from storage drive
Signed-off-by: sangluo <sangluo@pinduoduo.com>
2021-03-24 17:58:35 +08:00
Wang Yan
077b38b95e
Merge pull request #3364 from Agilicus/escape-json
fix: disable DisableHTMLEscape on logrus json logging
2021-03-24 11:16:55 +08:00
Derek McGowan
a01c71e247
Merge pull request #2815 from bainsy88/issue_2814
Add code to handle pagination of parts. Fixes max layer size of 10GB bug
2021-03-16 09:12:03 -07:00
Josh Dolitsky
32ccbf193d
Add configuration option for Redis TLS
Signed-off-by: Josh Dolitsky <josh@dolit.ski>
2021-03-01 18:55:56 -05:00
Adam Wolfe Gordon
43e502590f log: Include configured fields in all logs
It's possible to configure log fields in the configuration file, and we would
like these fields to be included in all logs. Previously these fields were
included only in logs produced using the main routine's context, meaning that
any logs from a request handler were missing the fields since those use a
context based on the HTTP request's context.

Add a configurable default logger to the `context` package, and set it when
configuring logging at startup time.

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
2021-03-01 14:44:56 -07:00
Milos Gajdos
17ab8afeec
Merge pull request #3174 from adamwg/manifest-delete-by-tag
manifests: Return UNSUPPORTED when deleting manifests by tag
2021-03-01 17:21:03 +00:00
Milos Gajdos
5a76dc8df1
Merge pull request #3169 from d-luu/configurable_ciphersuites
Added flag for user-configurable cipher suites

Thanks, @d-luu for putting the effort into this. Much appreciated!
2021-02-26 08:17:32 +00:00
David Luu
1e625d0076 Added flag for user configurable cipher suites
Configuration of list of cipher suites allows a user to disable use
of weak ciphers or continue to support them for legacy usage if they
so choose.

List of available cipher suites at:
https://golang.org/pkg/crypto/tls/#pkg-constants

Default cipher suites have been updated to:
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_AES_128_GCM_SHA256
- TLS_CHACHA20_POLY1305_SHA256
- TLS_AES_256_GCM_SHA384

MinimumTLS has also been updated to include TLS 1.3 as an option
and now defaults to TLS 1.2 since 1.0 and 1.1 have been deprecated.

Signed-off-by: David Luu <david@davidluu.info>
2021-02-25 14:19:56 -06:00
Don Bowman
9c43ba9dcc
fix: disable DisableHTMLEscape on logrus json logging
Fixes #3363

Without this, we emit illegal json logs, the user-agent
ends up as:

```
"http.request.useragent": "docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/5.10.10-051010-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.4 \(linux\))"
```

which is not valid according to [spec](https://www.json.org/json-en.html)

specifically, string: "<any codepoint except " or \ or control>*"

Signed-off-by: Don Bowman <don@agilicus.com>
2021-02-15 13:16:27 -05:00
olegburov
907e7be545 Bump Logstash hook for logrus to v1.0.0.
Signed-off-by: olegburov <oleg.burov@outlook.com>
2021-02-11 21:51:13 -08:00
Sebastiaan van Stijn
1d33874951
go.mod: change imports to github.com/distribution/distribution/v3
Go 1.13 and up enforce import paths to be versioned if a project
contains a go.mod and has released v2 or up.

The current v2.x branches (and releases) do not yet have a go.mod,
and therefore are still allowed to be imported with a non-versioned
import path (go modules add a `+incompatible` annotation in that case).

However, now that this project has a `go.mod` file, incompatible
import paths will not be accepted by go modules, and attempting
to use code from this repository will fail.

This patch uses `v3` for the import-paths (not `v2`), because changing
import paths itself is a breaking change, which means that  the
next release should increment the "major" version to comply with
SemVer (as go modules dictate).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-08 18:30:46 +01:00
João Pereira
6200038bc7
Merge pull request #3204 from stefannica/fsDriverRelaxedPermissions
Relax filesystem driver folder permissions to 0777
2021-02-06 16:12:02 +00:00
João Pereira
f82e1c2332
Merge pull request #3300 from jubalh/sp
Fix minor spelling mistakes
2021-02-06 16:03:09 +00:00
João Pereira
038a5060de
Merge pull request #2905 from 2opremio/master
Honor contexts passed to registry client methods
2021-02-06 15:25:03 +00:00
Michael Vetter
4d34a31762 Correct spelling: decription -> description
Signed-off-by: Michael Vetter <jubalh@iodoru.org>
2021-02-03 13:12:23 +01:00
Michael Vetter
084c0bd100 Fix typo in docu of NewURLBuilderFromString()
And one more minor typo.

Signed-off-by: Michael Vetter <jubalh@iodoru.org>
2021-02-03 13:12:16 +01:00
Michael Vetter
f0c93f65a2 Fix typo in NewSimpleManager() documentation
Signed-off-by: Michael Vetter <jubalh@iodoru.org>
2021-02-03 13:11:53 +01:00
Ricardo Maraschini
87cbd09fa7 Ignore self reference object on empty prefix
When a given prefix is empty and we attempt to list its content AWS
returns that the prefix contains one object with key defined as the
prefix with an extra "/" at the end.

e.g.

If we call ListObjects() passing to it an existing but empty prefix,
say "my/empty/prefix", AWS will return that "my/empty/prefix/" is an
object inside "my/empty/prefix" (ListObjectsOutput.Contents).

This extra "/" causes the upload purging process to panic. On normal
circunstances we never find empty prefixes on S3 but users may touch
it.

Signed-off-by: Ricardo Maraschini <rmarasch@redhat.com>
2020-11-30 13:04:14 +01:00
Arko Dasgupta
551158e600
Merge pull request #3289 from Smasherr/master
#3288 Remove empty Content-Type header
2020-11-13 18:03:30 -08:00
Arko Dasgupta
065aec5688
Merge pull request #3239 from olegburov/bump-redigo
Upgrade Redigo to `1.8.2`.
2020-11-04 18:19:51 -08:00
Smasherr
c8d90f904f Remove empty Content-Type header
Fixes #3288

Signed-off-by: Smasherr <soundcracker@gmail.com>
2020-11-03 12:24:02 +01:00
olegburov
34f1322664 Fix hardcoded credential provides.
Signed-off-by: olegburov <oleg.burov@outlook.com>
2020-09-02 17:32:34 -07:00
olegburov
264e26fd8c Bump Redigo to v1.8.2.
Signed-off-by: olegburov <oleg.burov@outlook.com>
2020-08-31 18:07:54 -07:00
Manish Tomar
2800ab0224
Merge pull request #2973 from redmatter/support-ecs-instance-profile-in-s3-driver
Support ECS TaskRole in S3 storage driver
2020-08-26 15:24:20 -07:00
Adam Wolfe Gordon
95f1eea5f5 manifests: Return UNSUPPORTED when deleting manifests by tag
The OCI distribution spec allows implementations to support deleting manifests
by tag, but also permits returning the `UNSUPPORTED` error code for such
requests. docker/distribution has never supported deleting manifests by tag, but
previously returned `DIGEST_INVALID`.

The `Tag` and `Digest` fields of the `manifestHandler` are already correctly
populated based on which kind of reference was given in the request URL. Return
`UNSUPPORTED` if the `Tag` field is populated.

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
2020-08-04 11:40:27 -06:00