Commit graph

744 commits

Author SHA1 Message Date
Milos Gajdos
5357c45703
Merge pull request #3788 from thaJeztah/deprecate_ReadSeekCloser 2022-11-18 08:53:15 +00:00
Milos Gajdos
3b8fbf9752
Merge pull request #3686 from m5i-work/exp 2022-11-11 17:07:14 +00:00
Wei Meng
35cae1099e Realloc slice exponentially in mfs
`registry/storage/driver/inmemory/driver_test.go` times out after ~10min. The slow test is `testsuites.go:TestWriteReadLargeStreams()` which writes a 5GB file.
Root cause is inefficient slice reallocation algorithm. The slice holding file bytes grows only 32K on each allocation. To fix it, this PR grows slice exponentially.

Signed-off-by: Wei Meng <wemeng@microsoft.com>
2022-11-11 18:18:08 +08:00
Sebastiaan van Stijn
019ead86f5
deprecate ReadSeekCloser in favor of io.ReadSeekCloser
Go's io package in stdlib now defines this interface, so we can switch
to using that instead.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:10:32 +01:00
Sebastiaan van Stijn
842d4c04f5
cloudfront: use strings.Equalfold()
Minor optimization :)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:16 +01:00
Sebastiaan van Stijn
3b391d3290
replace strings.Split(N) for strings.Cut() or alternatives
Go 1.18 and up now provides a strings.Cut() which is better suited for
splitting key/value pairs (and similar constructs), and performs better:

```go
func BenchmarkSplit(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_ = strings.SplitN(s, "=", 2)[0]
		}
	}
}

func BenchmarkCut(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_, _, _ = strings.Cut(s, "=")
		}
	}
}
```

    BenchmarkSplit
    BenchmarkSplit-10    	 8244206	       128.0 ns/op	     128 B/op	       4 allocs/op
    BenchmarkCut
    BenchmarkCut-10      	54411998	        21.80 ns/op	       0 B/op	       0 allocs/op

While looking at occurrences of `strings.Split()`, I also updated some for alternatives,
or added some constraints;

- for cases where an specific number of items is expected, I used `strings.SplitN()`
  with a suitable limit. This prevents (theoretical) unlimited splits.
- in some cases it we were using `strings.Split()`, but _actually_ were trying to match
  a prefix; for those I replaced the code to just match (and/or strip) the prefix.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:12 +01:00
Sebastiaan van Stijn
bf72536440
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
Both of these were deprecated in 55f675811a,
but the format of the GoDoc comments didn't follow the correct format, which
caused them not being picked up by tools as "deprecated".

This patch updates uses in the codebase to use the alternatives.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 13:51:11 +01:00
Sebastiaan van Stijn
f8b3af78fc
replace deprecated io/ioutil
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-04 23:47:52 +01:00
Hayley Swimelar
e3509fc1de
Merge pull request #3635 from milosgajdos/make-s3-driver-delete-faster
Delete S3 keys incrementally in batches
2022-11-04 16:56:41 +01:00
Hayley Swimelar
52d948a9f5
Merge pull request #3766 from thaJeztah/gofumpt
format code with gofumpt
2022-11-04 12:19:53 +01:00
Sebastiaan van Stijn
e0281dc609
format code with gofumpt
gofumpt (https://github.com/mvdan/gofumpt) provides a supserset of `gofmt` / `go fmt`,
and addresses various formatting issues that linters may be checking for.

We can consider enabling the `gofumpt` linter to verify the formatting in CI, although
not every developer may have it installed, so for now this runs it once to get formatting
in shape.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-03 22:48:20 +01:00
Sebastiaan van Stijn
f9ccd2c6ea
use http consts for request methods
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-02 23:31:47 +01:00
Milos Gajdos
ebc4234fd5
Delete S3 keys incrementally in batches
Instead of first collecting all keys and then batch deleting them,
we will do the incremental delete _online_ per max allowed batch.
Doing this prevents frequent allocations for large S3 keyspaces
and OOM-kills that might happen as a result of those.

This commit introduces storagedriver.Errors type that allows to return
multierrors as a single error from any storage driver implementation.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2022-10-30 19:10:24 +00:00
Milos Gajdos
fb2188868d
Merge pull request #3365 from brackendawson/3122-remove-workaround
Remove workaround from 2.1.1 for faulty 2.1.0 manifest links
2022-10-19 09:04:24 +01:00
Aaron Lehmann
e36cb0a5d8 registry/storage/cache/memory: Use LRU cache to bound cache size
Instead of letting the cache grow without bound, use a LRU to impose a
size limit.

The limit is configurable through a new `blobdescriptorsize` config key.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2022-09-07 07:20:06 -07:00
João Pereira
a7fc49b067
Merge pull request #3622 from ddelange/patch-1
Support all S3 instant retrieval storage classes
2022-04-26 10:23:14 +01:00
Bracken Dawson
b2b3f86039
Remove workaround from 2.1.1 for faulty 2.1.0 manifest links
This reverts commit 06a098c632

This changes the function of linkedBlobStatter.Clear(). It was either removing the first of two possible manifest links or returning nil if none were found. Now it once again it removes only the valid manifest link or returns an error if none are found.

Signed-off-by: Bracken Dawson <abdawson@gmail.com>
2022-04-25 13:01:44 +01:00
Milos Gajdos
27b5563245
Merge pull request #3624 from milosgajdos/aws-s3-listv2
Update s3 ListObjects to V2 API
2022-04-22 13:34:13 +01:00
duanhongyi
15de9e21ba Add forcepathstyle parameter for s3
Signed-off-by: duanhongyi <duanhongyi@doopai.com>
2022-04-20 08:44:12 +08:00
Milos Gajdos
48f3d9ad29
Fix typo
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2022-04-09 12:31:27 +01:00
Milos Gajdos
8eab5d1bd6
Update s3 ListObjects to V2 API
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2022-04-09 12:16:46 +01:00
Simone Locci
80952c9e2b
Rename s3accelerate parameter to accelerate
Signed-off-by: Simone Locci <simonelocci88@gmail.com>
2022-04-04 19:35:21 +02:00
Simone Locci
ea27621d4a
Fix review
Signed-off-by: Simone Locci <simonelocci88@gmail.com>
2022-04-04 19:35:09 +02:00
Kirat Singh
51c0c8148a
Add new parameter s3accelerate to S3 storage driver.
If s3accelerate is set to true then we turn on S3 Transfer
Acceleration via the AWS SDK.  It defaults to false since this is an
opt-in feature on the S3 bucket.

Signed-off-by: Kirat Singh <kirat.singh@wsq.io>
Signed-off-by: Simone Locci <simonelocci88@gmail.com>
2022-04-04 19:34:57 +02:00
ddelange
966fae5463
Add tests for all supported storage classes
Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>
2022-04-04 10:54:18 +02:00
ddelange
fb937deabf
Support all S3 instant retrieval storage classes
Signed-off-by: ddelange <14880945+ddelange@users.noreply.github.com>
2022-04-01 11:55:35 +02:00
Milos Gajdos
d2c9f72c6b
Merge pull request #3615 from zhsj/inmem-panic
Fix panic in inmemory driver
2022-03-27 16:20:31 +01:00
Shengjing Zhu
1a75c71907 Fix panic in inmemory driver
Signed-off-by: Shengjing Zhu <zhsj@debian.org>
2022-03-27 19:38:07 +08:00
James Hewitt
25bd1f704d
Incorrect variable in test output
Looks like a copy-paste bug from the same test for the image manifest.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2022-03-27 11:05:47 +01:00
João Pereira
514cbd71be
Merge pull request #3519 from jtherin/mpu-paginate
fix: paginate through s3 multipart uploads
2022-03-11 16:06:46 +00:00
Milos Gajdos
676691ce6d
Fix: Avoid a false type assertion in the inmemory driver
This issue was discovered by the following fuzzer:
https://github.com/cncf/cncf-fuzzing/blob/main/projects/distribution/inmemory_fuzzer.go#L24

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2022-01-28 11:00:37 +00:00
baojiangnan
4363fb1ef4 disable insecure cipher suites
This commit removes the following cipher suites that are known to be insecure:

TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
TLS_ECDHE_RSA_WITH_RC4_128_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256

And this commit deletes the tlsVersions of tls1.0 and tls1.1. The tls1.2 is the minimal supported tls version for creating a safer tls configuration.

Signed-off-by: david.bao <baojn1998@163.com>
2022-01-25 17:18:44 +08:00
libo.huang
117757a5cb feat: add option to disable combining the pending part
Signed-off-by: Libo Huang <huanglibo2010@gmail.com>
2022-01-07 18:20:31 +08:00
Adam Kaplan
e2caaf9cba Add dualstack option to S3 storage driver
Allow the storage driver to optionally use AWS SDK's dualstack mode.
This allows the registry to communicate with S3 in IPv6 environments.

Signed-off-by: Adam Kaplan <adam.kaplan@redhat.com>
2022-01-04 17:19:05 -05:00
Wang Yan
3f4c558dac bump up golang v1.17
Signed-off-by: Wang Yan <wangyan@vmware.com>
2021-10-27 15:51:30 +08:00
Jeremy THERIN
7736319f2e fix: paginate through s3 multipart uploads
Signed-off-by: Jeremy THERIN <jtherin@scaleway.com>
2021-10-26 17:55:03 +02:00
Milos Gajdos
1563384b69
Merge pull request #3480 from CollinShoop/optimize-s3-walk
Optimize storagedriver/s3 Walk (up to ~500x) + small bugfix
2021-09-26 10:24:39 +01:00
Oleg Bulatov
f5709b285a updatefrequency should not be saved into duration
When updatefrequency is set and is a string, its value should be saved
into updateFrequency, and it shouldn't override duration.

Signed-off-by: Oleg Bulatov <oleg@bulatov.me>
2021-09-21 11:12:34 +02:00
Collin Shoop
cf81f67a16 storagedriver/s3: Optimized Walk implementation + bugfix
Optimized S3 Walk impl by no longer listing files recursively. Overall gives a huge performance increase both in terms of runtime and S3 calls (up to ~500x).

Fixed a bug in WalkFallback where ErrSkipDir for was not handled as documented for non-directory.

Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-16 16:07:25 -04:00
Collin Shoop
9e873f31ec storagedriver/s3: Adding back missing import.
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
e625bc7160 storagedriver/s3: Removed temporary S3 test
(cherry picked from commit ce80e98cea1d15aa2a2ab931c8b9a1161fc6e218)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
dc5b77101d storagedriver/s3: Cleaning up tests
(cherry picked from commit 483ba26165ca66bcf18a1eaadf41ebe4d3bd5f85)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
6da7217b99 storagedriver/s3: Optimize s3 Delete test cleanup.
(cherry picked from commit e4af4dc3a6da6da724e7cff18cf5b6da6ef2a3fd)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
03f9eb3a18 storagedriver/s3: Fixed a Delete noop edgecase
Delete was not working when the subpath immediately followed the given path started with an ascii lower than "/" such as dash "-" and underscore "_" and requests no files to be deleted.

(cherry picked from commit 5d8fa0ce94b68cce70237805db92cdd8d40de282)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 11:56:13 -04:00
Collin Shoop
05a258e711 storagedriver/s3: Added Delete tests to s3_test
(cherry picked from commit 1e3b6b67a8e6d7f01307518370f0731212935d05)
Signed-off-by: Collin Shoop <cshoop@digitalocean.com>
2021-08-12 10:57:24 -04:00
João Pereira
6ae6df7d75
Add tag delete API
Signed-off-by: João Pereira <484633+joaodrp@users.noreply.github.com>
2021-05-27 23:27:02 +01:00
João Pereira
d80a63f1ea
Merge pull request #3143 from eyJhb/pagination
OCI: Add pagination on `/v2/<name>/tags/list`
2021-05-22 15:05:18 +01:00
eyjhb
4da2712b52
added pagination to v2/<name>/tags/list endpoint
Signed-off-by: eyjhb <eyjhbb@gmail.com>
2021-04-23 18:06:04 +02:00
Wei Fu
9e618c90c3 registry: verify digest and check blob presence when put manifest
According to OCI image spec, the descriptor's digest field is required.
For the normal config/layer blobs, the valivation should check the
presence of the blob when put manifest.

REF: https://github.com/opencontainers/image-spec/blob/v1.0.1/descriptor.md

Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2021-04-16 16:11:52 +08:00
sangluo
334a7e7ff6 close the io.ReadCloser from storage drive
Signed-off-by: sangluo <sangluo@pinduoduo.com>
2021-03-24 17:58:35 +08:00
Derek McGowan
a01c71e247
Merge pull request #2815 from bainsy88/issue_2814
Add code to handle pagination of parts. Fixes max layer size of 10GB bug
2021-03-16 09:12:03 -07:00
Sebastiaan van Stijn
1d33874951
go.mod: change imports to github.com/distribution/distribution/v3
Go 1.13 and up enforce import paths to be versioned if a project
contains a go.mod and has released v2 or up.

The current v2.x branches (and releases) do not yet have a go.mod,
and therefore are still allowed to be imported with a non-versioned
import path (go modules add a `+incompatible` annotation in that case).

However, now that this project has a `go.mod` file, incompatible
import paths will not be accepted by go modules, and attempting
to use code from this repository will fail.

This patch uses `v3` for the import-paths (not `v2`), because changing
import paths itself is a breaking change, which means that  the
next release should increment the "major" version to comply with
SemVer (as go modules dictate).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-08 18:30:46 +01:00
João Pereira
6200038bc7
Merge pull request #3204 from stefannica/fsDriverRelaxedPermissions
Relax filesystem driver folder permissions to 0777
2021-02-06 16:12:02 +00:00
Ricardo Maraschini
87cbd09fa7 Ignore self reference object on empty prefix
When a given prefix is empty and we attempt to list its content AWS
returns that the prefix contains one object with key defined as the
prefix with an extra "/" at the end.

e.g.

If we call ListObjects() passing to it an existing but empty prefix,
say "my/empty/prefix", AWS will return that "my/empty/prefix/" is an
object inside "my/empty/prefix" (ListObjectsOutput.Contents).

This extra "/" causes the upload purging process to panic. On normal
circunstances we never find empty prefixes on S3 but users may touch
it.

Signed-off-by: Ricardo Maraschini <rmarasch@redhat.com>
2020-11-30 13:04:14 +01:00
Arko Dasgupta
065aec5688
Merge pull request #3239 from olegburov/bump-redigo
Upgrade Redigo to `1.8.2`.
2020-11-04 18:19:51 -08:00
olegburov
34f1322664 Fix hardcoded credential provides.
Signed-off-by: olegburov <oleg.burov@outlook.com>
2020-09-02 17:32:34 -07:00
olegburov
264e26fd8c Bump Redigo to v1.8.2.
Signed-off-by: olegburov <oleg.burov@outlook.com>
2020-08-31 18:07:54 -07:00
Manish Tomar
2800ab0224
Merge pull request #2973 from redmatter/support-ecs-instance-profile-in-s3-driver
Support ECS TaskRole in S3 storage driver
2020-08-26 15:24:20 -07:00
Adam Wolfe Gordon
a784441b62 catalog: List repositories with no unique layers
A repository need not contain any unique layers, if its images use only layers
mounted from other repositories. But, the catalog endpoint was looking for the
_layers directory to indicate that a directory was a repository.

Use the _manifests directory as the marker instead, since any repository with
revisions will contain a _manifests directory.

Signed-off-by: Adam Wolfe Gordon <awg@digitalocean.com>
2020-08-04 11:38:49 -06:00
Stefan Nica
2672c0ebe2 Relax filesystem driver folder permissions to 0777 (cont)
There was a previous PR relaxing the filsystem driver permissions
for files and folders to 0666 and 0777 respectively [1][2], but it was
incomplete. This is required to get the registry to honor the umask
value.

[1] https://github.com/docker/distribution/pull/1304/
[2] https://github.com/docker/distribution/issues/1295

Signed-off-by: Stefan Nica <snica@suse.com>
2020-07-16 11:59:08 +02:00
Andrew Bulford
9690d843fa Support ECS TaskRole in S3 storage driver
Instead of constructing the list of credential providers manually, if we
use the default list we can take advantage of the AWS SDK checking the
environment and returning either the EC2RoleProvider or the generic HTTP
credentials provider, configured to use the ECS credentials endpoint.

Also, use the `defaults.Config()` function instead of `aws.NewConfig()`,
as this results in an initialised HTTP client which prevents a fatal
error when retrieving credentials from the ECS credentials endpoint.

Fixes #2960

Signed-off-by: Andrew Bulford <andrew.bulford@redmatter.com>
2020-07-01 08:42:56 +01:00
Oleg Bulatov
cdb4ba947a
Fix err shadowing in gcs driver
Signed-off-by: Oleg Bulatov <oleg@bulatov.me>
2020-03-18 11:41:34 +01:00
Derek McGowan
be29c05a1e
Remove deprecated cache metrics code
The metrics tracker in cached blob statter was replaced with prometheus
metrics and no longer needed.
Remove unused log wrapping.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-09 13:11:54 -07:00
Derek McGowan
495a4af7cf
Fix goimports
Separate fix for cherry-picked code

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Grant Watters
74d442a058
Consider redis.ErrNil as distribution.ErrBlobUnknown for Stat HGET
* Update redis.go

If the dgst key does not exist in the cache when calling HGET, `redis.String` will return an `ErrNil` which we need to translate into `distribution.ErrBlobUnknown` so that the error being returned can be properly handled. This will ensure that `SetDescriptor` is properly called from `cachedBlobStatter::Stat` for `repositoryScopedRedisBlobDescriptorService` which will update the redis cache and be considered as a Miss rather than an Error.

cc @manishtomar

* Update suite.go

Add unit test to ensure missing blobs for scoped repo properly return ErrBlobUnknown when HGET returns redis.ErrNil.

(cherry picked from commit dca6b9526a1d30dd218a9f321c4f84ecc4b5e62e)
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Manish Tomar
795892662b
redis metrics
* redis metrics

it is working but metrics are not very useful since default buckets
start from 5ms and almost all of them are in that range.

* remove extra comment

(cherry picked from commit ba1a1d74e7eb047dd1056548ccf0695e8846782c)
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Manish Tomar
ce101280fe
fix redis caching issue
* fix redis caching issue

earlier redis cache was updated when there was any error including any
temporary connectivity issue. This would trigger set calls which would
further increase load and possibly connectivity errors from redis
leaving the system with continuous errors and high latency. Now the
cache is updated only when it is genuine cache miss. Other errors do not
trigger a cache update.

* add back tracker Hit() and Miss() calls

*squashed commits*
(cherry picked from commit 6f3e1c10260ef59ba4e9c42e939329fad9fdd8c3)
(cherry picked from commit 6738ff3320cf82cc2df919a95a1bde2f7789a501)

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Derek McGowan
4c7c63b557
Add cache unit test
Test base functionality of the cache statter

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Derek McGowan
6b972e50fe
Merge pull request #2272 from naveedjamil/fips
Increase Unit Test Code Coverage
2020-02-22 17:40:41 -08:00
Derek McGowan
e65b3f1316
Fix CI for test updates
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-02-22 17:27:55 -08:00
Naveed Jamil
efdba4f210
Increase Unit Test Code Coverage
Unit test coverge was increased to cover the usages of crypto. This helps to ensure that everything is working fine with fips mode enabled.
Also updated sha1 to sha256 in registry/storage/driver/testsuites/testsuites.go because sha1 is not supported in fips mode.

Signed-off-by: Naveed Jamil <naveed.jamil@tenpearl.com>
2020-02-22 17:09:00 -08:00
Derek McGowan
bfa6b923e7
Merge pull request #3020 from kd7lxl/updatefrequency
Fix typo cloudfront updatefrenquency
2020-02-22 16:54:03 -08:00
sayboras
66809646d9 Migrate to golangci-lint
Signed-off-by: Tam Mach <sayboras@yahoo.com>
2020-02-14 08:11:16 +11:00
Tom Hayward
0f5e2753a6 Fix typo cloudfront updatefrenquency
Signed-off-by: Tom Hayward <thayward@infoblox.com>
2019-12-13 12:11:42 -08:00
Guillaume Rose
c9c3324300 Add unit tests for BlobEnumerator
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-06 09:38:13 +01:00
Manish Tomar
5538da4923 fixes to make layersPathSpec work
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-05 20:48:33 +01:00
Manish Tomar
fa7d949408 allow Repository.BlobStore to enumerate over blobs
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-05 20:48:33 +01:00
Manish Tomar
cf77113795 add pathspec for repo _layers directory
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-05 20:48:33 +01:00
Ryan Abrams
ae2e973db9
Merge pull request #2748 from manishtomar/tag-digests
API to retrive tag's digests
2019-10-08 12:14:56 -07:00
Terin Stock
b23dd1ef37 swift: correct segment path generation
When uploading segments to Swift, the registry generates a random file,
by taking the hash of the container path and 32-bytes of random data.
The registry attempts to shard across multiple directory paths, by
taking the first three hex characters as leader.

The implementation in registry, unfortunately, takes the hash of
nothing, and appends it to the path and random data. This results in all
segments being created in one directory.

Fixes: #2407
Fixes: #2311
Signed-off-by: Terin Stock <terinjokes@gmail.com>
2019-06-22 23:44:43 -07:00
Ryan Abrams
d80a17d8e0
Merge pull request #2879 from tbe/fix-s3-ceph
Fix s3 driver for supporting ceph radosgw
2019-06-19 19:09:07 -07:00
Ryan Abrams
55287010ce
Merge pull request #2894 from jabrown85/fix-offset-typos
Fix typo: offest -> offset
2019-06-19 19:08:22 -07:00
Ryan Abrams
3226863cbc
Merge pull request #2849 from Shawnpku/master
support Alibaba Cloud CDN storage middleware
2019-04-16 18:43:06 -07:00
Jesse Brown
74f429a5ad Fix typo: offest -> offset
Signed-off-by: Jesse Brown <jabrown85@gmail.com>
2019-04-05 14:20:20 -05:00
Shawn Chen
fd77cf43a6 change package name & format document
Signed-off-by: Shawn Chen <chen8132@gmail.com>
2019-03-18 11:35:46 +08:00
Thomas Berger
c18c6c33b2 S3 Driver: added comment for missing KeyCount workaround
Signed-off-by: Thomas Berger <loki@lokis-chaos.de>
2019-03-15 21:05:21 +01:00
Eohyung Lee
f877726503 Fix s3 driver for supporting ceph radosgw
Radosgw does not support S3 `GET Bucket` API v2 API but v1.
This API has backward compatibility, so most of this API is working
correctly but we can not get `KeyCount` in v1 API and which is only
for v2 API.

Signed-off-by: Eohyung Lee <liquidnuker@gmail.com>
2019-03-08 12:45:20 +01:00
Shawn Chen
3390f32aec fix Context issue
Signed-off-by: Shawn Chen <chen8132@gmail.com>
2019-03-04 17:48:32 +08:00
Shawnpku
6e10631d9c fix default cdn auth duration
Signed-off-by: Shawnpku <chen8132@gmail.com>
2019-03-04 14:53:48 +08:00
Vishesh Jindal
f9a0506191
Bugfix: Make ipfilteredby not required
Signed-off-by: Vishesh Jindal <vishesh92@gmail.com>
2019-03-02 08:58:52 +05:30
Shawnpku
bbc9885aa2 fix func name
Signed-off-by: Shawnpku <chen8132@gmail.com>
2019-02-20 15:54:21 +08:00
Shawnpku
3aa2a282f7 support alicdn middleware
Signed-off-by: Shawnpku <chen8132@gmail.com>
2019-02-11 15:11:26 +08:00
Manish Tomar
48818fdea7 Remove err nil check
since type checking nil will not panic and return appropriately

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2019-02-04 16:42:44 -08:00
Manish Tomar
da8db4666b Fix gometalint errors
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2019-02-04 16:01:04 -08:00
Jack Baines
bda79219b2 Add code to handle pagination of parts. Fixes max layer size of 10GB bug
Signed-off-by: Jack Baines <jack.baines@uk.ibm.com>
2019-01-14 14:54:55 +00:00
Li Yi
90bed67126 Support BYOK for OSS storage driver
Change-Id: I423ad03e63bd38aded3abfcba49079ff2fbb3b74
Signed-off-by: Li Yi <denverdino@gmail.com>
2018-12-25 08:30:40 +08:00
Manish Tomar
1251e51ad0 better name and updated tests
- use ManifestDigests name instead of Indexes
- update tests to validate against multiple tags

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-11-01 10:31:08 -07:00
Manish Tomar
9ebf151ac2 API to retrive tag's digests
Add an interface alongside TagStore that provides API to retreive
digests of all manifests that a tag historically pointed to. It also
includes currently linked tag.

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-10-29 18:19:05 -07:00
Rui Cao
569d18aef9 Fix some typos
Signed-off-by: Rui Cao <ruicao@alauda.io>
2018-09-24 09:05:44 +08:00
David Wu
a927fbdb9b track digest offset in blobwriter
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-12 14:53:27 -07:00
David Wu
166874ade9 fix gofmt and goimports
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 16:14:10 -07:00
David Wu
877d706b38 remove dependencies on resumable
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 08:21:27 -07:00
Derek McGowan
642075f42c
Merge pull request #2631 from whoshuu/feature/improve-gcs-driver
Improve gcs driver
2018-09-05 17:48:03 -07:00
Derek McGowan
15de837aa8
Merge pull request #2704 from dmcgowan/fix-2703
Fix registry stripping newlines from manifests
2018-09-05 17:46:56 -07:00
Huu Nguyen
7a195dd5ca Add back include_gcs build constraint
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-09-05 15:48:30 -07:00
Huu Nguyen
69299d93d9 Use existing jwtConf instead of creating a scoped one
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-09-05 15:48:30 -07:00
Tony Holdstock-Brown
f9187b2572 Add regulator to GCS
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-09-05 15:48:30 -07:00
Andrey Kostov
b424c3d870 Better error handling for GCS credential argument addition
Signed-off-by: Andrey Kostov <kostov.andrey@gmail.com>
2018-09-05 15:48:29 -07:00
Andrey Kostov
78238ef1a0 Add credentials argument for GCS driver
Signed-off-by: Andrey Kostov <kostov.andrey@gmail.com>
2018-09-05 15:48:29 -07:00
Derek McGowan
c88728f217
Fix registry stripping newlines from manifests
Content must be preserved exactly

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-05 13:40:42 -07:00
Ryan Abrams
6b73a9ab89 Ignore missing paths during enumeration
It's possible to run into a race condition in which the enumerator lists
lots of repositories and then starts the long process of enumerating through
them. In that time if someone deletes a repo, the enumerator may error out.

Signed-off-by: Ryan Abrams <rdabrams@gmail.com>
2018-09-05 10:17:08 -07:00
Derek McGowan
ef859e1b21
Merge pull request #2474 from vikstrous/disable-v1-master
disable schema1 by default, add a config flag to enable it
2018-08-24 10:58:39 -07:00
Olivier
53bd46af5c
Merge pull request #2651 from mikebrow/manifest-version-test-on-put
adds validation testing for schema version values
2018-08-20 12:19:40 -07:00
David Wu
0b0d470281 use aws sdk to validate regions
Signed-off-by: David Wu <david.wu@docker.com>
2018-08-20 11:02:14 -07:00
Mike Brown
2fdb2ac270 adds validation testing for schema version values
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-08-14 08:53:54 -05:00
Derek McGowan
88530ef7a0
Merge pull request #2671 from dmp42/goamz
Remove goamz
2018-08-10 15:54:04 -07:00
Olivier
7d9f067716
Merge pull request #2632 from whoshuu/feature/improve-s3-driver
Improve s3 driver
2018-08-10 15:01:11 -07:00
dmp
9caa7a81bc Remove goamz
Signed-off-by: Olivier <o+github@gambier.email>
2018-08-10 11:11:53 -07:00
Olivier
b7446e89bf
Merge pull request #2599 from legionus/storage-inmemory-deadlock
Fix deadlock in the inmemory storage driver
2018-08-09 15:26:12 -07:00
Olivier
4a75b72fd3
Merge pull request #2662 from dmcgowan/enable-structcheck
Enable static checks
2018-08-07 19:01:38 -07:00
Olivier
eefe9670bd
Merge pull request #2661 from dmcgowan/enable-goimports
Enable goimports check
2018-08-07 14:59:49 -07:00
Derek McGowan
003aa051b4
Merge pull request #2648 from manishtomar/tag-deleted-event
add repo and tag deletion event
2018-08-07 11:50:56 -07:00
Derek McGowan
db0a4ec1c8
Enable static checks
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-06 14:49:11 -07:00
Derek McGowan
795e11d5fb
Enable goimports check
Validates that goimports has been run on all files

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-06 14:37:44 -07:00
Manish Tomar
8c05756141 lint fix
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-06 09:46:42 -07:00
Manish Tomar
328069bb4d add support for repo deleted event also
by having another interface RepositoryRemover that is implemented by
registry instance and is injected in app context for event tracking

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-08-02 23:05:05 -07:00
Mike Brown
20aecf1d7b added test for initial oci schema version
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-07-19 19:41:31 -05:00
Mike Brown
5f588fbf9b address review comment regarding panic use
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-07-19 16:07:26 -05:00
Mike Brown
e8d7941ca6 address lint and gofmt issues
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-25 15:45:15 -05:00
Huu Nguyen
f5c6357c6d Update s3_test to account for SkipVerify parameter
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-06-22 12:00:28 -07:00
Huu Nguyen
7655a3d91f Add option to skip certificate verification for the s3 driver
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-06-22 12:00:21 -07:00
Owen W. Taylor
132abc6de5 Test storing OCI image manifests and indexes with/without a media type
OCI Image manifests and indexes are supported both with and without
an embeded MediaType (the field is reserved according to the spec).
Test storing and retrieving both types from the manifest store.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2018-06-19 11:24:25 -04:00
Owen W. Taylor
60d9c5dfad Handle OCI manifests and image indexes without a media type
In the OCI image specification, the MediaType field is reserved
and otherwise undefined; assume that manifests without a media
in storage are OCI images or image indexes, and determine which
by looking at what fields are in the JSON. We do keep a check
that when unmarshalling an OCI image or image index, if it has
a MediaType field, it must match that media type of the upload.

Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
2018-06-19 11:24:25 -04:00
Mike Brown
ad7ab0853c folow commit 9c88801a12
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
7b47fb13cf update url policy support; testing for annoations in index
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
ec2aa05cdf addressing comments from stevvooe
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
6bae7ca597 refactor adding enum for storage types
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
9e3f78b8c8 addresses minor debug comments
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:55:24 -05:00
Mike Brown
c94f28805e OCI media types; annotation support; oci index
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:48:58 -05:00
Mike Brown
6fcea22b0a add an ocischema manifest handler for the registry
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:48:58 -05:00
Mike Brown
9986e8ca7c adds support for oci manifests and manifestlists
Signed-off-by: Mike Brown <brownwm@us.ibm.com>
2018-06-18 13:48:58 -05:00
Yu Wang
62797237b9 closes #2496 and #2552
Update Azure SDK with release v16.2.1
Update Azure autorest SDK with release v10.8.1

Signed-off-by: Yu Wang <yuwa@microsoft.com>
2018-06-08 16:53:02 -07:00
Gladkov Alexey
fc7e8f42d7 Fix deadlock in the inmemory storage driver
According golang documentation [1]: no goroutine should expect to be
able to acquire a read lock until the initial read lock is released.

[1] https://golang.org/pkg/sync/#RWMutex

Signed-off-by: Gladkov Alexey <agladkov@redhat.com>
2018-05-30 18:01:08 +02:00
Michal Minář
492844e09c redis: delete right size attribute
Signed-off-by: Michal Minář <miminar@redhat.com>
2018-03-13 15:25:32 +01:00
Stephen Day
6664ec7039
Merge pull request #2466 from tifayuki/prometheus_go_metrics
add prometheus metrics
2018-02-09 15:49:04 -08:00
tifayuki
e3c37a46e2 Add Prometheus Metrics
at the first iteration, only the following metrics are collected:

  - HTTP metrics of each API endpoint
  - cache counter for request/hit/miss
  - histogram of storage actions, including:
    GetContent, PutContent, Stat, List, Move, and Delete

Signed-off-by: tifayuki <tifayuki@gmail.com>
2018-02-09 14:27:51 -08:00
Wenkai Yin
005c6e0236 ignore path not found error when look up tags
Signed-off-by: Wenkai Yin <yinw@vmware.com>
2018-02-08 17:32:25 +08:00
Stephen J Day
1ba5b3b553
registry/storage: ignore missing tag on delete
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2018-02-06 15:25:51 -08:00
Stephen Day
fea8bd5114
Merge pull request #2471 from GarageDeveloper/master
fixes #2249: sanitize tenant and tenantid
2018-02-02 16:29:34 -08:00
Sargun Dhillon
cbcbcb02c5 Remove old walk function
This removes the old global walk function, and changes all
the code to use the per-driver walk functions.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-18 13:06:07 -08:00
Sargun Dhillon
c7b0da2622 Use the new walk method for catalog enumeration
This change is primarily to make GC faster.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-18 11:58:46 -08:00
Sargun Dhillon
35b29a609e Use the New Driver Walk method for catalog enumeration
This changes the Walk Method used for catalog enumeration. Just to show
how much an effect this has on our s3 storage:
Original:
List calls: 6839

real    3m16.636s
user    0m0.000s
sys    0m0.016s

New:
ListObjectsV2 Calls: 1805

real    0m49.970s
user    0m0.008s
sys    0m0.000s

This is because it no longer performs a list and stat per item, and instead
is able to use the metadata gained from the list as a replacement to stat.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Sargun Dhillon
32ac467992 Introduce Walk Method Per Storage Driver
Move the Walk types into registry/storage/driver, and add a Walk method to each
storage driver. Although this is yet another API to implement, there is a fall
back implementation that relies on List and Stat. For some filesystems this is
very slow.

Also, this WalkDir Method conforms better do a traditional WalkDir (a la filepath).

This change is in preparation for refactoring.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Jesse Haka
ff87ad884c add possibility to clean untagged manifests
add tests

add possibility to clean untagged manifests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add dry tests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove underscores

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fixes

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

opts struct+use camelcase

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

doublecheck manifest in paths.go

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add gofmt

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix lint

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add log print

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

move log to dryrun as well

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove counter

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove manifest tag references

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add tag to tests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

manifestsWithoutTags -> removeUntagged

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove RemoveManifestTagReferences and use removemanifests

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove comment

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

remove pathfor

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

move removemanifest out of manifestenumerator, it does not work correctly if we delete stuff in it

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

add comment

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix context -> dcontext

Signed-off-by: Jesse Haka <haka.jesse@gmail.com>

fix gofmt
2018-01-05 12:20:01 +02:00
Viktor Stanchev
e9864ce8b9 disable schema1 by default, add a config flag to enable it
port of #2473

Signed-off-by: Viktor Stanchev <me@viktorstanchev.com>
2017-12-19 10:23:25 -08:00
Raphaël Enrici
8777e97b72 fixes #2249: sanitize tenant and tenantid
If tenant or tenantid are passed as env variables, we systematically use Sprint to make sure they are string and not integer as it would make mapstructure fail.

Signed-off-by: Raphaël Enrici <raphael@root-42.com>
2017-12-17 18:33:34 +01:00
tifayuki
e8ecc6dc55 add s3 region filters for cloudfront
Signed-off-by: tifayuki <tifayuki@gmail.com>
2017-12-01 15:58:58 -08:00
Stephen J Day
4abf680c76
registry: feed the linter by removing redundant err check
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-09-29 13:27:49 -07:00
Stephen J Day
860b28c5b9
registry: remove dependency on logrus for client
To simplify the vendoring story for the client, we have now removed the
requirement for `logrus` and the forked `context` package (usually
imported as `dcontext`). We inject the logger via the metrics tracker
for the blob cache and via options on the token handler. We preserve
logs on the proxy cache for that case. Clients expecting these log
messages may need to be updated accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-15 10:28:38 -07:00
Stephen J Day
9c88801a12
context: remove definition of Context
Back in the before time, the best practices surrounding usage of Context
weren't quite worked out. We defined our own type to make usage easier.
As this packaged was used elsewhere, it make it more and more
challenging to integrate with the forked `Context` type. Now that it is
available in the standard library, we can just use that one directly.

To make usage more consistent, we now use `dcontext` when referring to
the distribution context package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-11 15:53:31 -07:00
Derek McGowan
edc3ab29cd Merge pull request #2316 from igmor/logrus_case_dep_update
Update logrus and logrus-logstash-hook libraries
2017-07-26 10:46:10 -07:00
Stephen Day
91c507a39a Merge pull request #2340 from stevvooe/limit-payload-size
registry/{storage,handlers}: limit content sizes
2017-07-20 13:57:55 -07:00
fate-grand-order
a11fe173d5 fix misspelling "algorithm" for cache/redis/redis.go
Signed-off-by: Helen Chen <chenjg@harmonycloud.cn>
2017-07-18 16:02:42 +08:00
Stephen J Day
55ea440428
registry/{storage,handlers}: limit content sizes
Under certain circumstances, the use of `StorageDriver.GetContent` can
result in unbounded memory allocations. In particualr, this happens when
accessing a layer through the manifests endpoint.

This problem is mitigated by setting a 4MB limit when using to access
content that may have been accepted from a user. In practice, this means
setting the limit with the use of `BlobProvider.Get` by wrapping
`StorageDriver.GetContent` in a helper that uses `StorageDriver.Reader`
with a `limitReader` that returns an error.

When mitigating this security issue, we also noticed that the size of
manifests uploaded to the registry is also unlimited. We apply similar
logic to the request body of payloads that are full buffered.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-07-06 17:13:39 -07:00
Igor Morozov
a97d7c0c15 moved Sirupsen to sirupsen on a case sensitive system
Signed-off-by: Igor Morozov <igor@adhoc05-sjc1.prod.uber.internal>
2017-06-23 20:28:48 +00:00
Oleg Bulatov
258345ba0d Fix signalling Wait in regulator.enter
In some conditions, regulator.exit may not send a signal to blocked
regulator.enter.

Let's assume we are in the critical section of regulator.exit and r.available
is equal to 0. And there are three more gorotines. One goroutine also executes
regulator.exit and waits for the lock. Rest run regulator.enter and wait for
the signal.

We send the signal, and after releasing the lock, there will be lock
contention:

  1. Wait from regulator.enter
  2. Lock from regulator.exit

If the winner is Lock from regulator.exit, we will not send another signal to
unlock the second Wait.

Signed-off-by: Oleg Bulatov <obulatov@redhat.com>
2017-06-02 15:41:55 +02:00
william wei
37ca688dc0 Remove unused function
Signed-off-by: william wei <1342247033@qq.com>
2017-05-16 17:47:13 +08:00
Derek McGowan
1d7824702b Merge pull request #2247 from yuwaMSFT2/revendorazuresdk
closes #2224 re-vendor the latest Azure Storage SDK for better performance
2017-04-27 10:54:25 -07:00
Troels Thomsen
7f510ae9c9 Support session token
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2017-04-25 19:31:50 +02:00
Yu Wang (UC)
ac05d143d8 closes #2224: re-vendor the latest Azure Storage SDK for better performance
Signed-off-by: Yu Wang <yuwa@microsoft.com>
2017-04-14 14:20:18 -07:00
Alvin Feng
45bb7c9cc9 Remove expires tag from s3 upload
Signed-off-by: Alvin Feng <alvin4feng@yahoo.com>
2017-03-17 23:41:15 +00:00
Derek McGowan
4f87c80073 Merge pull request #2192 from uhayate/refactor-code-style
refactor the code style in distribution/registry/storage/driver/s3-goamz/s3.go
2017-02-15 17:12:16 -08:00
Aaron Lehmann
62d8d910b5 Merge pull request #2184 from sakeven/master
Check whether must use v4 auth in specific aws region ( storage driver s3-goamz )
2017-02-13 22:29:02 -08:00
sakeven
72bdf0e320 check whether must use v4 auth in specific aws region
Signed-off-by: sakeven <jc5930@sina.cn>
2017-02-14 10:42:20 +08:00
uhayate
75c2e524a1 refactor the code style in distribution/registry/storage/driver/s3-goamz/s3.go
Signed-off-by: uhayate <uhayate.gong@daocloud.io>
2017-02-13 17:29:08 +08:00
Eric Yang
2d20471cd7 fixing acronyms case
Signed-off-by: Eric Yang <qizhao.yang@daocloud.io>
2017-01-31 01:29:30 +08:00
Stephen J Day
7f565ed65a
registry/storage: clean up zero-length blob comments
A previous inspection of the code surrounding zero-length blobs led to
some interesting question. After inspection, it was found that the hash
was indeed for the empty string (""), and not an empty tar, so the code
was correct. The variable naming and comments have been updated
accordingly.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-27 15:28:30 -08:00
Derek McGowan
d8fcbeeb71
Update registry to use WithName for creating Named values
The registry uses partial Named values which the named parsers
no longer support. To allow the registry service to continue
to operate without canonicalization, switch to use WithName.
In the future, the registry should start using fully canonical
values on the backend and WithName should no longer support
creating partial values.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-18 13:56:56 -08:00
Michal Fojtik
9e510d67f5 Add more regions to registry S3 storage driver
Namely adding ca-central-1, ap-south-1 and eu-west-1.

Signed-off-by: Michal Fojtik <mfojtik@redhat.com>
2017-01-11 22:38:24 +01:00
Ahmet Alp Balkan
0a1ce58e2c
azure: revendor + remove hacky solution in is404
Removing the temporary workaround in is404() method by re-vendoring
the azure-sdk-for-go.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2017-01-09 17:22:28 -08:00
Stephen Day
7dba427612 Merge pull request #2111 from stevvooe/use-digest-package
digest: use new go-digest package
2017-01-06 16:25:57 -08:00
Stephen J Day
532ec9f036
digest: migrate to opencontainers/go-digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-06 15:42:03 -08:00
yixi zhang
8e915d69f4 Use app.driver.Stat for registry health check
`app.driver.List` on `"/"` is very expensive if registry contains significant amount of images. And the result isn't used anyways.
In most (if not all) storage drivers, `Stat` has a cheaper implementation, so use it instead to achieve the same goal.

Signed-off-by: yixi zhang <yixi@memsql.com>
2016-12-21 17:12:43 -08:00
Derek McGowan
729b8c5b91 Merge pull request #2105 from stevvooe/algorithm-own-file
digest: cleanup digester and verifier creation
2016-12-15 16:55:37 -08:00
Stephen J Day
9159833265
digest: remove error return from Digest.Verifier
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 16:43:59 -08:00
Stephen J Day
e37baed88e
digest: cleanup digester and verifier creation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 15:17:21 -08:00
Stephen J Day
01dfa0fcb9
digest: use digest.Parse over ParseDigest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 15:07:42 -08:00
Derek McGowan
8867e8fac3
Update schema2 builder to take media type
Modify manifest builder so it can be used to build
manifests with different configuration media types.
Rename config media type const to image config.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2016-12-14 16:17:20 -08:00
Derek McGowan
15dc1296af Merge pull request #2088 from ahmetalpbalkan/pr-upstream-azure-race-fix
azure: fix race condition in PutContent()
2016-12-06 14:07:53 -08:00
Ahmet Alp Balkan
78d0660319
azure: fix race condition in PutContent()
See #2077 for background.

The PR #1438 which was not reviewed by azure folks basically introduced
a race condition around uploads to the same blob by multiple clients
concurrently as it used the "writer" type for PutContent(), introduced in #1438.
This does chunked upload of blobs using "AppendBlob" type, which was not atomic.

Usage of "writer" type and thus AppendBlobs on metadata files is currently not
concurrency-safe and generally, they are not the right type of blob for the job.

This patch fixes PutContent() to use the atomic upload operation that works
for uploads smaller than 64 MB and creates blobs with "BlockBlob" type. To be
backwards compatible, we query the type of the blob first and if it is not
a "BlockBlob" we delete the blob first before doing an atomic PUT. This
creates a small inconsistency/race window "only once". Once the blob is made
"BlockBlob", it is overwritten with a single PUT atomicallly next time.

Therefore, going forward, PutContent() will be producing BlockBlobs and it
will silently migrate the AppendBlobs introduced in #1438 to BlockBlobs with
this patch.

Tested with existing code side by side, both registries with and without this
patch work fine without breaking each other. So this should be good from a
backwards/forward compatiblity perspective, with a cost of doing an extra
HEAD checking the blob type.

Fixes #2077.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-30 12:40:43 -08:00
Kira
4accc8f2be filter listResponse.Contents in driver.List()
Signed-off-by: Kira <me@imkira.com>
2016-11-17 10:38:56 +08:00
Derek McGowan
8b71d39c34
Remove newlines from end of error strings
Golint now checks for new lines at the end of go error strings,
remove these unneeded new lines.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-14 14:03:01 -08:00
Richard Scothern
6e62b39842 Merge pull request #2036 from pyr/fix/sort-v2-headers
v2 signer: correctly sort headers
2016-11-10 15:31:24 -08:00
Richard Scothern
4d65dd513e Merge pull request #2038 from spacexnice/master
fix: oss driver would get connection reset by peer when upload large image layer.
2016-11-10 14:44:32 -08:00
yaoyao.xyy
a4a227e351 oss native large file copy consume too much time which will eventually lead to client timeout because of no data transmit throughout native copy. change maxCopySize to 128MB, ensure only sm all medium size file use oss native copy to avoid connection reset by peer. And fix Move function with CopyLargeFileInParallel to optimize oss upload copy
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-11-08 12:14:13 +08:00
Derek McGowan
a2611c7520 Merge pull request #2027 from ahmetalpbalkan/pr-azure-memleak2
Update vendored azure-sdk-for-go
2016-11-04 10:08:40 -07:00
Ahmet Alp Balkan
2ab25288a2
Update vendored azure-sdk-for-go
Updating to a recent version of Azure Storage SDK to be
able to patch some memory leaks through configurable HTTP client
changes which were made possible by recent patches to it.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-03 13:24:57 -07:00
Pierre-Yves Ritschard
f1cf7de788 fixup! v2 signer: correctly sort headers
Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:07:02 +01:00
Pierre-Yves Ritschard
775cc6d632 v2 signer: correctly sort headers
The current code determines the header order for the
"string-to-sign" payload by sorting on the concatenation
of headers and values, whereas it should only happen on the
key.

During multipart uploads, since `x-amz-copy-source-range` and
`x-amz-copy-source` headers are present, V2 signatures fail to
validate since header order is swapped.

This patch reverts to the expected behavior.

Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:01:34 +01:00
Ahmet Alp Balkan
a994f35657
driver/swift: Fix go vet warning
Driver was passing connections by copying. Storing
`swift.Connection` as pointer to fix the warnings.

Ref: #2030.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-31 11:41:53 -07:00