Commit graph

532 commits

Author SHA1 Message Date
Oleg Bulatov
cdb4ba947a
Fix err shadowing in gcs driver
Signed-off-by: Oleg Bulatov <oleg@bulatov.me>
2020-03-18 11:41:34 +01:00
Derek McGowan
be29c05a1e
Remove deprecated cache metrics code
The metrics tracker in cached blob statter was replaced with prometheus
metrics and no longer needed.
Remove unused log wrapping.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-09 13:11:54 -07:00
Derek McGowan
495a4af7cf
Fix goimports
Separate fix for cherry-picked code

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Grant Watters
74d442a058
Consider redis.ErrNil as distribution.ErrBlobUnknown for Stat HGET
* Update redis.go

If the dgst key does not exist in the cache when calling HGET, `redis.String` will return an `ErrNil` which we need to translate into `distribution.ErrBlobUnknown` so that the error being returned can be properly handled. This will ensure that `SetDescriptor` is properly called from `cachedBlobStatter::Stat` for `repositoryScopedRedisBlobDescriptorService` which will update the redis cache and be considered as a Miss rather than an Error.

cc @manishtomar

* Update suite.go

Add unit test to ensure missing blobs for scoped repo properly return ErrBlobUnknown when HGET returns redis.ErrNil.

(cherry picked from commit dca6b9526a1d30dd218a9f321c4f84ecc4b5e62e)
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Manish Tomar
795892662b
redis metrics
* redis metrics

it is working but metrics are not very useful since default buckets
start from 5ms and almost all of them are in that range.

* remove extra comment

(cherry picked from commit ba1a1d74e7eb047dd1056548ccf0695e8846782c)
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Manish Tomar
ce101280fe
fix redis caching issue
* fix redis caching issue

earlier redis cache was updated when there was any error including any
temporary connectivity issue. This would trigger set calls which would
further increase load and possibly connectivity errors from redis
leaving the system with continuous errors and high latency. Now the
cache is updated only when it is genuine cache miss. Other errors do not
trigger a cache update.

* add back tracker Hit() and Miss() calls

*squashed commits*
(cherry picked from commit 6f3e1c10260ef59ba4e9c42e939329fad9fdd8c3)
(cherry picked from commit 6738ff3320cf82cc2df919a95a1bde2f7789a501)

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Derek McGowan
4c7c63b557
Add cache unit test
Test base functionality of the cache statter

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-03-04 17:51:37 -08:00
Derek McGowan
6b972e50fe
Merge pull request #2272 from naveedjamil/fips
Increase Unit Test Code Coverage
2020-02-22 17:40:41 -08:00
Derek McGowan
e65b3f1316
Fix CI for test updates
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2020-02-22 17:27:55 -08:00
Naveed Jamil
efdba4f210
Increase Unit Test Code Coverage
Unit test coverge was increased to cover the usages of crypto. This helps to ensure that everything is working fine with fips mode enabled.
Also updated sha1 to sha256 in registry/storage/driver/testsuites/testsuites.go because sha1 is not supported in fips mode.

Signed-off-by: Naveed Jamil <naveed.jamil@tenpearl.com>
2020-02-22 17:09:00 -08:00
Derek McGowan
bfa6b923e7
Merge pull request #3020 from kd7lxl/updatefrequency
Fix typo cloudfront updatefrenquency
2020-02-22 16:54:03 -08:00
sayboras
66809646d9 Migrate to golangci-lint
Signed-off-by: Tam Mach <sayboras@yahoo.com>
2020-02-14 08:11:16 +11:00
Tom Hayward
0f5e2753a6 Fix typo cloudfront updatefrenquency
Signed-off-by: Tom Hayward <thayward@infoblox.com>
2019-12-13 12:11:42 -08:00
Guillaume Rose
c9c3324300 Add unit tests for BlobEnumerator
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-06 09:38:13 +01:00
Manish Tomar
5538da4923 fixes to make layersPathSpec work
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-05 20:48:33 +01:00
Manish Tomar
fa7d949408 allow Repository.BlobStore to enumerate over blobs
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-05 20:48:33 +01:00
Manish Tomar
cf77113795 add pathspec for repo _layers directory
Signed-off-by: Guillaume Rose <guillaume.rose@docker.com>
2019-12-05 20:48:33 +01:00
Ryan Abrams
ae2e973db9
Merge pull request #2748 from manishtomar/tag-digests
API to retrive tag's digests
2019-10-08 12:14:56 -07:00
Terin Stock
b23dd1ef37 swift: correct segment path generation
When uploading segments to Swift, the registry generates a random file,
by taking the hash of the container path and 32-bytes of random data.
The registry attempts to shard across multiple directory paths, by
taking the first three hex characters as leader.

The implementation in registry, unfortunately, takes the hash of
nothing, and appends it to the path and random data. This results in all
segments being created in one directory.

Fixes: #2407
Fixes: #2311
Signed-off-by: Terin Stock <terinjokes@gmail.com>
2019-06-22 23:44:43 -07:00
Ryan Abrams
d80a17d8e0
Merge pull request #2879 from tbe/fix-s3-ceph
Fix s3 driver for supporting ceph radosgw
2019-06-19 19:09:07 -07:00
Ryan Abrams
55287010ce
Merge pull request #2894 from jabrown85/fix-offset-typos
Fix typo: offest -> offset
2019-06-19 19:08:22 -07:00
Ryan Abrams
3226863cbc
Merge pull request #2849 from Shawnpku/master
support Alibaba Cloud CDN storage middleware
2019-04-16 18:43:06 -07:00
Jesse Brown
74f429a5ad Fix typo: offest -> offset
Signed-off-by: Jesse Brown <jabrown85@gmail.com>
2019-04-05 14:20:20 -05:00
Shawn Chen
fd77cf43a6 change package name & format document
Signed-off-by: Shawn Chen <chen8132@gmail.com>
2019-03-18 11:35:46 +08:00
Thomas Berger
c18c6c33b2 S3 Driver: added comment for missing KeyCount workaround
Signed-off-by: Thomas Berger <loki@lokis-chaos.de>
2019-03-15 21:05:21 +01:00
Eohyung Lee
f877726503 Fix s3 driver for supporting ceph radosgw
Radosgw does not support S3 `GET Bucket` API v2 API but v1.
This API has backward compatibility, so most of this API is working
correctly but we can not get `KeyCount` in v1 API and which is only
for v2 API.

Signed-off-by: Eohyung Lee <liquidnuker@gmail.com>
2019-03-08 12:45:20 +01:00
Shawn Chen
3390f32aec fix Context issue
Signed-off-by: Shawn Chen <chen8132@gmail.com>
2019-03-04 17:48:32 +08:00
Shawnpku
6e10631d9c fix default cdn auth duration
Signed-off-by: Shawnpku <chen8132@gmail.com>
2019-03-04 14:53:48 +08:00
Vishesh Jindal
f9a0506191
Bugfix: Make ipfilteredby not required
Signed-off-by: Vishesh Jindal <vishesh92@gmail.com>
2019-03-02 08:58:52 +05:30
Shawnpku
bbc9885aa2 fix func name
Signed-off-by: Shawnpku <chen8132@gmail.com>
2019-02-20 15:54:21 +08:00
Shawnpku
3aa2a282f7 support alicdn middleware
Signed-off-by: Shawnpku <chen8132@gmail.com>
2019-02-11 15:11:26 +08:00
Manish Tomar
48818fdea7 Remove err nil check
since type checking nil will not panic and return appropriately

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2019-02-04 16:42:44 -08:00
Manish Tomar
da8db4666b Fix gometalint errors
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2019-02-04 16:01:04 -08:00
Li Yi
90bed67126 Support BYOK for OSS storage driver
Change-Id: I423ad03e63bd38aded3abfcba49079ff2fbb3b74
Signed-off-by: Li Yi <denverdino@gmail.com>
2018-12-25 08:30:40 +08:00
Manish Tomar
1251e51ad0 better name and updated tests
- use ManifestDigests name instead of Indexes
- update tests to validate against multiple tags

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-11-01 10:31:08 -07:00
Manish Tomar
9ebf151ac2 API to retrive tag's digests
Add an interface alongside TagStore that provides API to retreive
digests of all manifests that a tag historically pointed to. It also
includes currently linked tag.

Signed-off-by: Manish Tomar <manish.tomar@docker.com>
2018-10-29 18:19:05 -07:00
Rui Cao
569d18aef9 Fix some typos
Signed-off-by: Rui Cao <ruicao@alauda.io>
2018-09-24 09:05:44 +08:00
David Wu
a927fbdb9b track digest offset in blobwriter
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-12 14:53:27 -07:00
David Wu
166874ade9 fix gofmt and goimports
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 16:14:10 -07:00
David Wu
877d706b38 remove dependencies on resumable
Signed-off-by: David Wu <david.wu@docker.com>
2018-09-11 08:21:27 -07:00
Derek McGowan
642075f42c
Merge pull request #2631 from whoshuu/feature/improve-gcs-driver
Improve gcs driver
2018-09-05 17:48:03 -07:00
Derek McGowan
15de837aa8
Merge pull request #2704 from dmcgowan/fix-2703
Fix registry stripping newlines from manifests
2018-09-05 17:46:56 -07:00
Huu Nguyen
7a195dd5ca Add back include_gcs build constraint
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-09-05 15:48:30 -07:00
Huu Nguyen
69299d93d9 Use existing jwtConf instead of creating a scoped one
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-09-05 15:48:30 -07:00
Tony Holdstock-Brown
f9187b2572 Add regulator to GCS
Signed-off-by: Huu Nguyen <whoshuu@gmail.com>
2018-09-05 15:48:30 -07:00
Andrey Kostov
b424c3d870 Better error handling for GCS credential argument addition
Signed-off-by: Andrey Kostov <kostov.andrey@gmail.com>
2018-09-05 15:48:29 -07:00
Andrey Kostov
78238ef1a0 Add credentials argument for GCS driver
Signed-off-by: Andrey Kostov <kostov.andrey@gmail.com>
2018-09-05 15:48:29 -07:00
Derek McGowan
c88728f217
Fix registry stripping newlines from manifests
Content must be preserved exactly

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-09-05 13:40:42 -07:00
Ryan Abrams
6b73a9ab89 Ignore missing paths during enumeration
It's possible to run into a race condition in which the enumerator lists
lots of repositories and then starts the long process of enumerating through
them. In that time if someone deletes a repo, the enumerator may error out.

Signed-off-by: Ryan Abrams <rdabrams@gmail.com>
2018-09-05 10:17:08 -07:00
Derek McGowan
ef859e1b21
Merge pull request #2474 from vikstrous/disable-v1-master
disable schema1 by default, add a config flag to enable it
2018-08-24 10:58:39 -07:00