Commit graph

1259 commits

Author SHA1 Message Date
Milos Gajdos
17552d864d
Merge pull request #3685 from Jamstah/aws-paging
Work with the storage driver to minimise work when paging
2023-08-31 08:27:26 +01:00
bin liu
2513dd1f96 fix typos in registry/storage/paths.go
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-08-30 22:36:14 +08:00
Milos Gajdos
b9b19409cf
Merge pull request #4028 from liubin/delete-duplicated-code 2023-08-30 07:45:51 +01:00
bin liu
eda5fe2d67 remove duplicated code
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-08-30 07:56:56 +08:00
James Hewitt
e22f7cbc73
Pass the last paging flag to storage drivers
Storage drivers may be able to take advantage of the hint to start
their walk more efficiently.

For S3: The API takes a start-after parameter. Registries with many
repositories can drastically reduce calls to s3 by telling s3 to only
list results lexographically after the last parameter.

For the fallback: We can start deeper in the tree and avoid statting
the files and directories before the hint in a walk. For a filesystem
this improves performance a little, but many of the API based drivers
are currently treated like a filesystem, so this drastically improves
the performance of GCP and Azure blob.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-29 11:27:42 +01:00
Milos Gajdos
3a44c2e10e
Merge pull request #4022 from Jamstah/catalog-optimisation
Don't make a new buffer for catalog listing
2023-08-29 11:24:39 +01:00
James Hewitt
a41613ba3a
Don't make a new buffer for catalog listing
We are given a slice to fill, write catalog entries directly to the
slice until it is full.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-29 11:11:13 +01:00
bin liu
b889cc2eb4 Add username to create redis client
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-08-29 16:27:43 +08:00
Wang Yan
5f8b59177b
Merge pull request #4019 from milosgajdos/replace-redigo-redis
Replace redigo with redis-go
2023-08-29 09:53:32 +08:00
Milos Gajdos
884cf14d30
Merge pull request #4013 from milosgajdos/nonjson-error-client
Dont parse errors as JSON unless Content-Type is set to JSON
2023-08-28 14:28:48 +01:00
Milos Gajdos
45b7b9cec3
Dont parse errors as JSON unless Content-Type is set to JSON
Client attempts to parse the body of every error it receives as JSON
regardless of the content-type. This commit rectifies by only parsing
he error body as JSON if the Content-Type header is set to
either "application/json" or "application/vnd.api+json".

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-28 12:35:33 +01:00
David van der Spek
c7bdabadcf
add back getKeys + cleanup manifeststore test
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-28 12:44:49 +02:00
David van der Spek
f9bc9220eb
feat(storage)!: remove schema1 except manifeststore_test
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-28 12:44:46 +02:00
James Hewitt
1a3e73cb84
Handle rand deprecations in go 1.20
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-28 09:33:12 +01:00
Milos Gajdos
fcbc25e789
Replace redigo with redis-go
We are replacing the very outdated redigo Go module with the official
redis Go module, go-redis.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-26 07:44:02 +01:00
Milos Gajdos
4f7424c8eb
Merge pull request #4000 from DavidSpek/remove-bugsnag
Remove bugsnag
2023-08-23 15:21:18 +01:00
Milos Gajdos
4a85aab30f
Merge pull request #4012 from liubin/fix-typo
fix typo in comment and log
2023-08-23 15:20:05 +01:00
bin liu
1284c48781 fix typo in comment and log
Signed-off-by: bin liu <liubin0329@gmail.com>
2023-08-23 19:39:19 +08:00
Milos Gajdos
59dd684cc8
Merge pull request #3713 from Jamstah/s3-tests 2023-08-21 13:48:43 +01:00
Milos Gajdos
1a4638a06c
Merge pull request #4007 from milosgajdos/remove-schema1-client 2023-08-21 13:46:29 +01:00
Sebastiaan van Stijn
5b3be39870
s3: add interface assertion
This was added for the other drivers in 6b388b1ba6,
but it missed the s3 storage driver.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-08-21 13:54:13 +02:00
Milos Gajdos
df2787c6cf
Update test asserts
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-21 10:27:46 +01:00
David van der Spek
3e4c4ead4c
Remove bugsnag
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-21 10:24:36 +02:00
Milos Gajdos
11c341a369
Remove schema1 references from registry client
We've replaced all the schema1 references with OCI schema manifest.
Note, there are some TODO items that must be addressed at some point in
the future once the schema1 package is removed completely from the
codebase.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-20 09:07:59 +01:00
Milos Gajdos
3dbfbc7255
Enable bodyclose linter
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-19 09:45:44 +01:00
Milos Gajdos
597e8a6b34
Merge pull request #4001 from DavidSpek/remove-newrelic
Remove NewRelic
2023-08-18 22:35:16 +01:00
James Hewitt
37a213dc4b
Revert "optimize catalog last param"
This reverts commit 65f4ce4d93.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-18 13:52:37 +01:00
James Hewitt
8fd504debe
Revert "Rename catalog funcs and update their godocs."
This reverts commit 230cc72a8b.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-18 13:52:28 +01:00
James Hewitt
ad11105052
Revert "removed redundant check"
This reverts commit 0f846853fe.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-18 13:52:19 +01:00
James Hewitt
8e4a8517c5
Revert "fix: resolve most comments"
This reverts commit 6a5846b32e.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-18 13:52:06 +01:00
David van der Spek
77c33cd243
remove NewRelic
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-18 12:16:18 +02:00
Milos Gajdos
bf3c2df6b2
Merge pull request #3902 from pluralsh/catalog-opti-fix-rebase
Optimise catalog function rebase of #3145
2023-08-18 08:11:29 +01:00
Milos Gajdos
ac32466188
Merge pull request #3993 from DavidSpek/update-golang-lru
Update github.com/hashicorp/golang-lru to v2
2023-08-18 08:09:11 +01:00
Milos Gajdos
40c56bf1b6
Keep returning image for default arch to old clients fetching lists
This puts back the original flow where old clients are fetching manifest
lists schema1 images where we want to try returning some image for the
default architecture. This was incorrectly removed by one of the
previous commits.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-17 20:57:02 +01:00
David van der Spek
0f006548a1
update golang-lru to v2
Signed-off-by: David van der Spek <vanderspek.david@gmail.com>
2023-08-17 13:41:54 +02:00
Milos Gajdos
f517191da1
Add small update to api tests
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-17 09:37:26 +01:00
Milos Gajdos
7e39a7c6dc
Remove references to schema1 pacakge from handlers
schema1 package was deprecated a while ago so we are removing
any references to it from handlers. in preparation to
removing it from the codebase altogether.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-17 08:35:54 +01:00
Milos Gajdos
3f1859af26
Remove oss storage driver and alicdn storage driver middleware
This commit removes `oss` storage driver from distribution as well as
`alicdn` storage middleware which only works with the `oss` driver.

There are several reasons for it:
* no real-life expertise among the maintainers
* oss is compatible with S3 API operations required by S3 storage driver

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-16 08:39:20 +01:00
Milos Gajdos
65b57464f9
Merge pull request #3982 from milosgajdos/remove-swift-storage-driver
Remove SWIFT storage driver
2023-08-16 07:47:42 +01:00
Milos Gajdos
fff194fc46
Merge pull request #3983 from Jamstah/azure-tests
Fix Azure tests
2023-08-15 18:31:05 +01:00
James Hewitt
46ff5f8528
Fix Azure tests
The Azure tests fail if there is no Azure configuration available,
instead they should be skipped.

Also, one of the Azure tests is wrong and doesn't match the code.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:46:36 +01:00
James Hewitt
7622d0a453
Don't return the from of a walk
Other storage drivers will only return children and below, s3 should do
the same. The only reason it was returning was because of the addition
of a / to ensure we treat the from as a directory.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:26:37 +01:00
James Hewitt
f7bdd9127b
Don't test the OUTPOSTS storage class
This test will only work on an s3 bucket on an s3 outpost. Most
developers won't have access to one of these.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:52 +01:00
James Hewitt
6ceb904c3e
Don't check returned storage class if we use NONE
If we haven't set a storage class there's no point in checking the
storage class applied to the object - s3 will choose one.

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:51 +01:00
James Hewitt
2d316a12d3
We don't use gocheck in these tests
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:51 +01:00
James Hewitt
f78d81e78a
Remove test as S3 does not support empty directories
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2023-08-15 16:18:48 +01:00
Milos Gajdos
c6b9944ab1
Remove SWIFT storage driver
This commit removes swift storage driver from distribution.
There are several reasons for it:
* no real life expertise among the maintainers
* swift is compatible with S3 API operations required by S3 storage driver

This will also remove depedencies that are also hard to keep up with.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-15 09:14:11 +01:00
Milos Gajdos
24de708d22
Set Content-Type header in registry client ReadFrom
Client ReadFrom doesn't set Content-Type header leading to server
side implementor to assume it's application/octet-stream. This commit
makes this explicit on the client side.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-08-15 08:46:48 +01:00
Milos Gajdos
ff46bf1c41
Merge pull request #2897 from SuperQ/cache_metrics
Cleanup storage cache metrics
2023-08-15 07:44:14 +01:00
Hayley Swimelar
f1529a7782
Merge pull request #3977 from milosgajdos/remove-schema1-proxystore
Remove references to schema1 pacakge from proxy package
2023-08-14 16:24:45 -07:00