Commit graph

1385 commits

Author SHA1 Message Date
Milos Gajdos
2c6b6482fc
Include headers when serving blob through proxy (#4273) 2024-05-14 14:27:09 +01:00
Milos Gajdos
6a9b0cfb71
Add support for Basic Authentication to proxyingRegistry (#4263)
Merging despite CodeQL warnings. see this for more details, why we decided to merge: https://github.com/github/codeql/issues/16486
2024-05-14 10:43:56 +01:00
Milos Gajdos
56a020f7f1
Stop proxy scheduler on system exit (#4293) 2024-05-13 17:31:23 +01:00
Dimitar Kostadinov
062309c08b Stop proxy scheduler on system exit
Signed-off-by: Dimitar Kostadinov <dimitar.kostadinov@sap.com>
2024-05-13 17:01:35 +03:00
Milos Gajdos
c49220d492
Fix #2902: ‘autoRedirect’ hardcode ‘https’ scheme (#2903) 2024-05-04 15:32:25 +01:00
Sylvain DESGRAIS
f1875862cf Set readStartAtFile context aware for purge uploads
Signed-off-by: Sylvain DESGRAIS <sylvain.desgrais@gmail.com>
2024-05-02 11:06:39 +02:00
Robin Ketelbuters
16a305ebaf Add registry.Shutdown method for graceful shutdown of embedded registry
Signed-off-by: Robin Ketelbuters <robin.ketelbuters@gmail.com>
2024-04-29 20:18:58 +02:00
Liang Zheng
a2afe23f38 add concurrency limits for tag lookup and untag
Harbor is using the distribution for it's (harbor-registry) registry component.
The harbor GC will call into the registry to delete the manifest, which in turn
then does a lookup for all tags that reference the deleted manifest.
To find the tag references, the registry will iterate every tag in the repository
and read it's link file to check if it matches the deleted manifest (i.e. to see
if uses the same sha256 digest). So, the more tags in repository, the worse the
performance will be (as there will be more s3 API calls occurring for the tag
directory lookups and tag file reads).

Therefore, we can use concurrent lookup and untag to optimize performance as described in https://github.com/goharbor/harbor/issues/12948.

P.S. This optimization was originally contributed by @Antiarchitect, now I would like to take it over.
Thanks @Antiarchitect's efforts with PR https://github.com/distribution/distribution/pull/3890.

Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
2024-04-26 22:32:21 +08:00
Liang Zheng
112156321f fix: ignore error of manifest tag path not found in gc
it is reasonable to ignore the error that the manifest tag path does not exist when querying
all tags of the specified repository when executing gc.

Signed-off-by: Liang Zheng <zhengliang0901@gmail.com>
2024-04-25 17:13:06 +08:00
Milos Gajdos
e6d1d182bf
Allow setting s3 forcepathstyle without regionendpoint (#4291) 2024-04-24 08:34:01 +01:00
guoguangwu
2fe3442035 chore: fix some typos in comments
Signed-off-by: guoguangwu <guoguangwug@gmail.com>
2024-04-23 17:48:53 +08:00
Milos Gajdos
e8ea4e5951
chore: fix some typos in comments (#4332) 2024-04-23 09:03:51 +01:00
Milos Gajdos
bdd3d31fae
proxy: Do not configure HTTP secret for proxy registry (#4305) 2024-04-23 08:17:50 +01:00
goodactive
e0a1ce14a8 chore: fix some typos in comments
Signed-off-by: goodactive <goodactive@qq.com>
2024-04-23 12:04:03 +08:00
Anthony Ramahay
601b37d98b Handle OCI image index and V2 manifest list during garbage collection
Signed-off-by: Anthony Ramahay <thewolt@gmail.com>
2024-04-20 16:41:50 +02:00
Benjamin Schanzel
8654a0ee45
Allow setting s3 forcepathstyle without regionendpoint
Currently, the `forcepathstyle` parameter for the s3 storage driver is
considered only if the `regionendpoint` parameter is set. Since setting
a region endpoint explicitly is discouraged with AWS s3, it is not clear
how to enforce path style URLs with AWS s3.
This also means, that the default value (true) only applies if a region
endpoint is configured.

This change makes sure we always forward the `forcepathstyle` parameter
to the aws-sdk if present in the config. This is a breaking change where
a `regionendpoint` is configured but no explicit `forcepathstyle` value
is set.

Signed-off-by: Benjamin Schanzel <benjamin.schanzel@bmw.de>
2024-04-08 12:45:26 +02:00
xiaoxiangxianzi
2446e1102d chore: remove repetitive words in comments
Signed-off-by: xiaoxiangxianzi <zhaoyizheng@outlook.com>
2024-03-27 17:34:22 +08:00
Milos Gajdos
3cb985cac0
Initialize proxy prometheus counters values to 0 (#4283) 2024-03-18 14:34:28 +00:00
Ismail Alidzhikov
127fa7e057 proxy: Do not configure HTTP secret for proxy registry
Signed-off-by: Ismail Alidzhikov <i.alidjikov@gmail.com>
2024-03-15 18:27:08 +02:00
Milos Gajdos
d9815da9cb
Support redirects in gcs storage with default credentials (#4295) 2024-03-11 22:29:57 +00:00
Tadeusz Dudkiewicz
de450c903a update: support redirects in gcs storage with default credentials
Signed-off-by: Tadeusz Dudkiewicz <tadeusz.dudkiewicz@rtbhouse.com>
2024-03-11 21:05:03 +01:00
guoguangwu
6465b4cd08 fix: typo
Signed-off-by: guoguangwu <guoguangwug@gmail.com>
2024-03-07 10:08:58 +08:00
icefed
63eb22d74b
Fix: ‘autoRedirect’ hardcode ‘https’ scheme
Signed-off-by: icefed <zlwangel@gmail.com>
2024-03-05 20:50:09 +08:00
gotgelf
f690b3ebe2 Added Open Telemetry Tracing to Filesystem package
Signed-off-by: gotgelf <gotgelf@gmail.com>
2024-03-04 13:31:22 +01:00
Jaime Martinez
2763ba1eae
Do not write manifests on HEAD requests
Signed-off-by: Jaime Martinez <jmartinez@gitlab.com>
2024-02-29 11:16:11 +11:00
Dimitar Kostadinov
6ca646caad Initialize proxy prometheus counters values to 0 to prevent gaps after registry restart
Signed-off-by: Dimitar Kostadinov <dimitar.kostadinov@sap.com>
2024-02-21 14:35:49 +02:00
oliver-goetz
1e8ea03173
Add support for Basic Authentication to proxyingRegistry
Signed-off-by: oliver-goetz <o.goetz@sap.com>
2024-02-07 03:08:12 +01:00
Mikel Rychliski
041824555c Include headers when serving blob through proxy
In commit 17952924f3 we updated ServeBlob() to use an io.MultiWriter to
write simultaneously to the local store and the HTTP response.

However, copyContent was using a type assertion to only add headers if
the io.Writer was a http.ResponseWriter. Therefore, this change caused
us to stop sending the expected headers (i.e. Content-Length, Etag,
etc.) on the first request for a blob.

Resolve the issue by explicitly passing in http.Header and setting it
unconditionally.

Signed-off-by: Mikel Rychliski <mikel@mikelr.com>
2024-02-01 19:31:53 -05:00
Milos Gajdos
945eed71e1
feat: Add HTTP2 for unencrypted HTTP (v3) (#4248) 2024-01-18 20:51:58 +07:00
erezrokah
11f50c034e
feat: Add HTTP2 for unencrypted HTTP
Signed-off-by: erezrokah <erezrokah@users.noreply.github.com>
2024-01-17 20:59:02 +00:00
Eng Zer Jun
41161a6e12
refactor(storage/s3): remove redundant len check
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2024-01-17 18:27:05 +08:00
Wang Yan
14366a2dff
fix: load gcs credentials and client inside DriverConstructor (#4218) 2024-01-12 18:32:28 +08:00
Paul Meyer
5bd7f25880 fix: load gcs credentials and client inside DriverConstructor
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-12-27 11:22:27 +01:00
Paul Meyer
6908e0d5fa fix: add missing skip in s3 driver test
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-12-26 13:55:18 +01:00
Milos Gajdos
c5a887217e
version: export getter functions (#4204) 2023-12-19 23:24:35 +00:00
Cory Snider
a74cacff04 version: export getter functions
Future-proof the version package's exported interface by only making the
data available through getter functions. This affords us the flexibility
to e.g. implement them in terms of "runtime/debug".ReadBuildInfo() in
the future.

Signed-off-by: Cory Snider <csnider@mirantis.com>
2023-12-19 13:02:44 -05:00
Milos Gajdos
d59a570c3d
update: set User-Agent header in GCS storage driver
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-12-19 14:39:13 +00:00
Wang Yan
4a360f9da2
fix: remove disabling of multipart combine small parts (#4193) 2023-12-19 16:10:19 +08:00
Milos Gajdos
2f98b77171
fix: don't override storage driver useragent if it's set
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-12-18 14:21:38 +00:00
Milos Gajdos
7ba91015f5
fix: remove disabling of multipart combine small parts
This reverts https://github.com/distribution/distribution/pull/3556

This feature is currently broken and requires more fundamental changes
in the S3 driver. Until then it's better to remove it.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-12-18 09:52:19 +00:00
Milos Gajdos
def497a8aa
update: add tests for S3 driver client SkipVerify settings
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-12-16 12:48:55 +00:00
Milos Gajdos
8fa7a81cb2
fix: use http.DefaultTransport in S3 client
Unfortunately one of the changes we merged in broken the support for
http.ProxyFromEnvironment https://pkg.go.dev/net/http#ProxyFromEnvironment

This commit attempts to fix that by cloning the http.DefaultTransport
and updating it accordingly.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-12-15 09:34:06 +00:00
Milos Gajdos
3f3e61e299
fix: update incorrect godoc comment for (writer).Writer()
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-12-13 14:56:06 +00:00
Milos Gajdos
4baddbc608
fix: update S3 storage driver writer
This commit updates (writer).Writer() method in S3 storage driver to
handle the case where an append is attempted to a zer-size content.

S3 does not allow appending to already committed content, so we are
optiing to provide the following case as a narrowed down behaviour:
Writer can only append to zero byte content - in that case, a new S3
MultipartUpload is created that will be used for overriding the already
committed zero size content.

Appending to non-zero size content fails with error.

Co-authored-by: Cory Snider <corhere@gmail.com>
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-12-13 09:22:48 +00:00
Eng Zer Jun
80cbd744cc
refactor: apply suggestions from code review
This commit apply the following suggestions:

	1. https://github.com/distribution/distribution/pull/4185#discussion_r1419874037
	2. https://github.com/distribution/distribution/pull/4185#discussion_r1419876581
	3. https://github.com/distribution/distribution/pull/4185#discussion_r1419879450
	4. https://github.com/distribution/distribution/pull/4185#discussion_r1419886923

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-12-13 09:22:48 +00:00
Eng Zer Jun
ed5d493405
refactor: apply suggestions from code review
This commit apply the following suggestions:

	1. https://github.com/distribution/distribution/pull/4185#discussion_r1419694460
	2. https://github.com/distribution/distribution/pull/4185#discussion_r1419697921
	3. https://github.com/distribution/distribution/pull/4185#discussion_r1419699112
	4. https://github.com/distribution/distribution/pull/4185#discussion_r1419702609

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-12-13 09:22:48 +00:00
Eng Zer Jun
bcbf0431d1
testing: replace legacy gopkg.in/check.v1
This commit replaces the legacy `gopkg.in/check.v1` testing dependency
with `github.com/stretchr/testify`.

Closes https://github.com/distribution/distribution/issues/3884.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2023-12-13 09:22:43 +00:00
Milos Gajdos
bdf70a1e46
Otel tracing MVP (#4188) 2023-12-11 22:08:40 +00:00
gotgelf
fb7cdf8900 Otel tracing MVP
Signed-off-by: gotgelf <gotgelf@gmail.com>
2023-12-11 21:18:58 +01:00
Milos Gajdos
1054d157bf
update: remove gcs storage driver build tags
GCS storage driver used to be conditionally built due to its being
outdated and basically unmaintained. Recently the driver has gone
through a rework and updates. Let's remove the build tag so we have less
headaches dealing with it and try keeping it up to date.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2023-12-10 09:09:52 +00:00