Commit graph

5148 commits

Author SHA1 Message Date
Sebastiaan van Stijn
3b391d3290
replace strings.Split(N) for strings.Cut() or alternatives
Go 1.18 and up now provides a strings.Cut() which is better suited for
splitting key/value pairs (and similar constructs), and performs better:

```go
func BenchmarkSplit(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_ = strings.SplitN(s, "=", 2)[0]
		}
	}
}

func BenchmarkCut(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_, _, _ = strings.Cut(s, "=")
		}
	}
}
```

    BenchmarkSplit
    BenchmarkSplit-10    	 8244206	       128.0 ns/op	     128 B/op	       4 allocs/op
    BenchmarkCut
    BenchmarkCut-10      	54411998	        21.80 ns/op	       0 B/op	       0 allocs/op

While looking at occurrences of `strings.Split()`, I also updated some for alternatives,
or added some constraints;

- for cases where an specific number of items is expected, I used `strings.SplitN()`
  with a suitable limit. This prevents (theoretical) unlimited splits.
- in some cases it we were using `strings.Split()`, but _actually_ were trying to match
  a prefix; for those I replaced the code to just match (and/or strip) the prefix.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:12 +01:00
Sebastiaan van Stijn
2cd52d5c0c
simplify mocks
Embed the interface that we're mocking; calling any of it's methods
that are not implemented will panic, so should give the same result
as before.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 17:09:24 +01:00
Sebastiaan van Stijn
552b1526c6
reference: check for prefix instead of splitting, and use consts
- use strings.HasPrefix() to check for the prefix we're interested in instead
  of doing a strings.Split() without limits. This makes the code both easier
  to read, and prevents potential situations where we end up with a long slice.
- use consts for defaults; these should never be modified, so better to use
  consts for them to indicate they're fixed values.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-09 15:21:47 +01:00
Wang Yan
9d38ed78d2
Merge pull request #3764 from thaJeztah/bump_cobra
go.mod: github.com/spf13/cobra v1.6.1
2022-11-09 11:19:22 +08:00
Wang Yan
0071e46eee
Merge pull request #3781 from thaJeztah/fix_usage_of_deprecated_funcs
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
2022-11-09 11:03:33 +08:00
Milos Gajdos
9bb63e6e46
Merge pull request #3775 from thaJeztah/remove_digestset 2022-11-08 22:48:25 +00:00
Milos Gajdos
47c3610bff
Merge pull request #3679 from Jamstah/building-doc-updates 2022-11-08 22:46:09 +00:00
Milos Gajdos
75093e0aab
Merge pull request #3777 from thaJeztah/reference_add_sorting 2022-11-08 22:45:19 +00:00
Sebastiaan van Stijn
7b651a9692
digestset: deprecate package in favor of go-digest/digestset
This package was only used for the deprecated "shortid" syntax. Now that
support for this syntax was removed, we can also remove this package.

This patch deprecates and removes the package, adding temporary aliases pointing
to the new location to ease migration from docker/distribution to the new
distribution/distribution/v3. We should remove those aliases in a future update.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 23:17:10 +01:00
Derek McGowan
1052518d9f
reference: implement Sort()
This upstreams `Sort()` as originally implemented in containerd in
https://github.com/containerd/containerd/0886ceaea2470edc7339dfc5ebe0e3257ae84d06

From that commit:

> Fix reference ordering in CRI image store
>
> Currently image references end up being stored in a
> random order due to the way maps are iterated through
> in Go. This leads to inconsistent identifiers being
> resolved when a single reference is needed to identify
> an image and the ordering of the references is used for
> the selection.
>
> Sort references in a consistent and ranked manner,
> from higher information formats to lower.
>
> Note: A `name + tag` reference is considered higher
> information than a `name + digest` reference since a
> registry may be used to resolve the digest from a
> `name + tag` reference.

Co-Authored-by: Derek McGowan <derek@mcg.dev>
Co-Authored-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 22:38:47 +01:00
Milos Gajdos
fee8c686bb
Merge pull request #3776 from thaJeztah/ref_replace_deprecated
reference: replace deprecated function SplitHostname
2022-11-08 17:30:59 +00:00
Sebastiaan van Stijn
bf72536440
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
Both of these were deprecated in 55f675811a,
but the format of the GoDoc comments didn't follow the correct format, which
caused them not being picked up by tools as "deprecated".

This patch updates uses in the codebase to use the alternatives.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 13:51:11 +01:00
Hayley Swimelar
e9a25da7a4
Merge pull request #3772 from thaJeztah/registry_logging_simplify
registry: configureLogging() simplify logic a bit
2022-11-08 09:17:20 +01:00
Hayley Swimelar
c7758d1637
Merge pull request #3774 from thaJeztah/remove_deprecated_shortid
reference: remove support for deprecated "shortid" refs
2022-11-08 09:14:18 +01:00
Hayley Swimelar
4906c6ee88
Merge pull request #3767 from thaJeztah/replace_ioutils
replace deprecated io/ioutil
2022-11-08 09:13:08 +01:00
Milos Gajdos
ad6249fa92
Merge pull request #3681 from Jamstah/api-doc-fix
Fix API doc parameter placeholder: last
2022-11-07 15:34:07 +00:00
James Hewitt
30f5f41d8c
Use GitHub https url
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2022-11-07 15:06:46 +00:00
Milos Gajdos
08764d51bf
Merge pull request #3680 from Jamstah/env-var-clash
Raise a specific error for env var parsing issues
2022-11-07 11:26:09 +00:00
Sebastiaan van Stijn
3c71f4933d
referene: fix formatting of "deprecated" comment.
Go requires "deprecated" comments to have an empty line before them,
and to not be all-caps.

This updates to the comment so that it's correctly picked up as deprecated.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-06 22:52:01 +01:00
zounengren
79d1901549
replace deprecated function
Signed-off-by: Zou Nengren <zouyee1989@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-06 22:49:41 +01:00
Sebastiaan van Stijn
6d4f62d7fd
reference: remove support for deprecated "shortid" refs
The "shortid" syntax was added in d26a3b37a6,
and allowed for matching an image on its ID prefix (this is before images were
content-addressable). With the introduction of content-addressable references,
this syntax became problematic, and Docker deprecated this syntax in 2016
(Docker v1.13.0) through commit; 5fc71599a0

> The `repository:shortid` syntax for referencing images is very little used,
> collides with tag references, and can be confused with digest references.

Support for this syntax was removed in 2017 (Docker 17.12) through commit:
a942c92dd7

containerd uses a fork of the reference package with this syntax removed, and
does not support this syntax:
901bcb2231

This patch removes the deprecated syntax, the ParseAnyReferenceWithSet function,
and the ShortIdentifierRegexp regex.

As there are no external consumers for this function, nor the regexp, I'm
skipping a deprecation cycle for this;

- https://grep.app/search?q=.ShortIdentifierRegexp
- https://grep.app/search?q=.ParseAnyReferenceWithSet%28

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-06 20:23:30 +01:00
Sebastiaan van Stijn
b73c038000
registry: configureLogging() simplify logic a bit
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-05 13:26:13 +01:00
Sebastiaan van Stijn
f1dff3e434
registry: use consts for some defaults
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-05 13:26:05 +01:00
Sebastiaan van Stijn
f8b3af78fc
replace deprecated io/ioutil
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-04 23:47:52 +01:00
Hayley Swimelar
e3509fc1de
Merge pull request #3635 from milosgajdos/make-s3-driver-delete-faster
Delete S3 keys incrementally in batches
2022-11-04 16:56:41 +01:00
Hayley Swimelar
52d948a9f5
Merge pull request #3766 from thaJeztah/gofumpt
format code with gofumpt
2022-11-04 12:19:53 +01:00
Milos Gajdos
526d594ef6
Merge pull request #3770 from Jamstah/log-stderr-doc
Update doc to reflect that logs go to stderr
2022-11-03 22:19:51 +00:00
Sebastiaan van Stijn
e0281dc609
format code with gofumpt
gofumpt (https://github.com/mvdan/gofumpt) provides a supserset of `gofmt` / `go fmt`,
and addresses various formatting issues that linters may be checking for.

We can consider enabling the `gofumpt` linter to verify the formatting in CI, although
not every developer may have it installed, so for now this runs it once to get formatting
in shape.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-03 22:48:20 +01:00
James Hewitt
6dbb55ada5
Update doc to reflect that logs go to stderr
Closes #2855

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2022-11-03 16:37:00 +00:00
Hayley Swimelar
ebfa2a0ac0
Merge pull request #3768 from thaJeztah/http_consts
use http consts for request methods
2022-11-03 13:52:52 +01:00
Milos Gajdos
6a2594c5b0
Merge pull request #3754 from ndeloof/accept-encoding
Revert "registry/client: set Accept: identity header when getting layers
2022-11-03 11:06:17 +00:00
Sebastiaan van Stijn
f9ccd2c6ea
use http consts for request methods
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-02 23:31:47 +01:00
Sebastiaan van Stijn
b1285c33a8
go.mod: github.com/spf13/cobra v1.6.1
We were using v1.0.0 of Cobra as newer versions added spf13/viper as dependency,
which came with many indirect dependencies. Cobra v1.6.0 and up no longer depend
on viper, so we can now safely upgrade to the latest version.

full diff: https://github.com/spf13/cobra/compare/v1.0.0...v1.6.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-02 19:48:45 +01:00
Milos Gajdos
ebc4234fd5
Delete S3 keys incrementally in batches
Instead of first collecting all keys and then batch deleting them,
we will do the incremental delete _online_ per max allowed batch.
Doing this prevents frequent allocations for large S3 keyspaces
and OOM-kills that might happen as a result of those.

This commit introduces storagedriver.Errors type that allows to return
multierrors as a single error from any storage driver implementation.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2022-10-30 19:10:24 +00:00
Milos Gajdos
7f9f86c411
Merge pull request #3755 from wy65701436/fix-getstatus 2022-10-28 08:20:15 +01:00
Wang Yan
bad5dcb602 fit get status issue
1, return the right upload offset for client when asks.
2, do not call ResumeBlobUpload on getting status.
3, return 416 rather than 404 on failed to patch chunk blob.
4, add the missing upload close

Signed-off-by: Wang Yan <wangyan@vmware.com>
2022-10-26 23:33:39 +08:00
Milos Gajdos
c47a966fde
Merge pull request #3727 from thaJeztah/bump_x_net
vendor: golang.org/x/net v0.0.0-20220906165146-f3363e06e74c
2022-10-21 10:26:57 +01:00
Mark Sagi-Kazar
d6ea77ae65
refactor: rename WeakStringList to AudienceList
Signed-off-by: Mark Sagi-Kazar <mark.sagikazar@gmail.com>
2022-10-21 11:11:50 +02:00
Nicolas De Loof
9c04d0b30a
Revert "registry/client: set Accept: identity header when getting layers"
This reverts commit 16f086a0ec.

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
2022-10-20 14:44:10 +02:00
Milos Gajdos
fb2188868d
Merge pull request #3365 from brackendawson/3122-remove-workaround
Remove workaround from 2.1.1 for faulty 2.1.0 manifest links
2022-10-19 09:04:24 +01:00
Wang Yan
ff42b8bf8e
Merge pull request #3748 from gliptak/patch-1
Add Go 1.19 to GHA
2022-10-17 14:33:46 +08:00
Gábor Lipták
8cc5b4f5aa
Add Go 1.19 to GHA
Signed-off-by: Gábor Lipták gliptak@gmail.com
Signed-off-by: Gábor Lipták <gliptak@gmail.com>
2022-10-09 16:52:41 -04:00
Mark Sagi-Kazar
3472f7a8e3
feat: accept lists in the token audience claim
Signed-off-by: Mark Sagi-Kazar <mark.sagikazar@gmail.com>
2022-09-27 15:34:26 +02:00
Mark Sagi-Kazar
97fa1183bf
feat: add WeakStringList type to support lists in aud claim
Signed-off-by: Mark Sagi-Kazar <mark.sagikazar@gmail.com>
2022-09-27 15:31:01 +02:00
Alex
10975deab8 build: harden codeql-analysis.yml permissions
Signed-off-by: Alex <aleksandrosansan@gmail.com>
2022-09-24 08:57:02 +02:00
Alex
e09a9f2dc2 build: harden e2e.yml permissions
Signed-off-by: Alex <aleksandrosansan@gmail.com>
2022-09-24 08:56:40 +02:00
Alex
c26fe145ca build: harden conformance.yml permissions
Signed-off-by: Alex <aleksandrosansan@gmail.com>
2022-09-24 08:56:30 +02:00
Alex
1ca9af0184 build: harden fossa.yml permissions
Signed-off-by: Alex <aleksandrosansan@gmail.com>
2022-09-24 08:53:15 +02:00
Alex
feaa75c529 build: harden validate.yml permissions
Signed-off-by: Alex <aleksandrosansan@gmail.com>
2022-09-24 08:52:57 +02:00
Alex
1667a66856 build: harden build.yml permissions
Signed-off-by: Alex <aleksandrosansan@gmail.com>
2022-09-24 08:52:44 +02:00