Commit graph

4803 commits

Author SHA1 Message Date
Sebastiaan van Stijn
2bf5e1879e
reference: remove left-over occurrences of "short-identifier"
This was removed in 6d4f62d7fd, which forgot
to remove it from the grammar.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:24 +01:00
Sebastiaan van Stijn
8163e20c6f
reference: splitDockerDomain: remove incorrect "TODO"
My mistake; I added this TODO in 552b1526c6, but it
only applies to familiarizeName.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-20 15:07:21 +01:00
Milos Gajdos
85d4039064
Merge pull request #3786 from thaJeztah/reference_subtests 2022-11-19 09:36:43 +00:00
Sebastiaan van Stijn
b6a040faf4
reference: run tests with t.Parallel()
Not all tests have been rewritten to use sub-tests; for those
I enabled t.Parallel() for the parent test only.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-18 21:17:56 +01:00
Sebastiaan van Stijn
5703bcf17d
reference: use subtests for regex tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-18 21:16:51 +01:00
Sebastiaan van Stijn
f481f877f1
reference: use subtests for reference tests
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-18 21:13:49 +01:00
Milos Gajdos
21622ca699
Merge pull request #3787 from thaJeztah/simplify_mocks 2022-11-18 08:53:26 +00:00
Milos Gajdos
5357c45703
Merge pull request #3788 from thaJeztah/deprecate_ReadSeekCloser 2022-11-18 08:53:15 +00:00
AdamKorcz
9337b8df66 Fuzzing: Rewrite existing fuzzers to native go fuzzers
Signed-off-by: AdamKorcz <adam@adalogics.com>
2022-11-12 17:30:10 +00:00
Milos Gajdos
3b8fbf9752
Merge pull request #3686 from m5i-work/exp 2022-11-11 17:07:14 +00:00
Wei Meng
35cae1099e Realloc slice exponentially in mfs
`registry/storage/driver/inmemory/driver_test.go` times out after ~10min. The slow test is `testsuites.go:TestWriteReadLargeStreams()` which writes a 5GB file.
Root cause is inefficient slice reallocation algorithm. The slice holding file bytes grows only 32K on each allocation. To fix it, this PR grows slice exponentially.

Signed-off-by: Wei Meng <wemeng@microsoft.com>
2022-11-11 18:18:08 +08:00
Milos Gajdos
e67e40bfd2
Merge pull request #3769 from thaJeztah/strings_cut 2022-11-11 08:37:28 +00:00
Sebastiaan van Stijn
1d8cd5e443
registry/client: use struct literals
Remove some intermediate variables, and use struct literals instead.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:11:12 +01:00
Sebastiaan van Stijn
d71ad5b3a6
transport.NewHTTPReadSeeker: return concrete type, deprecate ReadSeekCloser
General convention is to define interfaces on the receiver side, and
to return concrete types.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:10:39 +01:00
Sebastiaan van Stijn
019ead86f5
deprecate ReadSeekCloser in favor of io.ReadSeekCloser
Go's io package in stdlib now defines this interface, so we can switch
to using that instead.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 23:10:32 +01:00
Sebastiaan van Stijn
842d4c04f5
cloudfront: use strings.Equalfold()
Minor optimization :)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:16 +01:00
Sebastiaan van Stijn
3b391d3290
replace strings.Split(N) for strings.Cut() or alternatives
Go 1.18 and up now provides a strings.Cut() which is better suited for
splitting key/value pairs (and similar constructs), and performs better:

```go
func BenchmarkSplit(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_ = strings.SplitN(s, "=", 2)[0]
		}
	}
}

func BenchmarkCut(b *testing.B) {
	b.ReportAllocs()
	data := []string{"12hello=world", "12hello=", "12=hello", "12hello"}
	for i := 0; i < b.N; i++ {
		for _, s := range data {
			_, _, _ = strings.Cut(s, "=")
		}
	}
}
```

    BenchmarkSplit
    BenchmarkSplit-10    	 8244206	       128.0 ns/op	     128 B/op	       4 allocs/op
    BenchmarkCut
    BenchmarkCut-10      	54411998	        21.80 ns/op	       0 B/op	       0 allocs/op

While looking at occurrences of `strings.Split()`, I also updated some for alternatives,
or added some constraints;

- for cases where an specific number of items is expected, I used `strings.SplitN()`
  with a suitable limit. This prevents (theoretical) unlimited splits.
- in some cases it we were using `strings.Split()`, but _actually_ were trying to match
  a prefix; for those I replaced the code to just match (and/or strip) the prefix.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 22:38:12 +01:00
Sebastiaan van Stijn
2cd52d5c0c
simplify mocks
Embed the interface that we're mocking; calling any of it's methods
that are not implemented will panic, so should give the same result
as before.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-10 17:09:24 +01:00
Sebastiaan van Stijn
552b1526c6
reference: check for prefix instead of splitting, and use consts
- use strings.HasPrefix() to check for the prefix we're interested in instead
  of doing a strings.Split() without limits. This makes the code both easier
  to read, and prevents potential situations where we end up with a long slice.
- use consts for defaults; these should never be modified, so better to use
  consts for them to indicate they're fixed values.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-09 15:21:47 +01:00
Wang Yan
9d38ed78d2
Merge pull request #3764 from thaJeztah/bump_cobra
go.mod: github.com/spf13/cobra v1.6.1
2022-11-09 11:19:22 +08:00
Wang Yan
0071e46eee
Merge pull request #3781 from thaJeztah/fix_usage_of_deprecated_funcs
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
2022-11-09 11:03:33 +08:00
Milos Gajdos
9bb63e6e46
Merge pull request #3775 from thaJeztah/remove_digestset 2022-11-08 22:48:25 +00:00
Milos Gajdos
47c3610bff
Merge pull request #3679 from Jamstah/building-doc-updates 2022-11-08 22:46:09 +00:00
Milos Gajdos
75093e0aab
Merge pull request #3777 from thaJeztah/reference_add_sorting 2022-11-08 22:45:19 +00:00
Sebastiaan van Stijn
7b651a9692
digestset: deprecate package in favor of go-digest/digestset
This package was only used for the deprecated "shortid" syntax. Now that
support for this syntax was removed, we can also remove this package.

This patch deprecates and removes the package, adding temporary aliases pointing
to the new location to ease migration from docker/distribution to the new
distribution/distribution/v3. We should remove those aliases in a future update.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 23:17:10 +01:00
Derek McGowan
1052518d9f
reference: implement Sort()
This upstreams `Sort()` as originally implemented in containerd in
https://github.com/containerd/containerd/0886ceaea2470edc7339dfc5ebe0e3257ae84d06

From that commit:

> Fix reference ordering in CRI image store
>
> Currently image references end up being stored in a
> random order due to the way maps are iterated through
> in Go. This leads to inconsistent identifiers being
> resolved when a single reference is needed to identify
> an image and the ordering of the references is used for
> the selection.
>
> Sort references in a consistent and ranked manner,
> from higher information formats to lower.
>
> Note: A `name + tag` reference is considered higher
> information than a `name + digest` reference since a
> registry may be used to resolve the digest from a
> `name + tag` reference.

Co-Authored-by: Derek McGowan <derek@mcg.dev>
Co-Authored-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 22:38:47 +01:00
Milos Gajdos
fee8c686bb
Merge pull request #3776 from thaJeztah/ref_replace_deprecated
reference: replace deprecated function SplitHostname
2022-11-08 17:30:59 +00:00
Sebastiaan van Stijn
bf72536440
Remove uses of deprecated go-digest.NewDigestFromHex, go-digest.Digest.Hex
Both of these were deprecated in 55f675811a,
but the format of the GoDoc comments didn't follow the correct format, which
caused them not being picked up by tools as "deprecated".

This patch updates uses in the codebase to use the alternatives.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-08 13:51:11 +01:00
Hayley Swimelar
e9a25da7a4
Merge pull request #3772 from thaJeztah/registry_logging_simplify
registry: configureLogging() simplify logic a bit
2022-11-08 09:17:20 +01:00
Hayley Swimelar
c7758d1637
Merge pull request #3774 from thaJeztah/remove_deprecated_shortid
reference: remove support for deprecated "shortid" refs
2022-11-08 09:14:18 +01:00
Hayley Swimelar
4906c6ee88
Merge pull request #3767 from thaJeztah/replace_ioutils
replace deprecated io/ioutil
2022-11-08 09:13:08 +01:00
Milos Gajdos
ad6249fa92
Merge pull request #3681 from Jamstah/api-doc-fix
Fix API doc parameter placeholder: last
2022-11-07 15:34:07 +00:00
James Hewitt
30f5f41d8c
Use GitHub https url
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2022-11-07 15:06:46 +00:00
Milos Gajdos
08764d51bf
Merge pull request #3680 from Jamstah/env-var-clash
Raise a specific error for env var parsing issues
2022-11-07 11:26:09 +00:00
Sebastiaan van Stijn
3c71f4933d
referene: fix formatting of "deprecated" comment.
Go requires "deprecated" comments to have an empty line before them,
and to not be all-caps.

This updates to the comment so that it's correctly picked up as deprecated.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-06 22:52:01 +01:00
zounengren
79d1901549
replace deprecated function
Signed-off-by: Zou Nengren <zouyee1989@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-06 22:49:41 +01:00
Sebastiaan van Stijn
6d4f62d7fd
reference: remove support for deprecated "shortid" refs
The "shortid" syntax was added in d26a3b37a6,
and allowed for matching an image on its ID prefix (this is before images were
content-addressable). With the introduction of content-addressable references,
this syntax became problematic, and Docker deprecated this syntax in 2016
(Docker v1.13.0) through commit; 5fc71599a0

> The `repository:shortid` syntax for referencing images is very little used,
> collides with tag references, and can be confused with digest references.

Support for this syntax was removed in 2017 (Docker 17.12) through commit:
a942c92dd7

containerd uses a fork of the reference package with this syntax removed, and
does not support this syntax:
901bcb2231

This patch removes the deprecated syntax, the ParseAnyReferenceWithSet function,
and the ShortIdentifierRegexp regex.

As there are no external consumers for this function, nor the regexp, I'm
skipping a deprecation cycle for this;

- https://grep.app/search?q=.ShortIdentifierRegexp
- https://grep.app/search?q=.ParseAnyReferenceWithSet%28

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-06 20:23:30 +01:00
Sebastiaan van Stijn
b73c038000
registry: configureLogging() simplify logic a bit
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-05 13:26:13 +01:00
Sebastiaan van Stijn
f1dff3e434
registry: use consts for some defaults
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-05 13:26:05 +01:00
Sebastiaan van Stijn
f8b3af78fc
replace deprecated io/ioutil
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-04 23:47:52 +01:00
Hayley Swimelar
e3509fc1de
Merge pull request #3635 from milosgajdos/make-s3-driver-delete-faster
Delete S3 keys incrementally in batches
2022-11-04 16:56:41 +01:00
Hayley Swimelar
52d948a9f5
Merge pull request #3766 from thaJeztah/gofumpt
format code with gofumpt
2022-11-04 12:19:53 +01:00
Milos Gajdos
526d594ef6
Merge pull request #3770 from Jamstah/log-stderr-doc
Update doc to reflect that logs go to stderr
2022-11-03 22:19:51 +00:00
Sebastiaan van Stijn
e0281dc609
format code with gofumpt
gofumpt (https://github.com/mvdan/gofumpt) provides a supserset of `gofmt` / `go fmt`,
and addresses various formatting issues that linters may be checking for.

We can consider enabling the `gofumpt` linter to verify the formatting in CI, although
not every developer may have it installed, so for now this runs it once to get formatting
in shape.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-03 22:48:20 +01:00
James Hewitt
6dbb55ada5
Update doc to reflect that logs go to stderr
Closes #2855

Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
2022-11-03 16:37:00 +00:00
Hayley Swimelar
ebfa2a0ac0
Merge pull request #3768 from thaJeztah/http_consts
use http consts for request methods
2022-11-03 13:52:52 +01:00
Milos Gajdos
6a2594c5b0
Merge pull request #3754 from ndeloof/accept-encoding
Revert "registry/client: set Accept: identity header when getting layers
2022-11-03 11:06:17 +00:00
Sebastiaan van Stijn
f9ccd2c6ea
use http consts for request methods
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-02 23:31:47 +01:00
Sebastiaan van Stijn
b1285c33a8
go.mod: github.com/spf13/cobra v1.6.1
We were using v1.0.0 of Cobra as newer versions added spf13/viper as dependency,
which came with many indirect dependencies. Cobra v1.6.0 and up no longer depend
on viper, so we can now safely upgrade to the latest version.

full diff: https://github.com/spf13/cobra/compare/v1.0.0...v1.6.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2022-11-02 19:48:45 +01:00
Milos Gajdos
ebc4234fd5
Delete S3 keys incrementally in batches
Instead of first collecting all keys and then batch deleting them,
we will do the incremental delete _online_ per max allowed batch.
Doing this prevents frequent allocations for large S3 keyspaces
and OOM-kills that might happen as a result of those.

This commit introduces storagedriver.Errors type that allows to return
multierrors as a single error from any storage driver implementation.

Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
2022-10-30 19:10:24 +00:00