This commit cleans up and attempts to optimise the performance of image push in S3 driver.
There are 2 main changes:
* we refactor the S3 driver Writer where instead of using separate bytes
slices for ready and pending parts which get constantly appended data
into them causing unnecessary allocations we use optimised bytes
buffers; we make sure these are used efficiently when written to.
* we introduce a memory pool that is used for allocating the byte
buffers introduced above
These changes should alleviate high memory pressure on the push path to S3.
Co-authored-by: Cory Snider <corhere@gmail.com>
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
The SuccessStatus acted on the response's status code, and was used to return
early, before checking the same status code with HandleErrorResponse.
This patch combines both functions into a HandleHTTPResponseError, which
returns an error for "non-success" status-codes, which simplifies handling
of responses, and makes some logic slightly more idiomatic.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
It was used for signing schema v1 manifests in tests which have now been
removed so there is no point in keeping these there anymore.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
Use the non-exported function to all errors; there's currently no external
consumers of this function (perhaps it should be deprecated).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In case drvr.PutContent fails and returns error we'd have
some extra memory allocated, though in this case
(test with known size of the slice being iterated), that's fine.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
Only some of the S3 storage driver calls were propagating context to the
S3 API calls. This commit updates the S3 storage drivers so the context
is propagated to all the S3 API calls.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
This integrates the new module, which was extracted from this repository
at commit b9b19409cf458dcb9e1253ff44ba75bd0620faa6;
# install filter-repo (https://github.com/newren/git-filter-repo/blob/main/INSTALL.md)
brew install git-filter-repo
# create a temporary clone of docker
cd ~/Projects
git clone https://github.com/distribution/distribution.git reference
cd reference
# commit taken from
git rev-parse --verify HEAD
b9b19409cf
# remove all code, except for general files, 'reference/', and rename to /
git filter-repo \
--path .github/workflows/codeql-analysis.yml \
--path .github/workflows/fossa.yml \
--path .golangci.yml \
--path distribution-logo.svg \
--path CODE-OF-CONDUCT.md \
--path CONTRIBUTING.md \
--path GOVERNANCE.md \
--path README.md \
--path LICENSE \
--path MAINTAINERS \
--path-glob 'reference/*.*' \
--path-rename reference/:
# initialize go.mod
go mod init github.com/distribution/reference
go mod tidy -go=1.20
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Storage drivers may be able to take advantage of the hint to start
their walk more efficiently.
For S3: The API takes a start-after parameter. Registries with many
repositories can drastically reduce calls to s3 by telling s3 to only
list results lexographically after the last parameter.
For the fallback: We can start deeper in the tree and avoid statting
the files and directories before the hint in a walk. For a filesystem
this improves performance a little, but many of the API based drivers
are currently treated like a filesystem, so this drastically improves
the performance of GCP and Azure blob.
Signed-off-by: James Hewitt <james.hewitt@uk.ibm.com>
Client attempts to parse the body of every error it receives as JSON
regardless of the content-type. This commit rectifies by only parsing
he error body as JSON if the Content-Type header is set to
either "application/json" or "application/vnd.api+json".
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
We are replacing the very outdated redigo Go module with the official
redis Go module, go-redis.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
We've replaced all the schema1 references with OCI schema manifest.
Note, there are some TODO items that must be addressed at some point in
the future once the schema1 package is removed completely from the
codebase.
Signed-off-by: Milos Gajdos <milosthegajdos@gmail.com>
