This deals with a memory leak, caused by goroutines, experienced when using the
s3 driver. Unfortunately, this section of the code leaks goroutines like a
sieve. There is probably some refactoring that could be done to avoid this but
instead, we have a done channel that will cause waiting goroutines to exit.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
- Change driver interface to take a context as its first argument
- Make newFileReader take a context as its first argument
- Make newFileWriter take a context as its first argument
- Make blobstore exists and delete take a context as a first argument
- Pass the layerreader's context to the storage layer
- Pass the app's context to purgeuploads
- Store the app's context into the blobstore (was previously null)
- Pass the trace'd context to the storage drivers
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
The code using values from the yaml package wasn't careful enought with the
possible incoming types. Turns out, it is just an int but we've made this
section somewhat bulletproof in case that package changes the behavior.
This code likely never worked. The configuration system should be decoupled
from the object instantiation.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Rather than accept the resulting of a layer validation, we retry up to three
times, backing off 100ms after each try. The thought is that we allow s3 files
to make their way into the correct location increasing the liklihood the
verification can proceed, if possible.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
When the registry starts a background timer will periodically
scan the upload directories on the file system every 24 hours
and delete any files older than 1 week. An initial jitter
intends to avoid contention on the filesystem where multiple
registries with the same storage driver are started
simultaneously.
Registry is intended to be used as a repository service than an abstract collection of repositories. Namespace better describes a collection of repositories retrievable by name.
The registry service serves any repository in the global scope.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
The original implementation wrote to different locations in a shared slice.
While this is theoretically okay, we end up thrashing the cpu cache since
multiple slice members may be on the same cache line. So, even though each
thread has its own memory location, there may be contention over the cache
line. This changes the code to aggregate to a slice in a single goroutine.
In reality, this change likely won't have any performance impact. The theory
proposed above hasn't really even been tested. Either way, we can consider it
and possibly go forward.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Rather than enforce lowercase paths for all drivers, support for
case-sensitivity has been deferred to the driver. There are a few caveats to
this approach:
1. There are possible security implications for tags that only differ in their
case. For instance, a tag "A" may be equivalent to tag "a" on certain file
system backends.
2. All system paths should not use case-sensitive identifiers where possible.
This might be problematic in a blob store that uses case-sensitive ids. For
now, since digest hex ids are all case-insensitive, this will not be an issue.
The recommend workaround is to not run the registry on a case-insensitive
filesystem driver in security sensitive applications.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
To avoid compounded round trips leading to slow retrieval of manifests with a
large number of signatures, the fetch of signatures has been parallelized. This
simply spawns a goroutine for each path, coordinated with a sync.WaitGroup.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
For consistency with other systems, the redis and caching monitoring data has
been moved under the "registry" section in expvar. This ensures the entire
registry state is kept to a single section.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset integrates the layer info cache with the registry webapp and
storage backend. The main benefit is to cache immutable layer meta data,
reducing backend roundtrips. The cache can be configured to use either redis or
an inmemory cache.
This provides massive performance benefits for HEAD http checks on layer blobs
and manifest verification.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This changeset defines the interface for layer info caches. Layer info caches
speed up access to layer meta data accessed in storage driver backends. The
two main operations are tests for repository membership and resolving path and
size information for backend blobs.
Two implementations are available. The main implementation leverages redis to
store layer info. An alternative implementation simply caches layer info in
maps, which should speed up resolution for less sophisticated implementations.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
By using a resumable digester and storing the state of upload digests between
subsequent upload chunks, finalizing an upload no longer requires reading back
all of the uploaded data to verify the client's expected digest.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
This chnage prevents a crash when moving from a non-existent directory that has
a file as a parent. To prevent this, we simply check that the node is a
directory and throws an error if it is not.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This enables Azure storage driver to be used with non-default
cloud endpoints like Azure China or Azure Government that does
not use `.blob.core.windows.net` FQDN suffix.
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
Convert middleware in the config to be a map of type->[]Middleware
Add support for registry & repository middleware.
Some naming updates as well.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
