Storage drivers can implement a method called URLFor which can return a direct
url for a given path. The functionality allows the registry to direct clients
to download content directly from the backend storage. This is commonly used
with s3 and cloudfront. Under certain conditions, such as when the registry is
not local to the backend, these redirects can hurt performance and waste
incoming bandwidth on pulls. This feature addition allows one to disable this
feature, if required.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Conflicts:
configuration/configuration.go
registry/handlers/app.go
registry/storage/catalog_test.go
registry/storage/manifeststore_test.go
registry/storage/registry.go
Implement the delete API by implementing soft delete for layers
and blobs by removing link files and updating the blob descriptor
cache. Deletion is configurable - if it is disabled API calls
will return an unsupported error.
We invalidate the blob descriptor cache by changing the linkedBlobStore's
blobStatter to a blobDescriptorService and naming it blobAccessController.
Delete() is added throughout the relevant API to support this functionality.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Signed-off-by: David Verhasselt <david@crowdway.com>
Add clarifying requirement of nginx 1.7.5 to authentication.md file.
Signed-off-by: David Verhasselt <david@crowdway.com>
Revert "Update authentication.md"
This reverts commit a79c28f4376fad333edd94e9b25211830e63abbd.
Add paragraph on manual set-up
To avoid confusion with the registry version number or other project versions,
the specification updates are now lettered.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Some missing descriptions and error code for tags pagination was cleaned up to
ensure clarity. Specifically, we ensure the request variations are named and
the proper error codes are included.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Move the specification to use a Link header, rather than a "next" entry in the
json results. This prevents requiring clients from parsing the request body to
issue the next request. It also ensures that the returned response body does
not change in between requests.
The ordering of the specification has been slightly tweaked, as well. Listing
image tags has been moved after the catalog specification. Tag pagination now
heavily references catalog pagination.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
This contains a proposal for a catalog API, provided access to the internal
contents of a registry instance. The API endpoint is prefixed with an
underscore, which is illegal in images names, to prevent collisions with
repositories names. To avoid issues with large result sets, a paginated version
of the API is proposed. We make an addition to the tags API to support
pagination to ensure the specification is conistent.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Since the actual port is 5003, it would make sense to name it local-5003 instead of local-8082
Signed-off-by: Florentin Raud <florentin.raud@gmail.com>
After consideration, we've changed the main descriptor field name to for number
of bytes to "size" to match convention. While this may be a subjective
argument, commonly we refer to files by their "size" rather than their
"length". This will match other conventions, like `(FileInfo).Size()` and
methods on `io.SizeReaderAt`. Under more broad analysis, this argument doesn't
necessarily hold up. If anything, "size" is shorter than "length".
Signed-off-by: Stephen J Day <stephen.day@docker.com>
yaml sections in the documentation does not display well on
docs.docker.com. This is due to the syntax highlighting
which uses highlight.js and does not support yaml
currently.
The fix is to remove triple back ticks and indent instead.
We loose yaml syntax highlighting on github, but it displays
an acceptable version on both github and docs.docker.com.
Signed-off-by: Olivier Jacques <olivier.jacques@hp.com>
The main goal of this changeset is to allow repository name components to
consist of a single character. The number of components allowed and the slash
separation requirements have also been clarified.
To go along with this simplification, errant constants and unneeded error types
have been removed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Intermediate certificates are issued by TLS providers who themselves are
an intermediate of a certificate in the trust store. Therefore, to prove
the chain of trust is valid, you need to include their certificate as
well as yours when you send your certificate to the client.
Contrary to what I said in issue #683, distribution can handle these
certificate bundles like nginx. As discussed in #docker-distribution,
I have updated the deployment documentation (which recommends the use of
a TLS certificate from a provider) to include instructions on how to
handle the intermediate certificate when a user is configuring
distribution.
Signed-off-by: Luke Carpenter <x@rubynerd.net>
This removes documentation and code related to IPC based storage driver
plugins. The existence of this functionality was an original feature goal but
is now not maintained and actively confusing incoming contributions. We will
likely explore some driver plugin mechanism in the future but we don't need
this laying around in the meantime.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Fixing heading not rendering
Fixing bad links:
deployed.md > deploying.md
spec/auth/token.md > /spec/auth/token.md
Signed-off-by: Mary Anthony <mary@docker.com>
Renaming to index.md;rereading of Hugo showed me my mistake; removing commented out/Markdown has no comment feature
Updating with Olivier. Yay! It looks great
Signed-off-by: Mary Anthony <mary@docker.com>
This change refreshes the updated version of Azure SDK
for Go that has the latest changes.
I manually vendored the new SDK (github.com/Azure/azure-sdk-for-go)
and I removed `management/` `core/` packages manually simply because
they're not used here and they have a fork of `net/http` and `crypto/tls`
for a particular reason. It was introducing a 44k SLOC change otherwise...
This also undoes the `include_azure` flag (actually Steven removed the
driver from imports but forgot to add the build flag apparently, so the
flag wasn't really including azure. 😄 ). This also must be obsolete
now.
Fixes#620, #175.
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
Updating for new docs build and tooling infrastructure 1.7 release; Thank you Olivier. I promise to look at all of this after Dockercon. We'll make it perfect.
Updating for tooling tests
Updating with the new sed scripts to protect links
updating with new image
Updating with comments
Signed-off-by: Mary Anthony <mary@docker.com>
- commenting out both the "JSON" and "token" specs, unless someone thinks they should be here
- added help, glossary, introduction and authentication documents
Signed-off-by: Olivier Gambier <olivier@docker.com>
- adding glossary
- removing empty "advanced"
- commenting out building and architecture for now
- minor text enhancements
Signed-off-by: Olivier Gambier <olivier@docker.com>
Simplified index again, to make access to information more obvious and more direct.
Added a TLDR for people in a hurry.
Signed-off-by: Olivier Gambier <olivier@docker.com>
Use cases, generalities, image naming overview.
Removed most of it from index, and some of it from other random pages.
Signed-off-by: Olivier Gambier <olivier@docker.com>
This clarifies the importance of properly setting a Go build environment when
building targets. Typically, users seem to editorialize the checkout location,
either ignoring the first section or have limited experience with the Go
development environment. We clarify the checkout requirements and point to the
documentation on how to setup Go.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
In the request parameters lists `tag` was used instead of
`reference` present in the HTTP requests paths
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
Catch up after 88795e0a (Lots of various golint fixes, 2014-11-17, #1)
and 65b0d73c (Move storagedriver package to registry/storage/driver,
2015-02-10, #167).
Signed-off-by: W. Trevor King <wking@tremily.us>
This PR is for issue of "email after registry webapp panic" #41, improving my
previous design (closed).
It use self setting up hooks, to catch panic in web application.
And, send email in hooks handle directly, to no use new http server and
handler.
Signed-off-by: xiekeyang <keyangxie@126.com>
This driver implements the storagedriver.StorageDriver interface and
uses Ceph Object Storage as storage backend.
Since RADOS is an object storage and no hierarchy notion, the
following convention is used to keep the filesystem notions stored in
this backend:
* All the objects data are stored with opaque UUID names prefixed
(e.g. "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164).
* All the hierarchy information are stored in rados omaps, where the
omap object identifier is the virtual directory name, the keys in
a specific are the relative filenames and the values the blob
object identifier (or empty value for a sub directory).
e.g. For the following hierarchy:
/directory1
/directory1/object1
/directory1/object2
/directory1/directory2/object3
The omap "/directory1" will contains the following key / values:
- "object1" "blob:d3d232ff-ab3a-4046-9ab7-930228d4c164"
- "object2" "blob:db2e359d-4af0-4bfb-ba1d-d2fd029866a0"
- "directory2" ""
The omap "/directory1/directory2" will contains:
- "object3" "blob:9ae2371c-81fc-4945-80ac-8bf7f566a5d9"
* The MOVE is implemented by changing the reference to a specific
blob in its parent virtual directory omap.
This driver stripes rados objects to a fixed size (e.g. 4M). The idea
is to keep small objects (as done by RBD on the top of RADOS) that
will be easily synchronized accross OSDs. The information of the
original object (i.e total size of the chunks) is stored as a Xattr
in the first chunk object.
Signed-off-by: Vincent Giersch <vincent.giersch@ovh.net>
This PR refactors the blob service API to be oriented around blob descriptors.
Identified by digests, blobs become an abstract entity that can be read and
written using a descriptor as a handle. This allows blobs to take many forms,
such as a ReadSeekCloser or a simple byte buffer, allowing blob oriented
operations to better integrate with blob agnostic APIs (such as the `io`
package). The error definitions are now better organized to reflect conditions
that can only be seen when interacting with the blob API.
The main benefit of this is to separate the much smaller metadata from large
file storage. Many benefits also follow from this. Reading and writing has
been separated into discrete services. Backend implementation is also
simplified, by reducing the amount of metadata that needs to be picked up to
simply serve a read. This also improves cacheability.
"Opening" a blob simply consists of an access check (Stat) and a path
calculation. Caching is greatly simplified and we've made the mapping of
provisional to canonical hashes a first-class concept. BlobDescriptorService
and BlobProvider can be combined in different ways to achieve varying effects.
Recommend Review Approach
-------------------------
This is a very large patch. While apologies are in order, we are getting a
considerable amount of refactoring. Most changes follow from the changes to
the root package (distribution), so start there. From there, the main changes
are in storage. Looking at (*repository).Blobs will help to understand the how
the linkedBlobStore is wired. One can explore the internals within and also
branch out into understanding the changes to the caching layer. Following the
descriptions below will also help to guide you.
To reduce the chances for regressions, it was critical that major changes to
unit tests were avoided. Where possible, they are left untouched and where
not, the spirit is hopefully captured. Pay particular attention to where
behavior may have changed.
Storage
-------
The primary changes to the `storage` package, other than the interface
updates, were to merge the layerstore and blobstore. Blob access is now
layered even further. The first layer, blobStore, exposes a global
`BlobStatter` and `BlobProvider`. Operations here provide a fast path for most
read operations that don't take access control into account. The
`linkedBlobStore` layers on top of the `blobStore`, providing repository-
scoped blob link management in the backend. The `linkedBlobStore` implements
the full `BlobStore` suite, providing access-controlled, repository-local blob
writers. The abstraction between the two is slightly broken in that
`linkedBlobStore` is the only channel under which one can write into the global
blob store. The `linkedBlobStore` also provides flexibility in that it can act
over different link sets depending on configuration. This allows us to use the
same code for signature links, manifest links and blob links. Eventually, we
will fully consolidate this storage.
The improved cache flow comes from the `linkedBlobStatter` component
of `linkedBlobStore`. Using a `cachedBlobStatter`, these combine together to
provide a simple cache hierarchy that should streamline access checks on read
and write operations, or at least provide a single path to optimize. The
metrics have been changed in a slightly incompatible way since the former
operations, Fetch and Exists, are no longer relevant.
The fileWriter and fileReader have been slightly modified to support the rest
of the changes. The most interesting is the removal of the `Stat` call from
`newFileReader`. This was the source of unnecessary round trips that were only
present to look up the size of the resulting reader. Now, one must simply pass
in the size, requiring the caller to decide whether or not the `Stat` call is
appropriate. In several cases, it turned out the caller already had the size
already. The `WriterAt` implementation has been removed from `fileWriter`,
since it is no longer required for `BlobWriter`, reducing the number of paths
which writes may take.
Cache
-----
Unfortunately, the `cache` package required a near full rewrite. It was pretty
mechanical in that the cache is oriented around the `BlobDescriptorService`
slightly modified to include the ability to set the values for individual
digests. While the implementation is oriented towards caching, it can act as a
primary store. Provisions are in place to have repository local metadata, in
addition to global metadata. Fallback is implemented as a part of the storage
package to maintain this flexibility.
One unfortunate side-effect is that caching is now repository-scoped, rather
than global. This should have little effect on performance but may increase
memory usage.
Handlers
--------
The `handlers` package has been updated to leverage the new API. For the most
part, the changes are superficial or mechanical based on the API changes. This
did expose a bug in the handling of provisional vs canonical digests that was
fixed in the unit tests.
Configuration
-------------
One user-facing change has been made to the configuration and is updated in
the associated documentation. The `layerinfo` cache parameter has been
deprecated by the `blobdescriptor` cache parameter. Both are equivalent and
configuration files should be backward compatible.
Notifications
-------------
Changes the `notification` package are simply to support the interface
changes.
Context
-------
A small change has been made to the tracing log-level. Traces have been moved
from "info" to "debug" level to reduce output when not needed.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Allow to use a unix socket as a listener.
To specify an endpoint type we use an optional configuration
field 'net', as there's no way to distinguish a relative
socket path from a hostname.
Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
- Ensures new uploads and resumed upload statuses always return an offset of 0. This allows future clients which support resumable upload to not attempt resumable upload on this version which does not support it.
- Add PATCH support for streaming data on upload.
- Add messaging to specification that PATCH with content range is currently not supported.
- Update PUT blob to only support full data or no data, no more last chunk messaging as it was not supported.
closes#470
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
- Add sed to Dockerfile; this sed exists on publish script; breaks headings/nav in files without metadata
- Ensure sed runs over storage-driver/ subdir
- Add metadata to all the files (including specs) that don't have it; this ensures they display correctly on publish
- Implement the fix for the showing up in Github
- Update template with GITHUB IGNORES
Signed-off-by: Mary Anthony <mary@docker.com>
The position was a bit too early. Fleshed out the example.
Also, using the _ underscore for emphasis was confusing because it is also used to indicate
a level change.
Signed-off-by: Mary Anthony <mary@docker.com>
Fixing typos and adding tables
Updating with testing material
Tweak after read through
Adding in Stephen's comments
Adding in Richard's comments. Fixing the broken images
closes issue #363
Another try
Signed-off-by: Mary Anthony <mary@docker.com>
- Explain why you would want to use a registry
- Explain difference between Docker Hub and Registry
- List some features of Registry
- Add table of contents for documentation - particularly "deploying a registry" call to action
- Use standard "Getting help" section from orchestration projects
Signed-off-by: Ben Firshman <ben@firshman.co.uk>
Registry is intended to be used as a repository service than an abstract collection of repositories. Namespace better describes a collection of repositories retrievable by name.
The registry service serves any repository in the global scope.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Adding new material
Adding in template chomped in error
Cover install/deploy in README
Adding in Stephen's comments
Fixing you tabs!
Updating with commentary from pr
Updating with last minute comments
Signed-off-by: Mary Anthony <mary@docker.com>
Adding docs build to the Makefile
Adding in Sven's changes to the Makefile
Removing DS_store file
Updating per Stephen's comments
Update with Stephen's final comment
Signed-off-by: Mary Anthony <mary@docker.com>
