Merge pull request #227 from stevvooe/readme-documentation-outline
README, doc: update README and outline documentation
This commit is contained in:
commit
5b72d32265
9 changed files with 303 additions and 213 deletions
290
README.md
290
README.md
|
@ -1,69 +1,249 @@
|
|||
> **Notice:** *This repository hosts experimental components that are currently under heavy and fast-paced development, not-ready for public consumption. If you are looking for the stable registry, please head over to [docker/docker-registry](https://github.com/docker/docker-registry) instead.*
|
||||
> **Notice:** *This repository hosts experimental components that are
|
||||
> currently under heavy and fast-paced development, not-ready for public
|
||||
> consumption. If you are looking for the stable registry, please head over to
|
||||
> [docker/docker-registry](https://github.com/docker/docker-registry)
|
||||
> instead.*
|
||||
|
||||
Distribution
|
||||
============
|
||||
|
||||
The Docker toolset to pack, ship, store, and deliver content.
|
||||
|
||||
Planned content for this repository:
|
||||
The main product of this repository is the new registry implementation for
|
||||
storing and distributing docker images. It supersedes the [docker/docker-
|
||||
registry](https://github.com/docker/docker-registry) project with a new API
|
||||
design, focused around security and performance.
|
||||
|
||||
* Distribution related specifications
|
||||
- Image format
|
||||
- JSON registry API
|
||||
* Registry implementation: a Golang implementation of the JSON API
|
||||
* Client libraries to consume conforming implementations of the JSON API
|
||||
The _Distribution_ project has the further long term goal of providing a
|
||||
secure tool chain for distributing content. The specifications, APIs and tools
|
||||
should be as useful with docker as they are without.
|
||||
|
||||
# Ongoing open sprint
|
||||
This repository contains the following components:
|
||||
|
||||
### What is an open sprint?
|
||||
|
||||
The open sprint is a focused effort of a small group of people to kick-off a new project, while commiting to becoming maintainers of the resulting work.
|
||||
|
||||
**Having a dedicated team work on the subject doesn't mean that you, the community, cannot contribute!** We need your input to make the best use of the sprint, and focus our work on what matters for you. For this particular topic:
|
||||
|
||||
* Come discuss on IRC: #docker-distribution on FreeNode
|
||||
* Come read and participate in the [Google Groups](https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution)
|
||||
* Submit your ideas, and upvote those you think matter the most on [Google Moderator](https://www.google.com/moderator/?authuser=1#16/e=2165c3)
|
||||
|
||||
### Goal of the distribution sprint
|
||||
|
||||
Design a professional grade and extensible content distribution system, that allow users to:
|
||||
|
||||
* Enjoy an efficient, secured and reliable way to store, manage, package and exchange content
|
||||
* Hack/roll their own on top of healthy open-source components
|
||||
* Implement their own home made solution through good specs, and solid extensions mechanism.
|
||||
|
||||
### Schedule and expected output
|
||||
|
||||
The Open Sprint will start on **Monday December 29th**, and end on **Friday January 16th**.
|
||||
|
||||
What we want to achieve as a result is:
|
||||
|
||||
* Tactical fixes of today's frustrations in the existing Docker codebase
|
||||
- This includes a throrough review of [docker/docker#9784](https://github.com/docker/docker/pull/9784) by core maintainers
|
||||
|
||||
* Laying the base of a new distribution subsystem, living independently, and with a well defined group of maintainers. This is the purpose of this repository, which aims at hosting:
|
||||
- A specification of the v2 image format
|
||||
- A specification of the JSON/HTTP protocol
|
||||
- Server-side Go implementation of the v2 registry
|
||||
- Client-side Go packages to consume this new API
|
||||
- Standalone binaries providing content distribution functionalities outside of Docker
|
||||
|
||||
* Constraints for interface provided by Distribution to Core:
|
||||
- The distribution repository is a collection of tools for packaging and
|
||||
shipping content with Docker
|
||||
- All tools are usable primarily as standalone command-line tools. They may
|
||||
also expose bindings in one or more programming languages. Typically the
|
||||
first available language is Go, but that is not automatically true and more
|
||||
languages will be supported over time
|
||||
- The distribution repository is still under incubation, any code layout,
|
||||
interface and name may change before it gets included in a stable release of
|
||||
Docker
|
||||
- **registry (beta):** An implementation of the [Docker Registry HTTP API
|
||||
V2](doc/spec/api.md) for use with docker 1.5+.
|
||||
- **libraries (unstable):** A rich set of libraries for interacting with
|
||||
distribution components. Please see
|
||||
[godoc](http://godoc.org/github.com/docker/distribution) for details. Note
|
||||
that the libraries *are not* considered stable.
|
||||
- **dist (experimental):** An experimental tool to provide distribution
|
||||
oriented functionality without the docker daemon.
|
||||
- **specifications**: _Distribution_ related specifications are available in
|
||||
[doc/spec](doc/spec).
|
||||
- **documentation:** Documentation is available in [doc](doc/overview.md).
|
||||
|
||||
### How will this integrate with Docker engine?
|
||||
|
||||
Building awesome, independent, and well maintained distribution tools should give Docker core maintainers enough incentive to switch to the newly develop subsystem. We make no assumptions on a given date or milestone as urgency should be fixed through [docker/docker#9784](https://github.com/docker/docker/pull/9784), and in order to maintain focus on producing a top quality alternative.
|
||||
This project should provide an implementation to a V2 API for use in the
|
||||
Docker core project. The API should be embeddable and simplify the process of
|
||||
securely pulling and pushing content from docker daemons.
|
||||
|
||||
### Relevant documents
|
||||
### What are the long term goals of the Distribution project?
|
||||
|
||||
* [Analysis of current state and goals](doc/opensprint/kickoff.md)
|
||||
Design a professional grade and extensible content distribution system, that
|
||||
allow users to:
|
||||
|
||||
* Enjoy an efficient, secured and reliable way to store, manage, package and
|
||||
exchange content
|
||||
* Hack/roll their own on top of healthy open-source components
|
||||
* Implement their own home made solution through good specs, and solid
|
||||
extensions mechanism.
|
||||
|
||||
Features
|
||||
--------
|
||||
|
||||
The new registry implementation provides the following benefits:
|
||||
|
||||
- faster push and pull
|
||||
- new, more efficient implementation
|
||||
- simplified deployment
|
||||
- pluggable storage backend
|
||||
- webhook notifications
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
**TODO(stevvooe):** Add the following here:
|
||||
- docker file
|
||||
- binary builds for non-docker environment (test installations, etc.)
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
The registry server can be configured with a yaml file. The following is a
|
||||
simple example that can used for local development:
|
||||
|
||||
```yaml
|
||||
version: 0.1
|
||||
loglevel: debug
|
||||
storage:
|
||||
filesystem:
|
||||
rootdirectory: /tmp/registry-dev
|
||||
http:
|
||||
addr: localhost:5000
|
||||
secret: asecretforlocaldevelopment
|
||||
debug:
|
||||
addr: localhost:5001
|
||||
```
|
||||
|
||||
The above configures the registry instance to run on port 5000, binding to
|
||||
"localhost", with the debug server enabled. Registry data will be stored in
|
||||
"/tmp/registry-dev". Logging will be in "debug" mode, which is the most
|
||||
verbose.
|
||||
|
||||
A similar simple configuration is available at [cmd/registry/config.yml],
|
||||
which is generally useful for local development.
|
||||
|
||||
**TODO(stevvooe): Need a "best practice" configuration overview. Perhaps, we
|
||||
can point to a documentation section.
|
||||
|
||||
For full details about configuring a registry server, please see [the
|
||||
documentation](doc/configuration.md).
|
||||
|
||||
### Upgrading
|
||||
|
||||
**TODO:** Add a section about upgrading from V1 registry along with link to
|
||||
migrating in documentation.
|
||||
|
||||
Build
|
||||
-----
|
||||
|
||||
If a go development environment is setup, one can use `go get` to install the
|
||||
`registry` command from the current latest:
|
||||
|
||||
```sh
|
||||
go get github.com/docker/distribution/cmd/registry
|
||||
```
|
||||
|
||||
The above will install the source repository into the `GOPATH`. The `registry`
|
||||
binary can then be run with the following:
|
||||
|
||||
```
|
||||
$ $GOPATH/bin/registry -version
|
||||
$GOPATH/bin/registry github.com/docker/distribution v2.0.0-alpha.1+unknown
|
||||
```
|
||||
|
||||
The registry can be run with the default config using the following
|
||||
incantantation:
|
||||
|
||||
```
|
||||
$ $GOPATH/bin/registry $GOPATH/src/github.com/docker/distribution/cmd/registry/config.yml
|
||||
INFO[0000] endpoint local-8082 disabled, skipping app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
|
||||
INFO[0000] endpoint local-8083 disabled, skipping app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
|
||||
INFO[0000] listening on :5000 app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
|
||||
INFO[0000] debug server listening localhost:5001
|
||||
```
|
||||
|
||||
If it is working, one should see the above log messages.
|
||||
|
||||
### Repeatable Builds
|
||||
|
||||
For the full development experience, one should `cd` into
|
||||
`$GOPATH/src/github.com/docker/distribution`. From there, the regular `go`
|
||||
commands, such as `go test`, should work per package (please see
|
||||
[Developing](#developing) if they don't work).
|
||||
|
||||
A `Makefile` has been provided as a convenience to support repeatable builds.
|
||||
Please install the following into `GOPATH` for it to work:
|
||||
|
||||
```
|
||||
go get github.com/tools/godep github.com/golang/lint/golint
|
||||
```
|
||||
|
||||
**TODO(stevvooe):** Add a `make setup` command to Makefile to run this. Have
|
||||
to think about how to interact with Godeps properly.
|
||||
|
||||
Once these commands are available in the `GOPATH`, run `make` to get a full
|
||||
build:
|
||||
|
||||
```
|
||||
$ GOPATH=`godep path`:$GOPATH make
|
||||
+ clean
|
||||
+ fmt
|
||||
+ vet
|
||||
+ lint
|
||||
+ build
|
||||
github.com/docker/docker/vendor/src/code.google.com/p/go/src/pkg/archive/tar
|
||||
github.com/Sirupsen/logrus
|
||||
github.com/docker/libtrust
|
||||
...
|
||||
github.com/yvasiyarov/gorelic
|
||||
github.com/docker/distribution/registry/handlers
|
||||
github.com/docker/distribution/cmd/registry
|
||||
+ test
|
||||
...
|
||||
ok github.com/docker/distribution/digest 7.875s
|
||||
ok github.com/docker/distribution/manifest 0.028s
|
||||
ok github.com/docker/distribution/notifications 17.322s
|
||||
? github.com/docker/distribution/registry [no test files]
|
||||
ok github.com/docker/distribution/registry/api/v2 0.101s
|
||||
? github.com/docker/distribution/registry/auth [no test files]
|
||||
ok github.com/docker/distribution/registry/auth/silly 0.011s
|
||||
...
|
||||
+ /Users/sday/go/src/github.com/docker/distribution/bin/registry
|
||||
+ /Users/sday/go/src/github.com/docker/distribution/bin/registry-api-descriptor-template
|
||||
+ /Users/sday/go/src/github.com/docker/distribution/bin/dist
|
||||
+ binaries
|
||||
```
|
||||
|
||||
The above provides a repeatable build using the contents of the vendored
|
||||
Godeps directory. This includes formatting, vetting, linting, building,
|
||||
testing and generating tagged binaries. We can verify this worked by running
|
||||
the registry binary generated in the "./bin" directory:
|
||||
|
||||
```sh
|
||||
$ ./bin/registry -version
|
||||
./bin/registry github.com/docker/distribution v2.0.0-alpha.2-80-g16d8b2c.m
|
||||
```
|
||||
|
||||
### Developing
|
||||
|
||||
The above approaches are helpful for small experimentation. If more complex
|
||||
tasks are at hand, it is recommended to employ the full power of `godep`.
|
||||
|
||||
The Makefile is designed to have its `GOPATH` defined externally. This allows
|
||||
one to experiment with various development environment setups. This is
|
||||
primarily useful when testing upstream bugfixes, by modifying local code. This
|
||||
can be demonstrated using `godep` to migrate the `GOPATH` to use the specified
|
||||
dependencies. The `GOPATH` can be migrated to the current package versions
|
||||
declared in `Godeps` with the following command:
|
||||
|
||||
```sh
|
||||
godep restore
|
||||
```
|
||||
|
||||
> **WARNING:** This command will checkout versions of the code specified in
|
||||
> Godeps/Godeps.json, modifying the contents of `GOPATH`. If this is
|
||||
> undesired, it is recommended to create a workspace devoted to work on the
|
||||
> _Distribution_ project.
|
||||
|
||||
With a successful run of the above command, one can now use `make` without
|
||||
specifying the `GOPATH`:
|
||||
|
||||
```sh
|
||||
$ make
|
||||
```
|
||||
|
||||
If that is successful, standard `go` commands, such as `go test` should work,
|
||||
per package, without issue.
|
||||
|
||||
Support
|
||||
-------
|
||||
|
||||
If any issues are encountered while using the _Distribution_ project, several
|
||||
avenues are available for support:
|
||||
|
||||
IRC: #docker-distribution on FreeNode
|
||||
Issue Tracker: github.com/docker/distribution/issues
|
||||
Google Groups: https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution
|
||||
Mailing List: docker@dockerproject.org
|
||||
|
||||
Contribute
|
||||
----------
|
||||
|
||||
Please see [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
This project is distributed under [Apache License, Version 2.0](LICENSE.md).
|
||||
|
|
4
doc/architecture.md
Normal file
4
doc/architecture.md
Normal file
|
@ -0,0 +1,4 @@
|
|||
# Architecture
|
||||
|
||||
**TODO(stevvooe):** Discuss the architecture of the registry, internally and
|
||||
externally, in a few different deployment scenarios.
|
4
doc/configuration.md
Normal file
4
doc/configuration.md
Normal file
|
@ -0,0 +1,4 @@
|
|||
# Configuration
|
||||
|
||||
**TODO(stevvooe): This should include an exhaustive account configuration
|
||||
parameters, including those for various subsystems.
|
6
doc/deploying.md
Normal file
6
doc/deploying.md
Normal file
|
@ -0,0 +1,6 @@
|
|||
# Deploying
|
||||
|
||||
**TODO(stevvooe):** This should discuss various deployment scenarios for
|
||||
production-ready deployments. These may be backed by ready-made docker images
|
||||
but this should explain how they were created and what considerations were
|
||||
present.
|
39
doc/glossary.md
Normal file
39
doc/glossary.md
Normal file
|
@ -0,0 +1,39 @@
|
|||
# Glossary
|
||||
|
||||
**TODO(stevvooe):** Define and describe distribution related terms. Ideally,
|
||||
we reference back to the actual documentation and specifications where
|
||||
appropriate.
|
||||
|
||||
**TODO(stevvooe):** The following list is a start but woefully incomplete.
|
||||
|
||||
<dl>
|
||||
<dt>Blob</dt>
|
||||
<dd>
|
||||
The primary unit of registry storage. A string of bytes identified by
|
||||
content-address, known as a _digest_.
|
||||
</dd>
|
||||
|
||||
<dt>Image</dt>
|
||||
<dd>An image is a collection of content from which a docker container can be created.</dd>
|
||||
|
||||
<dt>Layer</dt>
|
||||
<dd>
|
||||
A tar file representing the partial content of a filesystem. Several
|
||||
layers can be "stacked" to make up the root filesystem.
|
||||
</dd>
|
||||
|
||||
<dt>Manifest</dt>
|
||||
<dd>Describes a collection layers that make up an image.</dd>
|
||||
|
||||
<dt>Registry</dt>
|
||||
<dd>A registry is a collection of repositories.</dd>
|
||||
|
||||
<dt>Repository</dt>
|
||||
<dd>
|
||||
A repository is a collection of docker images, made up of manifests, tags
|
||||
and layers. The base unit of these components are blobs.
|
||||
</dd>
|
||||
|
||||
<dt>Tag</dt>
|
||||
<dd>Tag provides a common name to an image.</dd>
|
||||
</dl>
|
4
doc/notifications.md
Normal file
4
doc/notifications.md
Normal file
|
@ -0,0 +1,4 @@
|
|||
# Notifications
|
||||
|
||||
**TODO(stevvooe)** Cover use and deployment of webhook notifications. Link to
|
||||
description in architecture documentation.
|
|
@ -1,158 +0,0 @@
|
|||
Distribution
|
||||
=========================
|
||||
|
||||
## Project intentions
|
||||
|
||||
**Problem statement and requirements**
|
||||
|
||||
* What is the exact scope of the problem?
|
||||
|
||||
|
||||
Design a professional grade and extensible content distribution system, that allows docker users to:
|
||||
|
||||
... by default enjoy:
|
||||
|
||||
* an efficient, secured and reliable way to store, manage, package and exchange content
|
||||
|
||||
... optionally:
|
||||
|
||||
* can hack/roll their own on top of healthy open-source components
|
||||
|
||||
... with the liberty to:
|
||||
|
||||
* implement their own home made solution through good specs, and solid extensions mechanism
|
||||
|
||||
|
||||
* Who will the result be useful to?
|
||||
|
||||
* users
|
||||
* ISV (who distribute images or develop image distribution solutions)
|
||||
* docker
|
||||
|
||||
* What are the use cases (distinguish dev & ops population where applicable)?
|
||||
|
||||
* Everyone (... uses docker push/pull).
|
||||
|
||||
* Why does it matter that we build this now?
|
||||
|
||||
* Shortcomings of the existing codebase are the #1 pain point (by large) for users, partners and ISV, hence the most urgent thing to address (?)
|
||||
* That situation is getting worse everyday and killer competitors are going/have emerged.
|
||||
|
||||
* Who are the competitors?
|
||||
|
||||
* existing artifact storage solutions (eg: artifactory).
|
||||
* emerging products that aim at handling pull/push in place of docker.
|
||||
* ISV that are looking for alternatives to workaround this situation
|
||||
|
||||
**Current state: what do we have today?**
|
||||
|
||||
Problems of the existing system:
|
||||
|
||||
1. not reliable
|
||||
* registry goes down whenever the hub goes down
|
||||
* failing push result in broken repositories
|
||||
* concurrent push is not handled
|
||||
* python boto and gevent have a terrible history
|
||||
* organically grown, under-designed features are in a bad shape (search)
|
||||
2. inconsistent
|
||||
* discrepancies between duplicated API (and *duplicated APIs*)
|
||||
* unused features
|
||||
* missing essential features (proper SSL support)
|
||||
3. not reusable
|
||||
* tightly entangled with hub component makes it very difficult to use outside of docker
|
||||
* proper access-control is almost impossible to do right
|
||||
* not easily extensible
|
||||
4. not efficient
|
||||
* no parallel operations (by design)
|
||||
* sluggish client-side processing / bad pipeline design
|
||||
* poor reusability of content (random ids)
|
||||
* scalability issues (tags)
|
||||
* too many useless requests (protocol)
|
||||
* too much local space consumed (local garbage collection: broken + not efficient)
|
||||
* no squashing
|
||||
5. not resilient to errors
|
||||
* no resume
|
||||
* error handling is obscure or inexistent
|
||||
6. security
|
||||
* content is not verified
|
||||
* current tarsum is broken
|
||||
* random ids are a headache
|
||||
7. confusing
|
||||
* registry vs. registry.hub?
|
||||
* layer vs. image?
|
||||
8. broken features
|
||||
* mirroring is not done correctly (too complex, bug-laden, caching is hard)
|
||||
9. poor integration with the rest of the project
|
||||
* technology discrepancy (python vs. go)
|
||||
* poor testability
|
||||
* poor separation (API in the engine is not defined enough)
|
||||
10. missing features / prevents future
|
||||
* trust / image signing
|
||||
* naming / transport separation
|
||||
* discovery / layer federation
|
||||
* architecture + os support (eg: arm/windows)
|
||||
* quotas
|
||||
* alternative distribution methods (transport plugins)
|
||||
|
||||
**Future state: where do we want to get?**
|
||||
|
||||
* Deliverable
|
||||
* new JSON/HTTP protocol specification
|
||||
* new image format specification
|
||||
* (new image store in the engine)
|
||||
* new transport API between the engine and the distribution client code / new library
|
||||
* new registry in go
|
||||
* new authentication service on top of the trust graph in go
|
||||
|
||||
* What are the interactions with other components of the project?
|
||||
* critical interactions with docker push/pull mechanism
|
||||
* critical interactions with the way docker stores images locally
|
||||
|
||||
* In what way will the result be customizable?
|
||||
* transport plugins allowing for radically different transport methods (bittorent, direct S3 access, etc)
|
||||
* extensibility design for the registry allowing for complex integrations with other systems
|
||||
* backend storage drivers API
|
||||
|
||||
|
||||
## Kick-off output
|
||||
|
||||
**What is the expected output of the kick-off session?**
|
||||
|
||||
* draft specifications
|
||||
* separate binary tool for demo purpose
|
||||
* a mergeable PR that fixes 90% of the listed issues
|
||||
|
||||
|
||||
* agree on a vision that allows solving all that are deemed worthy
|
||||
* propose a long term battle plan with clear milestones that encompass all these
|
||||
* define a first milestone that is compatible with the future and does already deliver some of the solutions
|
||||
* deliver the specifications for image manifest format and transport API
|
||||
* deliver a working implementation that can be used as a drop-in replacement for the existing v1 with an equivalent feature-set
|
||||
|
||||
**How is the output going to be demoed?**
|
||||
|
||||
docker pull
|
||||
docker push
|
||||
|
||||
**Once demoed, what will be the path to shipping?**
|
||||
|
||||
A minimal PR that include the first subset of features to make docker work well with the new server side components.
|
||||
|
||||
## Pressing matters
|
||||
|
||||
* need a codename (ship, distribute)
|
||||
* new repository
|
||||
* new domains
|
||||
|
||||
* architecture / OS
|
||||
* persistent ids
|
||||
* registries discovery
|
||||
* naming (quay.io/foo/bar)
|
||||
* mirroring
|
||||
|
||||
|
||||
|
||||
## Assorted issues
|
||||
|
||||
* some devops want a docker engine that cannot do push/pull
|
||||
|
6
doc/overview.md
Normal file
6
doc/overview.md
Normal file
|
@ -0,0 +1,6 @@
|
|||
# Overview
|
||||
|
||||
**TODO(stevvooe):** Table of contents.
|
||||
|
||||
**TODO(stevvooe):** Include a full overview of each component and dispatch the
|
||||
user to the correct documentation.
|
5
doc/storagedrivers.md
Normal file
5
doc/storagedrivers.md
Normal file
|
@ -0,0 +1,5 @@
|
|||
# Storage Drivers
|
||||
|
||||
**TODO(stevvooe):** This should include detailed overviews of what a storage
|
||||
driver is and information about each storage driver implementation.
|
||||
Considerations when implementing a storage driver should also be present.
|
Loading…
Reference in a new issue