TrueCloudLab/distribution
13
0
Fork 3
Code Issues 3 Pull requests Projects Releases 2 Packages Wiki Activity Actions

Merge pull request #227 from stevvooe/readme-documentation-outline

Browse source 
README, doc: update README and outline documentation
This commit is contained in:
Stephen Day 2015-03-04 15:07:20 -08:00
commit 5b72d32265
9 changed files with 303 additions and 213 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

290
README.md
View file

@ -1,69 +1,249 @@
> **Notice:** *This repository hosts experimental components that are currently under heavy and fast-paced development, not-ready for public consumption. If you are looking for the stable registry, please head over to [docker/docker-registry](https://github.com/docker/docker-registry) instead.*
> **Notice:** *This repository hosts experimental components that are
> currently under heavy and fast-paced development, not-ready for public
> consumption. If you are looking for the stable registry, please head over to
> [docker/docker-registry](https://github.com/docker/docker-registry)
> instead.*
Distribution
============
The Docker toolset to pack, ship, store, and deliver content.
Planned content for this repository:
The main product of this repository is the new registry implementation for
storing and distributing docker images. It supersedes the [docker/docker-
registry](https://github.com/docker/docker-registry) project with a new API
design, focused around security and performance.
* Distribution related specifications
- Image format
- JSON registry API
* Registry implementation: a Golang implementation of the JSON API
* Client libraries to consume conforming implementations of the JSON API
The _Distribution_ project has the further long term goal of providing a
secure tool chain for distributing content. The specifications, APIs and tools
should be as useful with docker as they are without.
# Ongoing open sprint
This repository contains the following components:
### What is an open sprint?
The open sprint is a focused effort of a small group of people to kick-off a new project, while commiting to becoming maintainers of the resulting work.
**Having a dedicated team work on the subject doesn't mean that you, the community, cannot contribute!** We need your input to make the best use of the sprint, and focus our work on what matters for you. For this particular topic:
* Come discuss on IRC: #docker-distribution on FreeNode
* Come read and participate in the [Google Groups](https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution)
* Submit your ideas, and upvote those you think matter the most on [Google Moderator](https://www.google.com/moderator/?authuser=1#16/e=2165c3)
### Goal of the distribution sprint
Design a professional grade and extensible content distribution system, that allow users to:
* Enjoy an efficient, secured and reliable way to store, manage, package and exchange content
* Hack/roll their own on top of healthy open-source components
* Implement their own home made solution through good specs, and solid extensions mechanism.
### Schedule and expected output
The Open Sprint will start on **Monday December 29th**, and end on **Friday January 16th**.
What we want to achieve as a result is:
* Tactical fixes of today's frustrations in the existing Docker codebase
- This includes a throrough review of [docker/docker#9784](https://github.com/docker/docker/pull/9784) by core maintainers
* Laying the base of a new distribution subsystem, living independently, and with a well defined group of maintainers. This is the purpose of this repository, which aims at hosting:
- A specification of the v2 image format
- A specification of the JSON/HTTP protocol
- Server-side Go implementation of the v2 registry
- Client-side Go packages to consume this new API
- Standalone binaries providing content distribution functionalities outside of Docker
* Constraints for interface provided by Distribution to Core:
- The distribution repository is a collection of tools for packaging and
shipping content with Docker
- All tools are usable primarily as standalone command-line tools. They may
also expose bindings in one or more programming languages. Typically the
first available language is Go, but that is not automatically true and more
languages will be supported over time
- The distribution repository is still under incubation, any code layout,
interface and name may change before it gets included in a stable release of
Docker
- **registry (beta):** An implementation of the [Docker Registry HTTP API
V2](doc/spec/api.md) for use with docker 1.5+.
- **libraries (unstable):** A rich set of libraries for interacting with
distribution components. Please see
[godoc](http://godoc.org/github.com/docker/distribution) for details. Note
that the libraries *are not* considered stable.
- **dist (experimental):** An experimental tool to provide distribution
oriented functionality without the docker daemon.
- **specifications**: _Distribution_ related specifications are available in
[doc/spec](doc/spec).
- **documentation:** Documentation is available in [doc](doc/overview.md).
### How will this integrate with Docker engine?
Building awesome, independent, and well maintained distribution tools should give Docker core maintainers enough incentive to switch to the newly develop subsystem. We make no assumptions on a given date or milestone as urgency should be fixed through [docker/docker#9784](https://github.com/docker/docker/pull/9784), and in order to maintain focus on producing a top quality alternative.
This project should provide an implementation to a V2 API for use in the
Docker core project. The API should be embeddable and simplify the process of
securely pulling and pushing content from docker daemons.
### Relevant documents
### What are the long term goals of the Distribution project?
* [Analysis of current state and goals](doc/opensprint/kickoff.md)
Design a professional grade and extensible content distribution system, that
allow users to:
* Enjoy an efficient, secured and reliable way to store, manage, package and
exchange content
* Hack/roll their own on top of healthy open-source components
* Implement their own home made solution through good specs, and solid
extensions mechanism.
Features
--------
The new registry implementation provides the following benefits:
- faster push and pull
- new, more efficient implementation
- simplified deployment
- pluggable storage backend
- webhook notifications
Installation
------------
**TODO(stevvooe):** Add the following here:
- docker file
- binary builds for non-docker environment (test installations, etc.)
Configuration
-------------
The registry server can be configured with a yaml file. The following is a
simple example that can used for local development:
```yaml
version: 0.1
loglevel: debug
storage:
filesystem:
rootdirectory: /tmp/registry-dev
http:
addr: localhost:5000
secret: asecretforlocaldevelopment
debug:
addr: localhost:5001
```
The above configures the registry instance to run on port 5000, binding to
"localhost", with the debug server enabled. Registry data will be stored in
"/tmp/registry-dev". Logging will be in "debug" mode, which is the most
verbose.
A similar simple configuration is available at [cmd/registry/config.yml],
which is generally useful for local development.
**TODO(stevvooe): Need a "best practice" configuration overview. Perhaps, we
can point to a documentation section.
For full details about configuring a registry server, please see [the
documentation](doc/configuration.md).
### Upgrading
**TODO:** Add a section about upgrading from V1 registry along with link to
migrating in documentation.
Build
-----
If a go development environment is setup, one can use `go get` to install the
`registry` command from the current latest:
```sh
go get github.com/docker/distribution/cmd/registry
```
The above will install the source repository into the `GOPATH`. The `registry`
binary can then be run with the following:
```
$ $GOPATH/bin/registry -version
$GOPATH/bin/registry github.com/docker/distribution v2.0.0-alpha.1+unknown
```
The registry can be run with the default config using the following
incantantation:
```
$ $GOPATH/bin/registry $GOPATH/src/github.com/docker/distribution/cmd/registry/config.yml
INFO[0000] endpoint local-8082 disabled, skipping app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
INFO[0000] endpoint local-8083 disabled, skipping app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
INFO[0000] listening on :5000 app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown
INFO[0000] debug server listening localhost:5001
```
If it is working, one should see the above log messages.
### Repeatable Builds
For the full development experience, one should `cd` into
`$GOPATH/src/github.com/docker/distribution`. From there, the regular `go`
commands, such as `go test`, should work per package (please see
[Developing](#developing) if they don't work).
A `Makefile` has been provided as a convenience to support repeatable builds.
Please install the following into `GOPATH` for it to work:
```
go get github.com/tools/godep github.com/golang/lint/golint
```
**TODO(stevvooe):** Add a `make setup` command to Makefile to run this. Have
to think about how to interact with Godeps properly.
Once these commands are available in the `GOPATH`, run `make` to get a full
build:
```
$ GOPATH=`godep path`:$GOPATH make
+ clean
+ fmt
+ vet
+ lint
+ build
github.com/docker/docker/vendor/src/code.google.com/p/go/src/pkg/archive/tar
github.com/Sirupsen/logrus
github.com/docker/libtrust
...
github.com/yvasiyarov/gorelic
github.com/docker/distribution/registry/handlers
github.com/docker/distribution/cmd/registry
+ test
...
ok github.com/docker/distribution/digest 7.875s
ok github.com/docker/distribution/manifest 0.028s
ok github.com/docker/distribution/notifications 17.322s
? github.com/docker/distribution/registry [no test files]
ok github.com/docker/distribution/registry/api/v2 0.101s
? github.com/docker/distribution/registry/auth [no test files]
ok github.com/docker/distribution/registry/auth/silly 0.011s
...
+ /Users/sday/go/src/github.com/docker/distribution/bin/registry
+ /Users/sday/go/src/github.com/docker/distribution/bin/registry-api-descriptor-template
+ /Users/sday/go/src/github.com/docker/distribution/bin/dist
+ binaries
```
The above provides a repeatable build using the contents of the vendored
Godeps directory. This includes formatting, vetting, linting, building,
testing and generating tagged binaries. We can verify this worked by running
the registry binary generated in the "./bin" directory:
```sh
$ ./bin/registry -version
./bin/registry github.com/docker/distribution v2.0.0-alpha.2-80-g16d8b2c.m
```
### Developing
The above approaches are helpful for small experimentation. If more complex
tasks are at hand, it is recommended to employ the full power of `godep`.
The Makefile is designed to have its `GOPATH` defined externally. This allows
one to experiment with various development environment setups. This is
primarily useful when testing upstream bugfixes, by modifying local code. This
can be demonstrated using `godep` to migrate the `GOPATH` to use the specified
dependencies. The `GOPATH` can be migrated to the current package versions
declared in `Godeps` with the following command:
```sh
godep restore
```
> **WARNING:** This command will checkout versions of the code specified in
> Godeps/Godeps.json, modifying the contents of `GOPATH`. If this is
> undesired, it is recommended to create a workspace devoted to work on the
> _Distribution_ project.
With a successful run of the above command, one can now use `make` without
specifying the `GOPATH`:
```sh
$ make
```
If that is successful, standard `go` commands, such as `go test` should work,
per package, without issue.
Support
-------
If any issues are encountered while using the _Distribution_ project, several
avenues are available for support:
IRC: #docker-distribution on FreeNode
Issue Tracker: github.com/docker/distribution/issues
Google Groups: https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution
Mailing List: docker@dockerproject.org
Contribute
----------
Please see [CONTRIBUTING.md](CONTRIBUTING.md).
License
-------
This project is distributed under [Apache License, Version 2.0](LICENSE.md).

4
doc/architecture.md Normal file
View file

@ -0,0 +1,4 @@
# Architecture
**TODO(stevvooe):** Discuss the architecture of the registry, internally and
externally, in a few different deployment scenarios.

4
doc/configuration.md Normal file
View file

@ -0,0 +1,4 @@
# Configuration
**TODO(stevvooe): This should include an exhaustive account configuration
parameters, including those for various subsystems.

6
doc/deploying.md Normal file
View file

@ -0,0 +1,6 @@
# Deploying
**TODO(stevvooe):** This should discuss various deployment scenarios for
production-ready deployments. These may be backed by ready-made docker images
but this should explain how they were created and what considerations were
present.

39
doc/glossary.md Normal file
View file

@ -0,0 +1,39 @@
# Glossary
**TODO(stevvooe):** Define and describe distribution related terms. Ideally,
we reference back to the actual documentation and specifications where
appropriate.
**TODO(stevvooe):** The following list is a start but woefully incomplete.
<dl>
<dt>Blob</dt>
<dd>
The primary unit of registry storage. A string of bytes identified by
content-address, known as a _digest_.
</dd>
<dt>Image</dt>
<dd>An image is a collection of content from which a docker container can be created.</dd>
<dt>Layer</dt>
<dd>
A tar file representing the partial content of a filesystem. Several
layers can be "stacked" to make up the root filesystem.
</dd>
<dt>Manifest</dt>
<dd>Describes a collection layers that make up an image.</dd>
<dt>Registry</dt>
<dd>A registry is a collection of repositories.</dd>
<dt>Repository</dt>
<dd>
A repository is a collection of docker images, made up of manifests, tags
and layers. The base unit of these components are blobs.
</dd>
<dt>Tag</dt>
<dd>Tag provides a common name to an image.</dd>
</dl>

4
doc/notifications.md Normal file
View file

@ -0,0 +1,4 @@
# Notifications
**TODO(stevvooe)** Cover use and deployment of webhook notifications. Link to
description in architecture documentation.

158
doc/opensprint/kickoff.md
View file

@ -1,158 +0,0 @@
Distribution
=========================
## Project intentions
**Problem statement and requirements**
* What is the exact scope of the problem?
Design a professional grade and extensible content distribution system, that allows docker users to:
... by default enjoy:
* an efficient, secured and reliable way to store, manage, package and exchange content
... optionally:
* can hack/roll their own on top of healthy open-source components
... with the liberty to:
* implement their own home made solution through good specs, and solid extensions mechanism
* Who will the result be useful to?
* users
* ISV (who distribute images or develop image distribution solutions)
* docker
* What are the use cases (distinguish dev & ops population where applicable)?
* Everyone (... uses docker push/pull).
* Why does it matter that we build this now?
* Shortcomings of the existing codebase are the #1 pain point (by large) for users, partners and ISV, hence the most urgent thing to address (?)
* That situation is getting worse everyday and killer competitors are going/have emerged.
* Who are the competitors?
* existing artifact storage solutions (eg: artifactory).
* emerging products that aim at handling pull/push in place of docker.
* ISV that are looking for alternatives to workaround this situation
**Current state: what do we have today?**
Problems of the existing system:
1. not reliable
* registry goes down whenever the hub goes down
* failing push result in broken repositories
* concurrent push is not handled
* python boto and gevent have a terrible history
* organically grown, under-designed features are in a bad shape (search)
2. inconsistent
* discrepancies between duplicated API (and *duplicated APIs*)
* unused features
* missing essential features (proper SSL support)
3. not reusable
* tightly entangled with hub component makes it very difficult to use outside of docker
* proper access-control is almost impossible to do right
* not easily extensible
4. not efficient
* no parallel operations (by design)
* sluggish client-side processing / bad pipeline design
* poor reusability of content (random ids)
* scalability issues (tags)
* too many useless requests (protocol)
* too much local space consumed (local garbage collection: broken + not efficient)
* no squashing
5. not resilient to errors
* no resume
* error handling is obscure or inexistent
6. security
* content is not verified
* current tarsum is broken
* random ids are a headache
7. confusing
* registry vs. registry.hub?
* layer vs. image?
8. broken features
* mirroring is not done correctly (too complex, bug-laden, caching is hard)
9. poor integration with the rest of the project
* technology discrepancy (python vs. go)
* poor testability
* poor separation (API in the engine is not defined enough)
10. missing features / prevents future
* trust / image signing
* naming / transport separation
* discovery / layer federation
* architecture + os support (eg: arm/windows)
* quotas
* alternative distribution methods (transport plugins)
**Future state: where do we want to get?**
* Deliverable
* new JSON/HTTP protocol specification
* new image format specification
* (new image store in the engine)
* new transport API between the engine and the distribution client code / new library
* new registry in go
* new authentication service on top of the trust graph in go
* What are the interactions with other components of the project?
* critical interactions with docker push/pull mechanism
* critical interactions with the way docker stores images locally
* In what way will the result be customizable?
* transport plugins allowing for radically different transport methods (bittorent, direct S3 access, etc)
* extensibility design for the registry allowing for complex integrations with other systems
* backend storage drivers API
## Kick-off output
**What is the expected output of the kick-off session?**
* draft specifications
* separate binary tool for demo purpose
* a mergeable PR that fixes 90% of the listed issues
* agree on a vision that allows solving all that are deemed worthy
* propose a long term battle plan with clear milestones that encompass all these
* define a first milestone that is compatible with the future and does already deliver some of the solutions
* deliver the specifications for image manifest format and transport API
* deliver a working implementation that can be used as a drop-in replacement for the existing v1 with an equivalent feature-set
**How is the output going to be demoed?**
docker pull
docker push
**Once demoed, what will be the path to shipping?**
A minimal PR that include the first subset of features to make docker work well with the new server side components.
## Pressing matters
* need a codename (ship, distribute)
* new repository
* new domains
* architecture / OS
* persistent ids
* registries discovery
* naming (quay.io/foo/bar)
* mirroring
## Assorted issues
* some devops want a docker engine that cannot do push/pull

6
doc/overview.md Normal file
View file

@ -0,0 +1,6 @@
# Overview
**TODO(stevvooe):** Table of contents.
**TODO(stevvooe):** Include a full overview of each component and dispatch the
user to the correct documentation.

5
doc/storagedrivers.md Normal file
View file

@ -0,0 +1,5 @@
# Storage Drivers
**TODO(stevvooe):** This should include detailed overviews of what a storage
driver is and information about each storage driver implementation.
Considerations when implementing a storage driver should also be present.