From ad11768765805b0ae6ffb115cb91e61868d17c89 Mon Sep 17 00:00:00 2001 From: Stephen J Day Date: Mon, 2 Mar 2015 23:33:02 -0800 Subject: [PATCH] Update README and outline documentation Signed-off-by: Stephen J Day --- README.md | 290 ++++++++++++++++++++++++++++++-------- doc/architecture.md | 4 + doc/configuration.md | 4 + doc/deploying.md | 6 + doc/glossary.md | 39 +++++ doc/notifications.md | 4 + doc/opensprint/kickoff.md | 158 --------------------- doc/overview.md | 6 + doc/storagedrivers.md | 5 + 9 files changed, 303 insertions(+), 213 deletions(-) create mode 100644 doc/architecture.md create mode 100644 doc/configuration.md create mode 100644 doc/deploying.md create mode 100644 doc/glossary.md create mode 100644 doc/notifications.md delete mode 100644 doc/opensprint/kickoff.md create mode 100644 doc/overview.md create mode 100644 doc/storagedrivers.md diff --git a/README.md b/README.md index 64fba36fd..f0bc54d0e 100644 --- a/README.md +++ b/README.md @@ -1,69 +1,249 @@ -> **Notice:** *This repository hosts experimental components that are currently under heavy and fast-paced development, not-ready for public consumption. If you are looking for the stable registry, please head over to [docker/docker-registry](https://github.com/docker/docker-registry) instead.* +> **Notice:** *This repository hosts experimental components that are +> currently under heavy and fast-paced development, not-ready for public +> consumption. If you are looking for the stable registry, please head over to +> [docker/docker-registry](https://github.com/docker/docker-registry) +> instead.* Distribution ============ The Docker toolset to pack, ship, store, and deliver content. -Planned content for this repository: +The main product of this repository is the new registry implementation for +storing and distributing docker images. It supersedes the [docker/docker- +registry](https://github.com/docker/docker-registry) project with a new API +design, focused around security and performance. -* Distribution related specifications - - Image format - - JSON registry API -* Registry implementation: a Golang implementation of the JSON API -* Client libraries to consume conforming implementations of the JSON API +The _Distribution_ project has the further long term goal of providing a +secure tool chain for distributing content. The specifications, APIs and tools +should be as useful with docker as they are without. -# Ongoing open sprint +This repository contains the following components: -### What is an open sprint? - -The open sprint is a focused effort of a small group of people to kick-off a new project, while commiting to becoming maintainers of the resulting work. - -**Having a dedicated team work on the subject doesn't mean that you, the community, cannot contribute!** We need your input to make the best use of the sprint, and focus our work on what matters for you. For this particular topic: - -* Come discuss on IRC: #docker-distribution on FreeNode -* Come read and participate in the [Google Groups](https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution) -* Submit your ideas, and upvote those you think matter the most on [Google Moderator](https://www.google.com/moderator/?authuser=1#16/e=2165c3) - -### Goal of the distribution sprint - -Design a professional grade and extensible content distribution system, that allow users to: - -* Enjoy an efficient, secured and reliable way to store, manage, package and exchange content -* Hack/roll their own on top of healthy open-source components -* Implement their own home made solution through good specs, and solid extensions mechanism. - -### Schedule and expected output - -The Open Sprint will start on **Monday December 29th**, and end on **Friday January 16th**. - -What we want to achieve as a result is: - -* Tactical fixes of today's frustrations in the existing Docker codebase - - This includes a throrough review of [docker/docker#9784](https://github.com/docker/docker/pull/9784) by core maintainers - -* Laying the base of a new distribution subsystem, living independently, and with a well defined group of maintainers. This is the purpose of this repository, which aims at hosting: - - A specification of the v2 image format - - A specification of the JSON/HTTP protocol - - Server-side Go implementation of the v2 registry - - Client-side Go packages to consume this new API - - Standalone binaries providing content distribution functionalities outside of Docker - -* Constraints for interface provided by Distribution to Core: - - The distribution repository is a collection of tools for packaging and - shipping content with Docker - - All tools are usable primarily as standalone command-line tools. They may - also expose bindings in one or more programming languages. Typically the - first available language is Go, but that is not automatically true and more - languages will be supported over time - - The distribution repository is still under incubation, any code layout, - interface and name may change before it gets included in a stable release of - Docker +- **registry (beta):** An implementation of the [Docker Registry HTTP API + V2](doc/spec/api.md) for use with docker 1.5+. +- **libraries (unstable):** A rich set of libraries for interacting with + distribution components. Please see + [godoc](http://godoc.org/github.com/docker/distribution) for details. Note + that the libraries *are not* considered stable. +- **dist (experimental):** An experimental tool to provide distribution + oriented functionality without the docker daemon. +- **specifications**: _Distribution_ related specifications are available in + [doc/spec](doc/spec). +- **documentation:** Documentation is available in [doc](doc/overview.md). ### How will this integrate with Docker engine? -Building awesome, independent, and well maintained distribution tools should give Docker core maintainers enough incentive to switch to the newly develop subsystem. We make no assumptions on a given date or milestone as urgency should be fixed through [docker/docker#9784](https://github.com/docker/docker/pull/9784), and in order to maintain focus on producing a top quality alternative. +This project should provide an implementation to a V2 API for use in the +Docker core project. The API should be embeddable and simplify the process of +securely pulling and pushing content from docker daemons. -### Relevant documents +### What are the long term goals of the Distribution project? -* [Analysis of current state and goals](doc/opensprint/kickoff.md) +Design a professional grade and extensible content distribution system, that +allow users to: + +* Enjoy an efficient, secured and reliable way to store, manage, package and + exchange content +* Hack/roll their own on top of healthy open-source components +* Implement their own home made solution through good specs, and solid + extensions mechanism. + +Features +-------- + +The new registry implementation provides the following benefits: + +- faster push and pull +- new, more efficient implementation +- simplified deployment +- pluggable storage backend +- webhook notifications + +Installation +------------ + +**TODO(stevvooe):** Add the following here: +- docker file +- binary builds for non-docker environment (test installations, etc.) + +Configuration +------------- + +The registry server can be configured with a yaml file. The following is a +simple example that can used for local development: + +```yaml +version: 0.1 +loglevel: debug +storage: + filesystem: + rootdirectory: /tmp/registry-dev +http: + addr: localhost:5000 + secret: asecretforlocaldevelopment + debug: + addr: localhost:5001 +``` + +The above configures the registry instance to run on port 5000, binding to +"localhost", with the debug server enabled. Registry data will be stored in +"/tmp/registry-dev". Logging will be in "debug" mode, which is the most +verbose. + +A similar simple configuration is available at [cmd/registry/config.yml], +which is generally useful for local development. + +**TODO(stevvooe): Need a "best practice" configuration overview. Perhaps, we +can point to a documentation section. + +For full details about configuring a registry server, please see [the +documentation](doc/configuration.md). + +### Upgrading + +**TODO:** Add a section about upgrading from V1 registry along with link to +migrating in documentation. + +Build +----- + +If a go development environment is setup, one can use `go get` to install the +`registry` command from the current latest: + +```sh +go get github.com/docker/distribution/cmd/registry +``` + +The above will install the source repository into the `GOPATH`. The `registry` +binary can then be run with the following: + +``` +$ $GOPATH/bin/registry -version +$GOPATH/bin/registry github.com/docker/distribution v2.0.0-alpha.1+unknown +``` + +The registry can be run with the default config using the following +incantantation: + +``` +$ $GOPATH/bin/registry $GOPATH/src/github.com/docker/distribution/cmd/registry/config.yml +INFO[0000] endpoint local-8082 disabled, skipping app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown +INFO[0000] endpoint local-8083 disabled, skipping app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown +INFO[0000] listening on :5000 app.id=34bbec38-a91a-494a-9a3f-b72f9010081f version=v2.0.0-alpha.1+unknown +INFO[0000] debug server listening localhost:5001 +``` + +If it is working, one should see the above log messages. + +### Repeatable Builds + +For the full development experience, one should `cd` into +`$GOPATH/src/github.com/docker/distribution`. From there, the regular `go` +commands, such as `go test`, should work per package (please see +[Developing](#developing) if they don't work). + +A `Makefile` has been provided as a convenience to support repeatable builds. +Please install the following into `GOPATH` for it to work: + +``` +go get github.com/tools/godep github.com/golang/lint/golint +``` + +**TODO(stevvooe):** Add a `make setup` command to Makefile to run this. Have +to think about how to interact with Godeps properly. + +Once these commands are available in the `GOPATH`, run `make` to get a full +build: + +``` +$ GOPATH=`godep path`:$GOPATH make ++ clean ++ fmt ++ vet ++ lint ++ build +github.com/docker/docker/vendor/src/code.google.com/p/go/src/pkg/archive/tar +github.com/Sirupsen/logrus +github.com/docker/libtrust +... +github.com/yvasiyarov/gorelic +github.com/docker/distribution/registry/handlers +github.com/docker/distribution/cmd/registry ++ test +... +ok github.com/docker/distribution/digest 7.875s +ok github.com/docker/distribution/manifest 0.028s +ok github.com/docker/distribution/notifications 17.322s +? github.com/docker/distribution/registry [no test files] +ok github.com/docker/distribution/registry/api/v2 0.101s +? github.com/docker/distribution/registry/auth [no test files] +ok github.com/docker/distribution/registry/auth/silly 0.011s +... ++ /Users/sday/go/src/github.com/docker/distribution/bin/registry ++ /Users/sday/go/src/github.com/docker/distribution/bin/registry-api-descriptor-template ++ /Users/sday/go/src/github.com/docker/distribution/bin/dist ++ binaries +``` + +The above provides a repeatable build using the contents of the vendored +Godeps directory. This includes formatting, vetting, linting, building, +testing and generating tagged binaries. We can verify this worked by running +the registry binary generated in the "./bin" directory: + +```sh +$ ./bin/registry -version +./bin/registry github.com/docker/distribution v2.0.0-alpha.2-80-g16d8b2c.m +``` + +### Developing + +The above approaches are helpful for small experimentation. If more complex +tasks are at hand, it is recommended to employ the full power of `godep`. + +The Makefile is designed to have its `GOPATH` defined externally. This allows +one to experiment with various development environment setups. This is +primarily useful when testing upstream bugfixes, by modifying local code. This +can be demonstrated using `godep` to migrate the `GOPATH` to use the specified +dependencies. The `GOPATH` can be migrated to the current package versions +declared in `Godeps` with the following command: + +```sh +godep restore +``` + +> **WARNING:** This command will checkout versions of the code specified in +> Godeps/Godeps.json, modifying the contents of `GOPATH`. If this is +> undesired, it is recommended to create a workspace devoted to work on the +> _Distribution_ project. + +With a successful run of the above command, one can now use `make` without +specifying the `GOPATH`: + +```sh +$ make +``` + +If that is successful, standard `go` commands, such as `go test` should work, +per package, without issue. + +Support +------- + +If any issues are encountered while using the _Distribution_ project, several +avenues are available for support: + +IRC: #docker-distribution on FreeNode +Issue Tracker: github.com/docker/distribution/issues +Google Groups: https://groups.google.com/a/dockerproject.org/forum/#!forum/distribution +Mailing List: docker@dockerproject.org + +Contribute +---------- + +Please see [CONTRIBUTING.md](CONTRIBUTING.md). + +License +------- + +This project is distributed under [Apache License, Version 2.0](LICENSE.md). diff --git a/doc/architecture.md b/doc/architecture.md new file mode 100644 index 000000000..d8b96e8da --- /dev/null +++ b/doc/architecture.md @@ -0,0 +1,4 @@ +# Architecture + +**TODO(stevvooe):** Discuss the architecture of the registry, internally and +externally, in a few different deployment scenarios. \ No newline at end of file diff --git a/doc/configuration.md b/doc/configuration.md new file mode 100644 index 000000000..1367a5bcc --- /dev/null +++ b/doc/configuration.md @@ -0,0 +1,4 @@ +# Configuration + +**TODO(stevvooe): This should include an exhaustive account configuration +parameters, including those for various subsystems. \ No newline at end of file diff --git a/doc/deploying.md b/doc/deploying.md new file mode 100644 index 000000000..f30e32eb7 --- /dev/null +++ b/doc/deploying.md @@ -0,0 +1,6 @@ +# Deploying + +**TODO(stevvooe):** This should discuss various deployment scenarios for +production-ready deployments. These may be backed by ready-made docker images +but this should explain how they were created and what considerations were +present. \ No newline at end of file diff --git a/doc/glossary.md b/doc/glossary.md new file mode 100644 index 000000000..68fef2154 --- /dev/null +++ b/doc/glossary.md @@ -0,0 +1,39 @@ +# Glossary + +**TODO(stevvooe):** Define and describe distribution related terms. Ideally, +we reference back to the actual documentation and specifications where +appropriate. + +**TODO(stevvooe):** The following list is a start but woefully incomplete. + +
+
Blob
+
+ The primary unit of registry storage. A string of bytes identified by + content-address, known as a _digest_. +
+ +
Image
+
An image is a collection of content from which a docker container can be created.
+ +
Layer
+
+ A tar file representing the partial content of a filesystem. Several + layers can be "stacked" to make up the root filesystem. +
+ +
Manifest
+
Describes a collection layers that make up an image.
+ +
Registry
+
A registry is a collection of repositories.
+ +
Repository
+
+ A repository is a collection of docker images, made up of manifests, tags + and layers. The base unit of these components are blobs. +
+ +
Tag
+
Tag provides a common name to an image.
+
\ No newline at end of file diff --git a/doc/notifications.md b/doc/notifications.md new file mode 100644 index 000000000..5dbd8f347 --- /dev/null +++ b/doc/notifications.md @@ -0,0 +1,4 @@ +# Notifications + +**TODO(stevvooe)** Cover use and deployment of webhook notifications. Link to +description in architecture documentation. \ No newline at end of file diff --git a/doc/opensprint/kickoff.md b/doc/opensprint/kickoff.md deleted file mode 100644 index d2b69e48c..000000000 --- a/doc/opensprint/kickoff.md +++ /dev/null @@ -1,158 +0,0 @@ -Distribution -========================= - -## Project intentions - -**Problem statement and requirements** - -* What is the exact scope of the problem? - - -Design a professional grade and extensible content distribution system, that allows docker users to: - -... by default enjoy: - - * an efficient, secured and reliable way to store, manage, package and exchange content - -... optionally: - - * can hack/roll their own on top of healthy open-source components - -... with the liberty to: - - * implement their own home made solution through good specs, and solid extensions mechanism - - -* Who will the result be useful to? - - * users - * ISV (who distribute images or develop image distribution solutions) - * docker - -* What are the use cases (distinguish dev & ops population where applicable)? - - * Everyone (... uses docker push/pull). - -* Why does it matter that we build this now? - - * Shortcomings of the existing codebase are the #1 pain point (by large) for users, partners and ISV, hence the most urgent thing to address (?) - * That situation is getting worse everyday and killer competitors are going/have emerged. - -* Who are the competitors? - - * existing artifact storage solutions (eg: artifactory). - * emerging products that aim at handling pull/push in place of docker. - * ISV that are looking for alternatives to workaround this situation - -**Current state: what do we have today?** - -Problems of the existing system: - -1. not reliable - * registry goes down whenever the hub goes down - * failing push result in broken repositories - * concurrent push is not handled - * python boto and gevent have a terrible history - * organically grown, under-designed features are in a bad shape (search) -2. inconsistent - * discrepancies between duplicated API (and *duplicated APIs*) - * unused features - * missing essential features (proper SSL support) -3. not reusable - * tightly entangled with hub component makes it very difficult to use outside of docker - * proper access-control is almost impossible to do right - * not easily extensible -4. not efficient - * no parallel operations (by design) - * sluggish client-side processing / bad pipeline design - * poor reusability of content (random ids) - * scalability issues (tags) - * too many useless requests (protocol) - * too much local space consumed (local garbage collection: broken + not efficient) - * no squashing -5. not resilient to errors - * no resume - * error handling is obscure or inexistent -6. security - * content is not verified - * current tarsum is broken - * random ids are a headache -7. confusing - * registry vs. registry.hub? - * layer vs. image? -8. broken features - * mirroring is not done correctly (too complex, bug-laden, caching is hard) -9. poor integration with the rest of the project - * technology discrepancy (python vs. go) - * poor testability - * poor separation (API in the engine is not defined enough) -10. missing features / prevents future - * trust / image signing - * naming / transport separation - * discovery / layer federation - * architecture + os support (eg: arm/windows) - * quotas - * alternative distribution methods (transport plugins) - -**Future state: where do we want to get?** - -* Deliverable - * new JSON/HTTP protocol specification - * new image format specification - * (new image store in the engine) - * new transport API between the engine and the distribution client code / new library - * new registry in go - * new authentication service on top of the trust graph in go - -* What are the interactions with other components of the project? - * critical interactions with docker push/pull mechanism - * critical interactions with the way docker stores images locally - -* In what way will the result be customizable? - * transport plugins allowing for radically different transport methods (bittorent, direct S3 access, etc) - * extensibility design for the registry allowing for complex integrations with other systems - * backend storage drivers API - - -## Kick-off output - -**What is the expected output of the kick-off session?** - -* draft specifications -* separate binary tool for demo purpose -* a mergeable PR that fixes 90% of the listed issues - - -* agree on a vision that allows solving all that are deemed worthy -* propose a long term battle plan with clear milestones that encompass all these -* define a first milestone that is compatible with the future and does already deliver some of the solutions -* deliver the specifications for image manifest format and transport API -* deliver a working implementation that can be used as a drop-in replacement for the existing v1 with an equivalent feature-set - -**How is the output going to be demoed?** - -docker pull -docker push - -**Once demoed, what will be the path to shipping?** - -A minimal PR that include the first subset of features to make docker work well with the new server side components. - -## Pressing matters - - * need a codename (ship, distribute) - * new repository - * new domains - - * architecture / OS - * persistent ids - * registries discovery - * naming (quay.io/foo/bar) - * mirroring - - - -## Assorted issues - - * some devops want a docker engine that cannot do push/pull - diff --git a/doc/overview.md b/doc/overview.md new file mode 100644 index 000000000..65cf56b7f --- /dev/null +++ b/doc/overview.md @@ -0,0 +1,6 @@ +# Overview + +**TODO(stevvooe):** Table of contents. + +**TODO(stevvooe):** Include a full overview of each component and dispatch the +user to the correct documentation. diff --git a/doc/storagedrivers.md b/doc/storagedrivers.md new file mode 100644 index 000000000..f8e4d6751 --- /dev/null +++ b/doc/storagedrivers.md @@ -0,0 +1,5 @@ +# Storage Drivers + +**TODO(stevvooe):** This should include detailed overviews of what a storage +driver is and information about each storage driver implementation. +Considerations when implementing a storage driver should also be present. \ No newline at end of file