From 2df45044a66e2cb4a4e542d735636c6fd2db6920 Mon Sep 17 00:00:00 2001 From: Maria Bermudez Date: Wed, 29 Aug 2018 14:47:31 -0700 Subject: [PATCH 01/10] Revert "Merge branch 'master' of github.com:docker/docs-private into test-branch-2" This reverts commit af5f2fcc38c39c157180be7b9671fddd1ab3bfc5, reversing changes made to 338b690d26894aec370337caca1788eeaecbd8de. --- docs/deploying.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/deploying.md b/docs/deploying.md index 740adac2a..66a6374dd 100644 --- a/docs/deploying.md +++ b/docs/deploying.md @@ -322,15 +322,15 @@ $ docker service create \ --secret domain.key \ --constraint 'node.labels.registry==true' \ --mount type=bind,src=/mnt/registry,dst=/var/lib/registry \ - -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \ + -e REGISTRY_HTTP_ADDR=0.0.0.0:80 \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/run/secrets/domain.crt \ -e REGISTRY_HTTP_TLS_KEY=/run/secrets/domain.key \ - --publish published=443,target=443 \ + --publish published=80,target=80 \ --replicas 1 \ registry:2 ``` -You can access the service on port 443 of any swarm node. Docker sends the +You can access the service on port 80 of any swarm node. Docker sends the requests to the node which is running the service. ## Load balancing considerations @@ -458,8 +458,8 @@ secrets. You may want to leverage more advanced basic auth implementations by using a proxy in front of the registry. See the [recipes list](recipes/index.md). -The registry also supports delegated authentication which redirects users to a -specific trusted token server. This approach is more complicated to set up, and +The registry also supports delegated authentiation, which redirects users to a +specific, trusted token server. This approach is more complicated to set up, and only makes sense if you need to fully configure ACLs and need more control over the registry's integration into your global authorization and authentication systems. Refer to the following [background information](spec/auth/token.md) and From b9c4182eb6ce6a8a2489faffc785f1c4ea87d1ad Mon Sep 17 00:00:00 2001 From: Maria Bermudez Date: Wed, 29 Aug 2018 18:36:03 -0700 Subject: [PATCH 02/10] Revert "Revert "Merge branch 'master' of github.com:docker/docs-private into test-branch-2"" This reverts commit 2df45044a66e2cb4a4e542d735636c6fd2db6920. --- docs/deploying.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/deploying.md b/docs/deploying.md index 66a6374dd..740adac2a 100644 --- a/docs/deploying.md +++ b/docs/deploying.md @@ -322,15 +322,15 @@ $ docker service create \ --secret domain.key \ --constraint 'node.labels.registry==true' \ --mount type=bind,src=/mnt/registry,dst=/var/lib/registry \ - -e REGISTRY_HTTP_ADDR=0.0.0.0:80 \ + -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/run/secrets/domain.crt \ -e REGISTRY_HTTP_TLS_KEY=/run/secrets/domain.key \ - --publish published=80,target=80 \ + --publish published=443,target=443 \ --replicas 1 \ registry:2 ``` -You can access the service on port 80 of any swarm node. Docker sends the +You can access the service on port 443 of any swarm node. Docker sends the requests to the node which is running the service. ## Load balancing considerations @@ -458,8 +458,8 @@ secrets. You may want to leverage more advanced basic auth implementations by using a proxy in front of the registry. See the [recipes list](recipes/index.md). -The registry also supports delegated authentiation, which redirects users to a -specific, trusted token server. This approach is more complicated to set up, and +The registry also supports delegated authentication which redirects users to a +specific trusted token server. This approach is more complicated to set up, and only makes sense if you need to fully configure ACLs and need more control over the registry's integration into your global authorization and authentication systems. Refer to the following [background information](spec/auth/token.md) and From 88038ffd3a7e7063606f4f72b780b9af62823079 Mon Sep 17 00:00:00 2001 From: Maria Bermudez Date: Wed, 29 Aug 2018 18:37:44 -0700 Subject: [PATCH 03/10] Revert "Merge branch 'master' of github.com:docker/docs-private into test-branch-2" This reverts commit af5f2fcc38c39c157180be7b9671fddd1ab3bfc5, reversing changes made to 338b690d26894aec370337caca1788eeaecbd8de. --- docs/deploying.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/deploying.md b/docs/deploying.md index 740adac2a..66a6374dd 100644 --- a/docs/deploying.md +++ b/docs/deploying.md @@ -322,15 +322,15 @@ $ docker service create \ --secret domain.key \ --constraint 'node.labels.registry==true' \ --mount type=bind,src=/mnt/registry,dst=/var/lib/registry \ - -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \ + -e REGISTRY_HTTP_ADDR=0.0.0.0:80 \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/run/secrets/domain.crt \ -e REGISTRY_HTTP_TLS_KEY=/run/secrets/domain.key \ - --publish published=443,target=443 \ + --publish published=80,target=80 \ --replicas 1 \ registry:2 ``` -You can access the service on port 443 of any swarm node. Docker sends the +You can access the service on port 80 of any swarm node. Docker sends the requests to the node which is running the service. ## Load balancing considerations @@ -458,8 +458,8 @@ secrets. You may want to leverage more advanced basic auth implementations by using a proxy in front of the registry. See the [recipes list](recipes/index.md). -The registry also supports delegated authentication which redirects users to a -specific trusted token server. This approach is more complicated to set up, and +The registry also supports delegated authentiation, which redirects users to a +specific, trusted token server. This approach is more complicated to set up, and only makes sense if you need to fully configure ACLs and need more control over the registry's integration into your global authorization and authentication systems. Refer to the following [background information](spec/auth/token.md) and From f04f6208b9563111cfe4426fcdf9385f800dfac2 Mon Sep 17 00:00:00 2001 From: Maria Bermudez Date: Wed, 29 Aug 2018 19:01:03 -0700 Subject: [PATCH 04/10] Revert "Revert "Merge branch 'master' of github.com:docker/docs-private into test-branch-2"" This reverts commit 88038ffd3a7e7063606f4f72b780b9af62823079. --- docs/deploying.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/deploying.md b/docs/deploying.md index 66a6374dd..740adac2a 100644 --- a/docs/deploying.md +++ b/docs/deploying.md @@ -322,15 +322,15 @@ $ docker service create \ --secret domain.key \ --constraint 'node.labels.registry==true' \ --mount type=bind,src=/mnt/registry,dst=/var/lib/registry \ - -e REGISTRY_HTTP_ADDR=0.0.0.0:80 \ + -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \ -e REGISTRY_HTTP_TLS_CERTIFICATE=/run/secrets/domain.crt \ -e REGISTRY_HTTP_TLS_KEY=/run/secrets/domain.key \ - --publish published=80,target=80 \ + --publish published=443,target=443 \ --replicas 1 \ registry:2 ``` -You can access the service on port 80 of any swarm node. Docker sends the +You can access the service on port 443 of any swarm node. Docker sends the requests to the node which is running the service. ## Load balancing considerations @@ -458,8 +458,8 @@ secrets. You may want to leverage more advanced basic auth implementations by using a proxy in front of the registry. See the [recipes list](recipes/index.md). -The registry also supports delegated authentiation, which redirects users to a -specific, trusted token server. This approach is more complicated to set up, and +The registry also supports delegated authentication which redirects users to a +specific trusted token server. This approach is more complicated to set up, and only makes sense if you need to fully configure ACLs and need more control over the registry's integration into your global authorization and authentication systems. Refer to the following [background information](spec/auth/token.md) and From 71d02b105c93c4c4ec7219a5b481c35b7cf0d837 Mon Sep 17 00:00:00 2001 From: Maria Bermudez Date: Fri, 31 Aug 2018 22:08:13 -0700 Subject: [PATCH 05/10] Add online garbage collection feature and known limitation as described on DTR Workshop doc --- docs/garbage-collection.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/garbage-collection.md b/docs/garbage-collection.md index cc301c6a8..340b2e529 100644 --- a/docs/garbage-collection.md +++ b/docs/garbage-collection.md @@ -69,6 +69,9 @@ A -----> a \--> b ``` +### Online garbage collection +As of v2.6.0, the registry no longer has to be in read-only mode during garbage collection. This means +that you can push images while a garbage collection job is running. ### More details about garbage collection @@ -122,3 +125,6 @@ blob eligible for deletion: sha256:87192bdbe00f8f2a62527f36bb4c7c7f4eaf9307e4b87 blob eligible for deletion: sha256:b549a9959a664038fc35c155a95742cf12297672ca0ae35735ec027d55bf4e97 blob eligible for deletion: sha256:f251d679a7c61455f06d793e43c06786d7766c88b8c24edf242b2c08e3c3f599 ``` +## Known limitation +There is an issue with the "Do Not Repeat" setting which if selected will cause garbage collection to not be run. + From 50dacc554b901a3fd2167e11be6429e815d8c709 Mon Sep 17 00:00:00 2001 From: Maria Bermudez Date: Fri, 31 Aug 2018 22:28:14 -0700 Subject: [PATCH 06/10] Initial draft of product manual for tag-pruning --- docs/tag-pruning.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 docs/tag-pruning.md diff --git a/docs/tag-pruning.md b/docs/tag-pruning.md new file mode 100644 index 000000000..a6579086d --- /dev/null +++ b/docs/tag-pruning.md @@ -0,0 +1,23 @@ +--- +description: High level discussion of tag pruning +keywords: registry, pruning, images, tags, repository, distribution +title: Tag Pruning +--- + +As of v2.6.0 you can set tag pruning policies on individual repositories that you manage. Based on your specified rules, you can automatically delete unwanted images. In addition to a policy approach, you can also set repository tag limits which limit the number of tags in a specific repository. + +## About tag pruning + +In the context of the Docker registry, tag pruning is the process of deleting image tags but not actual blobs. A garbage collection job takes care of blob deletions. + +Additionally repository tag limits are processed in a first in first out manner. For example, if you set a tag limit of 2, adding a third tag would push out the first. + +## Tag pruning in practice + + +### Example + + + +### More details about tag pruning + From b0bb8437cf2bbe99c558710652459241975280f8 Mon Sep 17 00:00:00 2001 From: Maria Bermudez Date: Wed, 5 Sep 2018 07:12:12 -0700 Subject: [PATCH 07/10] Delete garbage-collection.md This relates to CE version of garbage collection. Can you confirm this, @davidswu? --- docs/garbage-collection.md | 130 ------------------------------------- 1 file changed, 130 deletions(-) delete mode 100644 docs/garbage-collection.md diff --git a/docs/garbage-collection.md b/docs/garbage-collection.md deleted file mode 100644 index 340b2e529..000000000 --- a/docs/garbage-collection.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -description: High level discussion of garbage collection -keywords: registry, garbage, images, tags, repository, distribution -title: Garbage collection ---- - -As of v2.4.0 a garbage collector command is included within the registry binary. -This document describes what this command does and how and why it should be used. - -## About garbage collection - -In the context of the Docker registry, garbage collection is the process of -removing blobs from the filesystem when they are no longer referenced by a -manifest. Blobs can include both layers and manifests. - -Registry data can occupy considerable amounts of disk space. In addition, -garbage collection can be a security consideration, when it is desirable to ensure -that certain layers no longer exist on the filesystem. - -## Garbage collection in practice - -Filesystem layers are stored by their content address in the Registry. This -has many advantages, one of which is that data is stored once and referred to by manifests. -See [here](compatibility.md#content-addressable-storage-cas) for more details. - -Layers are therefore shared amongst manifests; each manifest maintains a reference -to the layer. As long as a layer is referenced by one manifest, it cannot be garbage -collected. - -Manifests and layers can be `deleted` with the registry API (refer to the API -documentation [here](spec/api.md#deleting-a-layer) and -[here](spec/api.md#deleting-an-image) for details). This API removes references -to the target and makes them eligible for garbage collection. It also makes them -unable to be read via the API. - -If a layer is deleted, it is removed from the filesystem when garbage collection -is run. If a manifest is deleted the layers to which it refers are removed from -the filesystem if no other manifests refers to them. - - -### Example - -In this example manifest A references two layers: `a` and `b`. Manifest `B` references -layers `a` and `c`. In this state, nothing is eligible for garbage collection: - -``` -A -----> a <----- B - \--> b | - c <--/ -``` - -Manifest B is deleted via the API: - -``` -A -----> a B - \--> b - c -``` - -In this state layer `c` no longer has a reference and is eligible for garbage -collection. Layer `a` had one reference removed but not garbage -collected as it is still referenced by manifest `A`. The blob representing -manifest `B` is eligible for garbage collection. - -After garbage collection has been run, manifest `A` and its blobs remain. - -``` -A -----> a - \--> b -``` - -### Online garbage collection -As of v2.6.0, the registry no longer has to be in read-only mode during garbage collection. This means -that you can push images while a garbage collection job is running. - -### More details about garbage collection - -Garbage collection runs in two phases. First, in the 'mark' phase, the process -scans all the manifests in the registry. From these manifests, it constructs a -set of content address digests. This set is the 'mark set' and denotes the set -of blobs to *not* delete. Secondly, in the 'sweep' phase, the process scans all -the blobs and if a blob's content address digest is not in the mark set, the -process deletes it. - - -> **Note**: You should ensure that the registry is in read-only mode or not running at -> all. If you were to upload an image while garbage collection is running, there is the -> risk that the image's layers are mistakenly deleted leading to a corrupted image. - -This type of garbage collection is known as stop-the-world garbage collection. - -## Run garbage collection - -Garbage collection can be run as follows - -`bin/registry garbage-collect [--dry-run] /path/to/config.yml` - -The garbage-collect command accepts a `--dry-run` parameter, which prints the progress -of the mark and sweep phases without removing any data. Running with a log level of `info` -gives a clear indication of items eligible for deletion. - -The config.yml file should be in the following format: - -``` -version: 0.1 -storage: - filesystem: - rootdirectory: /registry/data -``` - -_Sample output from a dry run garbage collection with registry log level set to `info`_ - -``` -hello-world -hello-world: marking manifest sha256:fea8895f450959fa676bcc1df0611ea93823a735a01205fd8622846041d0c7cf -hello-world: marking blob sha256:03f4658f8b782e12230c1783426bd3bacce651ce582a4ffb6fbbfa2079428ecb -hello-world: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -hello-world: marking configuration sha256:690ed74de00f99a7d00a98a5ad855ac4febd66412be132438f9b8dbd300a937d -ubuntu - -4 blobs marked, 5 blobs eligible for deletion -blob eligible for deletion: sha256:28e09fddaacbfc8a13f82871d9d66141a6ed9ca526cb9ed295ef545ab4559b81 -blob eligible for deletion: sha256:7e15ce58ccb2181a8fced7709e9893206f0937cc9543bc0c8178ea1cf4d7e7b5 -blob eligible for deletion: sha256:87192bdbe00f8f2a62527f36bb4c7c7f4eaf9307e4b87e8334fb6abec1765bcb -blob eligible for deletion: sha256:b549a9959a664038fc35c155a95742cf12297672ca0ae35735ec027d55bf4e97 -blob eligible for deletion: sha256:f251d679a7c61455f06d793e43c06786d7766c88b8c24edf242b2c08e3c3f599 -``` -## Known limitation -There is an issue with the "Do Not Repeat" setting which if selected will cause garbage collection to not be run. - From 2fda032d489219100e72aa08d47d6306f9430893 Mon Sep 17 00:00:00 2001 From: Maria Bermudez Date: Thu, 6 Sep 2018 14:59:19 -0700 Subject: [PATCH 08/10] Delete tag-pruning.md Keeping tag pruning to one page for now --- docs/tag-pruning.md | 23 ----------------------- 1 file changed, 23 deletions(-) delete mode 100644 docs/tag-pruning.md diff --git a/docs/tag-pruning.md b/docs/tag-pruning.md deleted file mode 100644 index a6579086d..000000000 --- a/docs/tag-pruning.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -description: High level discussion of tag pruning -keywords: registry, pruning, images, tags, repository, distribution -title: Tag Pruning ---- - -As of v2.6.0 you can set tag pruning policies on individual repositories that you manage. Based on your specified rules, you can automatically delete unwanted images. In addition to a policy approach, you can also set repository tag limits which limit the number of tags in a specific repository. - -## About tag pruning - -In the context of the Docker registry, tag pruning is the process of deleting image tags but not actual blobs. A garbage collection job takes care of blob deletions. - -Additionally repository tag limits are processed in a first in first out manner. For example, if you set a tag limit of 2, adding a third tag would push out the first. - -## Tag pruning in practice - - -### Example - - - -### More details about tag pruning - From e92760a3a052931d6086237cbb8047f5a0dfdbad Mon Sep 17 00:00:00 2001 From: Oscar Caballero Date: Thu, 16 Aug 2018 14:09:59 +0200 Subject: [PATCH 09/10] Spelling revision --- docs/storage-drivers/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/storage-drivers/index.md b/docs/storage-drivers/index.md index e8f612982..9025bced1 100644 --- a/docs/storage-drivers/index.md +++ b/docs/storage-drivers/index.md @@ -31,7 +31,7 @@ validation of the `storagedriver.StorageDriver` interface. ## Driver selection and configuration -The preferred method of selecting a storage driver is using the `StorageDriverFactory` interface in the `storagedriver/factory` package. These factories provide a common interface for constructing storage drivers with a parameters map. The factory model is based off of the [Register](http://golang.org/pkg/database/sql/#Register) and [Open](http://golang.org/pkg/database/sql/#Open) methods in the builtin [database/sql](http://golang.org/pkg/database/sql) package. +The preferred method of selecting a storage driver is using the `StorageDriverFactory` interface in the `storagedriver/factory` package. These factories provide a common interface for constructing storage drivers with a parameters map. The factory model is based on the [Register](http://golang.org/pkg/database/sql/#Register) and [Open](http://golang.org/pkg/database/sql/#Open) methods in the builtin [database/sql](http://golang.org/pkg/database/sql) package. Storage driver factories may be registered by name using the `factory.Register` method, and then later invoked by calling `factory.Create` From 01ceef9f0ae6031f1243b5980ac275cd7fa7bccb Mon Sep 17 00:00:00 2001 From: Maria Bermudez Date: Thu, 15 Nov 2018 07:55:03 -0800 Subject: [PATCH 10/10] Restoring open source registry garbage collection page --- docs/garbage-collection.md | 124 +++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 docs/garbage-collection.md diff --git a/docs/garbage-collection.md b/docs/garbage-collection.md new file mode 100644 index 000000000..cc301c6a8 --- /dev/null +++ b/docs/garbage-collection.md @@ -0,0 +1,124 @@ +--- +description: High level discussion of garbage collection +keywords: registry, garbage, images, tags, repository, distribution +title: Garbage collection +--- + +As of v2.4.0 a garbage collector command is included within the registry binary. +This document describes what this command does and how and why it should be used. + +## About garbage collection + +In the context of the Docker registry, garbage collection is the process of +removing blobs from the filesystem when they are no longer referenced by a +manifest. Blobs can include both layers and manifests. + +Registry data can occupy considerable amounts of disk space. In addition, +garbage collection can be a security consideration, when it is desirable to ensure +that certain layers no longer exist on the filesystem. + +## Garbage collection in practice + +Filesystem layers are stored by their content address in the Registry. This +has many advantages, one of which is that data is stored once and referred to by manifests. +See [here](compatibility.md#content-addressable-storage-cas) for more details. + +Layers are therefore shared amongst manifests; each manifest maintains a reference +to the layer. As long as a layer is referenced by one manifest, it cannot be garbage +collected. + +Manifests and layers can be `deleted` with the registry API (refer to the API +documentation [here](spec/api.md#deleting-a-layer) and +[here](spec/api.md#deleting-an-image) for details). This API removes references +to the target and makes them eligible for garbage collection. It also makes them +unable to be read via the API. + +If a layer is deleted, it is removed from the filesystem when garbage collection +is run. If a manifest is deleted the layers to which it refers are removed from +the filesystem if no other manifests refers to them. + + +### Example + +In this example manifest A references two layers: `a` and `b`. Manifest `B` references +layers `a` and `c`. In this state, nothing is eligible for garbage collection: + +``` +A -----> a <----- B + \--> b | + c <--/ +``` + +Manifest B is deleted via the API: + +``` +A -----> a B + \--> b + c +``` + +In this state layer `c` no longer has a reference and is eligible for garbage +collection. Layer `a` had one reference removed but not garbage +collected as it is still referenced by manifest `A`. The blob representing +manifest `B` is eligible for garbage collection. + +After garbage collection has been run, manifest `A` and its blobs remain. + +``` +A -----> a + \--> b +``` + + +### More details about garbage collection + +Garbage collection runs in two phases. First, in the 'mark' phase, the process +scans all the manifests in the registry. From these manifests, it constructs a +set of content address digests. This set is the 'mark set' and denotes the set +of blobs to *not* delete. Secondly, in the 'sweep' phase, the process scans all +the blobs and if a blob's content address digest is not in the mark set, the +process deletes it. + + +> **Note**: You should ensure that the registry is in read-only mode or not running at +> all. If you were to upload an image while garbage collection is running, there is the +> risk that the image's layers are mistakenly deleted leading to a corrupted image. + +This type of garbage collection is known as stop-the-world garbage collection. + +## Run garbage collection + +Garbage collection can be run as follows + +`bin/registry garbage-collect [--dry-run] /path/to/config.yml` + +The garbage-collect command accepts a `--dry-run` parameter, which prints the progress +of the mark and sweep phases without removing any data. Running with a log level of `info` +gives a clear indication of items eligible for deletion. + +The config.yml file should be in the following format: + +``` +version: 0.1 +storage: + filesystem: + rootdirectory: /registry/data +``` + +_Sample output from a dry run garbage collection with registry log level set to `info`_ + +``` +hello-world +hello-world: marking manifest sha256:fea8895f450959fa676bcc1df0611ea93823a735a01205fd8622846041d0c7cf +hello-world: marking blob sha256:03f4658f8b782e12230c1783426bd3bacce651ce582a4ffb6fbbfa2079428ecb +hello-world: marking blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 +hello-world: marking configuration sha256:690ed74de00f99a7d00a98a5ad855ac4febd66412be132438f9b8dbd300a937d +ubuntu + +4 blobs marked, 5 blobs eligible for deletion +blob eligible for deletion: sha256:28e09fddaacbfc8a13f82871d9d66141a6ed9ca526cb9ed295ef545ab4559b81 +blob eligible for deletion: sha256:7e15ce58ccb2181a8fced7709e9893206f0937cc9543bc0c8178ea1cf4d7e7b5 +blob eligible for deletion: sha256:87192bdbe00f8f2a62527f36bb4c7c7f4eaf9307e4b87e8334fb6abec1765bcb +blob eligible for deletion: sha256:b549a9959a664038fc35c155a95742cf12297672ca0ae35735ec027d55bf4e97 +blob eligible for deletion: sha256:f251d679a7c61455f06d793e43c06786d7766c88b8c24edf242b2c08e3c3f599 +```