From 7e6b4e8c523a9352f0823dea16a998ca12816dd5 Mon Sep 17 00:00:00 2001 From: Stephen J Day Date: Thu, 4 Jun 2015 19:57:40 -0700 Subject: [PATCH] Add description of digests to API specification Signed-off-by: Stephen J Day --- docs/spec/api.md | 84 +++++++++++++++++++++++++++++++++++++++++++ docs/spec/api.md.tmpl | 84 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 168 insertions(+) diff --git a/docs/spec/api.md b/docs/spec/api.md index 9de340466..b18286f96 100644 --- a/docs/spec/api.md +++ b/docs/spec/api.md @@ -116,6 +116,12 @@ indicating what is different. Optionally, we may start marking parts of the specification to correspond with the versions enumerated here.
+
2.0.2
+
+
  • Added section covering digest format.
    • +
  • Added more clarification that manifest cannot be deleted by tag.
    • +
    +
    2.0.1
      @@ -238,6 +244,84 @@ When a `200 OK` or `401 Unauthorized` response is returned, the Clients may require this header value to determine if the endpoint serves this API. When this header is omitted, clients may fallback to an older API version. +### Content Digests + +This API design is driven heavily by [content addressability](http://en.wikipedia.org/wiki/Content-addressable_storage). +The core of this design is the concept of a content addressable identifier. It +uniquely identifies content by taking a collision-resistent hash of the bytes. +Such an identifier can be independently calculated and verified by selection +of a common _algorithm_. If such an identifier can be communicated in a secure +manner, one can retrieve the content from an insecure source, calculate it +independently and be certain that the correct content was obtained. Put simply, +the identifier is a property of the content. + +To disambiguate from other concepts, we call this identifier a _digest_. A +_digest_ is a serialized hash result, consisting of a _algorithm_ and _hex_ +portion. The _algorithm_ identifies the methodology used to calculate the +digest. The _hex_ portion is the hex-encoded result of the hash. + +We define a _digest_ string to match the following grammar: + + digest := algorithm ":" hex + algorithm := /[A-Fa-f0-9_+.-]+/ + hex := /[A-Fa-f0-9]+/ + +Some examples of _digests_ include the following: + +digest | description | +----------------------------------------------------------------------------------|------------------------------------------------ +sha256:6c3c624b58dbbcd3c0dd82b4c53f04194d1247c6eebdaab7c610cf7d66709b3b | Common sha256 based digest | +tarsum.v1+sha256:6c3c624b58dbbcd3c0dd82b4c53f04194d1247c6eebdaab7c610cf7d66709b3b | Tarsum digest, used for legacy layer digests. | + +> __NOTE:__ While we show an example of using a `tarsum` digest, the security +> of tarsum has not been verified. It is recommended that most implementations +> use sha256 for interoperability. + +While the _algorithm_ does allow one to implement a wide variety of +algorithms, compliant implementations should use sha256. Heavy processing of +input before calculating a hash is discouraged to avoid degrading the +uniqueness of the _digest_ but some canonicalization may be performed to +ensure consistent identifiers. + +Let's use a simple example in pseudo-code to demonstrate a digest calculation: + +``` +let C = 'a small string' +let B = sha256(C) +let D = 'sha256:' + EncodeHex(B) +let ID(C) = D +``` + +Above, we have bytestring _C_ passed into a function, _SHA256_, that returns a +bytestring B, which is the hash of _C_. _D_ gets the algorithm concatenated +with the hex encoding of _B_. We then define the identifier of _C_ to _ID(C)_ +as equal to _D_. A digest can be verified by independently calculating _D_ and +comparing it with identifier _ID(C)_ + +#### Digest Header + +To provide verification of http content, any response may include a `Docker- +Content-Digest` header. This will include the digest of the target entity +returned in the response. For blobs, this is the entire blob content. For +manifests, this is the manifest body without the signature content, also known +as the JWS payload. Note that the commonly used canonicalization for digest +calculation may be dependent on the mediatype of the content, such as with +manifests. + +The client may choose to ignore the header or may verify it to ensure content +integrity and transport security. This is most important when fetching by a +digest. To ensure security, the content should be verified against the digest +used to fetch the content. At times, the returned digest may differ from that +used to initiate a request. Such digests are considered to be from different +_domains_, meaning they have different values for _algorithm_. In such a case, +the client may choose to verify the digests in both domains or ignore the +server's digest. To maintain security, the client _must_ always verify the +content against the _digest_ used to fetch the content. + +> __IMPORTANT:__ If a _digest_ is used to fetch content, the client should use +> the same digest used to fetch the content to verify it. The header `Docker- +> Content-Digest` should not be trusted over the "local" digest. + ### Pulling An Image An "image" is a combination of a JSON manifest and individual layer files. The diff --git a/docs/spec/api.md.tmpl b/docs/spec/api.md.tmpl index 6f89c7d5c..b808a2ad2 100644 --- a/docs/spec/api.md.tmpl +++ b/docs/spec/api.md.tmpl @@ -116,6 +116,12 @@ indicating what is different. Optionally, we may start marking parts of the specification to correspond with the versions enumerated here.
      +
      2.0.2
      +
      +
    • Added section covering digest format.
      • +
    • Added more clarification that manifest cannot be deleted by tag.
      • +
      +
      2.0.1
        @@ -238,6 +244,84 @@ When a `200 OK` or `401 Unauthorized` response is returned, the Clients may require this header value to determine if the endpoint serves this API. When this header is omitted, clients may fallback to an older API version. +### Content Digests + +This API design is driven heavily by [content addressability](http://en.wikipedia.org/wiki/Content-addressable_storage). +The core of this design is the concept of a content addressable identifier. It +uniquely identifies content by taking a collision-resistent hash of the bytes. +Such an identifier can be independently calculated and verified by selection +of a common _algorithm_. If such an identifier can be communicated in a secure +manner, one can retrieve the content from an insecure source, calculate it +independently and be certain that the correct content was obtained. Put simply, +the identifier is a property of the content. + +To disambiguate from other concepts, we call this identifier a _digest_. A +_digest_ is a serialized hash result, consisting of a _algorithm_ and _hex_ +portion. The _algorithm_ identifies the methodology used to calculate the +digest. The _hex_ portion is the hex-encoded result of the hash. + +We define a _digest_ string to match the following grammar: + + digest := algorithm ":" hex + algorithm := /[A-Fa-f0-9_+.-]+/ + hex := /[A-Fa-f0-9]+/ + +Some examples of _digests_ include the following: + +digest | description | +----------------------------------------------------------------------------------|------------------------------------------------ +sha256:6c3c624b58dbbcd3c0dd82b4c53f04194d1247c6eebdaab7c610cf7d66709b3b | Common sha256 based digest | +tarsum.v1+sha256:6c3c624b58dbbcd3c0dd82b4c53f04194d1247c6eebdaab7c610cf7d66709b3b | Tarsum digest, used for legacy layer digests. | + +> __NOTE:__ While we show an example of using a `tarsum` digest, the security +> of tarsum has not been verified. It is recommended that most implementations +> use sha256 for interoperability. + +While the _algorithm_ does allow one to implement a wide variety of +algorithms, compliant implementations should use sha256. Heavy processing of +input before calculating a hash is discouraged to avoid degrading the +uniqueness of the _digest_ but some canonicalization may be performed to +ensure consistent identifiers. + +Let's use a simple example in pseudo-code to demonstrate a digest calculation: + +``` +let C = 'a small string' +let B = sha256(C) +let D = 'sha256:' + EncodeHex(B) +let ID(C) = D +``` + +Above, we have bytestring _C_ passed into a function, _SHA256_, that returns a +bytestring B, which is the hash of _C_. _D_ gets the algorithm concatenated +with the hex encoding of _B_. We then define the identifier of _C_ to _ID(C)_ +as equal to _D_. A digest can be verified by independently calculating _D_ and +comparing it with identifier _ID(C)_ + +#### Digest Header + +To provide verification of http content, any response may include a `Docker- +Content-Digest` header. This will include the digest of the target entity +returned in the response. For blobs, this is the entire blob content. For +manifests, this is the manifest body without the signature content, also known +as the JWS payload. Note that the commonly used canonicalization for digest +calculation may be dependent on the mediatype of the content, such as with +manifests. + +The client may choose to ignore the header or may verify it to ensure content +integrity and transport security. This is most important when fetching by a +digest. To ensure security, the content should be verified against the digest +used to fetch the content. At times, the returned digest may differ from that +used to initiate a request. Such digests are considered to be from different +_domains_, meaning they have different values for _algorithm_. In such a case, +the client may choose to verify the digests in both domains or ignore the +server's digest. To maintain security, the client _must_ always verify the +content against the _digest_ used to fetch the content. + +> __IMPORTANT:__ If a _digest_ is used to fetch content, the client should use +> the same digest used to fetch the content to verify it. The header `Docker- +> Content-Digest` should not be trusted over the "local" digest. + ### Pulling An Image An "image" is a combination of a JSON manifest and individual layer files. The