Missing manifest after distribution image restart #14

New issue

Closed

opened 2024-10-23 13:07:34 +00:00 by alexvanin · 3 comments

alexvanin commented

2024-10-23 13:07:34 +00:00

Owner

Description

According to report, the issue is reproduced by using storage components from AIO v1.6. Distribution is running in docker environment with this config

log:
  fields:
    service: registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
    Access-Control-Allow-Origin: ['*']
    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
    Access-Control-Allow-Headers: ['Authorization', 'Accept', 'Cache-Control']
    Access-Control-Max-Age: [1728000]
    Access-Control-Allow-Credentials: [true]
    Access-Control-Expose-Headers: ['Docker-Content-Digest']

  tls:
    certificate: /certs/public.pem
    key: /certs/private.key
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
storage:
  redirect:
    disable: true
  frostfs:
    wallet:
      path: /opt/wallet.json
      password: "S3rv!ce"
    peers:
      0:
        address: 10.78.70.189:8080
        weight: 1
        priority: 1
      1:
        address: 10.78.70.190:8080
        weight: 1
        priority: 1
      2:
        address: 10.78.70.191:8080
        weight: 1
        priority: 1
      3:
        address: 10.78.70.192:8080
        weight: 1
        priority: 1
    container: A7jS9FwnNSz1hs24LUUSibMht8Hsw6vgZQvg5sXpdVSR
    session_expiration_duration: 1000 # in blocks
    connection_timeout: 5s
    request_timeout: 5s
    rebalance_interval: 30s
  delete:
    enabled: true
auth:
  htpasswd:
    realm: basic-realm
    path: /etc/docker/registry/auth/htpasswd

Reproduce

build distribution image
run distribution image with config
run docker push <repo>
delete local image
run docker pull <repo>
stop distribution image
clean distribution image volumes
run distribution image once again
delete local image
run docker pull <repo>

Fails on last step with

Error response from daemon: manifest for <repo>:11.0.0 not found: manifest unknown: manifest unknown

Expected behavior

No response

registry version

v3.0.0-beta.1

Additional Info

No response

### Description According to report, the issue is reproduced by using storage components from AIO v1.6. Distribution is running in docker environment with this config ``` log: fields: service: registry http: addr: :5000 headers: X-Content-Type-Options: [nosniff] Access-Control-Allow-Origin: ['*'] Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE'] Access-Control-Allow-Headers: ['Authorization', 'Accept', 'Cache-Control'] Access-Control-Max-Age: [1728000] Access-Control-Allow-Credentials: [true] Access-Control-Expose-Headers: ['Docker-Content-Digest'] tls: certificate: /certs/public.pem key: /certs/private.key health: storagedriver: enabled: true interval: 10s threshold: 3 storage: redirect: disable: true frostfs: wallet: path: /opt/wallet.json password: "S3rv!ce" peers: 0: address: 10.78.70.189:8080 weight: 1 priority: 1 1: address: 10.78.70.190:8080 weight: 1 priority: 1 2: address: 10.78.70.191:8080 weight: 1 priority: 1 3: address: 10.78.70.192:8080 weight: 1 priority: 1 container: A7jS9FwnNSz1hs24LUUSibMht8Hsw6vgZQvg5sXpdVSR session_expiration_duration: 1000 # in blocks connection_timeout: 5s request_timeout: 5s rebalance_interval: 30s delete: enabled: true auth: htpasswd: realm: basic-realm path: /etc/docker/registry/auth/htpasswd ``` ### Reproduce 1. build distribution image 2. run distribution image with config 3. run `docker push <repo>` 4. delete local image 5. run `docker pull <repo>` 6. stop distribution image 7. clean distribution image volumes 8. run distribution image once again 9. delete local image 10. run `docker pull <repo>` Fails on last step with ``` Error response from daemon: manifest for <repo>:11.0.0 not found: manifest unknown: manifest unknown ``` ### Expected behavior _No response_ ### registry version v3.0.0-beta.1 ### Additional Info _No response_

alexvanin changed title from ~~Missing ma~~ to Missing manifest after distribution image restart

2024-10-23 13:08:00 +00:00

alexvanin added the

bug

label 2024-10-23 13:08:09 +00:00

r.loginov was assigned by alexvanin

2024-10-24 08:15:47 +00:00

r.loginov commented

2024-10-28 12:13:51 +00:00

Member

So far, it is not possible to reproduce the problem. I would like to see the full list of commands according to the steps described above to reproduce the bugs.

r.loginov commented

2024-11-18 10:52:16 +00:00

Member

Between releases v3.0.0-alpha.1-cl.1 and v3.0.0-beta.1-cl.1, there was a change in the Dockerfile regarding the path to the config. As a result, if the v3.0.0-beta.1-cl.1 version is raised in the same environment as the v3.0.0-alpha.1-cl.1 release, then eventually a situation may occur that will use the default config to store images in memory and not in frostfs. If you specify the correct path to the config in the environment (as in the current Dockerfile), then there will be no problem.

Between releases v3.0.0-alpha.1-cl.1 and v3.0.0-beta.1-cl.1, there was a change in the [Dockerfile](https://git.frostfs.info/TrueCloudLab/distribution/commit/975613d4a0b01418f2869ec6f78a17c5b464f61a) regarding the path to the config. As a result, if the v3.0.0-beta.1-cl.1 version is raised in the same environment as the v3.0.0-alpha.1-cl.1 release, then eventually a situation may occur that will use the default config to store images in memory and not in frostfs. If you specify the correct path to the config in the environment (as in the current Dockerfile), then there will be no problem.

r.loginov commented

2024-11-18 10:53:26 +00:00

Member

The solution is in the comment #14 (comment)

The solution is in the comment https://git.frostfs.info/TrueCloudLab/distribution/issues/14#issuecomment-58199

r.loginov closed this issue

2024-11-18 10:53:33 +00:00