Garbage collector failed to remove data in container after deleting the image. #4

New Issue

Closed

opened 2024-01-16 09:56:05 +00:00 by petrovskyav · 0 comments

petrovskyav commented

2024-01-16 09:56:05 +00:00

Push, pull, and push to an existing repository works well. The garbage collector failed to remove data in container after deleting the image.

Steps to reproduce:

Build a Docker image of the registry. The code is taken from the tcl/master branch..
Use the following part of config file:

storage:
 frostfs:
    wallet:
      path: /opt/wallet.json
      password: "*******"
    peers:
      0:
        address: 192.168.110.10:8080
        weight: 1
        priority: 1
      1:
        address: 192.168.110.20:8080
        weight: 1
        priority: 1
      2:
        address: 192.168.110.30:8080
        weight: 1
        priority: 1
      3:
        address: 192.168.110.40:8080
        weight: 1
        priority: 1
    container: CoXEgpTasteej4BEPbDBxdSXFyRsnJ6hf9fTX1ATQFAc
    session_expiration_duration: 1000 # in blocks
    connection_timeout: 5s
    request_timeout: 5s
    rebalance_interval: 30s
    rpc_endpoint: http://192.168.111.10:40333

  delete:
    enabled: true

  cache:
    blobdescriptor: redis
    blobdescriptorsize: 10000

Push some images to the registry.
Delete images using api or some web interface (I used joxit/docker-registry-ui).
Run garbage collector using command docker exec -it 6f7e458ee453 /bin/registry garbage-collect /etc/docker/registry/config.yml.

If the container was created using the command frostfs-cli container create --policy 'REP 2' --name <container name> -r 127.0.0.1:8080 --await --wallet wallet.json, the garbage collector returns messages:

alpine
alpine: marking manifest sha256:13b7e62e8df80264dbb747995705a986aa530415763a6c58f84a3ca8af9a5bcd
alpine: marking blob sha256:f8c20f8bbcb684055b4fea470fdd169c86e87786940b3262335b12ec3adef418
alpine: marking blob sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca
centos
ubuntu
 
3 blobs marked, 6 blobs and 0 manifests eligible for deletion
blob eligible for deletion: sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1
INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/a1/a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1  go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry
blob eligible for deletion: sha256:f958ac6f7075e036cdd6f4c99fe128955a301bcc5da654cd5b6c088cf1a5ef98
INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/f9/f958ac6f7075e036cdd6f4c99fe128955a301bcc5da654cd5b6c088cf1a5ef98  go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry
blob eligible for deletion: sha256:174c8c134b2a94b5bb0b37d9a2b6ba0663d82d23ebf62bd51f74a2fd457333da
INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/17/174c8c134b2a94b5bb0b37d9a2b6ba0663d82d23ebf62bd51f74a2fd457333da  go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry
blob eligible for deletion: sha256:3dd181f9be599de628e1bc6d868d517125e07f968824bcf7b7ed8d28ad1026b1
INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/3d/3dd181f9be599de628e1bc6d868d517125e07f968824bcf7b7ed8d28ad1026b1  go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry
blob eligible for deletion: sha256:5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6
INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/5d/5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6  go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry
blob eligible for deletion: sha256:a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc
INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/a1/a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc  go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry

But the deletion does not occur. When we run it again we get the same output

If we created a public container using the command aws --no-verify-ssl s3api create-bucket --bucket docker-registry-1 --acl public-read-write --endpoint https://192.168.110.10:443, the garbage collector returns messages:

WARN[0000] couldn't get list object '5NsFQ1DAdgaKyjRDTTjw9eFeb9ponsCQ8UDawQy7PG8H' in path '/docker/registry/v2/repositories': head object via client 192.168.110.40:8080: read object header via client: status: code = 2052 message = object already removed  go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry
alpine
alpine: marking manifest sha256:13b7e62e8df80264dbb747995705a986aa530415763a6c58f84a3ca8af9a5bcd
alpine: marking blob sha256:f8c20f8bbcb684055b4fea470fdd169c86e87786940b3262335b12ec3adef418
alpine: marking blob sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca
WARN[0000] couldn't get list object '5NsFQ1DAdgaKyjRDTTjw9eFeb9ponsCQ8UDawQy7PG8H' in path '/docker/registry/v2/repositories/centos': head object via client 192.168.110.20:8080: read object header via client: status: code = 2052 message = object already removed  go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry
centos
centos: marking manifest sha256:a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc
centos: marking blob sha256:5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6
centos: marking blob sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1
ubuntu
 
6 blobs marked, 3 blobs and 0 manifests eligible for deletion
blob eligible for deletion: sha256:174c8c134b2a94b5bb0b37d9a2b6ba0663d82d23ebf62bd51f74a2fd457333da
INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/17/174c8c134b2a94b5bb0b37d9a2b6ba0663d82d23ebf62bd51f74a2fd457333da  go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry
blob eligible for deletion: sha256:3dd181f9be599de628e1bc6d868d517125e07f968824bcf7b7ed8d28ad1026b1
INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/3d/3dd181f9be599de628e1bc6d868d517125e07f968824bcf7b7ed8d28ad1026b1  go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry
blob eligible for deletion: sha256:f958ac6f7075e036cdd6f4c99fe128955a301bcc5da654cd5b6c088cf1a5ef98
INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/f9/f958ac6f7075e036cdd6f4c99fe128955a301bcc5da654cd5b6c088cf1a5ef98  go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry

The deletion does not occur. When we run it again we get the same output.

Push, pull, and push to an existing repository works well. The garbage collector failed to remove data in container after deleting the image. ### Steps to reproduce: 1. Build a Docker image of the registry. The code is taken from the tcl/master branch.. 2. Use the following part of config file: ``` storage: frostfs: wallet: path: /opt/wallet.json password: "*******" peers: 0: address: 192.168.110.10:8080 weight: 1 priority: 1 1: address: 192.168.110.20:8080 weight: 1 priority: 1 2: address: 192.168.110.30:8080 weight: 1 priority: 1 3: address: 192.168.110.40:8080 weight: 1 priority: 1 container: CoXEgpTasteej4BEPbDBxdSXFyRsnJ6hf9fTX1ATQFAc session_expiration_duration: 1000 # in blocks connection_timeout: 5s request_timeout: 5s rebalance_interval: 30s rpc_endpoint: http://192.168.111.10:40333 delete: enabled: true cache: blobdescriptor: redis blobdescriptorsize: 10000 ``` 3. Push some images to the registry. 4. Delete images using api or some web interface (I used joxit/docker-registry-ui). 5. Run garbage collector using command `docker exec -it 6f7e458ee453 /bin/registry garbage-collect /etc/docker/registry/config.yml`. If the container was created using the command `frostfs-cli container create --policy 'REP 2' --name <container name> -r 127.0.0.1:8080 --await --wallet wallet.json`, the garbage collector returns messages: ``` alpine alpine: marking manifest sha256:13b7e62e8df80264dbb747995705a986aa530415763a6c58f84a3ca8af9a5bcd alpine: marking blob sha256:f8c20f8bbcb684055b4fea470fdd169c86e87786940b3262335b12ec3adef418 alpine: marking blob sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca centos ubuntu 3 blobs marked, 6 blobs and 0 manifests eligible for deletion blob eligible for deletion: sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1 INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/a1/a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1 go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry blob eligible for deletion: sha256:f958ac6f7075e036cdd6f4c99fe128955a301bcc5da654cd5b6c088cf1a5ef98 INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/f9/f958ac6f7075e036cdd6f4c99fe128955a301bcc5da654cd5b6c088cf1a5ef98 go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry blob eligible for deletion: sha256:174c8c134b2a94b5bb0b37d9a2b6ba0663d82d23ebf62bd51f74a2fd457333da INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/17/174c8c134b2a94b5bb0b37d9a2b6ba0663d82d23ebf62bd51f74a2fd457333da go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry blob eligible for deletion: sha256:3dd181f9be599de628e1bc6d868d517125e07f968824bcf7b7ed8d28ad1026b1 INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/3d/3dd181f9be599de628e1bc6d868d517125e07f968824bcf7b7ed8d28ad1026b1 go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry blob eligible for deletion: sha256:5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6 INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/5d/5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6 go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry blob eligible for deletion: sha256:a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/a1/a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc go.version=go1.19.10 instance.id=ef3f83d1-dfa5-498e-8e06-b7b4fcfba820 service=registry ``` But the deletion does not occur. When we run it again we get the same output If we created a public container using the command `aws --no-verify-ssl s3api create-bucket --bucket docker-registry-1 --acl public-read-write --endpoint https://192.168.110.10:443`, the garbage collector returns messages: ``` WARN[0000] couldn't get list object '5NsFQ1DAdgaKyjRDTTjw9eFeb9ponsCQ8UDawQy7PG8H' in path '/docker/registry/v2/repositories': head object via client 192.168.110.40:8080: read object header via client: status: code = 2052 message = object already removed go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry alpine alpine: marking manifest sha256:13b7e62e8df80264dbb747995705a986aa530415763a6c58f84a3ca8af9a5bcd alpine: marking blob sha256:f8c20f8bbcb684055b4fea470fdd169c86e87786940b3262335b12ec3adef418 alpine: marking blob sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca WARN[0000] couldn't get list object '5NsFQ1DAdgaKyjRDTTjw9eFeb9ponsCQ8UDawQy7PG8H' in path '/docker/registry/v2/repositories/centos': head object via client 192.168.110.20:8080: read object header via client: status: code = 2052 message = object already removed go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry centos centos: marking manifest sha256:a1801b843b1bfaf77c501e7a6d3f709401a1e0c83863037fa3aab063a7fdb9dc centos: marking blob sha256:5d0da3dc976460b72c77d94c8a1ad043720b0416bfc16c52c45d4847e53fadb6 centos: marking blob sha256:a1d0c75327776413fa0db9ed3adcdbadedc95a662eb1d360dad82bb913f8a1d1 ubuntu 6 blobs marked, 3 blobs and 0 manifests eligible for deletion blob eligible for deletion: sha256:174c8c134b2a94b5bb0b37d9a2b6ba0663d82d23ebf62bd51f74a2fd457333da INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/17/174c8c134b2a94b5bb0b37d9a2b6ba0663d82d23ebf62bd51f74a2fd457333da go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry blob eligible for deletion: sha256:3dd181f9be599de628e1bc6d868d517125e07f968824bcf7b7ed8d28ad1026b1 INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/3d/3dd181f9be599de628e1bc6d868d517125e07f968824bcf7b7ed8d28ad1026b1 go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry blob eligible for deletion: sha256:f958ac6f7075e036cdd6f4c99fe128955a301bcc5da654cd5b6c088cf1a5ef98 INFO[0001] Deleting blob: /docker/registry/v2/blobs/sha256/f9/f958ac6f7075e036cdd6f4c99fe128955a301bcc5da654cd5b6c088cf1a5ef98 go.version=go1.19.10 instance.id=8b7f0fcf-3877-4e21-8019-b2762cdfc75b service=registry ``` The deletion does not occur. When we run it again we get the same output.