TrueCloudLab/distribution
13
0
Fork 3
Code Issues 3 Pull requests Projects Releases 2 Packages Wiki Activity Actions

Change the filter operation when search objects #5

Merged
alexvanin merged 1 commit from :fix/delete_image into tcl/master 2024-01-31 11:28:26 +00:00
Conversation 0 Commits 1 Files changed 2 +26 -4
r.loginov commented 2024-01-25 06:33:33 +00:00
Member
Copy link

Close #4

Description of the problem:
The garbage collector was unable to delete data from containers after the image was deleted. This was due to the fact that the object deletion function of the frostfs driver did not delete objects recursively, that is, it did not delete objects in the path and its subpaths.

In fact, because of this, incorrect deletion of objects occurred not only at the garbage collection stage, but also at the object loading stage (after the image download is completed, temporary image loading objects are deleted), as well as at the stage of removing the object from the registry via the API (or another way).

Steps to verify the deletion functionality:

Step 1 - Create a container:

$ frostfs-cli container create --policy "REP 2" -r s01.frostfs.devenv:8080 -w services/storage/wallet01.json --await --name reg-cont
Enter password > 
container ID: G75iyQeDBBn5wWqR7SGQUqn5a6PckiwpAHNicDw5NxxT
awaiting...
container has been persisted on sidechain

Step 2 - Add the container's cid to the cmd/registry/config-dev-frostfs.yml config. The final config for running distribution with dev-env looks like this:

version: 0.1  
log:  
  level: info  
  fields:  
    service: registry  
    environment: development  
  hooks:  
    - type: mail  
      disabled: true  
      levels:  
        - panic  
      options:  
        smtp:  
          addr: mail.example.com:25  
          username: mailuser  
          password: password  
          insecure: true  
        from: sender@example.com  
        to:  
          - errors@example.com  
storage:  
  delete:  
    enabled: true  
  cache:  
    blobdescriptor: inmemory  
  maintenance:  
    uploadpurging:  
      enabled: false  
  
  frostfs:  
    wallet:  
      path: /home/roma/dev/public/upstream/frostfs-dev-env/services/storage/wallet01.json  
      password: ""  
    peers:  
      0:  
        address: s01.frostfs.devenv:8080  
        weight: 1  
        priority: 1  
      1:  
        address: s02.frostfs.devenv:8080  
        weight: 1  
        priority: 1  
      2:  
        address: s03.frostfs.devenv:8080  
        weight: 1  
        priority: 1  
      3:  
        address: s04.frostfs.devenv:8080  
        weight: 1  
        priority: 1  
    # container can be nicename (rpc_endpoint is required)  
    container: G75iyQeDBBn5wWqR7SGQUqn5a6PckiwpAHNicDw5NxxT  
    # the following params are optional  
    session_expiration_duration: 1000 # in blocks  
    connection_timeout: 5s  
    request_timeout: 5s  
    rebalance_interval: 30s  
    rpc_endpoint: http://morph-chain.frostfs.devenv:30333  
http:  
  addr: :5000  
  debug:  
    addr: :5001  
    prometheus:  
      enabled: true  
      path: /metrics  
  headers:  
    X-Content-Type-Options: [ nosniff ]  
redis:  
  addr: localhost:6379  
  pool:  
    maxidle: 16  
    maxactive: 64  
    idletimeout: 300s  
  dialtimeout: 10ms  
  readtimeout: 10ms  
  writetimeout: 10ms  
notifications:  
  events:  
    includereferences: true  
  endpoints:  
    - name: local-5003  
      url: http://localhost:5003/callback  
      headers:  
        Authorization: [ Bearer <an example token> ]  
      timeout: 1s  
      threshold: 10  
      backoff: 1s  
      disabled: true  
    - name: local-8083  
      url: http://localhost:8083/callback  
      timeout: 1s  
      threshold: 10  
      backoff: 1s  
      disabled: true  
health:  
  storagedriver:  
    enabled: true  
    interval: 30s  
    threshold: 3

Step 3 - Launch distribution:

$ ./bin/registry serve cmd/registry/config-dev-frostfs.yml

Step 4 - Push the image to the registry:

$ docker push localhost:5000/alpine:latest
The push refers to repository [localhost:5000/alpine]
4693057ce236: Pushed 
latest: digest: sha256:c5c5fda71656f28e49ac9c5416b3643eaa6a108a8093151d6d1afc9463be8e33 size: 528

Step 5 - Delete the image using the digest API:

$ curl -i -X DELETE http://localhost:5000/v2/alpine/manifests/sha256:c5c5fda71656f28e49ac9c5416b3643eaa6a108a8093151d6d1afc9463be8e33
HTTP/1.1 202 Accepted
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Thu, 25 Jan 2024 05:02:53 GMT
Content-Length: 0

Step 6 - Start the garbage collector:

$ ./registry garbage-collect ../cmd/registry/config-dev-frostfs.yml

0 blobs marked, 3 blobs and 0 manifests eligible for deletion
blob eligible for deletion: sha256:c5c5fda71656f28e49ac9c5416b3643eaa6a108a8093151d6d1afc9463be8e33
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/c5/c5c5fda71656f28e49ac9c5416b3643eaa6a108a8093151d6d1afc9463be8e33  environment=development go.version=go1.21.4 instance.id=616a70a0-5564-4fe5-a250-35167326d645 service=registry
blob eligible for deletion: sha256:7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/72/7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de  environment=development go.version=go1.21.4 instance.id=616a70a0-5564-4fe5-a250-35167326d645 service=registry
blob eligible for deletion: sha256:7e01a0d0a1dcd9e539f8e9bbd80106d59efbdf97293b3d38f5d7a34501526cdb
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/7e/7e01a0d0a1dcd9e539f8e9bbd80106d59efbdf97293b3d38f5d7a34501526cdb  environment=development go.version=go1.21.4 instance.id=616a70a0-5564-4fe5-a250-35167326d645 service=registry

When we call the garbage collector again, we will no longer see such an output, which indicates that the garbage collector has managed to delete the data:

./registry garbage-collect ../cmd/registry/config-dev-frostfs.yml

0 blobs marked, 0 blobs and 0 manifests eligible for deletion

We can also see how many objects are left in the container after deletion:

frostfs-cli container list-objects --cid G75iyQeDBBn5wWqR7SGQUqn5a6PckiwpAHNicDw5NxxT -r s01.frostfs.devenv:8080 -w services/storage/wallet01.json

It is important to note that some objects may remain in the container, namely objects indicating symbolic links to image layers - this is normal (for such objects FilePath contains '_layers' and FileName = link)

The same sequence of steps can be done with a bucket, if in step 1 you create a bucket like this:

aws --no-verify-ssl s3api create-bucket --bucket docker-registry-1 --acl public-read-write --endpoint https://s3.frostfs.devenv:8080

Signed-off-by: Roman Loginov r.loginov@yadro.com

Close #4 **Description of the problem:** The garbage collector was unable to delete data from containers after the image was deleted. This was due to the fact that the object deletion function of the frostfs driver did not delete objects recursively, that is, it did not delete objects in the path and its subpaths. In fact, because of this, incorrect deletion of objects occurred not only at the garbage collection stage, but also at the object loading stage (after the image download is completed, temporary image loading objects are deleted), as well as at the stage of removing the object from the registry via the API (or another way). **Steps to verify the deletion functionality:** Step 1 - Create a container: ```shell $ frostfs-cli container create --policy "REP 2" -r s01.frostfs.devenv:8080 -w services/storage/wallet01.json --await --name reg-cont Enter password > container ID: G75iyQeDBBn5wWqR7SGQUqn5a6PckiwpAHNicDw5NxxT awaiting... container has been persisted on sidechain ``` Step 2 - Add the container's cid to the cmd/registry/config-dev-frostfs.yml config. The final config for running distribution with dev-env looks like this: ```yaml version: 0.1 log: level: info fields: service: registry environment: development hooks: - type: mail disabled: true levels: - panic options: smtp: addr: mail.example.com:25 username: mailuser password: password insecure: true from: sender@example.com to: - errors@example.com storage: delete: enabled: true cache: blobdescriptor: inmemory maintenance: uploadpurging: enabled: false frostfs: wallet: path: /home/roma/dev/public/upstream/frostfs-dev-env/services/storage/wallet01.json password: "" peers: 0: address: s01.frostfs.devenv:8080 weight: 1 priority: 1 1: address: s02.frostfs.devenv:8080 weight: 1 priority: 1 2: address: s03.frostfs.devenv:8080 weight: 1 priority: 1 3: address: s04.frostfs.devenv:8080 weight: 1 priority: 1 # container can be nicename (rpc_endpoint is required) container: G75iyQeDBBn5wWqR7SGQUqn5a6PckiwpAHNicDw5NxxT # the following params are optional session_expiration_duration: 1000 # in blocks connection_timeout: 5s request_timeout: 5s rebalance_interval: 30s rpc_endpoint: http://morph-chain.frostfs.devenv:30333 http: addr: :5000 debug: addr: :5001 prometheus: enabled: true path: /metrics headers: X-Content-Type-Options: [ nosniff ] redis: addr: localhost:6379 pool: maxidle: 16 maxactive: 64 idletimeout: 300s dialtimeout: 10ms readtimeout: 10ms writetimeout: 10ms notifications: events: includereferences: true endpoints: - name: local-5003 url: http://localhost:5003/callback headers: Authorization: [ Bearer <an example token> ] timeout: 1s threshold: 10 backoff: 1s disabled: true - name: local-8083 url: http://localhost:8083/callback timeout: 1s threshold: 10 backoff: 1s disabled: true health: storagedriver: enabled: true interval: 30s threshold: 3 ``` Step 3 - Launch distribution: ```shell $ ./bin/registry serve cmd/registry/config-dev-frostfs.yml ``` Step 4 - Push the image to the registry: ```shell $ docker push localhost:5000/alpine:latest The push refers to repository [localhost:5000/alpine] 4693057ce236: Pushed latest: digest: sha256:c5c5fda71656f28e49ac9c5416b3643eaa6a108a8093151d6d1afc9463be8e33 size: 528 ``` Step 5 - Delete the image using the digest API: ```shell $ curl -i -X DELETE http://localhost:5000/v2/alpine/manifests/sha256:c5c5fda71656f28e49ac9c5416b3643eaa6a108a8093151d6d1afc9463be8e33 HTTP/1.1 202 Accepted Docker-Distribution-Api-Version: registry/2.0 X-Content-Type-Options: nosniff Date: Thu, 25 Jan 2024 05:02:53 GMT Content-Length: 0 ``` Step 6 - Start the garbage collector: ```shell $ ./registry garbage-collect ../cmd/registry/config-dev-frostfs.yml 0 blobs marked, 3 blobs and 0 manifests eligible for deletion blob eligible for deletion: sha256:c5c5fda71656f28e49ac9c5416b3643eaa6a108a8093151d6d1afc9463be8e33 INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/c5/c5c5fda71656f28e49ac9c5416b3643eaa6a108a8093151d6d1afc9463be8e33 environment=development go.version=go1.21.4 instance.id=616a70a0-5564-4fe5-a250-35167326d645 service=registry blob eligible for deletion: sha256:7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/72/7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de environment=development go.version=go1.21.4 instance.id=616a70a0-5564-4fe5-a250-35167326d645 service=registry blob eligible for deletion: sha256:7e01a0d0a1dcd9e539f8e9bbd80106d59efbdf97293b3d38f5d7a34501526cdb INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/7e/7e01a0d0a1dcd9e539f8e9bbd80106d59efbdf97293b3d38f5d7a34501526cdb environment=development go.version=go1.21.4 instance.id=616a70a0-5564-4fe5-a250-35167326d645 service=registry ``` When we call the garbage collector again, we will no longer see such an output, which indicates that the garbage collector has managed to delete the data: ```shell ./registry garbage-collect ../cmd/registry/config-dev-frostfs.yml 0 blobs marked, 0 blobs and 0 manifests eligible for deletion ``` We can also see how many objects are left in the container after deletion: ```shell frostfs-cli container list-objects --cid G75iyQeDBBn5wWqR7SGQUqn5a6PckiwpAHNicDw5NxxT -r s01.frostfs.devenv:8080 -w services/storage/wallet01.json ``` It is important to note that some objects may remain in the container, namely objects indicating symbolic links to image layers - this is normal (for such objects FilePath contains '\_layers' and FileName = link) The same sequence of steps can be done with a bucket, if in step 1 you create a bucket like this: ```shell aws --no-verify-ssl s3api create-bucket --bucket docker-registry-1 --acl public-read-write --endpoint https://s3.frostfs.devenv:8080 ``` Signed-off-by: Roman Loginov <r.loginov@yadro.com>
r.loginov added the
bug
 label 2024-01-25 06:33:33 +00:00
r.loginov self-assigned this 2024-01-25 06:33:33 +00:00
r.loginov force-pushed fix/delete_image from b03c45115e to 88f2546136 2024-01-25 06:36:17 +00:00 Compare
r.loginov requested review from alexvanin 2024-01-25 06:59:53 +00:00
r.loginov requested review from dkirillov 2024-01-25 06:59:53 +00:00
r.loginov requested review from storage-services-committers 2024-01-25 07:00:05 +00:00
r.loginov requested review from storage-services-developers 2024-01-25 07:00:06 +00:00
dkirillov reviewed 2024-01-25 08:57:41 +00:00
registry/storage/driver/frostfs/frostfs.go Outdated
@ -700,3 +700,3 @@
filters := object.NewSearchFilters()
filters.AddRootFilter()
filters.AddFilter(attributeFilePath, path, object.MatchStringEqual)
filters.AddFilter(attributeFilePath, path, object.MatchCommonPrefix)
dkirillov commented 2024-01-25 08:57:41 +00:00
Member
Copy link

Shouldn't we also do the same then ?

Shouldn't we also do [the same](https://git.frostfs.info/TrueCloudLab/distribution/src/commit/5c26d0c9c17e4149415a020579478cd0b97e438b/registry/storage/driver/s3-aws/s3.go#L944-L947) then ?
dkirillov marked this conversation as resolved
r.loginov force-pushed fix/delete_image from 88f2546136 to 101fea3485 2024-01-29 13:42:19 +00:00 Compare
dkirillov reviewed 2024-01-30 06:58:18 +00:00
registry/storage/driver/frostfs/frostfs.go Outdated
@ -733,0 +748,4 @@
}
fileInf := newFileInfo(ctx, obj, "")
if fileInf.Path() == path || fileInf.Path()[len(path)] == '/' {
dkirillov commented 2024-01-30 06:58:18 +00:00
Member
Copy link

If something goes wrong (e.g. in newFileInfo) we theoretically can get panic in fileInf.Path()[len(path)]. So we have to use fileInf.Path() <= path.

Also we can write

fileInf := newFileInfo(ctx, obj, "")
return fileInf.Path() <= path || fileInf.Path()[len(path)] == '/', nil

And finally let's keep the similar comment
// Check if a key is a subpath (so that deleting "/a" does not delete "/ab").

If something goes wrong (e.g. in `newFileInfo`) we theoretically can get panic in `fileInf.Path()[len(path)]`. So we have to use `fileInf.Path() <= path`. Also we can write ```golang fileInf := newFileInfo(ctx, obj, "") return fileInf.Path() <= path || fileInf.Path()[len(path)] == '/', nil ``` And finally let's keep the similar comment `// Check if a key is a subpath (so that deleting "/a" does not delete "/ab").`
dkirillov marked this conversation as resolved
r.loginov force-pushed fix/delete_image from 101fea3485 to 42eafd1fff 2024-01-30 09:03:58 +00:00 Compare
dkirillov approved these changes 2024-01-30 11:51:49 +00:00
dkirillov left a comment
Member
Copy link

LGTM

LGTM
alexvanin approved these changes 2024-01-31 11:28:07 +00:00
alexvanin merged commit 42eafd1fff into tcl/master 2024-01-31 11:28:25 +00:00
alexvanin deleted branch fix/delete_image 2024-01-31 11:28:35 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
dkirillov
alexvanin
TrueCloudLab/storage-services-developers
Labels
Clear labels
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
r.loginov
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/distribution#5
No description provided.
Delete branch ":fix/delete_image"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?