TrueCloudLab/distribution
13
0
Fork 3
Code Issues 2 Pull requests Projects Releases 2 Packages Wiki Activity Actions

feature/integration_with_frostfs #1

Merged
alexvanin merged 1 commit from :feature/integration_with_frostfs into tcl/master 2024-09-04 19:51:22 +00:00
Conversation 5 Commits 1 Files changed 8 +2392 -85
r.loginov commented 2023-08-30 06:38:53 +00:00
Member
Copy link

Signed-off-by: Roman Loginov r.loginov@yadro.com

Signed-off-by: Roman Loginov r.loginov@yadro.com
r.loginov force-pushed feature/integration_with_frostfs from 82139145ef to 972e99bf1e 2023-08-30 06:45:51 +00:00 Compare
r.loginov requested review from dkirillov 2023-08-30 07:23:51 +00:00
r.loginov force-pushed feature/integration_with_frostfs from 972e99bf1e to 472e2dfac6 2023-08-31 06:45:55 +00:00 Compare
dkirillov reviewed 2023-09-01 13:13:00 +00:00
go.mod Outdated
@ -32,3 +36,4 @@
gopkg.in/yaml.v2 v2.4.0
)
require (
dkirillov commented 2023-09-01 13:08:35 +00:00
Member
Copy link

Let's keep just two require section

Let's keep just two `require` section
👍 1
dkirillov commented 2023-09-01 13:13:39 +00:00
Member
Copy link

We have to register frostfs driver in https://git.frostfs.info/r.loginov/distribution/src/branch/feature/integration_with_frostfs/cmd/registry/main.go

We have to register `frostfs` driver in https://git.frostfs.info/r.loginov/distribution/src/branch/feature/integration_with_frostfs/cmd/registry/main.go
👍 1
dkirillov commented 2023-09-01 13:16:42 +00:00
Member
Copy link

Did you try to upload image to distribution using frostfs as driver?

I got the following error

ERRO[0048] response completed with error                 environment=development err.code=unknown err.detail="frostfs: couldn't parse split id as upload uuid '': invalid UUID length: 0" err.message="unknown error" go.version=go1.20.4 http.request.host="localhost:5000" http.request.id=c9e7d318-a4c0-4041-8bf8-6633f69efaa6 http.request.method=POST http.request.remoteaddr="[::1]:59248" http.request.uri=/v2/truecloudlab/frostfs-storage/blobs/uploads/ http.request.useragent="docker/20.10.5+dfsg1 go/go1.15.15 git-commit/363e9a8 kernel/6.4.5-x64v1-xanmod1 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5+dfsg1 \\(linux\\))" http.response.contenttype=application/json http.response.duration=67.028766ms http.response.status=500 http.response.written=106 instance.id=539b52fc-fe49-4238-8013-159d9da79d46 service=registry vars.name=truecloudlab/frostfs-storage version=v3.0.0+unknown

I ran distribution with the following config

version: 0.1
log:
  level: debug
  fields:
    service: registry
    environment: development
  hooks:
    - type: mail
      disabled: true
      levels:
        - panic
      options:
        smtp:
          addr: mail.example.com:25
          username: mailuser
          password: password
          insecure: true
        from: sender@example.com
        to:
          - errors@example.com
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  maintenance:
    uploadpurging:
      enabled: false

  frostfs:
    wallet:
      path: /home/denis/github/tcl/frostfs-dev-env/wallets/wallet.json
      password: ""
    peers:
      0:
        address: s01.frostfs.devenv:8080
        weight: 1
        priority: 1
      1:
        address: s02.frostfs.devenv:8080
        weight: 1
        priority: 1
      2:
        address: s03.frostfs.devenv:8080
        weight: 1
        priority: 1
      3:
        address: s04.frostfs.devenv:8080
        weight: 1
        priority: 1
    # container can be nicename (rpc_endpoint is required)
    container: 6horME4kUEUnN5bweqmLSU5YdYfkUw3kb24Y3FfDc76H
    # the following params are optional
    session_expiration_duration: 1000 # in blocks
    connection_timeout: 5s
    request_timeout: 5s
    rebalance_interval: 30s
    rpc_endpoint: http://morph_chain.frostfs.devenv:30333
http:
  addr: :5000
  debug:
    addr: :5001
    prometheus:
      enabled: true
      path: /metrics
  headers:
    X-Content-Type-Options: [ nosniff ]
redis:
  addr: localhost:6379
  pool:
    maxidle: 16
    maxactive: 64
    idletimeout: 300s
  dialtimeout: 10ms
  readtimeout: 10ms
  writetimeout: 10ms
notifications:
  events:
    includereferences: true
  endpoints:
    - name: local-5003
      url: http://localhost:5003/callback
      headers:
        Authorization: [ Bearer <an example token> ]
      timeout: 1s
      threshold: 10
      backoff: 1s
      disabled: true
    - name: local-8083
      url: http://localhost:8083/callback
      timeout: 1s
      threshold: 10
      backoff: 1s
      disabled: true
health:
  storagedriver:
    enabled: true
    interval: 30s
    threshold: 3

And tried to push image

$ docker tag truecloudlab/frostfs-storage:365a7ca0 localhost:5000/truecloudlab/frostfs-storage:365a7ca0
$ docker push localhost:5000/truecloudlab/frostfs-storage:365a7ca0
Did you try to upload image to distribution using frostfs as driver? I got the following error ``` ERRO[0048] response completed with error environment=development err.code=unknown err.detail="frostfs: couldn't parse split id as upload uuid '': invalid UUID length: 0" err.message="unknown error" go.version=go1.20.4 http.request.host="localhost:5000" http.request.id=c9e7d318-a4c0-4041-8bf8-6633f69efaa6 http.request.method=POST http.request.remoteaddr="[::1]:59248" http.request.uri=/v2/truecloudlab/frostfs-storage/blobs/uploads/ http.request.useragent="docker/20.10.5+dfsg1 go/go1.15.15 git-commit/363e9a8 kernel/6.4.5-x64v1-xanmod1 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5+dfsg1 \\(linux\\))" http.response.contenttype=application/json http.response.duration=67.028766ms http.response.status=500 http.response.written=106 instance.id=539b52fc-fe49-4238-8013-159d9da79d46 service=registry vars.name=truecloudlab/frostfs-storage version=v3.0.0+unknown ``` I ran distribution with the following config ```yaml version: 0.1 log: level: debug fields: service: registry environment: development hooks: - type: mail disabled: true levels: - panic options: smtp: addr: mail.example.com:25 username: mailuser password: password insecure: true from: sender@example.com to: - errors@example.com storage: delete: enabled: true cache: blobdescriptor: inmemory maintenance: uploadpurging: enabled: false frostfs: wallet: path: /home/denis/github/tcl/frostfs-dev-env/wallets/wallet.json password: "" peers: 0: address: s01.frostfs.devenv:8080 weight: 1 priority: 1 1: address: s02.frostfs.devenv:8080 weight: 1 priority: 1 2: address: s03.frostfs.devenv:8080 weight: 1 priority: 1 3: address: s04.frostfs.devenv:8080 weight: 1 priority: 1 # container can be nicename (rpc_endpoint is required) container: 6horME4kUEUnN5bweqmLSU5YdYfkUw3kb24Y3FfDc76H # the following params are optional session_expiration_duration: 1000 # in blocks connection_timeout: 5s request_timeout: 5s rebalance_interval: 30s rpc_endpoint: http://morph_chain.frostfs.devenv:30333 http: addr: :5000 debug: addr: :5001 prometheus: enabled: true path: /metrics headers: X-Content-Type-Options: [ nosniff ] redis: addr: localhost:6379 pool: maxidle: 16 maxactive: 64 idletimeout: 300s dialtimeout: 10ms readtimeout: 10ms writetimeout: 10ms notifications: events: includereferences: true endpoints: - name: local-5003 url: http://localhost:5003/callback headers: Authorization: [ Bearer <an example token> ] timeout: 1s threshold: 10 backoff: 1s disabled: true - name: local-8083 url: http://localhost:8083/callback timeout: 1s threshold: 10 backoff: 1s disabled: true health: storagedriver: enabled: true interval: 30s threshold: 3 ``` And tried to push image ```shell $ docker tag truecloudlab/frostfs-storage:365a7ca0 localhost:5000/truecloudlab/frostfs-storage:365a7ca0 $ docker push localhost:5000/truecloudlab/frostfs-storage:365a7ca0 ```
👍 1
dkirillov commented 2023-09-01 13:21:54 +00:00
Member
Copy link

I got the following error

Oh, I see. It seems we have to adopt the following fix for linkedblobstore.go 1ba38226ab (diff-1c30d8b7d3e65b3e66719176279abd0dd0bbb8402ac0482fc3cc6523e65520c4)

> I got the following error Oh, I see. It seems we have to adopt the following fix for `linkedblobstore.go` https://github.com/nspcc-dev/distribution/commit/1ba38226abd948e47e237ed13b49451d1990819b#diff-1c30d8b7d3e65b3e66719176279abd0dd0bbb8402ac0482fc3cc6523e65520c4
👍 1
dkirillov reviewed 2023-09-01 13:32:54 +00:00
registry/storage/driver/frostfs/transformer/types.go Outdated
@ -0,0 +1,111 @@
package transformer
dkirillov commented 2023-09-01 13:32:54 +00:00
Member
Copy link

It seems we can drop this file and use types from here

It seems we can drop this file and use types from [here](https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/src/commit/84e7e69f98ac8c0d6ef09fb90bb60ba035aade17/object/transformer/types.go#L13)
👍 1
dkirillov reviewed 2023-09-01 13:47:19 +00:00
registry/storage/driver/frostfs/fileWriter.go Outdated
@ -0,0 +1,461 @@
package frostfs
dkirillov commented 2023-09-01 13:47:19 +00:00
Member
Copy link

Probably, we could try to rewrite this to be more like 84e7e69f98/pool/object_put_pool_transformer.go or do this in follow up issues

Probably, we could try to rewrite this to be more like https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/src/commit/84e7e69f98ac8c0d6ef09fb90bb60ba035aade17/pool/object_put_pool_transformer.go or do this in follow up issues
dkirillov reviewed 2023-09-01 13:47:52 +00:00
dkirillov left a comment
Member
Copy link

Overall looks good.

As follow up we could consider using tree service

Overall looks good. As follow up we could consider using tree service
👍 1
r.loginov force-pushed feature/integration_with_frostfs from 472e2dfac6 to 9c08a6ecdf 2023-09-04 12:39:15 +00:00 Compare
r.loginov force-pushed feature/integration_with_frostfs from 9c08a6ecdf to 5c26d0c9c1 2023-09-05 06:41:12 +00:00 Compare
r.loginov changed title from WIP: feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-08 07:17:21 +00:00
dkirillov approved these changes 2023-09-20 13:14:09 +00:00
dkirillov commented 2023-09-20 13:16:07 +00:00
Member
Copy link

Maybe we should create new branch and merge to that new branch rather than upstream main? @alexvanin

Maybe we should create new branch and merge to that new branch rather than upstream main? @alexvanin
alexvanin commented 2023-09-20 14:38:56 +00:00
Owner
Copy link

@dkirillov Sure, like we did for certificates and lego.

@dkirillov Sure, like we did for [certificates](https://git.frostfs.info/TrueCloudLab/certificates) and [lego](https://git.frostfs.info/TrueCloudLab/lego).
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:40:40 +00:00
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:41:05 +00:00
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:42:06 +00:00
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:42:46 +00:00
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:44:18 +00:00
alexvanin changed target branch from main to tcl/master 2023-09-20 14:45:24 +00:00
r.loginov changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:47:22 +00:00
r.loginov changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:47:36 +00:00
alexvanin merged commit 5c26d0c9c1 into tcl/master 2023-09-20 14:48:53 +00:00
alexvanin deleted branch feature/integration_with_frostfs 2023-09-20 14:48:53 +00:00
alexvanin referenced this pull request from a commit 2023-09-20 14:48:55 +00:00
[#1] Remove vendor directory
alexvanin referenced this pull request from a commit 2023-09-20 14:48:55 +00:00
[#1] Add frostfs driver
r.loginov referenced this pull request from a commit 2024-04-03 13:36:14 +00:00
[#1] Remove vendor directory
r.loginov referenced this pull request from a commit 2024-04-03 13:36:14 +00:00
[#1] Add frostfs driver
Sign in to join this conversation.
Reviewers
No reviewers
dkirillov
Labels
Clear labels
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/distribution#1
No description provided.
Delete branch ":feature/integration_with_frostfs"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?