TrueCloudLab
/
distribution
13
0
Fork 3
Code Issues 2 Pull Requests Packages Projects Releases 1 Wiki Activity

feature/integration_with_frostfs #1

Merged
alexvanin merged 2 commits from :feature/integration_with_frostfs into tcl/master 2023-09-20 14:48:53 +00:00
Conversation 5 Commits 2 Files Changed 2178
r.loginov commented 2023-08-30 06:38:53 +00:00
Collaborator

Signed-off-by: Roman Loginov r.loginov@yadro.com

Signed-off-by: Roman Loginov r.loginov@yadro.com
r.loginov force-pushed feature/integration_with_frostfs from 82139145ef to 972e99bf1e 2023-08-30 06:45:51 +00:00 Compare
r.loginov requested review from dkirillov 2023-08-30 07:23:51 +00:00
r.loginov force-pushed feature/integration_with_frostfs from 972e99bf1e to 472e2dfac6 2023-08-31 06:45:55 +00:00 Compare
dkirillov reviewed 2023-09-01 13:13:00 +00:00
go.mod Outdated
@ -32,3 +36,4 @@
gopkg.in/yaml.v2 v2.4.0
)
require (
dkirillov commented 2023-09-01 13:08:35 +00:00
Collaborator

Let's keep just two require section

Let's keep just two `require` section
👍 1
dkirillov commented 2023-09-01 13:13:39 +00:00
Collaborator

We have to register frostfs driver in https://git.frostfs.info/r.loginov/distribution/src/branch/feature/integration_with_frostfs/cmd/registry/main.go

We have to register `frostfs` driver in https://git.frostfs.info/r.loginov/distribution/src/branch/feature/integration_with_frostfs/cmd/registry/main.go
👍 1
dkirillov commented 2023-09-01 13:16:42 +00:00
Collaborator

Did you try to upload image to distribution using frostfs as driver?

I got the following error

ERRO[0048] response completed with error                 environment=development err.code=unknown err.detail="frostfs: couldn't parse split id as upload uuid '': invalid UUID length: 0" err.message="unknown error" go.version=go1.20.4 http.request.host="localhost:5000" http.request.id=c9e7d318-a4c0-4041-8bf8-6633f69efaa6 http.request.method=POST http.request.remoteaddr="[::1]:59248" http.request.uri=/v2/truecloudlab/frostfs-storage/blobs/uploads/ http.request.useragent="docker/20.10.5+dfsg1 go/go1.15.15 git-commit/363e9a8 kernel/6.4.5-x64v1-xanmod1 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5+dfsg1 \\(linux\\))" http.response.contenttype=application/json http.response.duration=67.028766ms http.response.status=500 http.response.written=106 instance.id=539b52fc-fe49-4238-8013-159d9da79d46 service=registry vars.name=truecloudlab/frostfs-storage version=v3.0.0+unknown

I ran distribution with the following config

version: 0.1
log:
  level: debug
  fields:
    service: registry
    environment: development
  hooks:
    - type: mail
      disabled: true
      levels:
        - panic
      options:
        smtp:
          addr: mail.example.com:25
          username: mailuser
          password: password
          insecure: true
        from: sender@example.com
        to:
          - errors@example.com
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  maintenance:
    uploadpurging:
      enabled: false

  frostfs:
    wallet:
      path: /home/denis/github/tcl/frostfs-dev-env/wallets/wallet.json
      password: ""
    peers:
      0:
        address: s01.frostfs.devenv:8080
        weight: 1
        priority: 1
      1:
        address: s02.frostfs.devenv:8080
        weight: 1
        priority: 1
      2:
        address: s03.frostfs.devenv:8080
        weight: 1
        priority: 1
      3:
        address: s04.frostfs.devenv:8080
        weight: 1
        priority: 1
    # container can be nicename (rpc_endpoint is required)
    container: 6horME4kUEUnN5bweqmLSU5YdYfkUw3kb24Y3FfDc76H
    # the following params are optional
    session_expiration_duration: 1000 # in blocks
    connection_timeout: 5s
    request_timeout: 5s
    rebalance_interval: 30s
    rpc_endpoint: http://morph_chain.frostfs.devenv:30333
http:
  addr: :5000
  debug:
    addr: :5001
    prometheus:
      enabled: true
      path: /metrics
  headers:
    X-Content-Type-Options: [ nosniff ]
redis:
  addr: localhost:6379
  pool:
    maxidle: 16
    maxactive: 64
    idletimeout: 300s
  dialtimeout: 10ms
  readtimeout: 10ms
  writetimeout: 10ms
notifications:
  events:
    includereferences: true
  endpoints:
    - name: local-5003
      url: http://localhost:5003/callback
      headers:
        Authorization: [ Bearer <an example token> ]
      timeout: 1s
      threshold: 10
      backoff: 1s
      disabled: true
    - name: local-8083
      url: http://localhost:8083/callback
      timeout: 1s
      threshold: 10
      backoff: 1s
      disabled: true
health:
  storagedriver:
    enabled: true
    interval: 30s
    threshold: 3

And tried to push image

$ docker tag truecloudlab/frostfs-storage:365a7ca0 localhost:5000/truecloudlab/frostfs-storage:365a7ca0
$ docker push localhost:5000/truecloudlab/frostfs-storage:365a7ca0
Did you try to upload image to distribution using frostfs as driver? I got the following error ``` ERRO[0048] response completed with error environment=development err.code=unknown err.detail="frostfs: couldn't parse split id as upload uuid '': invalid UUID length: 0" err.message="unknown error" go.version=go1.20.4 http.request.host="localhost:5000" http.request.id=c9e7d318-a4c0-4041-8bf8-6633f69efaa6 http.request.method=POST http.request.remoteaddr="[::1]:59248" http.request.uri=/v2/truecloudlab/frostfs-storage/blobs/uploads/ http.request.useragent="docker/20.10.5+dfsg1 go/go1.15.15 git-commit/363e9a8 kernel/6.4.5-x64v1-xanmod1 os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.5+dfsg1 \\(linux\\))" http.response.contenttype=application/json http.response.duration=67.028766ms http.response.status=500 http.response.written=106 instance.id=539b52fc-fe49-4238-8013-159d9da79d46 service=registry vars.name=truecloudlab/frostfs-storage version=v3.0.0+unknown ``` I ran distribution with the following config ```yaml version: 0.1 log: level: debug fields: service: registry environment: development hooks: - type: mail disabled: true levels: - panic options: smtp: addr: mail.example.com:25 username: mailuser password: password insecure: true from: sender@example.com to: - errors@example.com storage: delete: enabled: true cache: blobdescriptor: inmemory maintenance: uploadpurging: enabled: false frostfs: wallet: path: /home/denis/github/tcl/frostfs-dev-env/wallets/wallet.json password: "" peers: 0: address: s01.frostfs.devenv:8080 weight: 1 priority: 1 1: address: s02.frostfs.devenv:8080 weight: 1 priority: 1 2: address: s03.frostfs.devenv:8080 weight: 1 priority: 1 3: address: s04.frostfs.devenv:8080 weight: 1 priority: 1 # container can be nicename (rpc_endpoint is required) container: 6horME4kUEUnN5bweqmLSU5YdYfkUw3kb24Y3FfDc76H # the following params are optional session_expiration_duration: 1000 # in blocks connection_timeout: 5s request_timeout: 5s rebalance_interval: 30s rpc_endpoint: http://morph_chain.frostfs.devenv:30333 http: addr: :5000 debug: addr: :5001 prometheus: enabled: true path: /metrics headers: X-Content-Type-Options: [ nosniff ] redis: addr: localhost:6379 pool: maxidle: 16 maxactive: 64 idletimeout: 300s dialtimeout: 10ms readtimeout: 10ms writetimeout: 10ms notifications: events: includereferences: true endpoints: - name: local-5003 url: http://localhost:5003/callback headers: Authorization: [ Bearer <an example token> ] timeout: 1s threshold: 10 backoff: 1s disabled: true - name: local-8083 url: http://localhost:8083/callback timeout: 1s threshold: 10 backoff: 1s disabled: true health: storagedriver: enabled: true interval: 30s threshold: 3 ``` And tried to push image ```shell $ docker tag truecloudlab/frostfs-storage:365a7ca0 localhost:5000/truecloudlab/frostfs-storage:365a7ca0 $ docker push localhost:5000/truecloudlab/frostfs-storage:365a7ca0 ```
👍 1
dkirillov commented 2023-09-01 13:21:54 +00:00
Collaborator

I got the following error

Oh, I see. It seems we have to adopt the following fix for linkedblobstore.go 1ba38226ab (diff-1c30d8b7d3e65b3e66719176279abd0dd0bbb8402ac0482fc3cc6523e65520c4)

> I got the following error Oh, I see. It seems we have to adopt the following fix for `linkedblobstore.go` https://github.com/nspcc-dev/distribution/commit/1ba38226abd948e47e237ed13b49451d1990819b#diff-1c30d8b7d3e65b3e66719176279abd0dd0bbb8402ac0482fc3cc6523e65520c4
👍 1
dkirillov reviewed 2023-09-01 13:32:54 +00:00
registry/storage/driver/frostfs/transformer/types.go Outdated
@ -0,0 +1,111 @@
package transformer
dkirillov commented 2023-09-01 13:32:54 +00:00
Collaborator

It seems we can drop this file and use types from here

It seems we can drop this file and use types from [here](https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/src/commit/84e7e69f98ac8c0d6ef09fb90bb60ba035aade17/object/transformer/types.go#L13)
👍 1
dkirillov reviewed 2023-09-01 13:47:19 +00:00
registry/storage/driver/frostfs/fileWriter.go Outdated
@ -0,0 +1,461 @@
package frostfs
dkirillov commented 2023-09-01 13:47:19 +00:00
Collaborator

Probably, we could try to rewrite this to be more like 84e7e69f98/pool/object_put_pool_transformer.go or do this in follow up issues

Probably, we could try to rewrite this to be more like https://git.frostfs.info/TrueCloudLab/frostfs-sdk-go/src/commit/84e7e69f98ac8c0d6ef09fb90bb60ba035aade17/pool/object_put_pool_transformer.go or do this in follow up issues
dkirillov reviewed 2023-09-01 13:47:52 +00:00
dkirillov left a comment
Collaborator

Overall looks good.

As follow up we could consider using tree service

Overall looks good. As follow up we could consider using tree service
👍 1
r.loginov force-pushed feature/integration_with_frostfs from 472e2dfac6 to 9c08a6ecdf 2023-09-04 12:39:15 +00:00 Compare
r.loginov force-pushed feature/integration_with_frostfs from 9c08a6ecdf to 5c26d0c9c1 2023-09-05 06:41:12 +00:00 Compare
r.loginov changed title from WIP: feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-08 07:17:21 +00:00
dkirillov approved these changes 2023-09-20 13:14:09 +00:00
dkirillov commented 2023-09-20 13:16:07 +00:00
Collaborator

Maybe we should create new branch and merge to that new branch rather than upstream main? @alexvanin

Maybe we should create new branch and merge to that new branch rather than upstream main? @alexvanin
alexvanin commented 2023-09-20 14:38:56 +00:00
Owner

@dkirillov Sure, like we did for certificates and lego.

@dkirillov Sure, like we did for [certificates](https://git.frostfs.info/TrueCloudLab/certificates) and [lego](https://git.frostfs.info/TrueCloudLab/lego).
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:40:40 +00:00
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:41:05 +00:00
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:42:06 +00:00
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:42:46 +00:00
alexvanin changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:44:18 +00:00
alexvanin changed target branch from main to tcl/master 2023-09-20 14:45:24 +00:00
r.loginov changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:47:22 +00:00
r.loginov changed title from feature/integration_with_frostfs to feature/integration_with_frostfs 2023-09-20 14:47:36 +00:00
alexvanin merged commit 5c26d0c9c1 into tcl/master 2023-09-20 14:48:53 +00:00
alexvanin deleted branch feature/integration_with_frostfs 2023-09-20 14:48:53 +00:00
alexvanin referenced this issue from a commit 2023-09-20 14:48:55 +00:00
[#1] Add frostfs driver
r.loginov referenced this issue from a commit 2024-04-03 13:36:14 +00:00
[#1] Add frostfs driver
Sign in to join this conversation.
Reviewers
No reviewers
dkirillov
Labels
Clear labels
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff   
blocked

The issue or PR is blocked by something   
bug

Something is not working   
config

Configuration format update or breaking change   
discussion

Talking about something in order to reach a decision or to exchange ideas   
documentation

Documentation related   
duplicate

This issue or pull request already exists   
enhancement

New feature   
go

Language features adoption   
help wanted

Extra attention is needed   
internal
   
invalid

Something is wrong   
kludge

This is a kludge and we know it   
observability

Related to metrics, tracing and logs   
perfomance

Perfomance related   
question

More information is needed   
refactoring

Refactoring   
wontfix

This won't be fixed
No Label
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/distribution#1
There is no content yet.
Delete Branch ":feature/integration_with_frostfs"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?