TrueCloudLab
/
frostfs-node
19
1
Fork 20
Code Issues 93 Pull Requests 9 Actions Releases Activity

Metabase keep opened after shard moved to degraded-read-only mode trough config #959

New Issue
Closed
opened 2024-02-05 09:53:21 +00:00 by anikeev-yadro · 5 comments
anikeev-yadro commented 2024-02-05 09:53:21 +00:00

Expected Behavior

Metabase should be closed after shard moved to degraded-read-only mode trough config in the same way as moved trough control command.

Current Behavior

Metabase keep opened after shard moved to degraded-read-only mode trough config.
Metabase closed after shard moved to degraded-read-only mode trough config control command.

Steps to Reproduce (for bugs)

  1. Set shard mode to degraded-read-only in config file
root@aanikeev-node1:~# cat /etc/frostfs/storage/conf.d/01_storage.yml
.
.
.
storage:
  shard:
    0:
      mode: degraded-read-only
      blobstor:
      - path: /srv/frostfs/data0/blobovnicza
        type: blobovnicza
      - path: /srv/frostfs/data0
        type: fstree
      metabase:
        path: /srv/frostfs/meta0/metabase0.db
      pilorama:
        path: /srv/frostfs/meta0/pilorama0.db
      writecache:
        path: /srv/frostfs/meta0/write_cache0
.
.
.
  1. Restart frostfs-storage
systemctl restart frostfs-storage
  1. See that metabase keep opened
root@aanikeev-node1:~# sudo lsof | grep metabase0
frostfs-n  310366                                 frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366                                 frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310368 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310368 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310369 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310369 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310370 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310370 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310371 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310371 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310373 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310373 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310374 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310374 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310375 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310375 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310385 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310385 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310386 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310386 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db

Regression

No

Version

FrostFS Storage node
Version: v0.37.0-rc.1-218-gf5160b27
GoVersion: go1.20.1

Your Environment

Cloud
4 nodes

## Expected Behavior Metabase should be closed after shard moved to degraded-read-only mode trough config in the same way as moved trough control command. ## Current Behavior Metabase keep opened after shard moved to degraded-read-only mode trough config. Metabase closed after shard moved to degraded-read-only mode trough config control command. ## Steps to Reproduce (for bugs) 1. Set shard mode to degraded-read-only in config file ``` root@aanikeev-node1:~# cat /etc/frostfs/storage/conf.d/01_storage.yml . . . storage: shard: 0: mode: degraded-read-only blobstor: - path: /srv/frostfs/data0/blobovnicza type: blobovnicza - path: /srv/frostfs/data0 type: fstree metabase: path: /srv/frostfs/meta0/metabase0.db pilorama: path: /srv/frostfs/meta0/pilorama0.db writecache: path: /srv/frostfs/meta0/write_cache0 . . . ``` 2. Restart frostfs-storage ``` systemctl restart frostfs-storage ``` 3. See that metabase keep opened ``` root@aanikeev-node1:~# sudo lsof | grep metabase0 frostfs-n 310366 frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310368 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310368 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310369 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310369 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310370 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310370 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310371 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310371 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310373 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310373 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310374 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310374 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310375 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310375 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310385 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310385 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310386 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310386 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db ``` ## Regression No ## Version ``` FrostFS Storage node Version: v0.37.0-rc.1-218-gf5160b27 GoVersion: go1.20.1 ``` ## Your Environment Cloud 4 nodes
anikeev-yadro added the
bug
triage
 labels 2024-02-05 09:53:21 +00:00
acid-ant self-assigned this 2024-02-06 14:39:20 +00:00
fyrchik commented 2024-02-14 11:03:38 +00:00
Owner 
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        debug        blobovniczatree/control.go:41        read-only mode, skip blobovniczas initialization...
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        info        shard/rebuilder.go:49        blobstore rebuild started        {"shard_id": "8kapvaeHbyfRTUQWZopoVL"}
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        error        blobstor/rebuild.go:32        failed to rebuild storages        {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "failed_storage_path": "/srv/frostfs/data0/blobovnicza", "failed_storage_type": "blobovnicza", "error": "opened as read-only"}
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        info        blobstor/rebuild.go:40        storages rebuild completed        {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "success": false, "total_files_removed": 0, "total_objects_moved": 0}
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        warn        shard/rebuilder.go:51        failed to rebuild blobstore        {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "error": "opened as read-only"}
``` фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z debug blobovniczatree/control.go:41 read-only mode, skip blobovniczas initialization... фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z info shard/rebuilder.go:49 blobstore rebuild started {"shard_id": "8kapvaeHbyfRTUQWZopoVL"} фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z error blobstor/rebuild.go:32 failed to rebuild storages {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "failed_storage_path": "/srv/frostfs/data0/blobovnicza", "failed_storage_type": "blobovnicza", "error": "opened as read-only"} фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z info blobstor/rebuild.go:40 storages rebuild completed {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "success": false, "total_files_removed": 0, "total_objects_moved": 0} фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z warn shard/rebuilder.go:51 failed to rebuild blobstore {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "error": "opened as read-only"} ```
fyrchik reopened this issue 2024-02-14 11:03:38 +00:00
anikeev-yadro commented 2024-02-14 12:06:18 +00:00
Poster

Panic after the following steps:

  1. Move shad to dergaded-read-only through config
Shard BCuU8MH9Mn46T3K79MxwJA:
Mode: degraded-read-only
Metabase: /srv/frostfs/meta0/metabase0.db
Blobstor:
        Path 0: /srv/frostfs/data0/blobovnicza
        Type 0: blobovnicza
        Path 1: /srv/frostfs/data0
        Type 1: fstree
Write-cache: /srv/frostfs/meta0/write_cache0
Pilorama: /srv/frostfs/meta0/pilorama0.db
Error count: 0
  1. Start the k6 preset script
service@loader:/etc/k6$ ./scenarios/preset/preset_grpc.py --size 6000 --containers 2 --out /tmp/grpc_1402_1.json --endpoint 10
Namespace(size='6000', containers='2', out='/tmp/grpc_1402_1.json', preload_obj='10', wallet='scenarios/files/wallet.json', coe, acl='public-read-write')
Create containers: 2
2024-02-14 10:25:23.281578 at 10.78.130.35:8080: Created container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd
2024-02-14 10:25:23.367903 at 10.78.130.35:8080: Created container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ
Create containers: Completed
 > Containers: ['9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ', '5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd']
Sleep for 8 seconds
Upload objects to each container: 10
 > Create random payload: Completed
2024-02-14 10:25:32.573482 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn
Object  has not been uploaded
Error: rpc error: client failure: rpc error: code = Unavailable desc = error reading from server: EOF

2024-02-14 10:25:32.573886 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn
Object  has not been uploaded
Error: rpc error: client failure: rpc error: code = Unavailable desc = error reading from server: EOF

2024-02-14 10:25:47.934352 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn
Object  has not been uploaded
Error: can't create API client: can't init SDK client: gRPC dial: context deadline exceeded

 > Uploaded object HFQBVyA8KjAPLCNgTjJg3UEM8GWaJe24TXL3jNVbDNuf for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object 8XWcw8fQYeapx6Z1fWf4zWwPpNqULY8mdqX7bniTcY19 for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object BUZ1HHCHKyFE3bxR391UA7EaX2AqEx6A7iRx8qwU6zWG for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object 23WL9XaPEoZ7WkMdTdDBJaXzsaENcSjFaQUjSiKTtUjD for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 5QAYf58o1WY3kxRinTic7eWjyrdypTa4CvjfzxqAkdqu for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object G2uztMkkEAJX2wnEa4J79HW5MGWURYbVRKo6avwW8gQ for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object EpwG3NaxHSHuA66jyzu73fG1vKTGBLaWiG7JbhMcNx6t for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object EbgXs5h9vjAeakripzoNXhg6mdrHQuhUmeGNroKEs5gs for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 4DuMA1RFfLheFMu1aEpbYrt36anCXw3YixXCynwoBJaB for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object 3Pen9xy7vNfCJbeVmb12uaQCJfEVDZ8GdefSKKFtx7nT for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 7eU1zLTQH9Jc3j2ucQdb7vqEYMLYFsc7pZT3wZALxLAQ for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object GSzoZZzRmPCzvsCPPx8Tfy7dqh4qS3ojT3VQMZDCUWxX for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object AzhALp7jgssELbBj9VBfDVoWcbq4sjJzJVYp3ne2SA21 for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object C1uXwmhXjxX4GzQbmCpLV1HfAEFP4DkDXVSG4J6niiXi for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 3Z6qsUhct1GKF427BBHyGvEXMd8KjADucfauiWYGq5F7 for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object 73Dw43dVsNwUeTPEJZJwitj6w4zUrY5tHsvpQaLg4Yfy for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 6ATeHcKbuUhkZ26kA9e1V1vEbw8YMBU7jBwC9qyw6T8K for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
Upload objects to each container: Completed
Objects mismatch in preset: expected 20, created 17
  1. Node cycle paniced after load was completed
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: panic: runtime error: invalid memory address or nil pointer der
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: goroutine 4185 [running]:
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).beginTx(0x0)
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         go.etcd.io/bbolt@v1.3.8/db.go:730 +0x27
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).Begin(0x1000000000000?, 0x58?)
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         go.etcd.io/bbolt@v1.3.8/db.go:723 +0x2c
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).View(0xc001043020?, 0xc0008b32f8)
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         go.etcd.io/bbolt@v1.3.8/db.go:901 +0x3b
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network/transpor
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object/grpc._Ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.16.1
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grp
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-observability@v0.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.getChainUnaryHandler.func1({0x15ccbf0, 0
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1205 +0xb9
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.Un
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.chainUnaryInterceptors.func1({0x15ccbf0,
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1196 +0x8f
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object/grpc._Ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.16.1
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000197400,
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1385 +0xe49
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).handleStream(0xc000197400, {0x
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1796 +0x1031
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).serveStreams.func2.1()
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1029 +0x91
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: created by google.golang.org/grpc.(*Server).serveStreams.func2
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1040 +0x145
фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Main process exited, code=exited, status=2/INVALI
фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Failed with result 'exit-code'.
фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Consumed 1.263s CPU time.
Panic after the following steps: 1. Move shad to dergaded-read-only through config ``` Shard BCuU8MH9Mn46T3K79MxwJA: Mode: degraded-read-only Metabase: /srv/frostfs/meta0/metabase0.db Blobstor: Path 0: /srv/frostfs/data0/blobovnicza Type 0: blobovnicza Path 1: /srv/frostfs/data0 Type 1: fstree Write-cache: /srv/frostfs/meta0/write_cache0 Pilorama: /srv/frostfs/meta0/pilorama0.db Error count: 0 ``` 2. Start the k6 preset script ``` service@loader:/etc/k6$ ./scenarios/preset/preset_grpc.py --size 6000 --containers 2 --out /tmp/grpc_1402_1.json --endpoint 10 Namespace(size='6000', containers='2', out='/tmp/grpc_1402_1.json', preload_obj='10', wallet='scenarios/files/wallet.json', coe, acl='public-read-write') Create containers: 2 2024-02-14 10:25:23.281578 at 10.78.130.35:8080: Created container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd 2024-02-14 10:25:23.367903 at 10.78.130.35:8080: Created container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ Create containers: Completed > Containers: ['9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ', '5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd'] Sleep for 8 seconds Upload objects to each container: 10 > Create random payload: Completed 2024-02-14 10:25:32.573482 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn Object has not been uploaded Error: rpc error: client failure: rpc error: code = Unavailable desc = error reading from server: EOF 2024-02-14 10:25:32.573886 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn Object has not been uploaded Error: rpc error: client failure: rpc error: code = Unavailable desc = error reading from server: EOF 2024-02-14 10:25:47.934352 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn Object has not been uploaded Error: can't create API client: can't init SDK client: gRPC dial: context deadline exceeded > Uploaded object HFQBVyA8KjAPLCNgTjJg3UEM8GWaJe24TXL3jNVbDNuf for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object 8XWcw8fQYeapx6Z1fWf4zWwPpNqULY8mdqX7bniTcY19 for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object BUZ1HHCHKyFE3bxR391UA7EaX2AqEx6A7iRx8qwU6zWG for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object 23WL9XaPEoZ7WkMdTdDBJaXzsaENcSjFaQUjSiKTtUjD for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 5QAYf58o1WY3kxRinTic7eWjyrdypTa4CvjfzxqAkdqu for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object G2uztMkkEAJX2wnEa4J79HW5MGWURYbVRKo6avwW8gQ for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object EpwG3NaxHSHuA66jyzu73fG1vKTGBLaWiG7JbhMcNx6t for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object EbgXs5h9vjAeakripzoNXhg6mdrHQuhUmeGNroKEs5gs for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 4DuMA1RFfLheFMu1aEpbYrt36anCXw3YixXCynwoBJaB for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object 3Pen9xy7vNfCJbeVmb12uaQCJfEVDZ8GdefSKKFtx7nT for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 7eU1zLTQH9Jc3j2ucQdb7vqEYMLYFsc7pZT3wZALxLAQ for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object GSzoZZzRmPCzvsCPPx8Tfy7dqh4qS3ojT3VQMZDCUWxX for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object AzhALp7jgssELbBj9VBfDVoWcbq4sjJzJVYp3ne2SA21 for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object C1uXwmhXjxX4GzQbmCpLV1HfAEFP4DkDXVSG4J6niiXi for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 3Z6qsUhct1GKF427BBHyGvEXMd8KjADucfauiWYGq5F7 for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object 73Dw43dVsNwUeTPEJZJwitj6w4zUrY5tHsvpQaLg4Yfy for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 6ATeHcKbuUhkZ26kA9e1V1vEbw8YMBU7jBwC9qyw6T8K for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via Upload objects to each container: Completed Objects mismatch in preset: expected 20, created 17 ``` 3. Node cycle paniced after load was completed ``` фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: panic: runtime error: invalid memory address or nil pointer der фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc= фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: goroutine 4185 [running]: фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).beginTx(0x0) фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt@v1.3.8/db.go:730 +0x27 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).Begin(0x1000000000000?, 0x58?) фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt@v1.3.8/db.go:723 +0x2c фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).View(0xc001043020?, 0xc0008b32f8) фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt@v1.3.8/db.go:901 +0x3b фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network/transpor фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object/grpc._Ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.16.1 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grp фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-observability@v0. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.getChainUnaryHandler.func1({0x15ccbf0, 0 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1205 +0xb9 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.Un фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.chainUnaryInterceptors.func1({0x15ccbf0, фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1196 +0x8f фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object/grpc._Ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.16.1 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000197400, фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1385 +0xe49 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).handleStream(0xc000197400, {0x фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1796 +0x1031 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).serveStreams.func2.1() фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1029 +0x91 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: created by google.golang.org/grpc.(*Server).serveStreams.func2 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1040 +0x145 фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Main process exited, code=exited, status=2/INVALI фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Failed with result 'exit-code'. фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Consumed 1.263s CPU time. ```
acid-ant commented 2024-02-20 14:17:12 +00:00
Collaborator

Need to initialize fstree in DEGRADED mode too:

...
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: 2024-02-20T13:13:27.710Z        warn        shard/get.go:137        fetching object without meta        {"shard_id": "M5bSgGo4qYNs5NcDQndB7x", "addr": "8XzcouaAUok2cUrMsCsqZAHcrEEY6eEqnrzRn7ZHaRB5/E1wMmzoE7EYgURdRKZWvw6AC4zWVvDwKp38rEbB6iMWC"}
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic: runtime error: invalid memory address or nil pointer dereference
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         panic: runtime error: invalid memory address or nil pointer dereference
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xd8a26a]
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: goroutine 4402 [running]:
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Get.func1()
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:354 +0x4a
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic({0x1188220, 0x1f49080})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         runtime/panic.go:890 +0x263
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Get(0x0, {0x15ccdd0, 0xc0007d4ea0}, {{{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}, ...})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:359 +0x115
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).getInternal(0xc0002fc120, {0x15ccdd0, 0xc0007d4ea0}, {0xc00102a300, 0x59}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, ...}, ...})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/get.go:52 +0x228
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).Get(0xc000e79080?, {0x15ccdd0, 0xc0007d4e40}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/get.go:32 +0x345
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Get.func2({0x15d84e0?, 0xc0002fc120?})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/get.go:102 +0x55
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).fetchObjectData(0xc0002fb110, {0x15ccdd0, 0xc0007d4e40}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}, ...)
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/get.go:141 +0x2b2
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Get(0xc0002fb110, {0x15ccdd0,
...
Need to initialize `fstree` in `DEGRADED` mode too: ``` ... фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: 2024-02-20T13:13:27.710Z warn shard/get.go:137 fetching object without meta {"shard_id": "M5bSgGo4qYNs5NcDQndB7x", "addr": "8XzcouaAUok2cUrMsCsqZAHcrEEY6eEqnrzRn7ZHaRB5/E1wMmzoE7EYgURdRKZWvw6AC4zWVvDwKp38rEbB6iMWC"} фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic: runtime error: invalid memory address or nil pointer dereference фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic: runtime error: invalid memory address or nil pointer dereference фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xd8a26a] фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: goroutine 4402 [running]: фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Get.func1() фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:354 +0x4a фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic({0x1188220, 0x1f49080}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: runtime/panic.go:890 +0x263 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Get(0x0, {0x15ccdd0, 0xc0007d4ea0}, {{{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}, ...}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:359 +0x115 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).getInternal(0xc0002fc120, {0x15ccdd0, 0xc0007d4ea0}, {0xc00102a300, 0x59}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, ...}, ...}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/get.go:52 +0x228 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).Get(0xc000e79080?, {0x15ccdd0, 0xc0007d4e40}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/get.go:32 +0x345 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Get.func2({0x15d84e0?, 0xc0002fc120?}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/get.go:102 +0x55 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).fetchObjectData(0xc0002fb110, {0x15ccdd0, 0xc0007d4e40}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}, ...) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/get.go:141 +0x2b2 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Get(0xc0002fb110, {0x15ccdd0, ... ```
acid-ant reopened this issue 2024-02-20 14:17:13 +00:00
fyrchik commented 2024-02-20 14:24:36 +00:00
Owner

Was #987 tested at all, or why didn't we catch it?

Was #987 tested at all, or why didn't we catch it?
acid-ant commented 2024-02-21 07:43:41 +00:00
Collaborator

Was #987 tested at all, or why didn't we catch it?

Was not tested under pressure - that is the reason. When the virtual cluster was checked, there were no errors in the log, that is why the fix was partial.

> Was #987 tested at all, or why didn't we catch it? Was not tested under pressure - that is the reason. When the virtual cluster was checked, there were no errors in the log, that is why the fix was partial.
👍 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
fix/cli_control_targets
master
support/v0.38
support/v0.37
support/v0.36
support/v0.34
support/v0.30
support/v0.27
v0.39.0
v0.38.6
v0.38.5
v0.38.4
v0.38.3
v0.38.2
v0.38.1
v0.38.0
v0.38.0-rc.2
v0.38.0-rc.1
v0.37.0
v0.37.0-rc.1
v0.36.0
v0.34.0
Labels
Clear labels   
P0

Highest   
P1

High   
P2

Medium   
P3

Low   
badger

Badger DB related issues   
frostfs-adm
   
frostfs-cli
   
frostfs-ir
   
frostfs-lens
   
frostfs-node
   
good first issue

Good for newcomers   
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff   
blocked

The issue or PR is blocked by something   
bug

Something is not working   
config

Configuration format update or breaking change   
discussion

Talking about something in order to reach a decision or to exchange ideas   
documentation

Documentation related   
duplicate

This issue or pull request already exists   
enhancement

New feature   
go

Language features adoption   
help wanted

Extra attention is needed   
internal
   
invalid

Something is wrong   
kludge

This is a kludge and we know it   
observability

Related to metrics, tracing and logs   
perfomance

Perfomance related   
question

More information is needed   
refactoring

Refactoring   
wontfix

This won't be fixed
No Label
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
acid-ant
3 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#959
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?