TrueCloudLab/frostfs-node
20
1
Fork 27
Code Issues 100 Pull requests 4 Projects Releases 1 Activity Actions

Metabase keep opened after shard moved to degraded-read-only mode trough config #959

New issue
Closed
opened 2024-02-05 09:53:21 +00:00 by anikeev-yadro · 5 comments
anikeev-yadro commented 2024-02-05 09:53:21 +00:00
Member
Copy link

Expected Behavior

Metabase should be closed after shard moved to degraded-read-only mode trough config in the same way as moved trough control command.

Current Behavior

Metabase keep opened after shard moved to degraded-read-only mode trough config.
Metabase closed after shard moved to degraded-read-only mode trough config control command.

Steps to Reproduce (for bugs)

  1. Set shard mode to degraded-read-only in config file
root@aanikeev-node1:~# cat /etc/frostfs/storage/conf.d/01_storage.yml
.
.
.
storage:
  shard:
    0:
      mode: degraded-read-only
      blobstor:
      - path: /srv/frostfs/data0/blobovnicza
        type: blobovnicza
      - path: /srv/frostfs/data0
        type: fstree
      metabase:
        path: /srv/frostfs/meta0/metabase0.db
      pilorama:
        path: /srv/frostfs/meta0/pilorama0.db
      writecache:
        path: /srv/frostfs/meta0/write_cache0
.
.
.
  1. Restart frostfs-storage
systemctl restart frostfs-storage
  1. See that metabase keep opened
root@aanikeev-node1:~# sudo lsof | grep metabase0
frostfs-n  310366                                 frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366                                 frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310368 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310368 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310369 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310369 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310370 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310370 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310371 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310371 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310373 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310373 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310374 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310374 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310375 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310375 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310385 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310385 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310386 frostfs-n               frostfs-storage  mem-W     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db
frostfs-n  310366  310386 frostfs-n               frostfs-storage   23uW     REG              8,224  151220224         11 /srv/frostfs/meta0/metabase0.db

Regression

No

Version

FrostFS Storage node
Version: v0.37.0-rc.1-218-gf5160b27
GoVersion: go1.20.1

Your Environment

Cloud
4 nodes

## Expected Behavior Metabase should be closed after shard moved to degraded-read-only mode trough config in the same way as moved trough control command. ## Current Behavior Metabase keep opened after shard moved to degraded-read-only mode trough config. Metabase closed after shard moved to degraded-read-only mode trough config control command. ## Steps to Reproduce (for bugs) 1. Set shard mode to degraded-read-only in config file ``` root@aanikeev-node1:~# cat /etc/frostfs/storage/conf.d/01_storage.yml . . . storage: shard: 0: mode: degraded-read-only blobstor: - path: /srv/frostfs/data0/blobovnicza type: blobovnicza - path: /srv/frostfs/data0 type: fstree metabase: path: /srv/frostfs/meta0/metabase0.db pilorama: path: /srv/frostfs/meta0/pilorama0.db writecache: path: /srv/frostfs/meta0/write_cache0 . . . ``` 2. Restart frostfs-storage ``` systemctl restart frostfs-storage ``` 3. See that metabase keep opened ``` root@aanikeev-node1:~# sudo lsof | grep metabase0 frostfs-n 310366 frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310368 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310368 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310369 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310369 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310370 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310370 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310371 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310371 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310373 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310373 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310374 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310374 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310375 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310375 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310385 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310385 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310386 frostfs-n frostfs-storage mem-W REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db frostfs-n 310366 310386 frostfs-n frostfs-storage 23uW REG 8,224 151220224 11 /srv/frostfs/meta0/metabase0.db ``` ## Regression No ## Version ``` FrostFS Storage node Version: v0.37.0-rc.1-218-gf5160b27 GoVersion: go1.20.1 ``` ## Your Environment Cloud 4 nodes
anikeev-yadro added the
bug
triage
 labels 2024-02-05 09:53:21 +00:00
acid-ant self-assigned this 2024-02-06 14:39:20 +00:00
fyrchik commented 2024-02-14 11:03:38 +00:00
Owner
Copy link
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        debug        blobovniczatree/control.go:41        read-only mode, skip blobovniczas initialization...
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        info        shard/rebuilder.go:49        blobstore rebuild started        {"shard_id": "8kapvaeHbyfRTUQWZopoVL"}
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        error        blobstor/rebuild.go:32        failed to rebuild storages        {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "failed_storage_path": "/srv/frostfs/data0/blobovnicza", "failed_storage_type": "blobovnicza", "error": "opened as read-only"}
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        info        blobstor/rebuild.go:40        storages rebuild completed        {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "success": false, "total_files_removed": 0, "total_objects_moved": 0}
фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z        warn        shard/rebuilder.go:51        failed to rebuild blobstore        {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "error": "opened as read-only"}
``` фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z debug blobovniczatree/control.go:41 read-only mode, skip blobovniczas initialization... фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z info shard/rebuilder.go:49 blobstore rebuild started {"shard_id": "8kapvaeHbyfRTUQWZopoVL"} фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z error blobstor/rebuild.go:32 failed to rebuild storages {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "failed_storage_path": "/srv/frostfs/data0/blobovnicza", "failed_storage_type": "blobovnicza", "error": "opened as read-only"} фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z info blobstor/rebuild.go:40 storages rebuild completed {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "success": false, "total_files_removed": 0, "total_objects_moved": 0} фев 14 10:25:56 aanikeev-node1 frostfs-node[2037080]: 2024-02-14T10:25:56.056Z warn shard/rebuilder.go:51 failed to rebuild blobstore {"shard_id": "8kapvaeHbyfRTUQWZopoVL", "error": "opened as read-only"} ```
fyrchik reopened this issue 2024-02-14 11:03:38 +00:00
anikeev-yadro commented 2024-02-14 12:06:18 +00:00
Author
Member
Copy link

Panic after the following steps:

  1. Move shad to dergaded-read-only through config
Shard BCuU8MH9Mn46T3K79MxwJA:
Mode: degraded-read-only
Metabase: /srv/frostfs/meta0/metabase0.db
Blobstor:
        Path 0: /srv/frostfs/data0/blobovnicza
        Type 0: blobovnicza
        Path 1: /srv/frostfs/data0
        Type 1: fstree
Write-cache: /srv/frostfs/meta0/write_cache0
Pilorama: /srv/frostfs/meta0/pilorama0.db
Error count: 0
  1. Start the k6 preset script
service@loader:/etc/k6$ ./scenarios/preset/preset_grpc.py --size 6000 --containers 2 --out /tmp/grpc_1402_1.json --endpoint 10
Namespace(size='6000', containers='2', out='/tmp/grpc_1402_1.json', preload_obj='10', wallet='scenarios/files/wallet.json', coe, acl='public-read-write')
Create containers: 2
2024-02-14 10:25:23.281578 at 10.78.130.35:8080: Created container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd
2024-02-14 10:25:23.367903 at 10.78.130.35:8080: Created container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ
Create containers: Completed
 > Containers: ['9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ', '5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd']
Sleep for 8 seconds
Upload objects to each container: 10
 > Create random payload: Completed
2024-02-14 10:25:32.573482 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn
Object  has not been uploaded
Error: rpc error: client failure: rpc error: code = Unavailable desc = error reading from server: EOF

2024-02-14 10:25:32.573886 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn
Object  has not been uploaded
Error: rpc error: client failure: rpc error: code = Unavailable desc = error reading from server: EOF

2024-02-14 10:25:47.934352 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn
Object  has not been uploaded
Error: can't create API client: can't init SDK client: gRPC dial: context deadline exceeded

 > Uploaded object HFQBVyA8KjAPLCNgTjJg3UEM8GWaJe24TXL3jNVbDNuf for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object 8XWcw8fQYeapx6Z1fWf4zWwPpNqULY8mdqX7bniTcY19 for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object BUZ1HHCHKyFE3bxR391UA7EaX2AqEx6A7iRx8qwU6zWG for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object 23WL9XaPEoZ7WkMdTdDBJaXzsaENcSjFaQUjSiKTtUjD for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 5QAYf58o1WY3kxRinTic7eWjyrdypTa4CvjfzxqAkdqu for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object G2uztMkkEAJX2wnEa4J79HW5MGWURYbVRKo6avwW8gQ for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object EpwG3NaxHSHuA66jyzu73fG1vKTGBLaWiG7JbhMcNx6t for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object EbgXs5h9vjAeakripzoNXhg6mdrHQuhUmeGNroKEs5gs for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 4DuMA1RFfLheFMu1aEpbYrt36anCXw3YixXCynwoBJaB for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object 3Pen9xy7vNfCJbeVmb12uaQCJfEVDZ8GdefSKKFtx7nT for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 7eU1zLTQH9Jc3j2ucQdb7vqEYMLYFsc7pZT3wZALxLAQ for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object GSzoZZzRmPCzvsCPPx8Tfy7dqh4qS3ojT3VQMZDCUWxX for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object AzhALp7jgssELbBj9VBfDVoWcbq4sjJzJVYp3ne2SA21 for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object C1uXwmhXjxX4GzQbmCpLV1HfAEFP4DkDXVSG4J6niiXi for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 3Z6qsUhct1GKF427BBHyGvEXMd8KjADucfauiWYGq5F7 for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
 > Uploaded object 73Dw43dVsNwUeTPEJZJwitj6w4zUrY5tHsvpQaLg4Yfy for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via
 > Uploaded object 6ATeHcKbuUhkZ26kA9e1V1vEbw8YMBU7jBwC9qyw6T8K for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via
Upload objects to each container: Completed
Objects mismatch in preset: expected 20, created 17
  1. Node cycle paniced after load was completed
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: panic: runtime error: invalid memory address or nil pointer der
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc=
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: goroutine 4185 [running]:
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).beginTx(0x0)
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         go.etcd.io/bbolt@v1.3.8/db.go:730 +0x27
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).Begin(0x1000000000000?, 0x58?)
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         go.etcd.io/bbolt@v1.3.8/db.go:723 +0x2c
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).View(0xc001043020?, 0xc0008b32f8)
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         go.etcd.io/bbolt@v1.3.8/db.go:901 +0x3b
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network/transpor
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object/grpc._Ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.16.1
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grp
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-observability@v0.
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.getChainUnaryHandler.func1({0x15ccbf0, 0
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1205 +0xb9
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.Un
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.chainUnaryInterceptors.func1({0x15ccbf0,
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1196 +0x8f
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object/grpc._Ob
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.16.1
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000197400,
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1385 +0xe49
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).handleStream(0xc000197400, {0x
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1796 +0x1031
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).serveStreams.func2.1()
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1029 +0x91
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: created by google.golang.org/grpc.(*Server).serveStreams.func2
фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]:         google.golang.org/grpc@v1.61.0/server.go:1040 +0x145
фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Main process exited, code=exited, status=2/INVALI
фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Failed with result 'exit-code'.
фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Consumed 1.263s CPU time.
Panic after the following steps: 1. Move shad to dergaded-read-only through config ``` Shard BCuU8MH9Mn46T3K79MxwJA: Mode: degraded-read-only Metabase: /srv/frostfs/meta0/metabase0.db Blobstor: Path 0: /srv/frostfs/data0/blobovnicza Type 0: blobovnicza Path 1: /srv/frostfs/data0 Type 1: fstree Write-cache: /srv/frostfs/meta0/write_cache0 Pilorama: /srv/frostfs/meta0/pilorama0.db Error count: 0 ``` 2. Start the k6 preset script ``` service@loader:/etc/k6$ ./scenarios/preset/preset_grpc.py --size 6000 --containers 2 --out /tmp/grpc_1402_1.json --endpoint 10 Namespace(size='6000', containers='2', out='/tmp/grpc_1402_1.json', preload_obj='10', wallet='scenarios/files/wallet.json', coe, acl='public-read-write') Create containers: 2 2024-02-14 10:25:23.281578 at 10.78.130.35:8080: Created container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd 2024-02-14 10:25:23.367903 at 10.78.130.35:8080: Created container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ Create containers: Completed > Containers: ['9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ', '5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd'] Sleep for 8 seconds Upload objects to each container: 10 > Create random payload: Completed 2024-02-14 10:25:32.573482 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn Object has not been uploaded Error: rpc error: client failure: rpc error: code = Unavailable desc = error reading from server: EOF 2024-02-14 10:25:32.573886 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn Object has not been uploaded Error: rpc error: client failure: rpc error: code = Unavailable desc = error reading from server: EOF 2024-02-14 10:25:47.934352 at 10.78.130.35:8080: frostfs-cli --rpc-endpoint 10.78.130.35:8080 object put --file /tmp/tmpl3y4wn Object has not been uploaded Error: can't create API client: can't init SDK client: gRPC dial: context deadline exceeded > Uploaded object HFQBVyA8KjAPLCNgTjJg3UEM8GWaJe24TXL3jNVbDNuf for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object 8XWcw8fQYeapx6Z1fWf4zWwPpNqULY8mdqX7bniTcY19 for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object BUZ1HHCHKyFE3bxR391UA7EaX2AqEx6A7iRx8qwU6zWG for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object 23WL9XaPEoZ7WkMdTdDBJaXzsaENcSjFaQUjSiKTtUjD for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 5QAYf58o1WY3kxRinTic7eWjyrdypTa4CvjfzxqAkdqu for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object G2uztMkkEAJX2wnEa4J79HW5MGWURYbVRKo6avwW8gQ for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object EpwG3NaxHSHuA66jyzu73fG1vKTGBLaWiG7JbhMcNx6t for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object EbgXs5h9vjAeakripzoNXhg6mdrHQuhUmeGNroKEs5gs for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 4DuMA1RFfLheFMu1aEpbYrt36anCXw3YixXCynwoBJaB for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object 3Pen9xy7vNfCJbeVmb12uaQCJfEVDZ8GdefSKKFtx7nT for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 7eU1zLTQH9Jc3j2ucQdb7vqEYMLYFsc7pZT3wZALxLAQ for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object GSzoZZzRmPCzvsCPPx8Tfy7dqh4qS3ojT3VQMZDCUWxX for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object AzhALp7jgssELbBj9VBfDVoWcbq4sjJzJVYp3ne2SA21 for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object C1uXwmhXjxX4GzQbmCpLV1HfAEFP4DkDXVSG4J6niiXi for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 3Z6qsUhct1GKF427BBHyGvEXMd8KjADucfauiWYGq5F7 for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via > Uploaded object 73Dw43dVsNwUeTPEJZJwitj6w4zUrY5tHsvpQaLg4Yfy for container 9wnWwjmj5TkxENVP8Z2C59CogzfsuX5KgJivfoVaR5yJ via > Uploaded object 6ATeHcKbuUhkZ26kA9e1V1vEbw8YMBU7jBwC9qyw6T8K for container 5ZE6YykBwYfYnQc8DbaPaa46VefPvNZFAe1KRVGhx4kd via Upload objects to each container: Completed Objects mismatch in preset: expected 20, created 17 ``` 3. Node cycle paniced after load was completed ``` фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: panic: runtime error: invalid memory address or nil pointer der фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x1c0 pc= фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: goroutine 4185 [running]: фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).beginTx(0x0) фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt@v1.3.8/db.go:730 +0x27 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).Begin(0x1000000000000?, 0x58?) фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt@v1.3.8/db.go:723 +0x2c фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt.(*DB).View(0xc001043020?, 0xc0008b32f8) фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: go.etcd.io/bbolt@v1.3.8/db.go:901 +0x3b фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_sto фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/object. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network/transpor фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/network/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object/grpc._Ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.16.1 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grp фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-observability@v0. фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.getChainUnaryHandler.func1({0x15ccbf0, 0 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1205 +0xb9 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.Un фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/ фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.chainUnaryInterceptors.func1({0x15ccbf0, фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1196 +0x8f фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/object/grpc._Ob фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.16.1 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000197400, фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1385 +0xe49 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).handleStream(0xc000197400, {0x фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1796 +0x1031 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc.(*Server).serveStreams.func2.1() фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1029 +0x91 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: created by google.golang.org/grpc.(*Server).serveStreams.func2 фев 14 10:28:35 aanikeev-node1 frostfs-node[2041725]: google.golang.org/grpc@v1.61.0/server.go:1040 +0x145 фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Main process exited, code=exited, status=2/INVALI фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Failed with result 'exit-code'. фев 14 10:28:35 aanikeev-node1 systemd[1]: frostfs-storage.service: Consumed 1.263s CPU time. ```
acid-ant commented 2024-02-20 14:17:12 +00:00
Member
Copy link

Need to initialize fstree in DEGRADED mode too:

...
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: 2024-02-20T13:13:27.710Z        warn        shard/get.go:137        fetching object without meta        {"shard_id": "M5bSgGo4qYNs5NcDQndB7x", "addr": "8XzcouaAUok2cUrMsCsqZAHcrEEY6eEqnrzRn7ZHaRB5/E1wMmzoE7EYgURdRKZWvw6AC4zWVvDwKp38rEbB6iMWC"}
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic: runtime error: invalid memory address or nil pointer dereference
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         panic: runtime error: invalid memory address or nil pointer dereference
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xd8a26a]
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: goroutine 4402 [running]:
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Get.func1()
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:354 +0x4a
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic({0x1188220, 0x1f49080})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         runtime/panic.go:890 +0x263
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Get(0x0, {0x15ccdd0, 0xc0007d4ea0}, {{{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}, ...})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:359 +0x115
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).getInternal(0xc0002fc120, {0x15ccdd0, 0xc0007d4ea0}, {0xc00102a300, 0x59}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, ...}, ...})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/get.go:52 +0x228
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).Get(0xc000e79080?, {0x15ccdd0, 0xc0007d4e40}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/get.go:32 +0x345
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Get.func2({0x15d84e0?, 0xc0002fc120?})
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/get.go:102 +0x55
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).fetchObjectData(0xc0002fb110, {0x15ccdd0, 0xc0007d4e40}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}, ...)
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/get.go:141 +0x2b2
фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Get(0xc0002fb110, {0x15ccdd0,
...
Need to initialize `fstree` in `DEGRADED` mode too: ``` ... фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: 2024-02-20T13:13:27.710Z warn shard/get.go:137 fetching object without meta {"shard_id": "M5bSgGo4qYNs5NcDQndB7x", "addr": "8XzcouaAUok2cUrMsCsqZAHcrEEY6eEqnrzRn7ZHaRB5/E1wMmzoE7EYgURdRKZWvw6AC4zWVvDwKp38rEbB6iMWC"} фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic: runtime error: invalid memory address or nil pointer dereference фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic: runtime error: invalid memory address or nil pointer dereference фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xd8a26a] фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: goroutine 4402 [running]: фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Get.func1() фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:354 +0x4a фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: panic({0x1188220, 0x1f49080}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: runtime/panic.go:890 +0x263 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree.(*FSTree).Get(0x0, {0x15ccdd0, 0xc0007d4ea0}, {{{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}, ...}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/blobstor/fstree/fstree.go:359 +0x115 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).getInternal(0xc0002fc120, {0x15ccdd0, 0xc0007d4ea0}, {0xc00102a300, 0x59}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, ...}, ...}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/get.go:52 +0x228 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache.(*cache).Get(0xc000e79080?, {0x15ccdd0, 0xc0007d4e40}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/writecache/get.go:32 +0x345 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Get.func2({0x15d84e0?, 0xc0002fc120?}) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/get.go:102 +0x55 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).fetchObjectData(0xc0002fb110, {0x15ccdd0, 0xc0007d4e40}, {{0x6f, 0xf2, 0x61, 0x85, 0xe7, 0x49, 0x44, ...}, ...}, ...) фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard/get.go:141 +0x2b2 фев 20 13:13:27 aanikeev-node1 frostfs-node[1796160]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/local_object_storage/shard.(*Shard).Get(0xc0002fb110, {0x15ccdd0, ... ```
acid-ant reopened this issue 2024-02-20 14:17:13 +00:00
fyrchik commented 2024-02-20 14:24:36 +00:00
Owner
Copy link

Was #987 tested at all, or why didn't we catch it?

Was #987 tested at all, or why didn't we catch it?
acid-ant commented 2024-02-21 07:43:41 +00:00
Member
Copy link

Was #987 tested at all, or why didn't we catch it?

Was not tested under pressure - that is the reason. When the virtual cluster was checked, there were no errors in the log, that is why the fix was partial.

> Was #987 tested at all, or why didn't we catch it? Was not tested under pressure - that is the reason. When the virtual cluster was checked, there were no errors in the log, that is why the fix was partial.
👍 1
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
support/v0.42
support/v0.38
support/v0.37
support/v0.36
support/v0.34
support/v0.30
support/v0.27
v0.44.0-rc.13
v0.44.0-rc.12
v0.44.0-rc.11
v0.42.16
v0.44.0-rc.10
v0.44.0-rc.9
v0.44.0-rc.8
v0.44.0-rc.7
v0.44.0-rc.6
v0.44.0-rc.5
v0.44.0-rc.4
v0.44.0-rc.3
v0.44.0-rc.2
v0.44.0-rc.1
v0.43.2
v0.43.1
v0.43.0
v0.42.15
v0.42.14
v0.42.13
v0.42.12
v0.42.11
v0.42.10
v0.42.9
v0.42.8
v0.42.7
v0.42.6
v0.42.5
v0.42.4
v0.42.3
v0.42.2
v0.42.1
v0.42.0
v0.42.0-rc.9
v0.42.0-rc.8
v0.42.0-rc.7
v0.38.9
v0.42.0-rc.6
v0.42.0-rc.5
v0.42.0-rc.4
v0.42.0-rc.3
v0.42.0-rc.2
v0.42.0-rc.1
v0.38.8
v0.41.0
v0.38.7
v0.40.0
v0.39.0
v0.38.6
v0.38.5
v0.38.4
v0.38.3
v0.38.2
v0.38.1
v0.38.0
v0.38.0-rc.2
v0.38.0-rc.1
v0.37.0
v0.37.0-rc.1
v0.36.0
v0.34.0
v0.22.1
v0.22.0
v0.21.1
v0.21.0
v0.20.0
v0.19.0
v0.18.0
v0.17.0
v0.16.0
v0.15.0
v0.14.3
v0.14.2
v0.14.1
v0.14.0
v0.14.0-rc.1
v0.13.2
v0.13.1
v0.13.0
v0.13.0-rc.1
v0.12.1
v0.12.0
v0.12.0-rc3
v0.12.0-rc2
v0.12.0-rc1
v0.11.0
v0.10.0
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
badger

Badger DB related issues

  
frostfs-adm

   
frostfs-cli

   
frostfs-ir

   
frostfs-lens

   
frostfs-node

   
good first issue

Good for newcomers

  
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
acid-ant
3 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#959
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?