TrueCloudLab/frostfs-node
20
1
Fork 27
Code Issues 104 Pull requests 5 Projects Releases 1 Activity Actions

Panic while tree sync #554

New issue
Closed
opened 2023-08-02 06:36:17 +00:00 by anikeev-yadro · 0 comments
anikeev-yadro commented 2023-08-02 06:36:17 +00:00
Member
Copy link

Current Behavior

Panic while tree sync

Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.516Z        info        frostfs-node/main.go:115        starting morph notification service...
Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.516Z        info        frostfs-node/main.go:115        starting pprof service...
Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.516Z        info        frostfs-node/main.go:115        starting metrics service...
Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.516Z        info        frostfs-node/main.go:115        starting control service...
Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.553Z        error        tree/sync.go:424        could not sync trees        {"cid": "4SKhejehUhjkojhLZro7zudmM3Qyf6hHMQbuBxY6qPFT", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44460->172.26.162.50:8080: use of closed network connection"}
Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z        error        tree/sync.go:424        could not sync trees        {"cid": "3V4CiPTnqnyrDGWtqmxKYCFdiwXqPpKqzGQfK3a1soBa", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44460->172.26.162.50:8080: use of closed network connection"}
Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z        error        tree/sync.go:424        could not sync trees        {"cid": "5qpgs8JjN8SV4no75NV5hdkop6B5QZAq38pYGmkdJ3EG", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:46756->172.26.162.209:8080: use of closed network connection"}
Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z        error        tree/sync.go:424        could not sync trees        {"cid": "3jW3FZEMD7qy5N3ddLz1oz6cNgZGmNjdgotJrtdW8Lx2", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44504->172.26.162.50:8080: use of closed network connection"}
Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z        error        tree/sync.go:424        could not sync trees        {"cid": "7TexYjPS94QLfYdmxSjYXi7DUeoeRrqj1s2izCwbkDiR", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44504->172.26.162.50:8080: use of closed network connection"}
Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z        error        tree/sync.go:424        could not sync trees        {"cid": "616CXxP8HG93KKPBnhnuW5vdSgKEjKajud4GNVXRC36c", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44504->172.26.162.50:8080: use of closed network connection"}
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: panic: runtime error: invalid memory address or nil pointer dereference
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xe64600]
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: goroutine 8342 [running]:
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*GetSubTreeRequest_Body_Order).StableSize(0x41?)
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_frostfs.pb.go:899
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/util/proto.NestedStructureSize[...](0x6?, 0x0)
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.15.1-0.20230726155259-7a5ee927c8a2/util/proto/marshal.go:280 +0x28
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*GetSubTreeRequest_Body).StableSize(0x42?)
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_frostfs.pb.go:932 +0x293
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*GetSubTreeRequest_Body).StableMarshal(0xc000a3a700, {0x0?, 0x270e?, 0xc00020912b?})
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_frostfs.pb.go:949 +0x38
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*GetSubTreeRequest).ReadSignedData(0x2?, {0x0?, 0x1da18c0?, 0x100c002266e70?})
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_frostfs.pb.go:1006 +0x2e
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.verifyMessage({0x14fbba8, 0xc00187b100})
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/signature.go:169 +0x46
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*Service).verifyClient(0xc000531e00, {0x14fbba8, 0xc00187b100}, {0x78, 0xf4, 0x7b, 0xbf, 0x26, 0x26, 0x9c, ...}, ...)
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/signature.go:50 +0x6c
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*Service).GetSubTree(0xc000531e00, 0xc00187b100, {0x1501b80, 0xc001211890})
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service.go:409 +0x1c5
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree._TreeService_GetSubTree_Handler({0x1273dc0?, 0xc000531e00}, {0x14ff410, 0xc001778ca0})
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_grpc.pb.go:376 +0xd0
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grpc.NewStreamServerInterceptor.func1({0x1273dc0, 0xc000531e00}, {0x14ffc80, 0xc001772d20}, 0xc002281998, 0x13b0ab0)
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         git.frostfs.info/TrueCloudLab/frostfs-observability@v0.0.0-20230531082742-c97d21411eb6/tracing/grpc/interceptors.go:110 +0x41d
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.getChainStreamHandler.func1({0x1273dc0, 0xc000531e00}, {0x14ffc80, 0xc001772d20})
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         google.golang.org/grpc@v1.56.1/server.go:1483 +0xb9
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1({0x1273dc0, 0xc000531e00}, {0x1500190, 0xc0001813b0}, 0xc002281998?, 0xc00187afc0)
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.5/interceptors/server.go:35 +0x31f
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.chainStreamInterceptors.func1({0x1273dc0, 0xc000531e00}, {0x1500190, 0xc0001813b0}, 0x1030120?, 0xc0012114f0?)
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         google.golang.org/grpc@v1.56.1/server.go:1474 +0x8f
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001783c0, {0x15036b8, 0xc000562d00}, 0xc000a3d0e0, 0xc000772450, 0x1df9c20, 0x0)
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         google.golang.org/grpc@v1.56.1/server.go:1638 +0x1363
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.(*Server).handleStream(0xc0001783c0, {0x15036b8, 0xc000562d00}, 0xc000a3d0e0, 0x0)
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         google.golang.org/grpc@v1.56.1/server.go:1718 +0x9f0
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.(*Server).serveStreams.func1.1()
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         google.golang.org/grpc@v1.56.1/server.go:959 +0x98
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]:         google.golang.org/grpc@v1.56.1/server.go:957 +0x18c
Aug 02 06:25:12 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 02 06:25:12 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Failed with result 'exit-code'.
Aug 02 06:25:12 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Consumed 2.761s CPU time.
Aug 02 06:25:17 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Scheduled restart job, restart counter is at 44.
Aug 02 06:25:17 tatlin-object-failover-node1 systemd[1]: Stopped frostfs Storage node.
Aug 02 06:25:17 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Consumed 2.761s CPU time.

Steps to Reproduce (for bugs)

1.Create bucket in s3-gw and put object

COMMAND: 'aws --no-verify-ssl --no-paginate s3api create-bucket --bucket 718ab48b-5dd3-418c-bbca-2a742fa529d8  --object-lock-enabled-for-bucket --endpoint https://172.26.161.22'
OUTPUT:
 urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.22'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
RC: 0
Start / End / Elapsed	 00:52:23.445751 / 00:52:39.497231 / 0:00:16.051480

COMMAND: 'aws --no-verify-ssl --no-paginate s3api put-object --bucket 718ab48b-5dd3-418c-bbca-2a742fa529d8 --key 06f8c5ff-7a6b-422d-9bca-fb6dbfdf9281 --body TemporaryDir/06f8c5ff-7a6b-422d-9bca-fb6dbfdf9281 --endpoint https://172.26.161.22'
OUTPUT:
 urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.22'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "ETag": "b4ee56ed0dbfff9216f5c2fb024831432065114123ae583d24a630dc42e845c0",
    "VersionId": "NavRgCufYeaByQgSqgaoVLgvx2b8MAZ4T1SFKTwjJdj"
}
RC: 0
Start / End / Elapsed	 00:52:44.503662 / 00:52:45.446094 / 0:00:00.942432

Aug 02 00:52:24 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:24.300Z        info        middleware/reqinfo.go:210        request start        {"request_id": "960c0cfc-cccc-47ae-860e-5460cb39ea93", "host": "172.26.161.22", "remote_host": "127.0.0.1"}
Aug 02 00:52:39 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:39.357Z        info        handler/put.go:781        bucket is created        {"request_id": "960c0cfc-cccc-47ae-860e-5460cb39ea93", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "container_id": "99A9UQLncdknmkq49ABC2N7XAQwuHdn3tnU8NHFuePFn"}
Aug 02 00:52:39 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:39.376Z        info        middleware/response.go:322        request end        {"request_id": "960c0cfc-cccc-47ae-860e-5460cb39ea93", "method": "CreateBucket", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "object": "", "status": 200, "description": "OK"}
Aug 02 00:52:45 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:45.273Z        info        middleware/reqinfo.go:210        request start        {"request_id": "10617f85-d369-4dc2-af3c-7771b88ae5a5", "host": "172.26.161.22", "remote_host": "127.0.0.1"}
Aug 02 00:52:45 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:45.329Z        info        middleware/response.go:322        request end        {"request_id": "10617f85-d369-4dc2-af3c-7771b88ae5a5", "method": "PutObject", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "object": "06f8c5ff-7a6b-422d-9bca-fb6dbfdf9281", "status": 200, "description": "OK"}
Aug 02 00:52:46 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:46.216Z        info        middleware/reqinfo.go:210        request start        {"request_id": "5d0e6000-a2a8-4840-8b69-36707be79ebc", "host": "172.26.161.22", "remote_host": "127.0.0.1"}
Aug 02 00:52:46 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:46.224Z        error        handler/util.go:39        request failed        {"status": 500, "request_id": "5d0e6000-a2a8-4840-8b69-36707be79ebc", "method": "ListObjectsV1", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "object": "", "description": "something went wrong", "error": "failed to get sub tree: rpc error: code = Unavailable desc = error reading from server: EOF"}
  1. List objects in trough s3-gw:
anikeev@NB-1670:~/neofs$ aws --no-verify-ssl --no-paginate s3api list-objects --bucket 718ab48b-5dd3-418c-bbca-2a742fa529d8 --endpoint https://172.26.161.22
urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.22'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

Aug 02 06:20:47 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:47.339Z        info        middleware/reqinfo.go:210        request start        {"request_id": "b57aaad6-07ef-46ff-a885-8ad7fb6b6229", "host": "172.26.161.22", "remote_host": "127.0.0.1"}
Aug 02 06:20:47 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:47.340Z        info        layer/layer.go:777        resolve bucket        {"request_id": "b57aaad6-07ef-46ff-a885-8ad7fb6b6229", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "cid": "99A9UQLncdknmkq49ABC2N7XAQwuHdn3tnU8NHFuePFn"}
Aug 02 06:20:47 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:47.347Z        error        handler/util.go:39        request failed        {"status": 500, "request_id": "b57aaad6-07ef-46ff-a885-8ad7fb6b6229", "method": "ListObjectsV1", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "object": "", "description": "something went wrong", "error": "failed to get sub tree: rpc error: code = Unavailable desc = error reading from server: EOF"}
Aug 02 06:20:53 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:53.812Z        debug        pool/pool.go:1716        health has changed        {"address": "localhost:8080", "healthy": false}
Aug 02 06:20:55 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:55.417Z        debug        tree/pool.go:625        tree health has changed        {"address": "localhost:8080", "healthy": false, "error": "healthcheck tree service: rpc error: code = Unknown desc = service is being synchronized"}

Aug 02 06:21:03 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:21:03.815Z        debug        pool/pool.go:1716        health has changed        {"address": "localhost:8080", "healthy": true}
Aug 02 06:21:05 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:21:05.419Z        debug        tree/pool.go:625        tree health has changed        {"address": "localhost:8080", "healthy": true}

Version

FrostFS Storage node
Version: v0.0.1-772-gec8a631d
GoVersion: go1.20.5

FrostFS S3 Gateway
Version: v0.27.0-rc.1-96-g52931663
GoVersion: go1.20.5

Regression

Yes

Your Environment

Cloud
4 nodes

## Current Behavior Panic while tree sync ``` Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.516Z info frostfs-node/main.go:115 starting morph notification service... Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.516Z info frostfs-node/main.go:115 starting pprof service... Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.516Z info frostfs-node/main.go:115 starting metrics service... Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.516Z info frostfs-node/main.go:115 starting control service... Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.553Z error tree/sync.go:424 could not sync trees {"cid": "4SKhejehUhjkojhLZro7zudmM3Qyf6hHMQbuBxY6qPFT", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44460->172.26.162.50:8080: use of closed network connection"} Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z error tree/sync.go:424 could not sync trees {"cid": "3V4CiPTnqnyrDGWtqmxKYCFdiwXqPpKqzGQfK3a1soBa", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44460->172.26.162.50:8080: use of closed network connection"} Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z error tree/sync.go:424 could not sync trees {"cid": "5qpgs8JjN8SV4no75NV5hdkop6B5QZAq38pYGmkdJ3EG", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:46756->172.26.162.209:8080: use of closed network connection"} Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z error tree/sync.go:424 could not sync trees {"cid": "3jW3FZEMD7qy5N3ddLz1oz6cNgZGmNjdgotJrtdW8Lx2", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44504->172.26.162.50:8080: use of closed network connection"} Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z error tree/sync.go:424 could not sync trees {"cid": "7TexYjPS94QLfYdmxSjYXi7DUeoeRrqj1s2izCwbkDiR", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44504->172.26.162.50:8080: use of closed network connection"} Aug 02 06:22:45 tatlin-object-failover-node1 frostfs-node[410999]: 2023-08-02T06:22:45.554Z error tree/sync.go:424 could not sync trees {"cid": "616CXxP8HG93KKPBnhnuW5vdSgKEjKajud4GNVXRC36c", "error": "could not fetch tree ID list: rpc error: code = Unavailable desc = error reading from server: read tcp 172.26.162.249:44504->172.26.162.50:8080: use of closed network connection"} Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: panic: runtime error: invalid memory address or nil pointer dereference Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xe64600] Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: goroutine 8342 [running]: Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*GetSubTreeRequest_Body_Order).StableSize(0x41?) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_frostfs.pb.go:899 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2/util/proto.NestedStructureSize[...](0x6?, 0x0) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-api-go/v2@v2.15.1-0.20230726155259-7a5ee927c8a2/util/proto/marshal.go:280 +0x28 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*GetSubTreeRequest_Body).StableSize(0x42?) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_frostfs.pb.go:932 +0x293 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*GetSubTreeRequest_Body).StableMarshal(0xc000a3a700, {0x0?, 0x270e?, 0xc00020912b?}) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_frostfs.pb.go:949 +0x38 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*GetSubTreeRequest).ReadSignedData(0x2?, {0x0?, 0x1da18c0?, 0x100c002266e70?}) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_frostfs.pb.go:1006 +0x2e Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.verifyMessage({0x14fbba8, 0xc00187b100}) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/signature.go:169 +0x46 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*Service).verifyClient(0xc000531e00, {0x14fbba8, 0xc00187b100}, {0x78, 0xf4, 0x7b, 0xbf, 0x26, 0x26, 0x9c, ...}, ...) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/signature.go:50 +0x6c Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree.(*Service).GetSubTree(0xc000531e00, 0xc00187b100, {0x1501b80, 0xc001211890}) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service.go:409 +0x1c5 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree._TreeService_GetSubTree_Handler({0x1273dc0?, 0xc000531e00}, {0x14ff410, 0xc001778ca0}) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-node/pkg/services/tree/service_grpc.pb.go:376 +0xd0 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-observability/tracing/grpc.NewStreamServerInterceptor.func1({0x1273dc0, 0xc000531e00}, {0x14ffc80, 0xc001772d20}, 0xc002281998, 0x13b0ab0) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: git.frostfs.info/TrueCloudLab/frostfs-observability@v0.0.0-20230531082742-c97d21411eb6/tracing/grpc/interceptors.go:110 +0x41d Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.getChainStreamHandler.func1({0x1273dc0, 0xc000531e00}, {0x14ffc80, 0xc001772d20}) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc@v1.56.1/server.go:1483 +0xb9 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.StreamServerInterceptor.func1({0x1273dc0, 0xc000531e00}, {0x1500190, 0xc0001813b0}, 0xc002281998?, 0xc00187afc0) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.0-rc.5/interceptors/server.go:35 +0x31f Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.chainStreamInterceptors.func1({0x1273dc0, 0xc000531e00}, {0x1500190, 0xc0001813b0}, 0x1030120?, 0xc0012114f0?) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc@v1.56.1/server.go:1474 +0x8f Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001783c0, {0x15036b8, 0xc000562d00}, 0xc000a3d0e0, 0xc000772450, 0x1df9c20, 0x0) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc@v1.56.1/server.go:1638 +0x1363 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.(*Server).handleStream(0xc0001783c0, {0x15036b8, 0xc000562d00}, 0xc000a3d0e0, 0x0) Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc@v1.56.1/server.go:1718 +0x9f0 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc.(*Server).serveStreams.func1.1() Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc@v1.56.1/server.go:959 +0x98 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: created by google.golang.org/grpc.(*Server).serveStreams.func1 Aug 02 06:25:12 tatlin-object-failover-node1 frostfs-node[410999]: google.golang.org/grpc@v1.56.1/server.go:957 +0x18c Aug 02 06:25:12 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Aug 02 06:25:12 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Failed with result 'exit-code'. Aug 02 06:25:12 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Consumed 2.761s CPU time. Aug 02 06:25:17 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Scheduled restart job, restart counter is at 44. Aug 02 06:25:17 tatlin-object-failover-node1 systemd[1]: Stopped frostfs Storage node. Aug 02 06:25:17 tatlin-object-failover-node1 systemd[1]: frostfs-storage.service: Consumed 2.761s CPU time. ``` ## Steps to Reproduce (for bugs) 1.Create bucket in s3-gw and put object ``` COMMAND: 'aws --no-verify-ssl --no-paginate s3api create-bucket --bucket 718ab48b-5dd3-418c-bbca-2a742fa529d8 --object-lock-enabled-for-bucket --endpoint https://172.26.161.22' OUTPUT: urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.22'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings RC: 0 Start / End / Elapsed 00:52:23.445751 / 00:52:39.497231 / 0:00:16.051480 COMMAND: 'aws --no-verify-ssl --no-paginate s3api put-object --bucket 718ab48b-5dd3-418c-bbca-2a742fa529d8 --key 06f8c5ff-7a6b-422d-9bca-fb6dbfdf9281 --body TemporaryDir/06f8c5ff-7a6b-422d-9bca-fb6dbfdf9281 --endpoint https://172.26.161.22' OUTPUT: urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.22'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings { "ETag": "b4ee56ed0dbfff9216f5c2fb024831432065114123ae583d24a630dc42e845c0", "VersionId": "NavRgCufYeaByQgSqgaoVLgvx2b8MAZ4T1SFKTwjJdj" } RC: 0 Start / End / Elapsed 00:52:44.503662 / 00:52:45.446094 / 0:00:00.942432 ``` ``` Aug 02 00:52:24 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:24.300Z info middleware/reqinfo.go:210 request start {"request_id": "960c0cfc-cccc-47ae-860e-5460cb39ea93", "host": "172.26.161.22", "remote_host": "127.0.0.1"} Aug 02 00:52:39 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:39.357Z info handler/put.go:781 bucket is created {"request_id": "960c0cfc-cccc-47ae-860e-5460cb39ea93", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "container_id": "99A9UQLncdknmkq49ABC2N7XAQwuHdn3tnU8NHFuePFn"} Aug 02 00:52:39 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:39.376Z info middleware/response.go:322 request end {"request_id": "960c0cfc-cccc-47ae-860e-5460cb39ea93", "method": "CreateBucket", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "object": "", "status": 200, "description": "OK"} Aug 02 00:52:45 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:45.273Z info middleware/reqinfo.go:210 request start {"request_id": "10617f85-d369-4dc2-af3c-7771b88ae5a5", "host": "172.26.161.22", "remote_host": "127.0.0.1"} Aug 02 00:52:45 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:45.329Z info middleware/response.go:322 request end {"request_id": "10617f85-d369-4dc2-af3c-7771b88ae5a5", "method": "PutObject", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "object": "06f8c5ff-7a6b-422d-9bca-fb6dbfdf9281", "status": 200, "description": "OK"} Aug 02 00:52:46 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:46.216Z info middleware/reqinfo.go:210 request start {"request_id": "5d0e6000-a2a8-4840-8b69-36707be79ebc", "host": "172.26.161.22", "remote_host": "127.0.0.1"} Aug 02 00:52:46 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T00:52:46.224Z error handler/util.go:39 request failed {"status": 500, "request_id": "5d0e6000-a2a8-4840-8b69-36707be79ebc", "method": "ListObjectsV1", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "object": "", "description": "something went wrong", "error": "failed to get sub tree: rpc error: code = Unavailable desc = error reading from server: EOF"} ``` 3. List objects in trough s3-gw: ``` anikeev@NB-1670:~/neofs$ aws --no-verify-ssl --no-paginate s3api list-objects --bucket 718ab48b-5dd3-418c-bbca-2a742fa529d8 --endpoint https://172.26.161.22 urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host '172.26.161.22'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings ``` ``` Aug 02 06:20:47 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:47.339Z info middleware/reqinfo.go:210 request start {"request_id": "b57aaad6-07ef-46ff-a885-8ad7fb6b6229", "host": "172.26.161.22", "remote_host": "127.0.0.1"} Aug 02 06:20:47 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:47.340Z info layer/layer.go:777 resolve bucket {"request_id": "b57aaad6-07ef-46ff-a885-8ad7fb6b6229", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "cid": "99A9UQLncdknmkq49ABC2N7XAQwuHdn3tnU8NHFuePFn"} Aug 02 06:20:47 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:47.347Z error handler/util.go:39 request failed {"status": 500, "request_id": "b57aaad6-07ef-46ff-a885-8ad7fb6b6229", "method": "ListObjectsV1", "bucket": "718ab48b-5dd3-418c-bbca-2a742fa529d8", "object": "", "description": "something went wrong", "error": "failed to get sub tree: rpc error: code = Unavailable desc = error reading from server: EOF"} Aug 02 06:20:53 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:53.812Z debug pool/pool.go:1716 health has changed {"address": "localhost:8080", "healthy": false} Aug 02 06:20:55 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:20:55.417Z debug tree/pool.go:625 tree health has changed {"address": "localhost:8080", "healthy": false, "error": "healthcheck tree service: rpc error: code = Unknown desc = service is being synchronized"} Aug 02 06:21:03 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:21:03.815Z debug pool/pool.go:1716 health has changed {"address": "localhost:8080", "healthy": true} Aug 02 06:21:05 tatlin-object-failover-node1 frostfs-s3-gw[50373]: 2023-08-02T06:21:05.419Z debug tree/pool.go:625 tree health has changed {"address": "localhost:8080", "healthy": true} ``` ## Version ``` FrostFS Storage node Version: v0.0.1-772-gec8a631d GoVersion: go1.20.5 FrostFS S3 Gateway Version: v0.27.0-rc.1-96-g52931663 GoVersion: go1.20.5 ``` ## Regression Yes ## Your Environment Cloud 4 nodes
anikeev-yadro added the
bug
triage
 labels 2023-08-02 06:36:17 +00:00
fyrchik self-assigned this 2023-08-02 07:23:47 +00:00
fyrchik added
P0
 and removed
triage
 labels 2023-08-02 07:33:56 +00:00
fyrchik added the
frostfs-node
 label 2023-08-02 08:19:35 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
support/v0.42
support/v0.44
support/v0.38
support/v0.37
support/v0.36
support/v0.34
support/v0.30
support/v0.27
v0.42.18
v0.44.3
v0.44.2
v0.42.17
v0.44.1
v0.44.0
v0.44.0-rc.13
v0.44.0-rc.12
v0.44.0-rc.11
v0.42.16
v0.44.0-rc.10
v0.44.0-rc.9
v0.44.0-rc.8
v0.44.0-rc.7
v0.44.0-rc.6
v0.44.0-rc.5
v0.44.0-rc.4
v0.44.0-rc.3
v0.44.0-rc.2
v0.44.0-rc.1
v0.43.2
v0.43.1
v0.43.0
v0.42.15
v0.42.14
v0.42.13
v0.42.12
v0.42.11
v0.42.10
v0.42.9
v0.42.8
v0.42.7
v0.42.6
v0.42.5
v0.42.4
v0.42.3
v0.42.2
v0.42.1
v0.42.0
v0.42.0-rc.9
v0.42.0-rc.8
v0.42.0-rc.7
v0.38.9
v0.42.0-rc.6
v0.42.0-rc.5
v0.42.0-rc.4
v0.42.0-rc.3
v0.42.0-rc.2
v0.42.0-rc.1
v0.38.8
v0.41.0
v0.38.7
v0.40.0
v0.39.0
v0.38.6
v0.38.5
v0.38.4
v0.38.3
v0.38.2
v0.38.1
v0.38.0
v0.38.0-rc.2
v0.38.0-rc.1
v0.37.0
v0.37.0-rc.1
v0.36.0
v0.34.0
v0.22.1
v0.22.0
v0.21.1
v0.21.0
v0.20.0
v0.19.0
v0.18.0
v0.17.0
v0.16.0
v0.15.0
v0.14.3
v0.14.2
v0.14.1
v0.14.0
v0.14.0-rc.1
v0.13.2
v0.13.1
v0.13.0
v0.13.0-rc.1
v0.12.1
v0.12.0
v0.12.0-rc3
v0.12.0-rc2
v0.12.0-rc1
v0.11.0
v0.10.0
Labels
Clear labels   
P0

Highest

  
P1

High

  
P2

Medium

  
P3

Low

  
badger

Badger DB related issues

  
frostfs-adm

   
frostfs-cli

   
frostfs-ir

   
frostfs-lens

   
frostfs-node

   
good first issue

Good for newcomers

  
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff

  
blocked

The issue or PR is blocked by something

  
bug

Something is not working

  
config

Configuration format update or breaking change

  
discussion

Talking about something in order to reach a decision or to exchange ideas

  
documentation

Documentation related

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
go

Language features adoption

  
help wanted

Extra attention is needed

  
internal

   
invalid

Something is wrong

  
kludge

This is a kludge and we know it

  
observability

Related to metrics, tracing and logs

  
perfomance

Perfomance related

  
question

More information is needed

  
refactoring

Refactoring

  
wontfix

This won't be fixed

No labels
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
fyrchik
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#554
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?