Tree syncronization misses some log operations #391

New issue

Closed

opened 2023-05-25 12:29:12 +00:00 by anikeev-yadro · 0 comments

anikeev-yadro commented

2023-05-25 12:29:12 +00:00

Member

Expected Behavior

Node should sync tree from high from the last sync.

Current Behavior

Node was synced tree from wrong high (maybe from the highest height among all container nodes).

Steps to Reproduce (for bugs)

Start load on all nodes except one
Reboot excepted node
After node started see in logs that conatner was synced from 4134 (have got CID as exapmle from load)

May 24 17:27:08 yest frostfs-node[8524]: 2023-05-24T17:27:08.128Z        debug        tree/sync.go:340        syncing container trees...        {"cid": "6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d"}
May 24 17:27:08 yest frostfs-node[8524]: 2023-05-24T17:27:08.129Z        debug        tree/sync.go:129        synchronize tree        {"cid": "6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d", "tree": "version", "from": 4134}
May 24 17:27:08 yest frostfs-node[8524]: 2023-05-24T17:27:08.131Z        debug        tree/sync.go:348        container trees have been synced        {"cid": "6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d"}

It leads that tree service objects not accessed

service@loader01:/var/log/autotests/proc_5156d15a-002c-4c33-b506-7cbefdecc89a$ aws s3api head-object --bucket f9c6edf6-7e7b-4a42-9a8f-050783f5c27d --key c8dde548-141b-49d8-854e-04521763965a  --endpoint-url http://10.78.70.107:8084

An error occurred (404) when calling the HeadObject operation: Not Found

But it accessed from other container nodes:

service@loader01:/var/log/autotests/proc_5156d15a-002c-4c33-b506-7cbefdecc89a$ aws s3api head-object --bucket f9c6edf6-7e7b-4a42-9a8f-050783f5c27d --key c8dde548-141b-49d8-854e-04521763965a  --endpoint-url http://10.78.70.106:8084
{
    "LastModified": "2023-05-24T17:20:16+00:00",
    "ContentLength": 1048576,
    "ETag": "79dc6c90bcf1b967ed40062c1675028e02a8dcf12725cf686bc82944c0508304",
    "ContentType": "application/octet-stream",
    "Metadata": {}
}

Also we checked container tree info from all nodes

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.106:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.107:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.108:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.109:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.110:8080 --latest --verbose                    

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.111:8080 --latest --verbose                    

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.112:8080 --latest --verbose                    
1023211176850576506:
        Parent ID:  0
        Timestamp:  3578
        Meta pairs: 
                OID: 5PWdjsgffhJpDguRqC1UmQCYWH5GQuzU7mCZDWLeKq7D
                FileName: c8dde548-141b-49d8-854e-04521763965a
                Size: 1048576
                ETag: 79dc6c90bcf1b967ed40062c1675028e02a8dcf12725cf686bc82944c0508304
                IsUnversioned: true

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.113:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.114:8080 --latest --verbose                    
1023211176850576506:
        Parent ID:  0
        Timestamp:  3578
        Meta pairs: 
                OID: 5PWdjsgffhJpDguRqC1UmQCYWH5GQuzU7mCZDWLeKq7D
                FileName: c8dde548-141b-49d8-854e-04521763965a
                Size: 1048576
                ETag: 79dc6c90bcf1b967ed40062c1675028e02a8dcf12725cf686bc82944c0508304
                IsUnversioned: true

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.115:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.116:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.117:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.118:8080 --latest --verbose                    
1023211176850576506:
        Parent ID:  0
        Timestamp:  3578
        Meta pairs: 
                OID: 5PWdjsgffhJpDguRqC1UmQCYWH5GQuzU7mCZDWLeKq7D
                FileName: c8dde548-141b-49d8-854e-04521763965a
                Size: 1048576
                ETag: 79dc6c90bcf1b967ed40062c1675028e02a8dcf12725cf686bc82944c0508304
                IsUnversioned: true

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.119:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.120:8080 --latest --verbose                    
The node is not found

$ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.121:8080 --latest --verbose                    
The node is not found

Info accessed from 3 nodes except one node which was rebooted.

Context

Failover test with reboot node while load.

Version

FrostFS Storage node
Version: v0.0.1-423-g1501f11e
GoVersion: go1.19.7

Your Environment

HW
16 nodes

## Expected Behavior Node should sync tree from high from the last sync. ## Current Behavior Node was synced tree from wrong high (maybe from the highest height among all container nodes). ## Steps to Reproduce (for bugs) 1. Start load on all nodes except one 2. Reboot excepted node 3. After node started see in logs that conatner was synced from 4134 (have got CID as exapmle from load) ``` May 24 17:27:08 yest frostfs-node[8524]: 2023-05-24T17:27:08.128Z debug tree/sync.go:340 syncing container trees... {"cid": "6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d"} May 24 17:27:08 yest frostfs-node[8524]: 2023-05-24T17:27:08.129Z debug tree/sync.go:129 synchronize tree {"cid": "6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d", "tree": "version", "from": 4134} May 24 17:27:08 yest frostfs-node[8524]: 2023-05-24T17:27:08.131Z debug tree/sync.go:348 container trees have been synced {"cid": "6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d"} ``` 4. It leads that tree service objects not accessed ``` service@loader01:/var/log/autotests/proc_5156d15a-002c-4c33-b506-7cbefdecc89a$ aws s3api head-object --bucket f9c6edf6-7e7b-4a42-9a8f-050783f5c27d --key c8dde548-141b-49d8-854e-04521763965a --endpoint-url http://10.78.70.107:8084 An error occurred (404) when calling the HeadObject operation: Not Found ``` 5. But it accessed from other container nodes: ``` service@loader01:/var/log/autotests/proc_5156d15a-002c-4c33-b506-7cbefdecc89a$ aws s3api head-object --bucket f9c6edf6-7e7b-4a42-9a8f-050783f5c27d --key c8dde548-141b-49d8-854e-04521763965a --endpoint-url http://10.78.70.106:8084 { "LastModified": "2023-05-24T17:20:16+00:00", "ContentLength": 1048576, "ETag": "79dc6c90bcf1b967ed40062c1675028e02a8dcf12725cf686bc82944c0508304", "ContentType": "application/octet-stream", "Metadata": {} } ``` 6. Also we checked container tree info from all nodes ``` $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.106:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.107:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.108:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.109:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.110:8080 --latest --verbose $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.111:8080 --latest --verbose $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.112:8080 --latest --verbose 1023211176850576506: Parent ID: 0 Timestamp: 3578 Meta pairs: OID: 5PWdjsgffhJpDguRqC1UmQCYWH5GQuzU7mCZDWLeKq7D FileName: c8dde548-141b-49d8-854e-04521763965a Size: 1048576 ETag: 79dc6c90bcf1b967ed40062c1675028e02a8dcf12725cf686bc82944c0508304 IsUnversioned: true $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.113:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.114:8080 --latest --verbose 1023211176850576506: Parent ID: 0 Timestamp: 3578 Meta pairs: OID: 5PWdjsgffhJpDguRqC1UmQCYWH5GQuzU7mCZDWLeKq7D FileName: c8dde548-141b-49d8-854e-04521763965a Size: 1048576 ETag: 79dc6c90bcf1b967ed40062c1675028e02a8dcf12725cf686bc82944c0508304 IsUnversioned: true $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.115:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.116:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.117:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.118:8080 --latest --verbose 1023211176850576506: Parent ID: 0 Timestamp: 3578 Meta pairs: OID: 5PWdjsgffhJpDguRqC1UmQCYWH5GQuzU7mCZDWLeKq7D FileName: c8dde548-141b-49d8-854e-04521763965a Size: 1048576 ETag: 79dc6c90bcf1b967ed40062c1675028e02a8dcf12725cf686bc82944c0508304 IsUnversioned: true $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.119:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.120:8080 --latest --verbose The node is not found $ frostfs-cli tree get-by-path --acckey 4BVN9ZHf9yBfG2VfCxH27B2QvF3K7BTia8aU7vf9sZEt04yemsrU1XDsEzVD2DGttgbXfRBK4q8zznLAy8phx3ywv --cid 6owJStxRC7KGVQoZA46X97oxfxSEbK1pPULEQD9GrJ1d --tid version -w ./s3w.json --path c8dde548-141b-49d8-854e-04521763965a -r 10.78.70.121:8080 --latest --verbose The node is not found ``` Info accessed from 3 nodes except one node which was rebooted. ## Context Failover test with reboot node while load. ## Version ``` FrostFS Storage node Version: v0.0.1-423-g1501f11e GoVersion: go1.19.7 ``` ## Your Environment HW 16 nodes