Expired object wasn't deleted after locks are expired for complex object (REOPEN #332) #731

New issue

Closed

opened 2023-10-09 10:20:01 +00:00 by anikeev-yadro · 1 comment

anikeev-yadro commented

2023-10-09 10:20:01 +00:00

Member

Autotest

testsuites.object.test_object_lock.TestObjectLockWithGrpc#test_expired_object_should_be_deleted_after_locks_are_expired

Expected Behavior

Expired object should be deleted after locks are expired for complex object

Current Behavior

Expired object wasn't deleted after locks are expired for complex object

Steps to Reproduce (for bugs)

Create object with expite at 116 epoch

16:41:48 [INFO] Executing command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.yaml object put --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --file 'TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5' --expire-at 116 --no-progress
16:41:58 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.yaml object put --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --file 'TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5' --expire-at 116 --no-progress
Success:
return code: 0 
Output: [TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5] Object successfully stored
  OID: 48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW
  CID: 564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL

Set up some locks

16:41:58 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 118
Success:
return code: 0 
Output: Lock object ID: 6zAzhRcxpXcZUBUFnSvLRDTnXFNc5qvB3mzghpRj1M6Z
Objects successfully locked.

16:42:19 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 120
Success:
return code: 0 
Output: Lock object ID: 2gHJruNx29nJvpjTfFRPNTgZwV1qHkpETBL72US6MGHE
Objects successfully locked.

/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 122
Success:
return code: 0 
Output: Lock object ID: EHUxMiQobSwZaijGVFMv3nzUVH8nhNjQpyzA2hj2LhS2
Objects successfully locked.

Tick epoch

HOST: 10.78.69.135
COMMAND:
 sudo frostfs-adm --config /home/service/config.yaml morph force-new-epoch 
RC:
 0
STDOUT:
 Current epoch: 122, increase to 123.
 Waiting for transactions to persist...

STDERR:
Start / End / Elapsed	 16:42:58.344732 / 16:43:06.205156 / 0:00:07.860424

But object was not deleted

COMMAND: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object head --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --json
RETCODE: 0

STDOUT:
{"objectID":{"value":"Ln7Yyj6bfa0uMAyXS1Gm+qDlKgrLkMZCCmDjIJNH6Gc="}, "signature":{"key":"AjUG/oxtypP4SHbyvQKfFnKm/yTKDwiuEu6WU6IJy0SV", "signature":"BC5DpCtAxmPcWUJ6yj3svBJmFlzNBC/BQaAP2xIGk5FhCK4l0Vt6bBaKURRMi66P0aulGqMoCP+H1AOXtRu+G5w=", "scheme":"ECDSA_SHA512"}, "header":{"version":{"major":2, "minor":13}, "containerID":{"value":"PLsAvmCcJpjb/MwlpcpPumcfafus2/SVQpQNaGtA8Pk="}, "ownerID":{"value":"NXZciZjiqh3ithFlk5QR7l1tHEhTGd5Nfw=="}, "creationEpoch":"115", "payloadLength":"201327592", "payloadHash":{"type":"SHA256", "sum":"JKoAL9yvTxq6hOeK0xgY7ScYjN2Jt6LsGf/vj2oWvOU="}, "objectType":"REGULAR", "homomorphicHash":null, "sessionToken":{"body":{"id":"+RnnnccKQjKi/45J4gVowg==", "ownerID":{"value":"NXZciZjiqh3ithFlk5QR7l1tHEhTGd5Nfw=="}, "lifetime":{"exp":"125", "nbf":"115", "iat":"115"}, "sessionKey":"AjUG/oxtypP4SHbyvQKfFnKm/yTKDwiuEu6WU6IJy0SV", "object":{"verb":"PUT", "target":{"container":{"value":"PLsAvmCcJpjb/MwlpcpPumcfafus2/SVQpQNaGtA8Pk="}, "objects":[]}}}, "signature":{"key":"AoxlzfnyDqNhgPgOM5Uc2Syv/gHZA5qi9D57tZL5BIk+", "signature":"BEjTM112fnTiwzHqcrNlbTs4Ri0f2Yne5oqZaQFgwKSLloR4m6UtsiKFcyzzNNyErLb2ztNmGKNqVWi/ercuEAw=", "scheme":"ECDSA_SHA512"}}, "attributes":[{"key":"FileName", "value":"b60799a3-d5eb-4c35-9478-fc1a774fcee5"}, {"key":"Timestamp", "value":"1696610509"}, {"key":"__SYSTEM__EXPIRATION_EPOCH", "value":"116"}], "split":null}, "payload":""}

STDERR:
Start / End / Elapsed	 16:44:21.162868 / 16:44:21.714287 / 0:00:00.551419

Regression

Yes

Version

0.37.0-rc.1-5-ge0f0b93b

Your Environment

## Autotest testsuites.object.test_object_lock.TestObjectLockWithGrpc#test_expired_object_should_be_deleted_after_locks_are_expired ## Expected Behavior Expired object should be deleted after locks are expired for complex object ## Current Behavior Expired object wasn't deleted after locks are expired for complex object ## Steps to Reproduce (for bugs) 1. Create object with expite at 116 epoch ``` 16:41:48 [INFO] Executing command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.yaml object put --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --file 'TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5' --expire-at 116 --no-progress 16:41:58 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.yaml object put --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --file 'TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5' --expire-at 116 --no-progress Success: return code: 0 Output: [TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5] Object successfully stored OID: 48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW CID: 564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL ``` 2. Set up some locks ``` 16:41:58 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 118 Success: return code: 0 Output: Lock object ID: 6zAzhRcxpXcZUBUFnSvLRDTnXFNc5qvB3mzghpRj1M6Z Objects successfully locked. 16:42:19 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 120 Success: return code: 0 Output: Lock object ID: 2gHJruNx29nJvpjTfFRPNTgZwV1qHkpETBL72US6MGHE Objects successfully locked. /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 122 Success: return code: 0 Output: Lock object ID: EHUxMiQobSwZaijGVFMv3nzUVH8nhNjQpyzA2hj2LhS2 Objects successfully locked. ``` 3. Tick epoch ``` HOST: 10.78.69.135 COMMAND: sudo frostfs-adm --config /home/service/config.yaml morph force-new-epoch RC: 0 STDOUT: Current epoch: 122, increase to 123. Waiting for transactions to persist... STDERR: Start / End / Elapsed 16:42:58.344732 / 16:43:06.205156 / 0:00:07.860424 ``` 4. But object was not deleted ``` COMMAND: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object head --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --json RETCODE: 0 STDOUT: {"objectID":{"value":"Ln7Yyj6bfa0uMAyXS1Gm+qDlKgrLkMZCCmDjIJNH6Gc="}, "signature":{"key":"AjUG/oxtypP4SHbyvQKfFnKm/yTKDwiuEu6WU6IJy0SV", "signature":"BC5DpCtAxmPcWUJ6yj3svBJmFlzNBC/BQaAP2xIGk5FhCK4l0Vt6bBaKURRMi66P0aulGqMoCP+H1AOXtRu+G5w=", "scheme":"ECDSA_SHA512"}, "header":{"version":{"major":2, "minor":13}, "containerID":{"value":"PLsAvmCcJpjb/MwlpcpPumcfafus2/SVQpQNaGtA8Pk="}, "ownerID":{"value":"NXZciZjiqh3ithFlk5QR7l1tHEhTGd5Nfw=="}, "creationEpoch":"115", "payloadLength":"201327592", "payloadHash":{"type":"SHA256", "sum":"JKoAL9yvTxq6hOeK0xgY7ScYjN2Jt6LsGf/vj2oWvOU="}, "objectType":"REGULAR", "homomorphicHash":null, "sessionToken":{"body":{"id":"+RnnnccKQjKi/45J4gVowg==", "ownerID":{"value":"NXZciZjiqh3ithFlk5QR7l1tHEhTGd5Nfw=="}, "lifetime":{"exp":"125", "nbf":"115", "iat":"115"}, "sessionKey":"AjUG/oxtypP4SHbyvQKfFnKm/yTKDwiuEu6WU6IJy0SV", "object":{"verb":"PUT", "target":{"container":{"value":"PLsAvmCcJpjb/MwlpcpPumcfafus2/SVQpQNaGtA8Pk="}, "objects":[]}}}, "signature":{"key":"AoxlzfnyDqNhgPgOM5Uc2Syv/gHZA5qi9D57tZL5BIk+", "signature":"BEjTM112fnTiwzHqcrNlbTs4Ri0f2Yne5oqZaQFgwKSLloR4m6UtsiKFcyzzNNyErLb2ztNmGKNqVWi/ercuEAw=", "scheme":"ECDSA_SHA512"}}, "attributes":[{"key":"FileName", "value":"b60799a3-d5eb-4c35-9478-fc1a774fcee5"}, {"key":"Timestamp", "value":"1696610509"}, {"key":"__SYSTEM__EXPIRATION_EPOCH", "value":"116"}], "split":null}, "payload":""} STDERR: Start / End / Elapsed 16:44:21.162868 / 16:44:21.714287 / 0:00:00.551419 ``` ## Regression Yes ## Version ``` 0.37.0-rc.1-5-ge0f0b93b ``` ## Your Environment HW