TrueCloudLab
/
frostfs-node
18
1
Fork 20
Code Issues 88 Pull Requests 8 Actions Releases Activity

Expired object wasn't deleted after locks are expired for complex object (REOPEN #332) #731

New Issue
Closed
opened 2023-10-09 10:20:01 +00:00 by anikeev-yadro · 1 comment
anikeev-yadro commented 2023-10-09 10:20:01 +00:00

Autotest

testsuites.object.test_object_lock.TestObjectLockWithGrpc#test_expired_object_should_be_deleted_after_locks_are_expired

Expected Behavior

Expired object should be deleted after locks are expired for complex object

Current Behavior

Expired object wasn't deleted after locks are expired for complex object

Steps to Reproduce (for bugs)

  1. Create object with expite at 116 epoch
16:41:48 [INFO] Executing command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.yaml object put --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --file 'TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5' --expire-at 116 --no-progress
16:41:58 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.yaml object put --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --file 'TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5' --expire-at 116 --no-progress
Success:
return code: 0 
Output: [TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5] Object successfully stored
  OID: 48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW
  CID: 564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL
  1. Set up some locks
16:41:58 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 118
Success:
return code: 0 
Output: Lock object ID: 6zAzhRcxpXcZUBUFnSvLRDTnXFNc5qvB3mzghpRj1M6Z
Objects successfully locked.

16:42:19 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 120
Success:
return code: 0 
Output: Lock object ID: 2gHJruNx29nJvpjTfFRPNTgZwV1qHkpETBL72US6MGHE
Objects successfully locked.

/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 122
Success:
return code: 0 
Output: Lock object ID: EHUxMiQobSwZaijGVFMv3nzUVH8nhNjQpyzA2hj2LhS2
Objects successfully locked.
  1. Tick epoch
HOST: 10.78.69.135
COMMAND:
 sudo frostfs-adm --config /home/service/config.yaml morph force-new-epoch 
RC:
 0
STDOUT:
 Current epoch: 122, increase to 123.
 Waiting for transactions to persist...

STDERR:
Start / End / Elapsed	 16:42:58.344732 / 16:43:06.205156 / 0:00:07.860424
  1. But object was not deleted
COMMAND: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object head --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --json
RETCODE: 0

STDOUT:
{"objectID":{"value":"Ln7Yyj6bfa0uMAyXS1Gm+qDlKgrLkMZCCmDjIJNH6Gc="}, "signature":{"key":"AjUG/oxtypP4SHbyvQKfFnKm/yTKDwiuEu6WU6IJy0SV", "signature":"BC5DpCtAxmPcWUJ6yj3svBJmFlzNBC/BQaAP2xIGk5FhCK4l0Vt6bBaKURRMi66P0aulGqMoCP+H1AOXtRu+G5w=", "scheme":"ECDSA_SHA512"}, "header":{"version":{"major":2, "minor":13}, "containerID":{"value":"PLsAvmCcJpjb/MwlpcpPumcfafus2/SVQpQNaGtA8Pk="}, "ownerID":{"value":"NXZciZjiqh3ithFlk5QR7l1tHEhTGd5Nfw=="}, "creationEpoch":"115", "payloadLength":"201327592", "payloadHash":{"type":"SHA256", "sum":"JKoAL9yvTxq6hOeK0xgY7ScYjN2Jt6LsGf/vj2oWvOU="}, "objectType":"REGULAR", "homomorphicHash":null, "sessionToken":{"body":{"id":"+RnnnccKQjKi/45J4gVowg==", "ownerID":{"value":"NXZciZjiqh3ithFlk5QR7l1tHEhTGd5Nfw=="}, "lifetime":{"exp":"125", "nbf":"115", "iat":"115"}, "sessionKey":"AjUG/oxtypP4SHbyvQKfFnKm/yTKDwiuEu6WU6IJy0SV", "object":{"verb":"PUT", "target":{"container":{"value":"PLsAvmCcJpjb/MwlpcpPumcfafus2/SVQpQNaGtA8Pk="}, "objects":[]}}}, "signature":{"key":"AoxlzfnyDqNhgPgOM5Uc2Syv/gHZA5qi9D57tZL5BIk+", "signature":"BEjTM112fnTiwzHqcrNlbTs4Ri0f2Yne5oqZaQFgwKSLloR4m6UtsiKFcyzzNNyErLb2ztNmGKNqVWi/ercuEAw=", "scheme":"ECDSA_SHA512"}}, "attributes":[{"key":"FileName", "value":"b60799a3-d5eb-4c35-9478-fc1a774fcee5"}, {"key":"Timestamp", "value":"1696610509"}, {"key":"__SYSTEM__EXPIRATION_EPOCH", "value":"116"}], "split":null}, "payload":""}

STDERR:
Start / End / Elapsed	 16:44:21.162868 / 16:44:21.714287 / 0:00:00.551419

Regression

Yes

Version

0.37.0-rc.1-5-ge0f0b93b

Your Environment

HW

## Autotest testsuites.object.test_object_lock.TestObjectLockWithGrpc#test_expired_object_should_be_deleted_after_locks_are_expired ## Expected Behavior Expired object should be deleted after locks are expired for complex object ## Current Behavior Expired object wasn't deleted after locks are expired for complex object ## Steps to Reproduce (for bugs) 1. Create object with expite at 116 epoch ``` 16:41:48 [INFO] Executing command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.yaml object put --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --file 'TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5' --expire-at 116 --no-progress 16:41:58 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.yaml object put --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --file 'TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5' --expire-at 116 --no-progress Success: return code: 0 Output: [TemporaryDir/b60799a3-d5eb-4c35-9478-fc1a774fcee5] Object successfully stored OID: 48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW CID: 564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL ``` 2. Set up some locks ``` 16:41:58 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 118 Success: return code: 0 Output: Lock object ID: 6zAzhRcxpXcZUBUFnSvLRDTnXFNc5qvB3mzghpRj1M6Z Objects successfully locked. 16:42:19 [INFO] Command: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 120 Success: return code: 0 Output: Lock object ID: 2gHJruNx29nJvpjTfFRPNTgZwV1qHkpETBL72US6MGHE Objects successfully locked. /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object lock --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --expire-at 122 Success: return code: 0 Output: Lock object ID: EHUxMiQobSwZaijGVFMv3nzUVH8nhNjQpyzA2hj2LhS2 Objects successfully locked. ``` 3. Tick epoch ``` HOST: 10.78.69.135 COMMAND: sudo frostfs-adm --config /home/service/config.yaml morph force-new-epoch RC: 0 STDOUT: Current epoch: 122, increase to 123. Waiting for transactions to persist... STDERR: Start / End / Elapsed 16:42:58.344732 / 16:43:06.205156 / 0:00:07.860424 ``` 4. But object was not deleted ``` COMMAND: frostfs-cli --config /jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/wallet_config.yml object head --rpc-endpoint '10.78.70.135:8080' --wallet '/jenkins/workspace/frostfs_tatlin_object_to_baremetal_test/tmp.LQbtmUHB42/frostfs-testcases/TemporaryDir/47450aad-7d38-4cea-8097-175617db792e.json' --cid '564rCHe4eF4hce2FYFp1NzUAnQhigm123x3wQJnv5yQL' --oid '48VwQi28SuRwtnjvzQEzTQF6dUirDCB6BiVrvhPADeGW' --json RETCODE: 0 STDOUT: {"objectID":{"value":"Ln7Yyj6bfa0uMAyXS1Gm+qDlKgrLkMZCCmDjIJNH6Gc="}, "signature":{"key":"AjUG/oxtypP4SHbyvQKfFnKm/yTKDwiuEu6WU6IJy0SV", "signature":"BC5DpCtAxmPcWUJ6yj3svBJmFlzNBC/BQaAP2xIGk5FhCK4l0Vt6bBaKURRMi66P0aulGqMoCP+H1AOXtRu+G5w=", "scheme":"ECDSA_SHA512"}, "header":{"version":{"major":2, "minor":13}, "containerID":{"value":"PLsAvmCcJpjb/MwlpcpPumcfafus2/SVQpQNaGtA8Pk="}, "ownerID":{"value":"NXZciZjiqh3ithFlk5QR7l1tHEhTGd5Nfw=="}, "creationEpoch":"115", "payloadLength":"201327592", "payloadHash":{"type":"SHA256", "sum":"JKoAL9yvTxq6hOeK0xgY7ScYjN2Jt6LsGf/vj2oWvOU="}, "objectType":"REGULAR", "homomorphicHash":null, "sessionToken":{"body":{"id":"+RnnnccKQjKi/45J4gVowg==", "ownerID":{"value":"NXZciZjiqh3ithFlk5QR7l1tHEhTGd5Nfw=="}, "lifetime":{"exp":"125", "nbf":"115", "iat":"115"}, "sessionKey":"AjUG/oxtypP4SHbyvQKfFnKm/yTKDwiuEu6WU6IJy0SV", "object":{"verb":"PUT", "target":{"container":{"value":"PLsAvmCcJpjb/MwlpcpPumcfafus2/SVQpQNaGtA8Pk="}, "objects":[]}}}, "signature":{"key":"AoxlzfnyDqNhgPgOM5Uc2Syv/gHZA5qi9D57tZL5BIk+", "signature":"BEjTM112fnTiwzHqcrNlbTs4Ri0f2Yne5oqZaQFgwKSLloR4m6UtsiKFcyzzNNyErLb2ztNmGKNqVWi/ercuEAw=", "scheme":"ECDSA_SHA512"}}, "attributes":[{"key":"FileName", "value":"b60799a3-d5eb-4c35-9478-fc1a774fcee5"}, {"key":"Timestamp", "value":"1696610509"}, {"key":"__SYSTEM__EXPIRATION_EPOCH", "value":"116"}], "split":null}, "payload":""} STDERR: Start / End / Elapsed 16:44:21.162868 / 16:44:21.714287 / 0:00:00.551419 ``` ## Regression Yes ## Version ``` 0.37.0-rc.1-5-ge0f0b93b ``` ## Your Environment HW
anikeev-yadro added the
bug
triage
 labels 2023-10-09 10:20:01 +00:00
dstepanov-yadro self-assigned this 2023-10-09 11:15:53 +00:00
fyrchik added this to the v0.37.0 milestone 2023-10-09 19:03:18 +00:00
fyrchik added
frostfs-node
 and removed
triage
 labels 2023-10-09 19:03:25 +00:00
dstepanov-yadro commented 2023-10-10 15:28:06 +00:00
Collaborator

It was not reproduced. The problem was probably an incorrect configuration.

It was not reproduced. The problem was probably an incorrect configuration.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
support/v0.38
support/v0.37
support/v0.36
support/v0.34
support/v0.30
support/v0.27
v0.42.0-rc.1
v0.38.8
v0.41.0
v0.38.7
v0.40.0
v0.39.0
v0.38.6
v0.38.5
v0.38.4
v0.38.3
v0.38.2
v0.38.1
v0.38.0
v0.38.0-rc.2
v0.38.0-rc.1
v0.37.0
v0.37.0-rc.1
v0.36.0
v0.34.0
Labels
Clear labels   
P0

Highest   
P1

High   
P2

Medium   
P3

Low   
badger

Badger DB related issues   
frostfs-adm
   
frostfs-cli
   
frostfs-ir
   
frostfs-lens
   
frostfs-node
   
good first issue

Good for newcomers   
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff   
blocked

The issue or PR is blocked by something   
bug

Something is not working   
config

Configuration format update or breaking change   
discussion

Talking about something in order to reach a decision or to exchange ideas   
documentation

Documentation related   
duplicate

This issue or pull request already exists   
enhancement

New feature   
go

Language features adoption   
help wanted

Extra attention is needed   
internal
   
invalid

Something is wrong   
kludge

This is a kludge and we know it   
observability

Related to metrics, tracing and logs   
perfomance

Perfomance related   
question

More information is needed   
refactoring

Refactoring   
wontfix

This won't be fixed
No Label
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No Milestone
v0.37.0
Assignees
Clear assignees
No Assignees
dstepanov-yadro
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#731
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?