TrueCloudLab
/
frostfs-node
18
1
Fork 20
Code Issues 88 Pull Requests 8 Actions Releases Activity

PUT objects through s3 gate failed with "incomplete object PUT by placement" and "access to object operation denied" #457

New Issue
Closed
opened 2023-06-21 07:53:25 +00:00 by anikeev-yadro · 4 comments
anikeev-yadro commented 2023-06-21 07:53:25 +00:00

Expected Behavior

PUT objects through s3 gate should PUT objects withpout any errors.

Current Behavior

PUT objects through s3 gate failed with "incomplete object PUT by placement" and "access to object operation denied".

Steps to Reproduce (for bugs)

1.Create s3 creds:

HOST: 10.78.69.104
COMMAND:
 frostfs-s3-authmate  issue-secret --wallet '/tmp/781607aa-cadd-40b0-848d-5e25e0aab84c.json' --peer '10.78.70.106:8080' --bearer-rules '/etc/k6/scenarios/files/rules.json' --gate-public-key '031f6a5a08afea03d26bca24ad729c735ba6e6e4a249de95d7157854c2b3218d8b' --gate-public-key '02e880fb3306b322de9e4d0f633f1042ac5454b9533ab3aa2126f5e1203238f035' --gate-public-key '03220fee34eb0d363e8d9f6a057b7ca5d42e5b7fe9050ba888facbf460b454279d' --gate-public-key '02abac27f257e99a6716eabf243f282110765fd1754aac913bf0030c8e0270ee0b' --gate-public-key '03d15a52949c581138a8bbee0a3ff543ff5983d029a31788ce30ac88dbcf0a0e62' --gate-public-key '0278a6b79988699dcc32d98056cb5cc72648967cffe03b3c501570518b3d7d635a' --gate-public-key '031f1d49bee67fbabecf7ca09fbcb8936b51a958e27aa280b3de6bec40128c7400' --gate-public-key '020b0e21e14b102e38a3130dc97c36b63d6326d6861cb1e051b5e17eaa307f8ae8' --gate-public-key '02e4c9e0d90dd1b38935c7b870939afacd3247748df150795da45abf2a6e428d90' --gate-public-key '03d00334d5ca94cfc69a925a76e30f0c8229e4c984acfc3a6d54bbda93eaa5c95c' --gate-public-key '0284d19d9bb1369d1c0f8c66cea3e0d9b9bd9c9c652480e9ba13c971d66c362591' --gate-public-key '024660361361fc86db414cceff03a69b113a27cbe9aa67c340a838861ed59746cc' --gate-public-key '02a3807d3f427be8198252f6aff03da3e8d7132b260690b77aa9950954743b79c6' --gate-public-key '02c1ab830efac81e01a3d1057889932da53dacc6641f8d24fcadaa70749ea3961c' --gate-public-key '0354f999ec5372821b51c4a66cf5cf93c4c863f7adaa374c6d1c197482630ef434' --gate-public-key '0242f8f40baa1e2ca1ca3eafa10a84b80aba18ebed160f49ae93695c80c88e5833' --container-placement-policy 'REP 2 IN X CBF 2 SELECT 2 FROM * AS X' --container-policy '/etc/k6/scenarios/files/policy.json'
RC:
 0
STDOUT:

 Enter password for /tmp/781607aa-cadd-40b0-848d-5e25e0aab84c.json > 
 {
   "access_key_id": "Hvi6ZuL78dR8hMRHYr3THRv5jQTq3TQwBrCKF52uNrT0FAKsWnGo8cQaYJwgfgXCzCyLgGiJPJKXv3fFpakWUznq",
   "secret_access_key": "9918d4505e5d799a3795be263f2a5645c8df1c57cd6d9dcec49a6b65155a218d",
   "owner_private_key": "3a10ca0f4f69b92fdaa66850761ed10de4211a38584c62ea29b55e88a3a7ac5f",
   "wallet_public_key": "02fa2a20c66e05c5ac344c7a5e7e3312ecfe6a43b3fc0dcdbad442bfd6ad47c63c",
   "container_id": "Hvi6ZuL78dR8hMRHYr3THRv5jQTq3TQwBrCKF52uNrT"
 }

STDERR:

Start / End / Elapsed	 17:11:51.623561 / 17:11:54.675287 / 0:00:03.051726

2.Start k6 preset of objects and containers. Preset failed to load data to some buckets

HOST: 10.78.69.104
COMMAND:
 /etc/k6/scenarios/preset/preset_s3.py --endpoint 10.78.70.115:8084 --size '64' --preload_obj '10' --out '/var/log/autotests/s3_bd52f455-f933-4366-a3c5-ab336f12a80f_2_0_prepare.json' --workers '50' --buckets '40' --location 'node-off'
RC:
 2
STDOUT:
 Namespace(size='64', buckets='40', out='/var/log/autotests/s3_bd52f455-f933-4366-a3c5-ab336f12a80f_2_0_prepare.json', preload_obj='10', endpoint='10.78.70.115:8084', update=None, location='node-off', versioning=None, ignore_errors=None, workers='50')
 Create buckets: 40
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 1ba7130c-baa1-4ca2-aba5-673811cdd2ba --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 35168777-cd28-484c-8a8d-c7c57b925c97 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 028ebfe1-c206-4ac9-beab-4548e022a17f --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 2426c517-4eb0-4cb6-ba79-6a0580cd1a7d --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket f6429cff-cc94-4427-8edd-678c99b0b9a1 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket e64d810f-5806-4c9e-acd3-e3e911d5db4e --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 6f8e92f6-4425-4515-a7c5-111827e80663 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket ebf816c5-c449-47dc-be66-0a75347d5cbc --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket bec4e7b8-21ca-4358-a952-fc5d15d96293 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket cc631339-3174-4573-838c-f2d0ce6349d3 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 4a024c96-ebb0-4f4d-b2b1-effc4c1eeba8 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket bab60d8e-1148-4862-8e2f-495b323c2ff8 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 5234c6dd-9ffa-487e-93f7-1343f18256d8 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 267c76ff-efba-4719-858c-12a7c4ab4807 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 0e76835e-aa48-436b-a034-d8732087c24e --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket d4dc6373-623e-4428-9e29-ae4be66c644b --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 618911b4-f101-48d4-a793-ce13fb0735ae --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket b3f4e3d7-a92b-4b7d-a448-80dc65b04731 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket a1369c11-92fa-411c-89e5-666ea65599fa --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket e252ccb5-7c78-488e-8be5-4bc7f7ee35a5 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket bf00ff14-b224-4d8d-b9c8-3c6fa3baa5fc --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket d180f547-2fd0-40df-86eb-d2201c80d029 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 6d70cdad-a75e-4568-96a7-f9941a1b672e --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 7ee083c8-c645-458c-8fed-9ed6a3bb787e --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket a28a2e2a-e920-46b5-8b7e-b5d04f5bed15 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket afb73de2-10d8-44b0-b9ec-921b68db3a76 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket d3561fa6-26f9-4ef2-ab2b-c75046f6e71f --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 4dd64609-2bc7-4f5a-a8cb-f397d625f8f6 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket f1f7f3b1-80dd-4c0d-94a0-bd0f43b0f3e0 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 1e758e55-0900-4fed-bbef-f48f9f4cbfe7 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket c0d48ac8-cddb-47cd-afe7-bdcf2c9f773a --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket c8354c94-184a-4ce1-8238-8667a28028a3 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket ac116755-0608-427b-8d9e-73935053ea57 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket d48d7a1c-f108-4179-8149-8bf1347a5c64 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket d459f8c2-d309-4854-a0d9-bba7f2af2d54 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 67a01030-17ab-48c8-b2b3-4ce4b264ce29 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 47cce4e8-e4be-4eef-bf47-414c5cf8c7ec --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket da399a9a-4ba3-4380-b8dc-6bca8274df71 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket b251ee47-f075-4610-804a-ee893be028a3 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 cmd: aws --no-verify-ssl s3api create-bucket --bucket 840363bb-82ff-4e2e-8e39-2f236c7cd65b --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off'
 Create buckets: Completed
  > Buckets: ['ac116755-0608-427b-8d9e-73935053ea57', 'c0d48ac8-cddb-47cd-afe7-bdcf2c9f773a', '840363bb-82ff-4e2e-8e39-2f236c7cd65b', 'd459f8c2-d309-4854-a0d9-bba7f2af2d54', 'd48d7a1c-f108-4179-8149-8bf1347a5c64', 'd4dc6373-623e-4428-9e29-ae4be66c644b', 'da399a9a-4ba3-4380-b8dc-6bca8274df71', 'd3561fa6-26f9-4ef2-ab2b-c75046f6e71f', '47cce4e8-e4be-4eef-bf47-414c5cf8c7ec', 'b3f4e3d7-a92b-4b7d-a448-80dc65b04731', '1e758e55-0900-4fed-bbef-f48f9f4cbfe7', 'ebf816c5-c449-47dc-be66-0a75347d5cbc', 'c8354c94-184a-4ce1-8238-8667a28028a3', '7ee083c8-c645-458c-8fed-9ed6a3bb787e', '67a01030-17ab-48c8-b2b3-4ce4b264ce29', '618911b4-f101-48d4-a793-ce13fb0735ae', 'b251ee47-f075-4610-804a-ee893be028a3', '5234c6dd-9ffa-487e-93f7-1343f18256d8', 'f6429cff-cc94-4427-8edd-678c99b0b9a1', '267c76ff-efba-4719-858c-12a7c4ab4807', '0e76835e-aa48-436b-a034-d8732087c24e', '028ebfe1-c206-4ac9-beab-4548e022a17f', 'cc631339-3174-4573-838c-f2d0ce6349d3', '35168777-cd28-484c-8a8d-c7c57b925c97', 'e64d810f-5806-4c9e-acd3-e3e911d5db4e', '6f8e92f6-4425-4515-a7c5-111827e80663', 'bab60d8e-1148-4862-8e2f-495b323c2ff8', '2426c517-4eb0-4cb6-ba79-6a0580cd1a7d', 'bec4e7b8-21ca-4358-a952-fc5d15d96293', 'afb73de2-10d8-44b0-b9ec-921b68db3a76', '1ba7130c-baa1-4ca2-aba5-673811cdd2ba', '4a024c96-ebb0-4f4d-b2b1-effc4c1eeba8', 'f1f7f3b1-80dd-4c0d-94a0-bd0f43b0f3e0', '6d70cdad-a75e-4568-96a7-f9941a1b672e', '4dd64609-2bc7-4f5a-a8cb-f397d625f8f6', 'e252ccb5-7c78-488e-8be5-4bc7f7ee35a5', 'a1369c11-92fa-411c-89e5-666ea65599fa', 'a28a2e2a-e920-46b5-8b7e-b5d04f5bed15', 'd180f547-2fd0-40df-86eb-d2201c80d029', 'bf00ff14-b224-4d8d-b9c8-3c6fa3baa5fc']
 Upload objects to each bucket: 10 
  > Create random payload: Completed
  > Upload objects for bucket ac116755-0608-427b-8d9e-73935053ea57
  > Upload objects for bucket ac116755-0608-427b-8d9e-73935053ea57: Completed
  > Upload objects for bucket c0d48ac8-cddb-47cd-afe7-bdcf2c9f773a
  > Upload objects for bucket c0d48ac8-cddb-47cd-afe7-bdcf2c9f773a: Completed
  > Upload objects for bucket 840363bb-82ff-4e2e-8e39-2f236c7cd65b
  > Upload objects for bucket 840363bb-82ff-4e2e-8e39-2f236c7cd65b: Completed
  > Upload objects for bucket d459f8c2-d309-4854-a0d9-bba7f2af2d54
  > Upload objects for bucket d459f8c2-d309-4854-a0d9-bba7f2af2d54: Completed
  > Upload objects for bucket d48d7a1c-f108-4179-8149-8bf1347a5c64
  > Upload objects for bucket d48d7a1c-f108-4179-8149-8bf1347a5c64: Completed
  > Upload objects for bucket d4dc6373-623e-4428-9e29-ae4be66c644b
  > Upload objects for bucket d4dc6373-623e-4428-9e29-ae4be66c644b: Completed
  > Upload objects for bucket da399a9a-4ba3-4380-b8dc-6bca8274df71
  > Upload objects for bucket da399a9a-4ba3-4380-b8dc-6bca8274df71: Completed
  > Upload objects for bucket d3561fa6-26f9-4ef2-ab2b-c75046f6e71f
  > Upload objects for bucket d3561fa6-26f9-4ef2-ab2b-c75046f6e71f: Completed
  > Upload objects for bucket 47cce4e8-e4be-4eef-bf47-414c5cf8c7ec
  > Upload objects for bucket 47cce4e8-e4be-4eef-bf47-414c5cf8c7ec: Completed
  > Upload objects for bucket b3f4e3d7-a92b-4b7d-a448-80dc65b04731
  > Object b54a65fe-27bf-4c59-aa74-1c936ac93ffa has not been uploaded.
  > Object a41f5c11-f3ba-4dec-9108-8972ee16b189 has not been uploaded.
  > Object 1d6da043-c213-4a25-9efb-aa575dc6ebd2 has not been uploaded.
  > Object 79721c7f-5bcb-485c-97d0-7e4e5a835c91 has not been uploaded.
  > Object ff64b481-67d8-4c16-8bf1-c55c7f57085d has not been uploaded.
  > Object b9f24372-1a52-4133-a13d-0006faf9e205 has not been uploaded.
  > Object 6ec29919-6baa-4769-84b8-d1ac429c5dc0 has not been uploaded.
  > Object 03bd939f-ac8f-46c4-94fe-a16c8cb555c6 has not been uploaded.
  > Object 5121e18e-cb95-45ff-ad71-459db4d5e51d has not been uploaded.
  > Object f7e8743c-851a-462e-b753-dc2e96ac4e8e has not been uploaded.
  > Upload objects for bucket b3f4e3d7-a92b-4b7d-a448-80dc65b04731: Completed
  > Upload objects for bucket 1e758e55-0900-4fed-bbef-f48f9f4cbfe7
  > Upload objects for bucket 1e758e55-0900-4fed-bbef-f48f9f4cbfe7: Completed
  > Upload objects for bucket ebf816c5-c449-47dc-be66-0a75347d5cbc
  > Upload objects for bucket ebf816c5-c449-47dc-be66-0a75347d5cbc: Completed
  > Upload objects for bucket c8354c94-184a-4ce1-8238-8667a28028a3
  > Upload objects for bucket c8354c94-184a-4ce1-8238-8667a28028a3: Completed
  > Upload objects for bucket 7ee083c8-c645-458c-8fed-9ed6a3bb787e
  > Upload objects for bucket 7ee083c8-c645-458c-8fed-9ed6a3bb787e: Completed
  > Upload objects for bucket 67a01030-17ab-48c8-b2b3-4ce4b264ce29
  > Upload objects for bucket 67a01030-17ab-48c8-b2b3-4ce4b264ce29: Completed
  > Upload objects for bucket 618911b4-f101-48d4-a793-ce13fb0735ae
  > Upload objects for bucket 618911b4-f101-48d4-a793-ce13fb0735ae: Completed
  > Upload objects for bucket b251ee47-f075-4610-804a-ee893be028a3
  > Upload objects for bucket b251ee47-f075-4610-804a-ee893be028a3: Completed
  > Upload objects for bucket 5234c6dd-9ffa-487e-93f7-1343f18256d8
  > Object 0b4c227d-5fc4-4db1-90e9-13c514e53539 has not been uploaded.
  > Object eb202178-0899-4fd1-9650-3a6725d9f3b9 has not been uploaded.
  > Object 4703030d-bae6-46a6-9882-8c3dc4159175 has not been uploaded.
  > Object 0b5b30a9-f7ee-4a40-bcc7-dbacae789a6f has not been uploaded.
  > Object 10312b5a-8da5-4223-bc5c-271a642fea87 has not been uploaded.
  > Object 30a22245-0d09-4045-89e0-542724734c1e has not been uploaded.
  > Object a1057c30-d8fb-4656-ba07-2b105c97aba0 has not been uploaded.
  > Object 00451f5c-5df2-4698-965b-db3c88592371 has not been uploaded.
  > Object 076461d9-c9f0-4a9b-8c06-780ccd8e93dc has not been uploaded.
  > Object 7efff5f9-7792-4743-8c14-ff5fd9722813 has not been uploaded.
  > Upload objects for bucket 5234c6dd-9ffa-487e-93f7-1343f18256d8: Completed
  > Upload objects for bucket f6429cff-cc94-4427-8edd-678c99b0b9a1
  > Upload objects for bucket f6429cff-cc94-4427-8edd-678c99b0b9a1: Completed
  > Upload objects for bucket 267c76ff-efba-4719-858c-12a7c4ab4807
  > Upload objects for bucket 267c76ff-efba-4719-858c-12a7c4ab4807: Completed
  > Upload objects for bucket 0e76835e-aa48-436b-a034-d8732087c24e
  > Upload objects for bucket 0e76835e-aa48-436b-a034-d8732087c24e: Completed
  > Upload objects for bucket 028ebfe1-c206-4ac9-beab-4548e022a17f
  > Upload objects for bucket 028ebfe1-c206-4ac9-beab-4548e022a17f: Completed
  > Upload objects for bucket cc631339-3174-4573-838c-f2d0ce6349d3
  > Upload objects for bucket cc631339-3174-4573-838c-f2d0ce6349d3: Completed
  > Upload objects for bucket 35168777-cd28-484c-8a8d-c7c57b925c97
  > Upload objects for bucket 35168777-cd28-484c-8a8d-c7c57b925c97: Completed
  > Upload objects for bucket e64d810f-5806-4c9e-acd3-e3e911d5db4e
  > Upload objects for bucket e64d810f-5806-4c9e-acd3-e3e911d5db4e: Completed
  > Upload objects for bucket 6f8e92f6-4425-4515-a7c5-111827e80663
  > Upload objects for bucket 6f8e92f6-4425-4515-a7c5-111827e80663: Completed
  > Upload objects for bucket bab60d8e-1148-4862-8e2f-495b323c2ff8
  > Upload objects for bucket bab60d8e-1148-4862-8e2f-495b323c2ff8: Completed
  > Upload objects for bucket 2426c517-4eb0-4cb6-ba79-6a0580cd1a7d
  > Upload objects for bucket 2426c517-4eb0-4cb6-ba79-6a0580cd1a7d: Completed
  > Upload objects for bucket bec4e7b8-21ca-4358-a952-fc5d15d96293
  > Upload objects for bucket bec4e7b8-21ca-4358-a952-fc5d15d96293: Completed
  > Upload objects for bucket afb73de2-10d8-44b0-b9ec-921b68db3a76
  > Upload objects for bucket afb73de2-10d8-44b0-b9ec-921b68db3a76: Completed
  > Upload objects for bucket 1ba7130c-baa1-4ca2-aba5-673811cdd2ba
  > Upload objects for bucket 1ba7130c-baa1-4ca2-aba5-673811cdd2ba: Completed
  > Upload objects for bucket 4a024c96-ebb0-4f4d-b2b1-effc4c1eeba8
  > Upload objects for bucket 4a024c96-ebb0-4f4d-b2b1-effc4c1eeba8: Completed
  > Upload objects for bucket f1f7f3b1-80dd-4c0d-94a0-bd0f43b0f3e0
  > Upload objects for bucket f1f7f3b1-80dd-4c0d-94a0-bd0f43b0f3e0: Completed
  > Upload objects for bucket 6d70cdad-a75e-4568-96a7-f9941a1b672e
  > Upload objects for bucket 6d70cdad-a75e-4568-96a7-f9941a1b672e: Completed
  > Upload objects for bucket 4dd64609-2bc7-4f5a-a8cb-f397d625f8f6
  > Upload objects for bucket 4dd64609-2bc7-4f5a-a8cb-f397d625f8f6: Completed
  > Upload objects for bucket e252ccb5-7c78-488e-8be5-4bc7f7ee35a5
  > Upload objects for bucket e252ccb5-7c78-488e-8be5-4bc7f7ee35a5: Completed
  > Upload objects for bucket a1369c11-92fa-411c-89e5-666ea65599fa
  > Upload objects for bucket a1369c11-92fa-411c-89e5-666ea65599fa: Completed
  > Upload objects for bucket a28a2e2a-e920-46b5-8b7e-b5d04f5bed15
  > Upload objects for bucket a28a2e2a-e920-46b5-8b7e-b5d04f5bed15: Completed
  > Upload objects for bucket d180f547-2fd0-40df-86eb-d2201c80d029
  > Upload objects for bucket d180f547-2fd0-40df-86eb-d2201c80d029: Completed
  > Upload objects for bucket bf00ff14-b224-4d8d-b9c8-3c6fa3baa5fc
  > Upload objects for bucket bf00ff14-b224-4d8d-b9c8-3c6fa3baa5fc: Completed
 Upload objects to each bucket: Completed
 Objects mismatch in preset: expected 400, created 380

STDERR:

Start / End / Elapsed	 17:11:56.321103 / 17:12:56.396507 / 0:01:00.075404

3.See corresponding errors on logs:

Jun 20 17:12:14 kako frostfs-s3-gw[7541]: 2023-06-20T17:12:14.019Z        error        handler/util.go:29        call method        {"status": 500, "request_id": "bed32f03-0aff-4112-891e-f2bc26a8f021", "method": "PutObject", "bucket": "b3f4e3d7-a92b-4b7d-a448-80dc65b04731", "object": "b54a65fe-27bf-4c59-aa74-1c936ac93ffa", "description": "could not upload object", "error": "save object via connection pool: init writing on API client: client failure: status: code = 1024 message = incomplete object PUT by placement: could not close object stream: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.14/tcp/8080 /ip4/192.168.201.114/tcp/8080]: write object via client: status: code = 2048 message = access to object operation denied", "body close errors": []}

Jun 20 17:12:14 kako frostfs-s3-gw[7541]: 2023-06-20T17:12:14.315Z        error        handler/util.go:29        call method        {"status": 500, "request_id": "08343099-9780-4031-9452-727e9a7eb5a0", "method": "PutObject", "bucket": "b3f4e3d7-a92b-4b7d-a448-80dc65b04731", "object": "b54a65fe-27bf-4c59-aa74-1c936ac93ffa", "description": "could not upload object", "error": "save object via connection pool: init writing on API client: client failure: status: code = 1024 message = incomplete object PUT by placement: could not close object stream: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.20/tcp/8080 /ip4/192.168.201.120/tcp/8080]: write object via client: status: code = 2048 message = access to object operation denied", "body close errors": []}

Jun 20 17:12:14 kako frostfs-s3-gw[7541]: 2023-06-20T17:12:14.906Z        error        handler/util.go:29        call method        {"status": 500, "request_id": "ea3045fc-b313-46d5-b0ee-f3f472d4e11e", "method": "PutObject", "bucket": "b3f4e3d7-a92b-4b7d-a448-80dc65b04731", "object": "b54a65fe-27bf-4c59-aa74-1c936ac93ffa", "description": "could not upload object", "error": "save object via connection pool: init writing on API client: client failure: status: code = 1024 message = incomplete object PUT by placement: could not close object stream: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.19/tcp/8080 /ip4/192.168.201.119/tcp/8080]: write object via client: status: code = 2048 message = access to object operation denied", "body close errors": []}

4.Get CID corresponding to bucket

service@ize:~$ curl --head http://10.78.70.114:8084/b3f4e3d7-a92b-4b7d-a448-80dc65b04731
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 0
Server: FrostFS-S3-GW/v0.27.0-rc.2-6-g7edd8277
X-Amz-Bucket-Region: node-off
X-Amz-Request-Id: e5c146ca-8e1a-4e08-8fc0-a4060325f4da
X-Container-Id: DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n
X-Container-Name: b3f4e3d7-a92b-4b7d-a448-80dc65b04731
X-Container-Zone: container
X-Owner-Id: NXGWt8wJXGsWZZoYTm5PK7LMph49RmTCjR
Date: Wed, 21 Jun 2023 06:13:47 GMT

5.At the same time on the nodes in errors bellow I cannot see any errors like "access denied"

service@slovo:~$ journalctl --utc -S  "2023-06-20 17:12:14" --no-pager --grep=DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n -n 20
-- Journal begins at Thu 2023-06-15 19:42:12 UTC, ends at Wed 2023-06-21 06:21:24 UTC. --
Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z        debug        get/local.go:32        local get failed        {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/DBbNsMMJTfn2yu4GRBXBkmZRKcYZZU2CfAoVp6CTW5DD", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z        debug        get/local.go:32        local get failed        {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7aHjfQ5LAWSxpeeWvj5XJe2GLGiC5euz68YX7EQ4uZRK", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z        debug        get/get.go:107        operation finished with error        {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7aHjfQ5LAWSxpeeWvj5XJe2GLGiC5euz68YX7EQ4uZRK", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z        debug        get/container.go:12        return result directly        {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7aHjfQ5LAWSxpeeWvj5XJe2GLGiC5euz68YX7EQ4uZRK", "raw": false, "local": true, "with session": false, "with bearer": false}
Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z        debug        get/get.go:107        operation finished with error        {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7aHjfQ5LAWSxpeeWvj5XJe2GLGiC5euz68YX7EQ4uZRK", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z        debug        get/get.go:107        operation finished with error        {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/DBbNsMMJTfn2yu4GRBXBkmZRKcYZZU2CfAoVp6CTW5DD", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z        debug        get/container.go:12        return result directly        {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/DBbNsMMJTfn2yu4GRBXBkmZRKcYZZU2CfAoVp6CTW5DD", "raw": false, "local": true, "with session": false, "with bearer": false}
Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z        debug        get/get.go:107        operation finished with error        {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/DBbNsMMJTfn2yu4GRBXBkmZRKcYZZU2CfAoVp6CTW5DD", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}
Jun 20 17:12:16 slovo frostfs-node[8504]: 2023-06-20T17:12:16.414Z        debug        get/local.go:32        local get failed        {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7VKbiiYogZcrnzZRsHTgK6339rCgt1cTRccoyqxMvpnH", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"}

Version

FrostFS Storage node
Version: v0.0.1-434-g0252875a
GoVersion: go1.19.7

Regression

no

Your Environment

HW
16 nodes (8 alphabet in first datacenter and other 8 storage nodes in second datacenter)
Network profile between datacenters: normal (Bandwith: 1Gb, Packect loss: 0.5% with 50% dependency from previous loss)

## Expected Behavior PUT objects through s3 gate should PUT objects withpout any errors. ## Current Behavior PUT objects through s3 gate failed with "incomplete object PUT by placement" and "access to object operation denied". ## Steps to Reproduce (for bugs) 1.Create s3 creds: ``` HOST: 10.78.69.104 COMMAND: frostfs-s3-authmate issue-secret --wallet '/tmp/781607aa-cadd-40b0-848d-5e25e0aab84c.json' --peer '10.78.70.106:8080' --bearer-rules '/etc/k6/scenarios/files/rules.json' --gate-public-key '031f6a5a08afea03d26bca24ad729c735ba6e6e4a249de95d7157854c2b3218d8b' --gate-public-key '02e880fb3306b322de9e4d0f633f1042ac5454b9533ab3aa2126f5e1203238f035' --gate-public-key '03220fee34eb0d363e8d9f6a057b7ca5d42e5b7fe9050ba888facbf460b454279d' --gate-public-key '02abac27f257e99a6716eabf243f282110765fd1754aac913bf0030c8e0270ee0b' --gate-public-key '03d15a52949c581138a8bbee0a3ff543ff5983d029a31788ce30ac88dbcf0a0e62' --gate-public-key '0278a6b79988699dcc32d98056cb5cc72648967cffe03b3c501570518b3d7d635a' --gate-public-key '031f1d49bee67fbabecf7ca09fbcb8936b51a958e27aa280b3de6bec40128c7400' --gate-public-key '020b0e21e14b102e38a3130dc97c36b63d6326d6861cb1e051b5e17eaa307f8ae8' --gate-public-key '02e4c9e0d90dd1b38935c7b870939afacd3247748df150795da45abf2a6e428d90' --gate-public-key '03d00334d5ca94cfc69a925a76e30f0c8229e4c984acfc3a6d54bbda93eaa5c95c' --gate-public-key '0284d19d9bb1369d1c0f8c66cea3e0d9b9bd9c9c652480e9ba13c971d66c362591' --gate-public-key '024660361361fc86db414cceff03a69b113a27cbe9aa67c340a838861ed59746cc' --gate-public-key '02a3807d3f427be8198252f6aff03da3e8d7132b260690b77aa9950954743b79c6' --gate-public-key '02c1ab830efac81e01a3d1057889932da53dacc6641f8d24fcadaa70749ea3961c' --gate-public-key '0354f999ec5372821b51c4a66cf5cf93c4c863f7adaa374c6d1c197482630ef434' --gate-public-key '0242f8f40baa1e2ca1ca3eafa10a84b80aba18ebed160f49ae93695c80c88e5833' --container-placement-policy 'REP 2 IN X CBF 2 SELECT 2 FROM * AS X' --container-policy '/etc/k6/scenarios/files/policy.json' RC: 0 STDOUT: Enter password for /tmp/781607aa-cadd-40b0-848d-5e25e0aab84c.json > { "access_key_id": "Hvi6ZuL78dR8hMRHYr3THRv5jQTq3TQwBrCKF52uNrT0FAKsWnGo8cQaYJwgfgXCzCyLgGiJPJKXv3fFpakWUznq", "secret_access_key": "9918d4505e5d799a3795be263f2a5645c8df1c57cd6d9dcec49a6b65155a218d", "owner_private_key": "3a10ca0f4f69b92fdaa66850761ed10de4211a38584c62ea29b55e88a3a7ac5f", "wallet_public_key": "02fa2a20c66e05c5ac344c7a5e7e3312ecfe6a43b3fc0dcdbad442bfd6ad47c63c", "container_id": "Hvi6ZuL78dR8hMRHYr3THRv5jQTq3TQwBrCKF52uNrT" } STDERR: Start / End / Elapsed 17:11:51.623561 / 17:11:54.675287 / 0:00:03.051726 ``` 2.Start k6 preset of objects and containers. Preset failed to load data to some buckets ``` HOST: 10.78.69.104 COMMAND: /etc/k6/scenarios/preset/preset_s3.py --endpoint 10.78.70.115:8084 --size '64' --preload_obj '10' --out '/var/log/autotests/s3_bd52f455-f933-4366-a3c5-ab336f12a80f_2_0_prepare.json' --workers '50' --buckets '40' --location 'node-off' RC: 2 STDOUT: Namespace(size='64', buckets='40', out='/var/log/autotests/s3_bd52f455-f933-4366-a3c5-ab336f12a80f_2_0_prepare.json', preload_obj='10', endpoint='10.78.70.115:8084', update=None, location='node-off', versioning=None, ignore_errors=None, workers='50') Create buckets: 40 cmd: aws --no-verify-ssl s3api create-bucket --bucket 1ba7130c-baa1-4ca2-aba5-673811cdd2ba --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 35168777-cd28-484c-8a8d-c7c57b925c97 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 028ebfe1-c206-4ac9-beab-4548e022a17f --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 2426c517-4eb0-4cb6-ba79-6a0580cd1a7d --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket f6429cff-cc94-4427-8edd-678c99b0b9a1 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket e64d810f-5806-4c9e-acd3-e3e911d5db4e --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 6f8e92f6-4425-4515-a7c5-111827e80663 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket ebf816c5-c449-47dc-be66-0a75347d5cbc --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket bec4e7b8-21ca-4358-a952-fc5d15d96293 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket cc631339-3174-4573-838c-f2d0ce6349d3 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 4a024c96-ebb0-4f4d-b2b1-effc4c1eeba8 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket bab60d8e-1148-4862-8e2f-495b323c2ff8 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 5234c6dd-9ffa-487e-93f7-1343f18256d8 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 267c76ff-efba-4719-858c-12a7c4ab4807 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 0e76835e-aa48-436b-a034-d8732087c24e --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket d4dc6373-623e-4428-9e29-ae4be66c644b --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 618911b4-f101-48d4-a793-ce13fb0735ae --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket b3f4e3d7-a92b-4b7d-a448-80dc65b04731 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket a1369c11-92fa-411c-89e5-666ea65599fa --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket e252ccb5-7c78-488e-8be5-4bc7f7ee35a5 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket bf00ff14-b224-4d8d-b9c8-3c6fa3baa5fc --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket d180f547-2fd0-40df-86eb-d2201c80d029 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 6d70cdad-a75e-4568-96a7-f9941a1b672e --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 7ee083c8-c645-458c-8fed-9ed6a3bb787e --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket a28a2e2a-e920-46b5-8b7e-b5d04f5bed15 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket afb73de2-10d8-44b0-b9ec-921b68db3a76 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket d3561fa6-26f9-4ef2-ab2b-c75046f6e71f --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 4dd64609-2bc7-4f5a-a8cb-f397d625f8f6 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket f1f7f3b1-80dd-4c0d-94a0-bd0f43b0f3e0 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 1e758e55-0900-4fed-bbef-f48f9f4cbfe7 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket c0d48ac8-cddb-47cd-afe7-bdcf2c9f773a --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket c8354c94-184a-4ce1-8238-8667a28028a3 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket ac116755-0608-427b-8d9e-73935053ea57 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket d48d7a1c-f108-4179-8149-8bf1347a5c64 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket d459f8c2-d309-4854-a0d9-bba7f2af2d54 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 67a01030-17ab-48c8-b2b3-4ce4b264ce29 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 47cce4e8-e4be-4eef-bf47-414c5cf8c7ec --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket da399a9a-4ba3-4380-b8dc-6bca8274df71 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket b251ee47-f075-4610-804a-ee893be028a3 --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' cmd: aws --no-verify-ssl s3api create-bucket --bucket 840363bb-82ff-4e2e-8e39-2f236c7cd65b --cli-read-timeout 120 --cli-connect-timeout 120 --endpoint http://10.78.70.115:8084 --create-bucket-configuration 'LocationConstraint=node-off' Create buckets: Completed > Buckets: ['ac116755-0608-427b-8d9e-73935053ea57', 'c0d48ac8-cddb-47cd-afe7-bdcf2c9f773a', '840363bb-82ff-4e2e-8e39-2f236c7cd65b', 'd459f8c2-d309-4854-a0d9-bba7f2af2d54', 'd48d7a1c-f108-4179-8149-8bf1347a5c64', 'd4dc6373-623e-4428-9e29-ae4be66c644b', 'da399a9a-4ba3-4380-b8dc-6bca8274df71', 'd3561fa6-26f9-4ef2-ab2b-c75046f6e71f', '47cce4e8-e4be-4eef-bf47-414c5cf8c7ec', 'b3f4e3d7-a92b-4b7d-a448-80dc65b04731', '1e758e55-0900-4fed-bbef-f48f9f4cbfe7', 'ebf816c5-c449-47dc-be66-0a75347d5cbc', 'c8354c94-184a-4ce1-8238-8667a28028a3', '7ee083c8-c645-458c-8fed-9ed6a3bb787e', '67a01030-17ab-48c8-b2b3-4ce4b264ce29', '618911b4-f101-48d4-a793-ce13fb0735ae', 'b251ee47-f075-4610-804a-ee893be028a3', '5234c6dd-9ffa-487e-93f7-1343f18256d8', 'f6429cff-cc94-4427-8edd-678c99b0b9a1', '267c76ff-efba-4719-858c-12a7c4ab4807', '0e76835e-aa48-436b-a034-d8732087c24e', '028ebfe1-c206-4ac9-beab-4548e022a17f', 'cc631339-3174-4573-838c-f2d0ce6349d3', '35168777-cd28-484c-8a8d-c7c57b925c97', 'e64d810f-5806-4c9e-acd3-e3e911d5db4e', '6f8e92f6-4425-4515-a7c5-111827e80663', 'bab60d8e-1148-4862-8e2f-495b323c2ff8', '2426c517-4eb0-4cb6-ba79-6a0580cd1a7d', 'bec4e7b8-21ca-4358-a952-fc5d15d96293', 'afb73de2-10d8-44b0-b9ec-921b68db3a76', '1ba7130c-baa1-4ca2-aba5-673811cdd2ba', '4a024c96-ebb0-4f4d-b2b1-effc4c1eeba8', 'f1f7f3b1-80dd-4c0d-94a0-bd0f43b0f3e0', '6d70cdad-a75e-4568-96a7-f9941a1b672e', '4dd64609-2bc7-4f5a-a8cb-f397d625f8f6', 'e252ccb5-7c78-488e-8be5-4bc7f7ee35a5', 'a1369c11-92fa-411c-89e5-666ea65599fa', 'a28a2e2a-e920-46b5-8b7e-b5d04f5bed15', 'd180f547-2fd0-40df-86eb-d2201c80d029', 'bf00ff14-b224-4d8d-b9c8-3c6fa3baa5fc'] Upload objects to each bucket: 10 > Create random payload: Completed > Upload objects for bucket ac116755-0608-427b-8d9e-73935053ea57 > Upload objects for bucket ac116755-0608-427b-8d9e-73935053ea57: Completed > Upload objects for bucket c0d48ac8-cddb-47cd-afe7-bdcf2c9f773a > Upload objects for bucket c0d48ac8-cddb-47cd-afe7-bdcf2c9f773a: Completed > Upload objects for bucket 840363bb-82ff-4e2e-8e39-2f236c7cd65b > Upload objects for bucket 840363bb-82ff-4e2e-8e39-2f236c7cd65b: Completed > Upload objects for bucket d459f8c2-d309-4854-a0d9-bba7f2af2d54 > Upload objects for bucket d459f8c2-d309-4854-a0d9-bba7f2af2d54: Completed > Upload objects for bucket d48d7a1c-f108-4179-8149-8bf1347a5c64 > Upload objects for bucket d48d7a1c-f108-4179-8149-8bf1347a5c64: Completed > Upload objects for bucket d4dc6373-623e-4428-9e29-ae4be66c644b > Upload objects for bucket d4dc6373-623e-4428-9e29-ae4be66c644b: Completed > Upload objects for bucket da399a9a-4ba3-4380-b8dc-6bca8274df71 > Upload objects for bucket da399a9a-4ba3-4380-b8dc-6bca8274df71: Completed > Upload objects for bucket d3561fa6-26f9-4ef2-ab2b-c75046f6e71f > Upload objects for bucket d3561fa6-26f9-4ef2-ab2b-c75046f6e71f: Completed > Upload objects for bucket 47cce4e8-e4be-4eef-bf47-414c5cf8c7ec > Upload objects for bucket 47cce4e8-e4be-4eef-bf47-414c5cf8c7ec: Completed > Upload objects for bucket b3f4e3d7-a92b-4b7d-a448-80dc65b04731 > Object b54a65fe-27bf-4c59-aa74-1c936ac93ffa has not been uploaded. > Object a41f5c11-f3ba-4dec-9108-8972ee16b189 has not been uploaded. > Object 1d6da043-c213-4a25-9efb-aa575dc6ebd2 has not been uploaded. > Object 79721c7f-5bcb-485c-97d0-7e4e5a835c91 has not been uploaded. > Object ff64b481-67d8-4c16-8bf1-c55c7f57085d has not been uploaded. > Object b9f24372-1a52-4133-a13d-0006faf9e205 has not been uploaded. > Object 6ec29919-6baa-4769-84b8-d1ac429c5dc0 has not been uploaded. > Object 03bd939f-ac8f-46c4-94fe-a16c8cb555c6 has not been uploaded. > Object 5121e18e-cb95-45ff-ad71-459db4d5e51d has not been uploaded. > Object f7e8743c-851a-462e-b753-dc2e96ac4e8e has not been uploaded. > Upload objects for bucket b3f4e3d7-a92b-4b7d-a448-80dc65b04731: Completed > Upload objects for bucket 1e758e55-0900-4fed-bbef-f48f9f4cbfe7 > Upload objects for bucket 1e758e55-0900-4fed-bbef-f48f9f4cbfe7: Completed > Upload objects for bucket ebf816c5-c449-47dc-be66-0a75347d5cbc > Upload objects for bucket ebf816c5-c449-47dc-be66-0a75347d5cbc: Completed > Upload objects for bucket c8354c94-184a-4ce1-8238-8667a28028a3 > Upload objects for bucket c8354c94-184a-4ce1-8238-8667a28028a3: Completed > Upload objects for bucket 7ee083c8-c645-458c-8fed-9ed6a3bb787e > Upload objects for bucket 7ee083c8-c645-458c-8fed-9ed6a3bb787e: Completed > Upload objects for bucket 67a01030-17ab-48c8-b2b3-4ce4b264ce29 > Upload objects for bucket 67a01030-17ab-48c8-b2b3-4ce4b264ce29: Completed > Upload objects for bucket 618911b4-f101-48d4-a793-ce13fb0735ae > Upload objects for bucket 618911b4-f101-48d4-a793-ce13fb0735ae: Completed > Upload objects for bucket b251ee47-f075-4610-804a-ee893be028a3 > Upload objects for bucket b251ee47-f075-4610-804a-ee893be028a3: Completed > Upload objects for bucket 5234c6dd-9ffa-487e-93f7-1343f18256d8 > Object 0b4c227d-5fc4-4db1-90e9-13c514e53539 has not been uploaded. > Object eb202178-0899-4fd1-9650-3a6725d9f3b9 has not been uploaded. > Object 4703030d-bae6-46a6-9882-8c3dc4159175 has not been uploaded. > Object 0b5b30a9-f7ee-4a40-bcc7-dbacae789a6f has not been uploaded. > Object 10312b5a-8da5-4223-bc5c-271a642fea87 has not been uploaded. > Object 30a22245-0d09-4045-89e0-542724734c1e has not been uploaded. > Object a1057c30-d8fb-4656-ba07-2b105c97aba0 has not been uploaded. > Object 00451f5c-5df2-4698-965b-db3c88592371 has not been uploaded. > Object 076461d9-c9f0-4a9b-8c06-780ccd8e93dc has not been uploaded. > Object 7efff5f9-7792-4743-8c14-ff5fd9722813 has not been uploaded. > Upload objects for bucket 5234c6dd-9ffa-487e-93f7-1343f18256d8: Completed > Upload objects for bucket f6429cff-cc94-4427-8edd-678c99b0b9a1 > Upload objects for bucket f6429cff-cc94-4427-8edd-678c99b0b9a1: Completed > Upload objects for bucket 267c76ff-efba-4719-858c-12a7c4ab4807 > Upload objects for bucket 267c76ff-efba-4719-858c-12a7c4ab4807: Completed > Upload objects for bucket 0e76835e-aa48-436b-a034-d8732087c24e > Upload objects for bucket 0e76835e-aa48-436b-a034-d8732087c24e: Completed > Upload objects for bucket 028ebfe1-c206-4ac9-beab-4548e022a17f > Upload objects for bucket 028ebfe1-c206-4ac9-beab-4548e022a17f: Completed > Upload objects for bucket cc631339-3174-4573-838c-f2d0ce6349d3 > Upload objects for bucket cc631339-3174-4573-838c-f2d0ce6349d3: Completed > Upload objects for bucket 35168777-cd28-484c-8a8d-c7c57b925c97 > Upload objects for bucket 35168777-cd28-484c-8a8d-c7c57b925c97: Completed > Upload objects for bucket e64d810f-5806-4c9e-acd3-e3e911d5db4e > Upload objects for bucket e64d810f-5806-4c9e-acd3-e3e911d5db4e: Completed > Upload objects for bucket 6f8e92f6-4425-4515-a7c5-111827e80663 > Upload objects for bucket 6f8e92f6-4425-4515-a7c5-111827e80663: Completed > Upload objects for bucket bab60d8e-1148-4862-8e2f-495b323c2ff8 > Upload objects for bucket bab60d8e-1148-4862-8e2f-495b323c2ff8: Completed > Upload objects for bucket 2426c517-4eb0-4cb6-ba79-6a0580cd1a7d > Upload objects for bucket 2426c517-4eb0-4cb6-ba79-6a0580cd1a7d: Completed > Upload objects for bucket bec4e7b8-21ca-4358-a952-fc5d15d96293 > Upload objects for bucket bec4e7b8-21ca-4358-a952-fc5d15d96293: Completed > Upload objects for bucket afb73de2-10d8-44b0-b9ec-921b68db3a76 > Upload objects for bucket afb73de2-10d8-44b0-b9ec-921b68db3a76: Completed > Upload objects for bucket 1ba7130c-baa1-4ca2-aba5-673811cdd2ba > Upload objects for bucket 1ba7130c-baa1-4ca2-aba5-673811cdd2ba: Completed > Upload objects for bucket 4a024c96-ebb0-4f4d-b2b1-effc4c1eeba8 > Upload objects for bucket 4a024c96-ebb0-4f4d-b2b1-effc4c1eeba8: Completed > Upload objects for bucket f1f7f3b1-80dd-4c0d-94a0-bd0f43b0f3e0 > Upload objects for bucket f1f7f3b1-80dd-4c0d-94a0-bd0f43b0f3e0: Completed > Upload objects for bucket 6d70cdad-a75e-4568-96a7-f9941a1b672e > Upload objects for bucket 6d70cdad-a75e-4568-96a7-f9941a1b672e: Completed > Upload objects for bucket 4dd64609-2bc7-4f5a-a8cb-f397d625f8f6 > Upload objects for bucket 4dd64609-2bc7-4f5a-a8cb-f397d625f8f6: Completed > Upload objects for bucket e252ccb5-7c78-488e-8be5-4bc7f7ee35a5 > Upload objects for bucket e252ccb5-7c78-488e-8be5-4bc7f7ee35a5: Completed > Upload objects for bucket a1369c11-92fa-411c-89e5-666ea65599fa > Upload objects for bucket a1369c11-92fa-411c-89e5-666ea65599fa: Completed > Upload objects for bucket a28a2e2a-e920-46b5-8b7e-b5d04f5bed15 > Upload objects for bucket a28a2e2a-e920-46b5-8b7e-b5d04f5bed15: Completed > Upload objects for bucket d180f547-2fd0-40df-86eb-d2201c80d029 > Upload objects for bucket d180f547-2fd0-40df-86eb-d2201c80d029: Completed > Upload objects for bucket bf00ff14-b224-4d8d-b9c8-3c6fa3baa5fc > Upload objects for bucket bf00ff14-b224-4d8d-b9c8-3c6fa3baa5fc: Completed Upload objects to each bucket: Completed Objects mismatch in preset: expected 400, created 380 STDERR: Start / End / Elapsed 17:11:56.321103 / 17:12:56.396507 / 0:01:00.075404 ``` 3.See corresponding errors on logs: ``` Jun 20 17:12:14 kako frostfs-s3-gw[7541]: 2023-06-20T17:12:14.019Z error handler/util.go:29 call method {"status": 500, "request_id": "bed32f03-0aff-4112-891e-f2bc26a8f021", "method": "PutObject", "bucket": "b3f4e3d7-a92b-4b7d-a448-80dc65b04731", "object": "b54a65fe-27bf-4c59-aa74-1c936ac93ffa", "description": "could not upload object", "error": "save object via connection pool: init writing on API client: client failure: status: code = 1024 message = incomplete object PUT by placement: could not close object stream: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.14/tcp/8080 /ip4/192.168.201.114/tcp/8080]: write object via client: status: code = 2048 message = access to object operation denied", "body close errors": []} Jun 20 17:12:14 kako frostfs-s3-gw[7541]: 2023-06-20T17:12:14.315Z error handler/util.go:29 call method {"status": 500, "request_id": "08343099-9780-4031-9452-727e9a7eb5a0", "method": "PutObject", "bucket": "b3f4e3d7-a92b-4b7d-a448-80dc65b04731", "object": "b54a65fe-27bf-4c59-aa74-1c936ac93ffa", "description": "could not upload object", "error": "save object via connection pool: init writing on API client: client failure: status: code = 1024 message = incomplete object PUT by placement: could not close object stream: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.20/tcp/8080 /ip4/192.168.201.120/tcp/8080]: write object via client: status: code = 2048 message = access to object operation denied", "body close errors": []} Jun 20 17:12:14 kako frostfs-s3-gw[7541]: 2023-06-20T17:12:14.906Z error handler/util.go:29 call method {"status": 500, "request_id": "ea3045fc-b313-46d5-b0ee-f3f472d4e11e", "method": "PutObject", "bucket": "b3f4e3d7-a92b-4b7d-a448-80dc65b04731", "object": "b54a65fe-27bf-4c59-aa74-1c936ac93ffa", "description": "could not upload object", "error": "save object via connection pool: init writing on API client: client failure: status: code = 1024 message = incomplete object PUT by placement: could not close object stream: (*putsvc.remoteTarget) could not put object to [/ip4/192.168.201.19/tcp/8080 /ip4/192.168.201.119/tcp/8080]: write object via client: status: code = 2048 message = access to object operation denied", "body close errors": []} ``` 4.Get CID corresponding to bucket ``` service@ize:~$ curl --head http://10.78.70.114:8084/b3f4e3d7-a92b-4b7d-a448-80dc65b04731 HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 0 Server: FrostFS-S3-GW/v0.27.0-rc.2-6-g7edd8277 X-Amz-Bucket-Region: node-off X-Amz-Request-Id: e5c146ca-8e1a-4e08-8fc0-a4060325f4da X-Container-Id: DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n X-Container-Name: b3f4e3d7-a92b-4b7d-a448-80dc65b04731 X-Container-Zone: container X-Owner-Id: NXGWt8wJXGsWZZoYTm5PK7LMph49RmTCjR Date: Wed, 21 Jun 2023 06:13:47 GMT ``` 5.At the same time on the nodes in errors bellow I cannot see any errors like "access denied" ``` service@slovo:~$ journalctl --utc -S "2023-06-20 17:12:14" --no-pager --grep=DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n -n 20 -- Journal begins at Thu 2023-06-15 19:42:12 UTC, ends at Wed 2023-06-21 06:21:24 UTC. -- Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z debug get/local.go:32 local get failed {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/DBbNsMMJTfn2yu4GRBXBkmZRKcYZZU2CfAoVp6CTW5DD", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"} Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z debug get/local.go:32 local get failed {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7aHjfQ5LAWSxpeeWvj5XJe2GLGiC5euz68YX7EQ4uZRK", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"} Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z debug get/get.go:107 operation finished with error {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7aHjfQ5LAWSxpeeWvj5XJe2GLGiC5euz68YX7EQ4uZRK", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"} Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z debug get/container.go:12 return result directly {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7aHjfQ5LAWSxpeeWvj5XJe2GLGiC5euz68YX7EQ4uZRK", "raw": false, "local": true, "with session": false, "with bearer": false} Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z debug get/get.go:107 operation finished with error {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7aHjfQ5LAWSxpeeWvj5XJe2GLGiC5euz68YX7EQ4uZRK", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"} Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z debug get/get.go:107 operation finished with error {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/DBbNsMMJTfn2yu4GRBXBkmZRKcYZZU2CfAoVp6CTW5DD", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"} Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z debug get/container.go:12 return result directly {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/DBbNsMMJTfn2yu4GRBXBkmZRKcYZZU2CfAoVp6CTW5DD", "raw": false, "local": true, "with session": false, "with bearer": false} Jun 20 17:12:15 slovo frostfs-node[8504]: 2023-06-20T17:12:15.076Z debug get/get.go:107 operation finished with error {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/DBbNsMMJTfn2yu4GRBXBkmZRKcYZZU2CfAoVp6CTW5DD", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"} Jun 20 17:12:16 slovo frostfs-node[8504]: 2023-06-20T17:12:16.414Z debug get/local.go:32 local get failed {"component": "Object.Get service", "request": "HEAD", "address": "DixaS7CKXuwfGDsKVnmJ5GcFqQdWJELcYyH5aZaQ5N3n/7VKbiiYogZcrnzZRsHTgK6339rCgt1cTRccoyqxMvpnH", "raw": false, "local": true, "with session": false, "with bearer": false, "error": "status: code = 2049 message = object not found"} ``` ## Version ``` FrostFS Storage node Version: v0.0.1-434-g0252875a GoVersion: go1.19.7 ``` ## Regression no ## Your Environment HW 16 nodes (8 alphabet in first datacenter and other 8 storage nodes in second datacenter) Network profile between datacenters: normal (Bandwith: 1Gb, Packect loss: 0.5% with 50% dependency from previous loss)
anikeev-yadro added the
bug
triage
 labels 2023-06-21 07:53:25 +00:00
anikeev-yadro changed title from PUT objects trough s3 gate failed with "incomplete object PUT by placement" and "access to object operation denied" to PUT objects through s3 gate failed with "incomplete object PUT by placement" and "access to object operation denied" 2023-06-21 07:55:08 +00:00
dstepanov-yadro was assigned by fyrchik 2023-06-21 08:10:27 +00:00
fyrchik added this to the v0.37.0 milestone 2023-06-21 08:10:29 +00:00
fyrchik added the
frostfs-node
 label 2023-06-21 08:10:34 +00:00
fyrchik commented 2023-06-21 08:11:30 +00:00
Owner

Can be related to 16 nodes, may be not a regression, as we didn't test 16 nodes with a previous release, right? @anikeev-yadro

Can be related to 16 nodes, may be not a regression, as we didn't test 16 nodes with a previous release, right? @anikeev-yadro
anikeev-yadro commented 2023-06-21 09:21:29 +00:00
Poster

ok, lets set to regression=no.

ok, lets set to `regression=no`.
fyrchik commented 2023-09-15 14:37:01 +00:00
Owner

Supposedly fixed with neo-go update, need retest

Supposedly fixed with neo-go update, need retest
anikeev-yadro commented 2023-10-23 07:42:09 +00:00
Poster

Not reproduced on v0.37.0-rc.1-6-g5acc13fa
Suggest to close.

Not reproduced on v0.37.0-rc.1-6-g5acc13fa Suggest to close.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
support/v0.38
support/v0.37
support/v0.36
support/v0.34
support/v0.30
support/v0.27
v0.42.0-rc.1
v0.38.8
v0.41.0
v0.38.7
v0.40.0
v0.39.0
v0.38.6
v0.38.5
v0.38.4
v0.38.3
v0.38.2
v0.38.1
v0.38.0
v0.38.0-rc.2
v0.38.0-rc.1
v0.37.0
v0.37.0-rc.1
v0.36.0
v0.34.0
Labels
Clear labels   
P0

Highest   
P1

High   
P2

Medium   
P3

Low   
badger

Badger DB related issues   
frostfs-adm
   
frostfs-cli
   
frostfs-ir
   
frostfs-lens
   
frostfs-node
   
good first issue

Good for newcomers   
triage
  
Infrastructure

Infrastructure, Automation, CI/CD or DevOps related stuff   
blocked

The issue or PR is blocked by something   
bug

Something is not working   
config

Configuration format update or breaking change   
discussion

Talking about something in order to reach a decision or to exchange ideas   
documentation

Documentation related   
duplicate

This issue or pull request already exists   
enhancement

New feature   
go

Language features adoption   
help wanted

Extra attention is needed   
internal
   
invalid

Something is wrong   
kludge

This is a kludge and we know it   
observability

Related to metrics, tracing and logs   
perfomance

Perfomance related   
question

More information is needed   
refactoring

Refactoring   
wontfix

This won't be fixed
No Label
P0
P1
P2
P3
badger
frostfs-adm
frostfs-cli
frostfs-ir
frostfs-lens
frostfs-node
good first issue
triage
Infrastructure
blocked
bug
config
discussion
documentation
duplicate
enhancement
go
help wanted
internal
invalid
kludge
observability
perfomance
question
refactoring
wontfix
Milestone
Clear milestone
No items
No Milestone
v0.37.0
Assignees
Clear assignees
No Assignees
dstepanov-yadro
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#457
There is no content yet.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may exist for a short time before cleaning up, in most cases it CANNOT be undone. Continue?