Small object is not available during put load #27

New issue

Closed

opened 2023-01-24 13:51:22 +00:00 by alexvanin · 1 comment

alexvanin commented

2023-01-24 13:51:22 +00:00

(Migrated from github.com)

Expected Behavior

S3 Gateway receives small (under 10 KiB) objects with S3 credentials when it is evicted from the cache during put object load from 25 or 50 workers per node and 1MiB object size.

Current Behavior

S3 Gateway can't receive credential objects and skips some requests for a while

Steps to Reproduce (for bugs)

Prepare 4-node cluster
Create single container
Start uploading 1 MiB objects with 25 to 50 workers to this single container through S3
See some interrupts during this load.

Context

Cluster behavior seems very strange. FrostFS CLI receives object not found error while trying to HEAD the credentials objects.
On GET requests, it receives timeouts and sometimes credential object is returned

All 4 nodes behave the same way. Their shards are not degraded at that moment.

Your Environment

Hard metal cluster

## Expected Behavior S3 Gateway receives small (under 10 KiB) objects with S3 credentials when it is evicted from the cache during put object load from 25 or 50 workers per node and 1MiB object size. ## Current Behavior S3 Gateway can't receive credential objects and skips some requests for a while ## Steps to Reproduce (for bugs) 1. Prepare 4-node cluster 2. Create single container 3. Start uploading 1 MiB objects with 25 to 50 workers to this single container through S3 4. See some interrupts during this load. ## Context Cluster behavior seems very strange. FrostFS CLI receives `object not found` error while trying to HEAD the credentials objects. On GET requests, it receives timeouts and sometimes credential object is returned All 4 nodes behave the same way. Their shards are not degraded at that moment. ## Your Environment Hard metal cluster