Error "could not put object to BLOB storage: context canceled" should not increase shard errors #596

Closed
opened 2023-08-10 14:20:53 +00:00 by anikeev-yadro · 1 comment
Member

Expected Behavior

Error "could not put object to BLOB storage: context canceled" increases shard errors.

Current Behavior

Error "could not put object to BLOB storage: context canceled" should not increase shard errors, because potential network problems can lead to change mode of shard to degraded.

Steps to Reproduce (for bugs)

1.Start k6 load to system
2.See errors in log that increases errors count on shards:

Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "Q3bXcFeSTqYyzH1uN7jDSL", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "BpWzNVsyF2Gzb2Y4weQSQg", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "FG5Amewcesh5WmPSNsPa9W", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "3jGNVtLBdKBHqghHbmaMEg", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "6uDgyPqK7BuMLqQtCcFos7", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "MAnN2B47bAyJGUn9DaP4Gn", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "F96Bmy5BCBk56K9pTxhSZZ", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "XsU91t6ozwjtraWFD2jdAC", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "4xuG4akTWbMAHY2A5dkFmN", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "8bVQHPEp2tM88Z4eZZ8Npx", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "4g2NuEWZqWgtE8xR5gQNkp", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "2gD85agBavThVgQ1EePnkB", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}

Version

FrostFS Storage node
Version: v0.0.1-783-g7da4306e
GoVersion: go1.20.5

Your Environment

HW

## Expected Behavior Error "could not put object to BLOB storage: context canceled" increases shard errors. ## Current Behavior Error "could not put object to BLOB storage: context canceled" should not increase shard errors, because potential network problems can lead to change mode of shard to `degraded`. ## Steps to Reproduce (for bugs) 1.Start k6 load to system 2.See errors in log that increases errors count on shards: ``` Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "Q3bXcFeSTqYyzH1uN7jDSL", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "BpWzNVsyF2Gzb2Y4weQSQg", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "FG5Amewcesh5WmPSNsPa9W", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "3jGNVtLBdKBHqghHbmaMEg", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "6uDgyPqK7BuMLqQtCcFos7", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "MAnN2B47bAyJGUn9DaP4Gn", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "F96Bmy5BCBk56K9pTxhSZZ", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "XsU91t6ozwjtraWFD2jdAC", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "4xuG4akTWbMAHY2A5dkFmN", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "8bVQHPEp2tM88Z4eZZ8Npx", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "4g2NuEWZqWgtE8xR5gQNkp", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "2gD85agBavThVgQ1EePnkB", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} ``` ## Version ``` FrostFS Storage node Version: v0.0.1-783-g7da4306e GoVersion: go1.20.5 ``` ## Your Environment HW
anikeev-yadro added the
bug
triage
labels 2023-08-10 14:20:53 +00:00
fyrchik added the
frostfs-node
label 2023-08-10 14:22:57 +00:00
fyrchik added this to the v0.37.0 milestone 2023-08-14 07:13:03 +00:00
Owner

It seems impossible to cleanly convert them to logical errors everywhere, I think we can check it in the engine.reportError, right close to where we check the "logicality".

It seems impossible to cleanly convert them to logical errors everywhere, I think we can check it in the `engine.reportError`, right close to where we check the "logicality".
ale64bit was assigned by fyrchik 2023-08-14 07:41:58 +00:00
fyrchik self-assigned this 2023-08-15 12:35:49 +00:00
ale64bit was unassigned by fyrchik 2023-08-15 12:35:49 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: TrueCloudLab/frostfs-node#596
No description provided.