Error "could not put object to BLOB storage: context canceled" should not increase shard errors #596

New Issue

Closed

opened 2023-08-10 14:20:53 +00:00 by anikeev-yadro · 1 comment

anikeev-yadro commented

2023-08-10 14:20:53 +00:00

Expected Behavior

Error "could not put object to BLOB storage: context canceled" increases shard errors.

Current Behavior

Error "could not put object to BLOB storage: context canceled" should not increase shard errors, because potential network problems can lead to change mode of shard to degraded.

Steps to Reproduce (for bugs)

1.Start k6 load to system
2.See errors in log that increases errors count on shards:

Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "Q3bXcFeSTqYyzH1uN7jDSL", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "BpWzNVsyF2Gzb2Y4weQSQg", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "FG5Amewcesh5WmPSNsPa9W", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "3jGNVtLBdKBHqghHbmaMEg", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "6uDgyPqK7BuMLqQtCcFos7", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "MAnN2B47bAyJGUn9DaP4Gn", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "F96Bmy5BCBk56K9pTxhSZZ", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "XsU91t6ozwjtraWFD2jdAC", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "4xuG4akTWbMAHY2A5dkFmN", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "8bVQHPEp2tM88Z4eZZ8Npx", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "4g2NuEWZqWgtE8xR5gQNkp", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}
Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z        warn        engine/engine.go:166        could not put object to shard        {"shard_id": "2gD85agBavThVgQ1EePnkB", "error count": 6, "error": "could not put object to BLOB storage: context canceled"}

Version

FrostFS Storage node
Version: v0.0.1-783-g7da4306e
GoVersion: go1.20.5

Your Environment

## Expected Behavior Error "could not put object to BLOB storage: context canceled" increases shard errors. ## Current Behavior Error "could not put object to BLOB storage: context canceled" should not increase shard errors, because potential network problems can lead to change mode of shard to `degraded`. ## Steps to Reproduce (for bugs) 1.Start k6 load to system 2.See errors in log that increases errors count on shards: ``` Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "Q3bXcFeSTqYyzH1uN7jDSL", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "BpWzNVsyF2Gzb2Y4weQSQg", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "FG5Amewcesh5WmPSNsPa9W", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "3jGNVtLBdKBHqghHbmaMEg", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.085Z warn engine/engine.go:166 could not put object to shard {"shard_id": "6uDgyPqK7BuMLqQtCcFos7", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "MAnN2B47bAyJGUn9DaP4Gn", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "F96Bmy5BCBk56K9pTxhSZZ", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "XsU91t6ozwjtraWFD2jdAC", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "4xuG4akTWbMAHY2A5dkFmN", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "8bVQHPEp2tM88Z4eZZ8Npx", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "4g2NuEWZqWgtE8xR5gQNkp", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} Aug 10 13:19:17 vedi frostfs-node[14801]: 2023-08-10T13:19:17.086Z warn engine/engine.go:166 could not put object to shard {"shard_id": "2gD85agBavThVgQ1EePnkB", "error count": 6, "error": "could not put object to BLOB storage: context canceled"} ``` ## Version ``` FrostFS Storage node Version: v0.0.1-783-g7da4306e GoVersion: go1.20.5 ``` ## Your Environment HW

anikeev-yadro added the

bug

triage

labels 2023-08-10 14:20:53 +00:00

fyrchik added the

frostfs-node

label 2023-08-10 14:22:57 +00:00

fyrchik added this to the v0.37.0 milestone 2023-08-14 07:13:03 +00:00

fyrchik commented

2023-08-14 07:41:48 +00:00

Owner

It seems impossible to cleanly convert them to logical errors everywhere, I think we can check it in the engine.reportError, right close to where we check the "logicality".

It seems impossible to cleanly convert them to logical errors everywhere, I think we can check it in the `engine.reportError`, right close to where we check the "logicality".