b2: Successful delete if file does not exist

When deleting a file, B2 sometimes returns a "500 Service Unavailable"
error but nevertheless correctly deletes the file. Due to retries in
the B2 library blazer, we sometimes also see a "400 File not present"
error. The retries of restic for the delete request then fail with
"404 File with such name does not exist.".

As we have to rely on request retries in a distributed system to handle
temporary errors, also consider a delete request to be successful if the
file is reported as not existing. This should be safe as B2 claims to
provide a strongly consistent bucket listing and thus a missing file
shouldn't mysteriously show up again later on.

changelog/unreleased/issue-3541

@@ -0,0 +1,9 @@
+Bugfix: Improve handling of temporary B2 delete errors
+
+Deleting files on B2 can sometimes fail temporarily, which requires restic to
+retry the delete operation. In some cases the file was deleted nevertheless
+causing the retries and ultimately the restic command to fail. This has been
+fixed.
+
+https://github.com/restic/restic/issues/3541
+https://github.com/restic/restic/pull/3544

internal/backend/b2/b2.go

@@ -265,7 +265,13 @@ func (be *b2Backend) Remove(ctx context.Context, h restic.Handle) error {
 	defer be.sem.ReleaseToken()
 
 	obj := be.bucket.Object(be.Filename(h))
-	return errors.Wrap(obj.Delete(ctx), "Delete")
+	err := obj.Delete(ctx)
+	// consider a file as removed if b2 informs us that it does not exist
+	if b2.IsNotExist(err) {
+		return nil
+	}
+
+	return errors.Wrap(err, "Delete")
 }
 
 type semLocker struct {