Verify data integrity in parallel with response streaming #1499

New issue

Open

opened 2024-11-15 10:31:19 +00:00 by potyarkin · 0 comments

potyarkin commented

2024-11-15 10:31:19 +00:00

Member

Currently frostfs-node does not perform any data integrity verification when it streams objects to client from disk. As far as I know, we had this feature previously and have disabled it for performance reasons (iirc, memory usage and latency considerations). By streaming data directly from disk to network we avoid putting large objects into memory and minimize the time to first byte.

Responsibility for bitrot protection is currently delegated to client (app developer): discussion in private chat.

Describe the solution you'd like

Let's discuss whether it's possible to add integrity verification back:

Can we send an error (disk i/o error, integrity verification error) not before the first byte but after/during data stream?
If that would be possible we could run hash calculation in a parallel subroutine: read data from disk just once, in small chunks, and then send each chunk both to the client connection and to the hasher.
If hasher detects an error the data stream would be interrupted/restarted from another replica (that's up to discussion).

This approach would introduce some CPU overhead for hashing but would be pretty cheap in terms of memory and latency. TTFB would not be affected at all.

This suggestion is inspired in part by structured replies in NBD protocol. They were introduced to solve a similar problem there.

Describe alternatives you've considered

Keep things as they are: no bitrot protection, data integrity verification delegated to client.

## Is your feature request related to a problem? Please describe. Currently frostfs-node does not perform any data integrity verification when it streams objects to client from disk. As far as I know, we had this feature previously and have disabled it for performance reasons (iirc, memory usage and latency considerations). By streaming data directly from disk to network we avoid putting large objects into memory and minimize the time to first byte. Responsibility for bitrot protection is currently delegated to client (app developer): [discussion in private chat](https://chat.yadro.com/yadro/pl/t4nn3quwcpf9zy3n67c7uad9nc). ## Describe the solution you'd like Let's discuss whether it's possible to add integrity verification back: - Can we send an error (disk i/o error, integrity verification error) not before the first byte but after/during data stream? - If that would be possible we could run hash calculation in a parallel subroutine: read data from disk just once, in small chunks, and then send each chunk both to the client connection and to the hasher. - If hasher detects an error the data stream would be interrupted/restarted from another replica (that's up to discussion). This approach would introduce some CPU overhead for hashing but would be pretty cheap in terms of memory and latency. TTFB would not be affected at all. This suggestion is inspired in part by [structured replies](https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md#structured-reply-chunk-message) in NBD protocol. They were introduced to solve a similar problem there. ## Describe alternatives you've considered Keep things as they are: no bitrot protection, data integrity verification delegated to client.