Commit graph

88 commits

Author SHA1 Message Date
Michael Eischer
ac805d6838 cache: cleanup debug logs 2024-05-18 21:28:54 +02:00
Michael Eischer
5214af88e2 cache: test forget behavior 2024-05-18 21:28:54 +02:00
Michael Eischer
e734746f75 cache: forget cached file at most once
This is inspired by the circuit breaker pattern used for distributed
systems. If too many requests fails, then it is better to immediately
fail new requests for a limited time to give the backend time to
recover.

By only forgetting a file in the cache at most once, we can ensure that
a broken file is only retrieved once again from the backend. If the file
stored there is broken, previously it would be cached and deleted
continuously. Now, it is retrieved only once again, all later requests
just use the cached copy and either succeed or fail immediately.
2024-05-18 21:28:54 +02:00
Michael Eischer
97a307df1a cache: Always use cached file if it exists
A file is always cached whole. Thus, any out of bounds access will also
fail when directed at the backend. To handle case in which the cached
file is broken, then caller must call Cache.Forget(h) for the file in
question.
2024-05-18 21:28:54 +02:00
Michael Eischer
2ace242f36 repository: make reloading broken files explicit 2024-05-18 21:28:54 +02:00
Michael Eischer
e9390352a7 cache: code cleanups 2024-05-18 21:26:00 +02:00
Michael Eischer
1d6d3656b0 repository: move backend.LoadAll to repository.LoadRaw
LoadRaw also includes improved context cancellation handling similar to the
implementation in repository.LoadUnpacked.

The removed cache backend test will be added again later on.
2024-05-18 21:26:00 +02:00
Michael Eischer
47232bf8b0 backend: move LimitReadCloser to util package
The helper is only intended for usage by backend implementations.
2024-05-18 21:26:00 +02:00
Michael Eischer
6328b7e1f5 replace "too small" with "too short" in error messages 2024-05-18 19:59:26 +02:00
Michael Eischer
0747cf5319 cache: ignore ErrNotExist during cleanup of old files
Two restic processes running concurrently can try to remove the same
files from the cache. This could cause one process to fail with an error
if the other one has already remove a file that the current process also
tries to delete.
2024-04-10 19:25:51 +02:00
Michael Eischer
1b8a67fe76 move Backend interface to backend package 2023-10-25 23:00:18 +02:00
Michael Eischer
b6d79bdf6f restic: decouple restic.Handle 2023-10-25 22:54:07 +02:00
Michael Eischer
17446da5fd
Merge pull request from greatroar/cache
cache: Restructure New to remove redundant operations
2023-06-02 22:41:17 +02:00
greatroar
da419be43c cache: Restructure New to remove redundant operations
New and its helpers used to create the cache directories several times
over. They now only do so once. The added test ensures that the cache is
produced in a consistent state when parts are deleted.
2023-05-27 10:32:08 +02:00
Michael Eischer
1514593f22 Remove unused context or testing parameters 2023-05-18 21:17:53 +02:00
Michael Eischer
5e4e268bdc Use _ as parameter name for unused Context
The context is required by the implemented interface.
2023-05-18 21:15:45 +02:00
Michael Eischer
e01baeabba Use either test or rtest to refer to internal test helpers
A single test file should not use both names.
2023-05-18 21:15:45 +02:00
Michael Eischer
78a1757e5a Cancel current command if cache becomes unusable
If the cache suddenly disappears, the current command will now fail.
2023-04-23 11:31:15 +02:00
Michael Eischer
6042df075f migrations: Fix S3 backend detection 2023-04-14 22:32:16 +02:00
greatroar
72922a79ed cache: Replace readCloser+LimitedReader by backend.LimitedReadCloser 2023-01-03 19:03:36 +01:00
greatroar
f90bf84ba7 test: Use testing.T.Cleanup to remove tempdirs 2022-12-09 14:23:55 +01:00
greatroar
d45a2475e1 cache: Rewrite unnecessary if-else 2022-12-03 12:13:54 +01:00
Michael Eischer
ff7ef5007e Replace most usages of ioutil with the underlying function
The ioutil functions are deprecated since Go 1.17 and only wrap another
library function. Thus directly call the underlying function.

This commit only mechanically replaces the function calls.
2022-12-02 19:36:43 +01:00
greatroar
9adae5521d cache: Call interface method once 2022-10-21 14:32:46 +02:00
greatroar
22147e1e02 all: Minor cleanups
if x { return true } return false => return x

	fmt.Sprintf("%v", x) => fmt.Sprint(x) or x.String()

The fmt.Sprintf idiom is still used in the SecretString tests, where it
serves security hardening.
2022-10-16 10:50:39 +02:00
Michael Eischer
430ab32941 cache: Fix file descriptor leak in TestBackendRemoveBroken 2022-10-03 00:06:44 +02:00
Michael Eischer
34c1a83340 cache: Drop cache entry if it cannot be processed
Failing to process data requested from the cache usually indicates a
problem with the returned data. Assume that the cache entry is somehow
damaged and retry downloading it once.
2022-09-25 11:55:09 +02:00
Michael Eischer
aa3b1925b4 cache: Simplify loadFromCacheOrDelegate 2022-09-25 11:35:35 +02:00
Michael Eischer
822422ef03 retry key loading on hash mismatch 2022-09-25 11:35:35 +02:00
Michael Eischer
7fc178aaf4 internal/cache: extend description of cache sharing test failure 2022-09-24 13:07:01 +02:00
greatroar
1220fe9650 internal/cache: Concurrent use of cache not working on Windows 2022-09-17 19:49:44 +02:00
Michael Eischer
ce902aac67 cache: Just try to open cache entry without calling stat first
Instead of first checking whether a file is in the repository cache and
then opening it, we just can open the file. This saves one stat call. If
the file is in the cache, everything is fine and otherwise the code
follows its normal fallback path.
2022-08-19 20:59:06 +02:00
Michael Eischer
8c11fc3ec9 crypto: move crypto buffer helpers 2022-07-17 13:42:23 +02:00
greatroar
2da377c582 cache: Don't Lstat before creating the tag file
The tag file is opened with O_CREATE|O_EXCL and ErrExist is handled, so
we don't need to check for existence first.
2022-05-10 18:52:39 +02:00
Michael Eischer
243698680a crypto: Use helpers for size calculations 2022-03-28 22:09:49 +02:00
MichaelEischer
cb81ee9396
Merge pull request from kitone/fix-issue-3382
Honor RESTIC_CACHE_DIR environment variable
2021-11-07 17:57:54 +01:00
kitone
1b23675f21 cache --cleanup should handle directories created by restic check.
Because there is no guarantee that a cleanup of these directories will occur
after the "restic check", we extend the behavior to detect and manage these
specific cache directories and allow their cleanup too.
2021-11-04 15:10:38 +01:00
kitone
95eb859b54 Honor RESTIC_CACHE_DIR environment variable
Fix : restic check doesn't obey the RESTIC_CACHE_DIR environment variable
2021-10-10 16:00:02 +02:00
Alexander Weiss
81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
Michael Eischer
9aa2eff384 Add plumbing to calculate backend specific file hash for upload
This enables the backends to request the calculation of a
backend-specific hash. For the currently supported backends this will
always be MD5. The hash calculation happens as early as possible, for
pack files this is during assembly of the pack file. That way the hash
would even capture corruptions of the temporary pack file on disk.
2021-08-04 22:17:46 +02:00
greatroar
6586e90acf Modernize internal/cache error handling 2021-08-04 22:02:42 +02:00
greatroar
ea04f40eb3 Save cached files to a temporary location first 2021-08-04 22:02:42 +02:00
greatroar
f9b6f8fd45 Replace duplicate type checking in cache with a function 2021-08-04 22:02:42 +02:00
Alexander Neumann
0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
greatroar
673dda77c0 Less repetitive error messages in internal/cache
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
2020-10-05 20:28:54 +02:00
greatroar
7d0fa1a686 Simplify cache directory creation 2020-10-05 10:46:45 +02:00
greatroar
cb3f531050 Make Cache.Path private
It's only used inside the package.
2020-09-18 11:17:29 +02:00
greatroar
23fcbb275a Remove restic.Cache interface
It was used in one code path, which then asserted its concrete type as
*cache.Cache.

Privatised some of the interface methods.
2020-09-18 10:48:13 +02:00
greatroar
6e3215a80d Remove unused error type from internal/cache 2020-09-18 10:38:14 +02:00
aawsome
0fed6a8dfc
Use "pack file" instead of "data file" ()
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00