TrueCloudLab/restic
16
0
Fork 2
Code Issues 1 Pull requests 1 Projects Releases Packages Wiki Activity Actions 1
3411 commits 9 branches 49 tags 79 MiB
Commit graph

268 commits

Author SHA1 Message Date
Michael Pratt
5f4f997126 gs: minor comment cleanups 
* Remove a reference to S3.
* Config can only be used for GCS, not other "gcs compatibile servers".
* Make comments complete sentences.
 2017-09-24 10:10:56 -07:00
Alexander Neumann
9c6b7f688e Merge pull request #1270 from restic/sftp-allow-password-prompt 
sftp: Allow password entry
 2017-09-23 22:13:04 +02:00
Alexander Neumann
429106340f Merge pull request #1267 from harshavardhana/possible-fix-memory 
Implement Size() and Len() to know the optimal size.
 2017-09-23 14:04:15 +02:00
Alexander Neumann
fb9729fdb9 sftp: Allow password entry 
This was a bit tricky: We start the ssh binary, but we want it to ignore
SIGINT. In contrast, restic itself should process SIGINT and clean up
properly. Before, we used `setsid()` to give the ssh process its own
process group, but that means it cannot prompt the user for a password
because the tty is gone.

So, now we're passing in two functions that ignore SIGINT just before
the ssh process is started and re-install it after start.
 2017-09-23 11:43:33 +02:00
Harshavardhana
98369f6a5d Implement Size() and Len() to know the optimal size. 2017-09-22 12:09:17 -07:00
Alexander Neumann
3541d06d07 repo: Split packers for tree and data 
The code now bundles tree blobs and data blobs into different pack
files, so we'll end up with pack files that either only contain data or
trees. This is in preparation to adding a cache (#1040), because
tree-only pack files can easily be cached later on.
 2017-09-22 15:36:47 +02:00
Alexander Neumann
db0e3cd772 repo: Remove packer limits 
This commit simplifies finding a packer: The first open packer is taken,
and the upper limit for the pack file is removed.
 2017-09-22 15:36:47 +02:00
Alexander Neumann
d610c60991 repo: Remove unused sync.Pool 2017-09-22 12:37:10 +02:00
Alexander Neumann
3f6e11d26e Allow sorting nodes in trees 2017-09-22 12:37:05 +02:00
Alexander Neumann
9842eff887 local/sftp: Remove unneeded stat() call 2017-09-21 21:47:03 +02:00
Tobias Klein
1a83a739dc fuse: added symlink 'latest' to snapshots-dir 2017-09-21 16:41:20 +02:00
Alexander Neumann
5479daa6d4 Merge pull request #1247 from restic/fix-1246 
Fix backend List()
 2017-09-20 11:04:05 +02:00
Alexander Neumann
4c6b626db6 backend: Improve TestList 2017-09-18 13:18:42 +02:00
Alexander Neumann
835ba16c27 b2: Add pagination for List() 2017-09-18 12:13:35 +02:00
Alexander Neumann
3b6a580b32 backend: Make pagination for List configurable 2017-09-18 12:01:54 +02:00
Tobias Klein
9b38980ed9 fuse: typo 2017-09-17 17:39:28 +02:00
Alexander Neumann
649c536250 backend: Improve test for pagination in list 2017-09-17 11:36:45 +02:00
Alexander Neumann
dd49e2b12d Azure: Fix List(), use pagination marker 2017-09-17 11:32:05 +02:00
Alexander Neumann
f61dab1774 backend: Add test for List() 2017-09-17 11:09:16 +02:00
Alexander Neumann
40edf00182 gs: implement pagination 2017-09-17 11:08:51 +02:00
Alexander Neumann
c35518a865 Azure/GS: Remove ReadDir() 2017-09-17 11:05:30 +02:00
Alexander Neumann
c784a15aaa Merge pull request #1244 from restic/fix-swift-backend-tests 
Ignore "not exist" errors for swift backend tests
 2017-09-16 14:54:32 +02:00
Alexander Neumann
ce180de9b8 Merge pull request #1243 from restic/improve-error-reporting 
Improve error reporting
 2017-09-16 14:54:30 +02:00
Alexander Neumann
2a1633621b Ignore "not exist" errors for swift backend tests 2017-09-16 13:59:55 +02:00
Alexander Neumann
d4e994de7b Improve error reporting 
This will print the error (including a stack trace) if available before
exiting.
 2017-09-16 10:55:13 +02:00
Alexander Neumann
a60e751217 Use .Equal() instead of == for time.Time 
Closes #1238
 2017-09-15 20:57:35 +02:00
Alexander Neumann
5bf2228596 local: Fix creating data dirs 2017-09-11 21:48:25 +02:00
Alexander Neumann
227b01395f local: Add test for open non-existing dir 2017-09-11 21:34:26 +02:00
Alexander Neumann
36eaa22ed0 Merge pull request #1205 from mungomat/backup_time 
Backup time
 2017-09-11 19:01:41 +02:00
Alexander Neumann
62df316356 Merge pull request #1194 from mungomat/bugfix_bucker_always 
fix: bucker
 2017-09-11 19:01:23 +02:00
Alexander Neumann
9fa909ccd6 Update golden files 2017-09-11 17:52:22 +02:00
Alexander Neumann
e66adc42da Always use long name for keys 
Otherwise the code panics if a file with a short name is tried.
 2017-09-10 15:35:10 +02:00
Tobias Klein
9924c311c9 added test cases 2017-09-10 12:23:28 +02:00
Alexander Neumann
e846e14965 Ignore files with invalid name in the repo 2017-09-10 11:00:07 +02:00
Alexander Neumann
36e70228f2 Handle invalid key file 2017-09-10 10:55:01 +02:00
Tobias Klein
a677f1139a removed unnacessary line 2017-09-10 10:41:07 +02:00
Tobias Klein
f26c0cb70f testcase updated 2017-09-09 15:33:12 +02:00
Tobias Klein
087c3fe1dc tests updated 2017-09-09 13:26:35 +02:00
Tobias Klein
43ff971dfd new sub-option for backup: time 
New option to specify the timestamp for a backup
 2017-09-09 13:26:35 +02:00
Alexander Neumann
1e48141648 Fix panic when file name is too short 
Closes #1204
 2017-09-09 10:50:32 +02:00
Alexander Neumann
5dfb4d1195 Merge pull request #1209 from restic/handle-colliding-names 
Resolve name collisions
 2017-09-05 22:33:32 +02:00
Alexander Neumann
83eb075e3a Resolve name collisions 
At the moment when two items to be saved have the same directory name,
restic only saves the first one to the repo. Let's say we have a
structure like this:

    dir1
    └── subdir
        └── file
    dir2
    └── subdir
        └── file

When restic is run on `dir1/subdir` and `dir2/subdir`, it will only save
the first `subdir`:

    $ restic backup dir1/subdir dir2/subdir
    [...]

    $ restic ls -l latest
    drwxr-xr-x  1000   100      0 2017-08-27 20:56:39 /subdir
    -rw-r--r--  1000   100     17 2017-08-27 20:56:39 /subdir/file

That's obviously a bad thing, caused by an early decision to strip the
full path to the files/dirs to save and only leave the last directory.

This commit partly resolves this by handling colliding names and
resolving the conflicts. Restic will now append a counter to the file
(`-123`) until the conflict is resolved. So in the example above, we'll
end up with the following structure:

    $ restic ls -l latest
    drwxr-xr-x  1000   100      0 2017-08-27 20:56:39 /subdir
    -rw-r--r--  1000   100     17 2017-08-27 20:56:39 /subdir/file
    drwxr-xr-x  1000   100      0 2017-08-27 20:56:46 /subdir-1
    -rw-r--r--  1000   100     17 2017-08-27 20:56:46 /subdir-1/file

This partly addresses #549 and closes #1179.

At first I thought that the obvious correction would be to archive the
full path. But it turns out that collisions may still occur: Suppose you
have a file named `foo` in the current directory, and the parent directory
also contains a file `foo`. Archiving these with restic also causes a
collision, since restic strips the `../` from the first file:

    $ restic backup ../foo foo

This also happens with `tar`, which does not handle the collision and
will happily archive two files called `foo`.

So, the best way forward is to handle name collisions and archive the
whole path. The latter will be tackled in a separate PR.
 2017-09-05 21:47:02 +02:00
Alexander Neumann
2444522243 Add test for colliding names 2017-09-05 21:10:02 +02:00
Michael Pratt
92eb1cbffd filter: document recursive wildcards 
Match/ChildMatch accept a ** pattern which is not noted in the doc
string, nor do any of the docs or tests specify whether the match is
greedy (i.e., can 'foo/**/bar' match paths with additional intermediate
bar directories?).

Add a note to the doc string and add test cases for greedy matches.
 2017-09-04 14:38:48 -07:00
Alexander Neumann
fa2ee78a5c Merge pull request #1044 from lloeki/982-improve-restore 
Improve restore
 2017-09-04 21:51:12 +02:00
Tobias Klein
2d73a273af saving a variable 2017-09-03 17:09:55 +02:00
Tobias Klein
761af08889 fix: bucker 
bucker "always" does not return a unique id in case of exact same timestamps
 2017-09-03 17:09:55 +02:00
Greg V
3fbdd12b04 Handle SIGINFO on all supported platforms 
Not just darwin
 2017-09-02 22:06:31 +03:00
Alexander Neumann
85055d1c68 Merge pull request #1187 from FiloSottile/patch-1 
internal/crypto: small simplifications
 2017-08-29 20:43:15 +02:00
Filippo Valsorda
9940e8d9f1 internal/crypto: small simplifications 
* append operates on len, not cap (not a bug since len is set to cap above, but let's avoid the confusion)
* no need to extend ciphertext again to cap after we made it big enough
* make consistent use of ciphertext[:ivSize] vs iv[:]
* make all input problems errors and impossible/catastrophic cases panics
 2017-08-29 00:30:06 +02:00
Alexander Neumann
48b1ab5aaf Merge pull request #1182 from restic/fix-1167 
local: do not create dirs below data/ for non-existing dir
 2017-08-28 21:13:24 +02:00
Michael Pratt
9537bc561d gs: fix nil dereference 
info can be nil if err != nil, resulting in a nil dereference while
logging:

$ # GCS config
$ ./restic init
debug enabled
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x935947]

goroutine 1 [running]:
github.com/restic/restic/internal/backend/gs.(*Backend).Save(0xc420012690, 0xe84e80, 0xc420010448, 0xb57149, 0x3, 0xc4203fc140, 0x40, 0xe7be40, 0xc4201d8f90, 0xa0, ...)
	src/github.com/restic/restic/internal/backend/gs/gs.go:226 +0x6d7
github.com/restic/restic/internal/repository.AddKey(0xe84e80, 0xc420010448, 0xc4202f0360, 0xc42000a1b0, 0x4, 0x0, 0xa55b60, 0xc4203043e0, 0xa55420)
	src/github.com/restic/restic/internal/repository/key.go:235 +0x4a1
github.com/restic/restic/internal/repository.createMasterKey(0xc4202f0360, 0xc42000a1b0, 0x4, 0xa55420, 0xc420304370, 0x6a6070)
	src/github.com/restic/restic/internal/repository/key.go:62 +0x60
github.com/restic/restic/internal/repository.(*Repository).init(0xc4202f0360, 0xe84e80, 0xc420010448, 0xc42000a1b0, 0x4, 0x1, 0xc42030a440, 0x40, 0x32a4573d3d9eb5, 0x0, ...)
	src/github.com/restic/restic/internal/repository/repository.go:403 +0x5d
github.com/restic/restic/internal/repository.(*Repository).Init(0xc4202f0360, 0xe84e80, 0xc420010448, 0xc42000a1b0, 0x4, 0xe84e40, 0xc42004ad80)
	src/github.com/restic/restic/internal/repository/repository.go:397 +0x12c
main.runInit(0xc420018072, 0x16, 0x0, 0x0, 0x0, 0xe84e40, 0xc42004ad80, 0xc42000a1b0, 0x4, 0xe7dac0, ...)
	src/github.com/restic/restic/cmd/restic/cmd_init.go:47 +0x2a4
main.glob..func9(0xeb5000, 0xedad70, 0x0, 0x0, 0x0, 0x0)
	src/github.com/restic/restic/cmd/restic/cmd_init.go:20 +0x8e
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).execute(0xeb5000, 0xedad70, 0x0, 0x0, 0xeb5000, 0xedad70)
	src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:649 +0x457
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xeb3e00, 0xc420011650, 0xa55b60, 0xc420011660)
	src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:728 +0x339
github.com/restic/restic/vendor/github.com/spf13/cobra.(*Command).Execute(0xeb3e00, 0x25, 0xc4201a7eb8)
	src/github.com/restic/restic/vendor/github.com/spf13/cobra/command.go:687 +0x2b
main.main()
	src/github.com/restic/restic/cmd/restic/main.go:72 +0x268

(The error was likely because I had just enabled the GCS API. Subsequent
runs were fine.)
 2017-08-27 21:36:04 -07:00
Alexander Neumann
f9a934759f sftp: Improve error handling for non-existing dir 2017-08-27 20:53:04 +02:00
Alexander Neumann
3686b1ffe5 local: Create directories below data/ if it exists 2017-08-27 20:52:58 +02:00
Alexander Neumann
ea017a49c3 local: Add test for #1167 
It was discovered that restic creates directories when a non-existing
directory is specified as a local repository.
 2017-08-27 20:38:46 +02:00
Loic Nageleisen
f880ff21aa Fixing restore with excluded 
An exclude filter is basically a 'wildcard but foo', so even if a
childMayMatch, other children of a dir may not, therefore childMayMatch
does not matter, but we should not go down unless the dir is selected
for restore.
 2017-08-16 15:25:02 +02:00
Loic Nageleisen
4a36993c19 Smarter filter when children won't match 
This improves restore performance by several orders of magniture by not
going through the whole tree recursively when we can anticipate that no
match will ever occur.
 2017-08-16 15:25:02 +02:00
Emil Hessman
c2ff7150aa internal: check error before deferring file Close() 
If there is an error, file will be `nil`. We should check the returned error before deferring file `Close()`.
 2017-08-13 19:28:13 +02:00
Alexander Neumann
b67c178672 Merge pull request #1149 from restic/azure-support 
Add Azure blob storage as backend
 2017-08-09 21:30:35 +02:00
Alexander Neumann
d9a5b9178e gs: Rework path initialization 2017-08-06 21:47:56 +02:00
Dipta Das
ba75a3884c Add Google Cloud Storage as backend 
Environment variables:
GOOGLE_PROJECT_ID=gcp-project-id
GOOGLE_APPLICATION_CREDENTIALS=path-to-json-file

Environment variables for test:
RESTIC_TEST_GS_PROJECT_ID=gcp-project-id
RESTIC_TEST_GS_APPLICATION_CREDENTIALS=path-to-json-file
RESTIC_TEST_GS_REPOSITORY=gs:us-central1/test-bucket

Init repository:
$ restic -r gs🪣/[prefix] init
 2017-08-06 21:47:55 +02:00
Alexander Neumann
a726c91116 azure: Rework path initialization 2017-08-06 21:47:04 +02:00
Alexander Neumann
072b7a014e azure: User internal errors package 2017-08-06 21:47:04 +02:00
Alexander Neumann
618ce115d7 Azure: Use default HTTP transport 2017-08-06 21:47:04 +02:00
Dipta Das
3a85b6b7c6 Add Azure Blob Storage as backend 
Environment variables:
AZURE_ACCOUNT_NAME=storage-account-name
AZURE_ACCOUNT_KEY=storage-account-key

Environment variables for test:
RESTIC_TEST_AZURE_ACCOUNT_NAME=storage-account-name
RESTIC_TEST_AZURE_ACCOUNT_KEY=storage-account-key
RESTIC_TEST_AZURE_REPOSITORY=azure:restic-test-container

Init repository:
$ restic -r azure:container-name:/prefix/dir init
 2017-08-06 21:47:04 +02:00
Alexander Neumann
23c903074c Move restic package to internal/restic 2017-07-24 17:43:32 +02:00
Alexander Neumann
6caeff2408 Run goimports 2017-07-23 14:21:03 +02:00
Alexander Neumann
83d1a46526 Moves files 2017-07-23 14:19:13 +02:00