Compare commits

..

2 commits

Author SHA1 Message Date
Alexander Neumann
271eabd239 Add crypto debug benchmark 2020-12-23 10:46:53 +01:00
Alexander Neumann
0333ef3325 Add code to debug wrong hash issue 2020-12-20 14:51:15 +01:00
1008 changed files with 32584 additions and 75717 deletions

2
.gitattributes vendored
View file

@ -1,2 +0,0 @@
# Workaround for https://github.com/golang/go/issues/52268.
**/testdata/fuzz/*/* eol=lf

View file

@ -32,30 +32,23 @@ Output of `restic version`
--------------------------
What backend/service did you use to store the repository?
---------------------------------------------------------
Problem description / Steps to reproduce
----------------------------------------
How did you run restic exactly?
-------------------------------
<!--
This section should include at least:
* A description of the problem you are having with restic.
* The complete command line and any environment variables you used to
configure restic's backend access. Make sure to replace sensitive values!
* The output of the commands, what restic prints gives may give us much
information to diagnose the problem!
* The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
What backend/server/service did you use to store the repository?
----------------------------------------------------------------
Expected behavior
-----------------
@ -72,12 +65,22 @@ In this section, please try to concentrate on observations, so only describe
what you observed directly.
-->
Steps to reproduce the behavior
-------------------------------
<!--
The more time you spend describing an easy way to reproduce the behavior (if
this is possible), the easier it is for the project developers to fix it!
-->
Do you have any idea what may have caused this?
-----------------------------------------------
<!--
Did something noteworthy happen on your system, Internet connection, backend services, etc?
-->
Do you have an idea how to solve the issue?
-------------------------------------------
Did restic help you today? Did it make you happy in any way?

View file

@ -1,7 +1,13 @@
<!--
Thank you very much for contributing code or documentation to restic! Please
fill out the following questions to make it easier for us to review your
changes.
You do not need to check all the boxes below all at once, feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box.
-->
What does this PR change? What problem does it solve?
@ -11,8 +17,8 @@ What does this PR change? What problem does it solve?
Describe the changes and their purpose here, as detailed as needed.
-->
Was the change previously discussed in an issue or on the forum?
----------------------------------------------------------------
Was the change discussed in an issue or in the forum before?
------------------------------------------------------------
<!--
Link issues and relevant forum posts here.
@ -24,19 +30,11 @@ is closed automatically when this PR is merged.
Checklist
---------
<!--
You do not need to check all the boxes below all at once. Feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box. Enable a checkbox by replacing [ ] with [x].
Please always follow these steps:
- Read the [contribution guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches).
- Enable [maintainer edits](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
- Run `gofmt` on the code in all commits.
- Format all commit messages in the same style as [the other commits in the repository](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits).
-->
- [ ] I have added tests for all code changes.
- [ ] I have added documentation for relevant changes (in the manual).
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (see [template](https://github.com/restic/restic/blob/master/changelog/TEMPLATE)).
- [ ] I'm done! This pull request is ready for review.
- [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
- [ ] I have enabled [maintainer edits for this PR](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
- [ ] I have added tests for all changes in this PR
- [ ] I have added documentation for the changes (in the manual)
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/TEMPLATE))
- [ ] I have run `gofmt` on the code in all commits
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits)
- [ ] I'm done, this Pull Request is ready for review

View file

@ -1,13 +0,0 @@
version: 2
updates:
# Dependencies listed in go.mod
- package-ecosystem: "gomod"
directory: "/" # Location of package manifests
schedule:
interval: "monthly"
# Dependencies listed in .github/workflows/*.yml
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"

View file

@ -1,66 +0,0 @@
name: Create and publish a Docker image
on:
push:
tags:
- 'v*'
branches:
- 'master'
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
jobs:
build-and-push-image:
if: github.repository == 'restic/restic'
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Log in to the Container registry
uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@8e5442c4ef9f78752691e2d8f8d19755c6f78e81
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
- name: Set up QEMU
uses: docker/setup-qemu-action@68827325e0b33c7199eb31dd4e31fbe9023e06e3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@2b51285047da1547ffb1b2203d8be4c0af6b1f20
- name: Ensure consistent binaries
run: |
echo "removing git directory for consistency with release binaries"
rm -rf .git
# remove VCS information from release builds, keep VCS for nightly builds on master
if: github.ref != 'refs/heads/master'
- name: Build and push Docker image
uses: docker/build-push-action@15560696de535e4014efeff63c48f16952e52dd1
with:
push: true
context: .
file: docker/Dockerfile.release
platforms: linux/386,linux/amd64,linux/arm,linux/arm64
pull: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

View file

@ -1,330 +0,0 @@
name: test
on:
# run tests on push to master, but not when other branches are pushed to
push:
branches:
- master
# run tests for all pull requests
pull_request:
merge_group:
permissions:
contents: read
env:
latest_go: "1.23.x"
GO111MODULE: on
jobs:
test:
strategy:
matrix:
# list of jobs to run:
include:
- job_name: Windows
go: 1.23.x
os: windows-latest
- job_name: macOS
go: 1.23.x
os: macOS-latest
test_fuse: false
- job_name: Linux
go: 1.23.x
os: ubuntu-latest
test_cloud_backends: true
test_fuse: true
check_changelog: true
- job_name: Linux (race)
go: 1.23.x
os: ubuntu-latest
test_fuse: true
test_opts: "-race"
- job_name: Linux
go: 1.22.x
os: ubuntu-latest
test_fuse: true
- job_name: Linux
go: 1.21.x
os: ubuntu-latest
test_fuse: true
name: ${{ matrix.job_name }} Go ${{ matrix.go }}
runs-on: ${{ matrix.os }}
env:
GOPROXY: https://proxy.golang.org
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Set up Go ${{ matrix.go }}
uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go }}
- name: Get programs (Linux/macOS)
run: |
echo "build Go tools"
go install github.com/restic/rest-server/cmd/rest-server@master
echo "install minio server"
mkdir $HOME/bin
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/darwin-amd64/minio
else
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
fi
chmod 755 $HOME/bin/minio
echo "install rclone"
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-osx-amd64.zip
else
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-linux-amd64.zip
fi
unzip rclone.zip
cp rclone*/rclone $HOME/bin
chmod 755 $HOME/bin/rclone
rm -rf rclone*
# add $HOME/bin to path ($GOBIN was already added to the path by setup-go@v3)
echo $HOME/bin >> $GITHUB_PATH
if: matrix.os == 'ubuntu-latest' || matrix.os == 'macOS-latest'
- name: Get programs (Windows)
shell: powershell
run: |
$ProgressPreference = 'SilentlyContinue'
echo "build Go tools"
go install github.com/restic/rest-server/cmd/rest-server@master
echo "install minio server"
mkdir $Env:USERPROFILE/bin
Invoke-WebRequest https://dl.minio.io/server/minio/release/windows-amd64/minio.exe -OutFile $Env:USERPROFILE/bin/minio.exe
echo "install rclone"
Invoke-WebRequest https://downloads.rclone.org/rclone-current-windows-amd64.zip -OutFile rclone.zip
unzip rclone.zip
copy rclone*/rclone.exe $Env:USERPROFILE/bin
# add $USERPROFILE/bin to path ($GOBIN was already added to the path by setup-go@v3)
echo $Env:USERPROFILE\bin >> $Env:GITHUB_PATH
echo "install tar"
cd $env:USERPROFILE
mkdir tar
cd tar
# install exactly these versions of tar and the libraries, other combinations might not work!
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/tar-1.13-1-bin.zip -OutFile tar.zip
unzip tar.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libintl-0.11.5-2-bin.zip -OutFile libintl.zip
unzip libintl.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libiconv-1.8-1-bin.zip -OutFile libiconv.zip
unzip libiconv.zip
# add $USERPROFILE/tar/bin to path
echo $Env:USERPROFILE\tar\bin >> $Env:GITHUB_PATH
if: matrix.os == 'windows-latest'
- name: Build with build.go
run: |
go run build.go
- name: Minimal test
run: |
./restic init
./restic backup .
env:
RESTIC_REPOSITORY: ../testrepo
RESTIC_PASSWORD: password
- name: Run local Tests
env:
RESTIC_TEST_FUSE: ${{ matrix.test_fuse }}
run: |
go test -cover ${{matrix.test_opts}} ./...
- name: Test cloud backends
env:
RESTIC_TEST_S3_KEY: ${{ secrets.RESTIC_TEST_S3_KEY }}
RESTIC_TEST_S3_SECRET: ${{ secrets.RESTIC_TEST_S3_SECRET }}
RESTIC_TEST_S3_REPOSITORY: ${{ secrets.RESTIC_TEST_S3_REPOSITORY }}
RESTIC_TEST_AZURE_ACCOUNT_NAME: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_NAME }}
RESTIC_TEST_AZURE_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_KEY }}
RESTIC_TEST_AZURE_REPOSITORY: ${{ secrets.RESTIC_TEST_AZURE_REPOSITORY }}
RESTIC_TEST_B2_ACCOUNT_ID: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_ID }}
RESTIC_TEST_B2_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_KEY }}
RESTIC_TEST_B2_REPOSITORY: ${{ secrets.RESTIC_TEST_B2_REPOSITORY }}
RESTIC_TEST_GS_REPOSITORY: ${{ secrets.RESTIC_TEST_GS_REPOSITORY }}
RESTIC_TEST_GS_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
GOOGLE_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64: ${{ secrets.RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 }}
RESTIC_TEST_OS_AUTH_URL: ${{ secrets.RESTIC_TEST_OS_AUTH_URL }}
RESTIC_TEST_OS_TENANT_NAME: ${{ secrets.RESTIC_TEST_OS_TENANT_NAME }}
RESTIC_TEST_OS_USERNAME: ${{ secrets.RESTIC_TEST_OS_USERNAME }}
RESTIC_TEST_OS_PASSWORD: ${{ secrets.RESTIC_TEST_OS_PASSWORD }}
RESTIC_TEST_OS_REGION_NAME: ${{ secrets.RESTIC_TEST_OS_REGION_NAME }}
RESTIC_TEST_SWIFT: ${{ secrets.RESTIC_TEST_SWIFT }}
# fail if any of the following tests cannot be run
RESTIC_TEST_DISALLOW_SKIP: "restic/backend/rest.TestBackendREST,\
restic/backend/sftp.TestBackendSFTP,\
restic/backend/s3.TestBackendMinio,\
restic/backend/rclone.TestBackendRclone,\
restic/backend/s3.TestBackendS3,\
restic/backend/swift.TestBackendSwift,\
restic/backend/b2.TestBackendB2,\
restic/backend/gs.TestBackendGS,\
restic/backend/azure.TestBackendAzure"
run: |
# prepare credentials for Google Cloud Storage tests in a temp file
export GOOGLE_APPLICATION_CREDENTIALS=$(mktemp --tmpdir restic-gcs-auth-XXXXXXX)
echo $RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
go test -cover -parallel 4 ./internal/backend/...
# only run cloud backend tests for pull requests from and pushes to our
# own repo, otherwise the secrets are not available
# Skip for Dependabot pull requests as these are run without secrets
# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#responding-to-events
if: ((github.repository == 'restic/restic' && github.event_name == 'push') || github.event.pull_request.head.repo.full_name == github.repository) && (github.actor != 'dependabot[bot]') && matrix.test_cloud_backends
- name: Check changelog files with calens
run: |
echo "install calens"
go install github.com/restic/calens@latest
echo "check changelog files"
calens
if: matrix.check_changelog
cross_compile:
strategy:
matrix:
# run cross-compile in three batches parallel so the overall tests run faster
subset:
- "0/3"
- "1/3"
- "2/3"
env:
GOPROXY: https://proxy.golang.org
runs-on: ubuntu-latest
name: Cross Compile for subset ${{ matrix.subset }}
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v5
with:
go-version: ${{ env.latest_go }}
- name: Cross-compile for subset ${{ matrix.subset }}
run: |
mkdir build-output build-output-debug
go run ./helpers/build-release-binaries/main.go -o build-output -s . --platform-subset ${{ matrix.subset }}
go run ./helpers/build-release-binaries/main.go -o build-output-debug -s . --platform-subset ${{ matrix.subset }} --tags debug
lint:
name: lint
runs-on: ubuntu-latest
permissions:
contents: read
# allow annotating code in the PR
checks: write
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v5
with:
go-version: ${{ env.latest_go }}
- name: golangci-lint
uses: golangci/golangci-lint-action@v6
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.61.0
args: --verbose --timeout 5m
# only run golangci-lint for pull requests, otherwise ALL hints get
# reported. We need to slowly address all issues until we can enable
# linting the master branch :)
if: github.event_name == 'pull_request'
- name: Check go.mod/go.sum
run: |
echo "check if go.mod and go.sum are up to date"
go mod tidy
git diff --exit-code go.mod go.sum
analyze:
name: Analyze results
needs: [test, cross_compile, lint]
if: always()
permissions: # no need to access code
contents: none
runs-on: ubuntu-latest
steps:
- name: Decide whether the needed jobs succeeded or failed
uses: re-actors/alls-green@05ac9388f0aebcb5727afa17fcccfecd6f8ec5fe
with:
jobs: ${{ toJSON(needs) }}
docker:
name: docker
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v4
- name: Docker meta
id: meta
uses: docker/metadata-action@v5
with:
# list of Docker images to use as base name for tags
images: |
restic/restic
# generate Docker tags based on the following events/attributes
tags: |
type=schedule
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and push
id: docker_build
uses: docker/build-push-action@v6
with:
push: false
context: .
file: docker/Dockerfile
pull: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

2
.gitignore vendored
View file

@ -1,5 +1,3 @@
/.idea
/restic
/restic.exe
/.vagrant
/.vscode

View file

@ -1,69 +0,0 @@
# This is the configuration for golangci-lint for the restic project.
#
# A sample config with all settings is here:
# https://github.com/golangci/golangci-lint/blob/master/.golangci.example.yml
linters:
# only enable the linters listed below
disable-all: true
enable:
# make sure all errors returned by functions are handled
- errcheck
# show how code can be simplified
- gosimple
# make sure code is formatted
- gofmt
# examine code and report suspicious constructs, such as Printf calls whose
# arguments do not align with the format string
- govet
# make sure names and comments are used according to the conventions
- revive
# detect when assignments to existing variables are not used
- ineffassign
# run static analysis and find errors
- staticcheck
# find unused variables, functions, structs, types, etc.
- unused
# parse and typecheck code
- typecheck
# ensure that http response bodies are closed
- bodyclose
- importas
issues:
# don't use the default exclude rules, this hides (among others) ignored
# errors from Close() calls
exclude-use-default: false
# list of things to not warn about
exclude:
# revive: do not warn about missing comments for exported stuff
- exported (function|method|var|type|const) .* should have comment or be unexported
# revive: ignore constants in all caps
- don't use ALL_CAPS in Go names; use CamelCase
# revive: lots of packages don't have such a comment
- "package-comments: should have a package comment"
# staticcheck: there's no easy way to replace these packages
- "SA1019: \"golang.org/x/crypto/poly1305\" is deprecated"
- "SA1019: \"golang.org/x/crypto/openpgp\" is deprecated"
exclude-rules:
# revive: ignore unused parameters in tests
- path: (_test\.go|testing\.go|backend/.*/tests\.go)
text: "unused-parameter:"
linters-settings:
importas:
alias:
- pkg: github.com/restic/restic/internal/test
alias: rtest

2
.hound.yml Normal file
View file

@ -0,0 +1,2 @@
go:
enabled: true

View file

@ -1,22 +0,0 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
version: 2
build:
os: ubuntu-22.04
tools:
python: "3.11"
# Build HTMLZip
formats:
- htmlzip
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: doc/conf.py
# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
install:
- requirements: doc/requirements.txt

58
.travis.yml Normal file
View file

@ -0,0 +1,58 @@
language: go
sudo: false
matrix:
include:
- os: linux
go: "1.13.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: linux
go: "1.14.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
# only run fuse and cloud backends tests on Travis for the latest Go on Linux
- os: linux
go: "1.15.x"
sudo: true
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: osx
go: "1.15.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/Library/Caches/go-build
- $HOME/gopath/pkg/mod
branches:
only:
- master
notifications:
irc:
channels:
- "chat.freenode.net#restic"
on_success: change
on_failure: change
skip_join: true
install:
- go version
- export GOBIN="$GOPATH/bin"
- export PATH="$PATH:$GOBIN"
- go env
script:
- go run run_integration_tests.go

File diff suppressed because it is too large Load diff

View file

@ -6,18 +6,18 @@ Ways to Help Out
Thank you for your contribution! Please **open an issue first** (or add a
comment to an existing issue) if you plan to work on any code or add a new
feature. This way, duplicate work is prevented and we can discuss your ideas
and design first. Small bugfixes are an exception to this rule, just open a
pull request in this case.
and design first.
There are several ways you can help us out. First of all code contributions and
bug fixes are most welcome. However even "minor" details as fixing spelling
errors, improving documentation or pointing out usability issues are a great
help also.
The restic project uses the GitHub infrastructure (see the
[project page](https://github.com/restic/restic)) for all related discussions
as well as the [forum](https://forum.restic.net/) and the `#restic` channel
on [irc.libera.chat](https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:6697/#restic).
on [irc.freenode.net](https://kiwiirc.com/nextclient/irc.freenode.net/restic).
If you want to find an area that currently needs improving have a look at the
open issues listed at the
@ -49,8 +49,9 @@ environment was used and so on. Please tell us at least the following things:
Remember, the easier it is for us to reproduce the bug, the earlier it will be
corrected!
In addition, you can instruct restic to create a debug log by setting the
environment variable `DEBUG_LOG` to a file, e.g. like this:
In addition, you can compile restic with debug support by running
`go run build.go -tags debug` and instructing it to create a debug
log by setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
$ export DEBUG_LOG=/tmp/restic-debug.log
$ restic backup ~/work
@ -59,19 +60,6 @@ Please be aware that the debug log file will contain potentially sensitive
things like file and directory names, so please either redact it before
uploading it somewhere or post only the parts that are really relevant.
If restic gets stuck, please also include a stacktrace in the description.
On non-Windows systems, you can send a SIGQUIT signal to restic or press
`Ctrl-\` to achieve the same result. This causes restic to print a stacktrace
and then exit immediately. This will not damage your repository, however,
it might be necessary to manually clean up stale lock files using
`restic unlock`.
On Windows, please set the environment variable `RESTIC_DEBUG_STACKTRACE_SIGINT`
to `true` and press `Ctrl-C` to create a stacktrace.
If you think restic uses too much memory or a too large cache directory, then
please include the output of `restic stats --mode debug`.
Development Environment
=======================
@ -79,8 +67,8 @@ Development Environment
The repository contains the code written for restic in the directories
`cmd/` and `internal/`.
Make sure you have the minimum required Go version installed. Clone the repo
(without having `$GOPATH` set) and `cd` into the directory:
Restic requires Go version 1.13 or later for compiling. Clone the repo (without
having `$GOPATH` set) and `cd` into the directory:
$ unset GOPATH
$ git clone https://github.com/restic/restic
@ -90,42 +78,12 @@ Then use the `go` tool to build restic:
$ go build ./cmd/restic
$ ./restic version
restic 0.14.0-dev (compiled manually) compiled with go1.19 on linux/amd64
To create a debug build use:
$ go build -tags debug ./cmd/restic
restic 0.10.0-dev (compiled manually) compiled with go1.15.2 on linux/amd64
You can run all tests with the following command:
$ go test ./...
Performance and Memory Usage Issues
===================================
Debug builds of restic support the `--block-profile`, `--cpu-profile`,
`--mem-profile`, and `--trace-profile` options which collect performance data
that later on can be analyzed using the go tools:
$ restic --cpu-profile . [...]
$ go tool pprof -http localhost:12345 cpu.pprof
To analyze a trace profile use `go tool trace -http=localhost:12345 trace.out`.
As the memory usage of restic changes over time, it may be useful to capture a
snapshot of the current heap. This is possible using then `--listen-profile`
option. Then while restic runs you can query and afterwards analyze the heap statistics.
$ restic --listen-profile localhost:12345 [...]
$ curl http://localhost:12345/debug/pprof/heap -o heap.pprof
$ go tool pprof -http localhost:12345 heap.pprof
Further useful tools are setting the environment variable `GODEBUG=gctrace=1`,
which provides information about garbage collector runs. For a graphical variant
combine this with gcvis.
Providing Patches
=================
@ -166,10 +124,7 @@ down to the following steps:
writing, ask yourself: If I were the user, what would I need to be aware
of with this change?
8. Do not edit the man pages under `doc/man` or `doc/manual_rest.rst` -
these are autogenerated before new releases.
9. Once your code looks good and passes all the tests, we'll merge it. Thanks
8. Once your code looks good and passes all the tests, we'll merge it. Thanks
a lot for your contribution!
Please provide the patches for each bug or feature in a separate branch and
@ -186,14 +141,6 @@ Installing the script `fmt-check` from https://github.com/edsrzf/gofmt-git-hook
locally as a pre-commit hook checks formatting before committing automatically,
just copy this script to `.git/hooks/pre-commit`.
The project is using the program
[`golangci-lint`](https://github.com/golangci/golangci-lint) to run a list of
linters and checkers. It will be run on the code when you submit a PR. In order
to check your code beforehand, you can run `golangci-lint run` manually.
Eventually, we will enable `golangci-lint` for the whole code base. For now,
you can ignore warnings printed for lines you did not modify, those will be
ignored by the CI.
For each pull request, several different systems run the integration tests on
Linux, macOS and Windows. We won't merge any code that does not pass all tests
for all systems, so when a tests fails, try to find out what's wrong and fix

View file

@ -1,5 +1,6 @@
[![Documentation](https://readthedocs.org/projects/restic/badge/?version=latest)](https://restic.readthedocs.io/en/latest/?badge=latest)
[![Build Status](https://github.com/restic/restic/workflows/test/badge.svg)](https://github.com/restic/restic/actions?query=workflow%3Atest)
[![Build Status Travis](https://travis-ci.com/restic/restic.svg?branch=master)](https://travis-ci.com/restic/restic)
[![Build Status AppVeyor](https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true)](https://ci.appveyor.com/project/fd0/restic/branch/master)
[![Go Report Card](https://goreportcard.com/badge/github.com/restic/restic)](https://goreportcard.com/report/github.com/restic/restic)
# Introduction
@ -10,7 +11,8 @@ For detailed usage and installation instructions check out the [documentation](h
You can ask questions in our [Discourse forum](https://forum.restic.net).
## Quick start
Quick start
-----------
Once you've [installed](https://restic.readthedocs.io/en/latest/020_installation.html) restic, start
off with creating a repository for your backups:
@ -45,8 +47,8 @@ Therefore, restic supports the following backends for storing backups natively:
- [Local directory](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local)
- [sftp server (via SSH)](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp)
- [HTTP REST server](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) ([protocol](https://restic.readthedocs.io/en/latest/100_references.html#rest-backend), [rest-server](https://github.com/restic/rest-server))
- [Amazon S3](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) (either from Amazon or using the [Minio](https://minio.io) server)
- [HTTP REST server](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) ([protocol](doc/100_references.rst#rest-backend), [rest-server](https://github.com/restic/rest-server))
- [AWS S3](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) (either from Amazon or using the [Minio](https://minio.io) server)
- [OpenStack Swift](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift)
- [BackBlaze B2](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2)
- [Microsoft Azure Blob Storage](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage)
@ -58,7 +60,7 @@ Therefore, restic supports the following backends for storing backups natively:
Restic is a program that does backups right and was designed with the
following principles in mind:
- **Easy**: Doing backups should be a frictionless process, otherwise
- **Easy:** Doing backups should be a frictionless process, otherwise
you might be tempted to skip it. Restic should be easy to configure
and use, so that, in the event of a data loss, you can just restore
it. Likewise, restoring data should not be complicated.
@ -91,17 +93,20 @@ reproduce a byte identical version from the source code for that
release. Instructions on how to do that are contained in the
[builder repository](https://github.com/restic/builder).
## News
News
----
You can follow the restic project on Mastodon [@resticbackup](https://fosstodon.org/@restic) or subscribe to
You can follow the restic project on Twitter [@resticbackup](https://twitter.com/resticbackup) or by subscribing to
the [project blog](https://restic.net/blog/).
## License
License
-------
Restic is licensed under [BSD 2-Clause License](https://opensource.org/licenses/BSD-2-Clause). You can find the
complete text in [`LICENSE`](LICENSE).
complete text in [``LICENSE``](LICENSE).
## Sponsorship
Sponsorship
-----------
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by [AppsCode](https://appscode.com)!

View file

@ -1 +1 @@
0.17.3-dev
0.11.0

32
appveyor.yml Normal file
View file

@ -0,0 +1,32 @@
clone_folder: c:\restic
environment:
GOPATH: c:\gopath
branches:
only:
- master
cache:
- '%LocalAppData%\go-build'
init:
- ps: >-
$app = Get-WmiObject -Class Win32_Product -Filter "Vendor = 'http://golang.org'"
if ($app) {
$app.Uninstall()
}
install:
- rmdir c:\go /s /q
- appveyor DownloadFile https://dl.google.com/go/go1.15.2.windows-amd64.msi
- msiexec /i go1.15.2.windows-amd64.msi /q
- go version
- go env
- appveyor DownloadFile https://sourceforge.netcologne.de/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip
- 7z x tar.zip bin/tar.exe
- set PATH=bin/;%PATH%
build_script:
- go run run_integration_tests.go

View file

@ -3,8 +3,8 @@
// This program aims to make building Go programs for end users easier by just
// calling it with `go run`, without having to setup a GOPATH.
//
// This program checks for a minimum Go version. It will use Go modules for
// compilation. It builds the package configured as Main in the Config struct.
// This program needs Go >= 1.12. It'll use Go modules for compilation. It
// builds the package configured as Main in the Config struct.
// BSD 2-Clause License
//
@ -35,7 +35,6 @@
// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//go:build ignore_build_go
// +build ignore_build_go
package main
@ -43,6 +42,7 @@ package main
import (
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
"path/filepath"
@ -58,7 +58,7 @@ var config = Config{
Main: "./cmd/restic", // package name for the main package
DefaultBuildTags: []string{"selfupdate"}, // specify build tags which are always used
Tests: []string{"./..."}, // tests to run
MinVersion: GoVersion{Major: 1, Minor: 21, Patch: 0}, // minimum Go version supported
MinVersion: GoVersion{Major: 1, Minor: 11, Patch: 0}, // minimum Go version supported
}
// Config configures the build.
@ -123,8 +123,17 @@ func printEnv(env []string) {
// build runs "go build args..." with GOPATH set to gopath.
func build(cwd string, env map[string]string, args ...string) error {
// -trimpath removes all absolute paths from the binary.
a := []string{"build", "-trimpath"}
a := []string{"build"}
// try to remove all absolute paths from resulting binary
if goVersion.AtLeast(GoVersion{1, 13, 0}) {
// use the new flag introduced by Go 1.13
a = append(a, "-trimpath")
} else {
// otherwise try to trim as many paths as possible
a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", cwd))
a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", cwd))
}
if enablePIE {
a = append(a, "-buildmode=pie")
@ -178,7 +187,7 @@ func test(cwd string, env map[string]string, args ...string) error {
// getVersion returns the version string from the file VERSION in the current
// directory.
func getVersionFromFile() string {
buf, err := os.ReadFile("VERSION")
buf, err := ioutil.ReadFile("VERSION")
if err != nil {
verbosePrintf("error reading file VERSION: %v\n", err)
return ""
@ -318,8 +327,12 @@ func (v GoVersion) String() string {
}
func main() {
if !goVersion.AtLeast(GoVersion{1, 12, 0}) {
die("Go version (%v) is too old, restic requires Go >= 1.12\n", goVersion)
}
if !goVersion.AtLeast(config.MinVersion) {
fmt.Fprintf(os.Stderr, "Detected version %s is too old, restic requires at least %s\n", goVersion, config.MinVersion)
fmt.Fprintf(os.Stderr, "%s detected, this program requires at least %s\n", goVersion, config.MinVersion)
os.Exit(1)
}
@ -380,12 +393,6 @@ func main() {
}
}
solarisMinVersion := GoVersion{Major: 1, Minor: 20, Patch: 0}
if env["GOARCH"] == "solaris" && !goVersion.AtLeast(solarisMinVersion) {
fmt.Fprintf(os.Stderr, "Detected version %s is too old, restic requires at least %s for Solaris\n", goVersion, solarisMinVersion)
os.Exit(1)
}
verbosePrintf("detected Go version %v\n", goVersion)
preserveSymbols := false

View file

@ -10,7 +10,7 @@ https://github.com/restic/restic/issues/2244
NOTE: This new implementation does not guarantee order in which blobs
are written to the target files and, for example, the last blob of a
file can be written to the file before any of the preceding file blobs.
file can be written to the file before any of the preceeding file blobs.
It is therefore possible to have gaps in the data written to the target
files if restore fails or interrupted by the user.

View file

@ -1,6 +1,6 @@
Bugfix: Don't abort the stats command when data blobs are missing
Running the stats command in the blobs-per-file mode on a repository with
Runing the stats command in the blobs-per-file mode on a repository with
missing data blobs previously resulted in a crash.
https://github.com/restic/restic/pull/2668

View file

@ -1,6 +1,6 @@
Bugfix: Mark repository files as read-only when using the local backend
Files stored in a local repository were marked as writable on the
Files stored in a local repository were marked as writeable on the
filesystem for non-Windows systems, which did not prevent accidental file
modifications outside of restic. In addition, the local backend did not work
with certain filesystems and network mounts which do not permit modifications

View file

@ -5,7 +5,7 @@ another process using an exclusive lock through a filesystem snapshot. Restic
was unable to backup those files before. This update enables backing up these
files.
This needs to be enabled explicitly using the --use-fs-snapshot option of the
This needs to be enabled explicitely using the --use-fs-snapshot option of the
backup command.
https://github.com/restic/restic/issues/340

View file

@ -1,10 +0,0 @@
Bugfix: Make `mount` not create missing mount point directory
When specifying a non-existent directory as mount point for the `mount`
command, restic used to create the specified directory automatically.
This has now changed such that restic instead gives an error when the
specified directory for the mount point does not exist.
https://github.com/restic/restic/issues/1681
https://github.com/restic/restic/pull/3008

View file

@ -1,8 +0,0 @@
Bugfix: Ignore `no data available` filesystem error during backup
Restic was unable to backup files on some filesystems, for example certain
configurations of CIFS on Linux which return a `no data available` error
when reading extended attributes. These errors are now ignored.
https://github.com/restic/restic/issues/1800
https://github.com/restic/restic/pull/3034

View file

@ -1,8 +0,0 @@
Enhancement: Allow specifying percentage in `check --read-data-subset`
We've enhanced the `check` command's `--read-data-subset` option to also accept
a percentage (e.g. `2.5%` or `10%`). This will check the given percentage of
pack files (which are randomly selected on each run).
https://github.com/restic/restic/issues/2186
https://github.com/restic/restic/pull/3038

View file

@ -1,14 +0,0 @@
Enhancement: Report permanent/fatal backend errors earlier
When encountering errors in reading from or writing to storage backends,
restic retries the failing operation up to nine times (for a total of ten
attempts). It used to retry all backend operations, but now detects some
permanent error conditions so that it can report fatal errors earlier.
Permanent failures include local disks being full, SSH connections
dropping and permission errors.
https://github.com/restic/restic/issues/2453
https://github.com/restic/restic/pull/3170
https://github.com/restic/restic/issues/3180
https://github.com/restic/restic/pull/3181

View file

@ -1,21 +0,0 @@
Enhancement: Add Alibaba/Aliyun OSS support in the `s3` backend
A new extended option `s3.bucket-lookup` has been added to support
Alibaba/Aliyun OSS in the `s3` backend. The option can be set to one
of the following values:
- `auto` - Existing behaviour
- `dns` - Use DNS style bucket access
- `path` - Use path style bucket access
To make the `s3` backend work with Alibaba/Aliyun OSS you must set
`s3.bucket-lookup` to `dns` and set the `s3.region` parameter. For
example:
restic -o s3.bucket-lookup=dns -o s3.region=oss-eu-west-1 -r s3:https://oss-eu-west-1.aliyuncs.com/bucketname init
Note that `s3.region` must be set, otherwise the MinIO SDK tries to
look it up and it seems that Alibaba doesn't support that properly.
https://github.com/restic/restic/issues/2528
https://github.com/restic/restic/pull/2535

View file

@ -1,9 +0,0 @@
Bugfix: Report the correct owner of directories in FUSE mounts
Restic 0.10.0 changed the FUSE mount to always report the current user
as the owner of directories within the FUSE mount, which is incorrect.
This is now changed back to reporting the correct owner of a directory.
https://github.com/restic/restic/issues/2563
https://github.com/restic/restic/pull/3141

View file

@ -1,31 +0,0 @@
Bugfix: Make `backup` and `tag` commands separate tags by comma
Running `restic backup --tag foo,bar` previously created snapshots with one
single tag containing a comma (`foo,bar`) instead of two tags (`foo`, `bar`).
Similarly, the `tag` command's `--set`, `--add` and `--remove` options would
treat `foo,bar` as one tag instead of two tags. This was inconsistent with
other commands and often unexpected when one intended `foo,bar` to mean two
tags.
To be consistent in all commands, restic now interprets `foo,bar` to mean two
separate tags (`foo` and `bar`) instead of one tag (`foo,bar`) everywhere,
including in the `backup` and `tag` commands.
NOTE: This change might result in unexpected behavior in cases where you use
the `forget` command and filter on tags like `foo,bar`. Snapshots previously
backed up with `--tag foo,bar` will still not match that filter, but snapshots
saved from now on will match that filter.
To replace `foo,bar` tags with `foo` and `bar` tags in old snapshots, you can
first generate a list of the relevant snapshots using a command like:
restic snapshots --json --quiet | jq '.[] | select(contains({tags: ["foo,bar"]})) | .id'
and then use `restic tag --set foo --set bar snapshotID [...]` to set the new
tags. Please adjust the commands to include real tag names and any additional
tags, as well as the list of snapshots to process.
https://github.com/restic/restic/issues/2688
https://github.com/restic/restic/pull/2690
https://github.com/restic/restic/pull/3197

View file

@ -1,17 +0,0 @@
Enhancement: Configurable progress reports for non-interactive terminals
The `backup`, `check` and `prune` commands never printed any progress
reports on non-interactive terminals. This behavior is now configurable
using the `RESTIC_PROGRESS_FPS` environment variable. Use for example a
value of `1` for an update every second, or `0.01666` for an update every
minute.
The `backup` command now also prints the current progress when restic
receives a `SIGUSR1` signal.
Setting the `RESTIC_PROGRESS_FPS` environment variable or sending a `SIGUSR1`
signal prints a status report even when `--quiet` was specified.
https://github.com/restic/restic/issues/2706
https://github.com/restic/restic/issues/3194
https://github.com/restic/restic/pull/3199

View file

@ -1,5 +0,0 @@
Bugfix: Make the `cat` command respect the `--no-lock` option
The `cat` command would not respect the `--no-lock` flag. This is now fixed.
https://github.com/restic/restic/issues/2739

View file

@ -1,13 +0,0 @@
Enhancement: Add `backup` options `--files-from-{verbatim,raw}`
The new `backup` options `--files-from-verbatim` and `--files-from-raw` read a
list of files to back up from a file. Unlike the existing `--files-from`
option, these options do not interpret the listed filenames as glob patterns;
instead, whitespace in filenames is preserved as-is and no pattern expansion is
done. Please see the documentation for specifics.
These new options are highly recommended over `--files-from`, when using a
script to generate the list of files to back up.
https://github.com/restic/restic/issues/2944
https://github.com/restic/restic/issues/3013

View file

@ -1,18 +0,0 @@
Enhancement: Allow usage of deprecated S3 `ListObjects` API
Some S3 API implementations, e.g. Ceph before version 14.2.5, have a broken
`ListObjectsV2` implementation which causes problems for restic when using
their API endpoints. When a broken server implementation is used, restic prints
errors similar to the following:
List() returned error: Truncated response should have continuation token set
As a temporary workaround, restic now allows using the older `ListObjects`
endpoint by setting the `s3.list-objects-v1` extended option, for instance:
restic -o s3.list-objects-v1=true snapshots
Please note that this option may be removed in future versions of restic.
https://github.com/restic/restic/issues/3083
https://github.com/restic/restic/pull/3085

View file

@ -1,10 +0,0 @@
Bugfix: The `--use-fs-snapshot` option now works on windows/386
Restic failed to create VSS snapshots on windows/386 with the following error:
GetSnapshotProperties() failed: E_INVALIDARG (0x80070057)
This is now fixed.
https://github.com/restic/restic/issues/3087
https://github.com/restic/restic/pull/3090

View file

@ -1,17 +0,0 @@
Change: Deleting files on Google Drive now moves them to the trash
When deleting files on Google Drive via the `rclone` backend, restic used to
bypass the trash folder required that one used the `-o rclone.args` option to
enable usage of the trash folder. This ensured that deleted files in Google
Drive were not kept indefinitely in the trash folder. However, since Google
Drive's trash retention policy changed to deleting trashed files after 30 days,
this is no longer needed.
Restic now leaves it up to rclone and its configuration to use or not use the
trash folder when deleting files. The default is to use the trash folder, as
of rclone 1.53.2. To re-enable the restic 0.11 behavior, set the
`RCLONE_DRIVE_USE_TRASH` environment variable or change the rclone
configuration. See the rclone documentation for more details.
https://github.com/restic/restic/issues/3095
https://github.com/restic/restic/pull/3102

View file

@ -1,10 +0,0 @@
Bugfix: Do not require gs bucket permissions when running `init`
Restic used to require bucket level permissions for the `gs` backend
in order to initialize a restic repository.
It now allows a `gs` service account to initialize a repository if the
bucket does exist and the service account has permissions to write/read
to that bucket.
https://github.com/restic/restic/issues/3100

View file

@ -1,9 +0,0 @@
Bugfix: Correctly detect output redirection for `backup` command on Windows
On Windows, since restic 0.10.0 the `backup` command did not properly detect
when the output was redirected to a file. This caused restic to output
terminal control characters. This has been fixed by correcting the terminal
detection.
https://github.com/restic/restic/issues/3111
https://github.com/restic/restic/pull/3150

View file

@ -1,11 +0,0 @@
Enhancement: Support additional environment variables for Swift authentication
The `swift` backend now supports the following additional environment variables
for passing authentication details to restic:
`OS_USER_ID`, `OS_USER_DOMAIN_ID`, `OS_PROJECT_DOMAIN_ID` and `OS_TRUST_ID`
Depending on the `openrc` configuration file these might be required when the
user and project domains differ from one another.
https://github.com/restic/restic/issues/3147
https://github.com/restic/restic/pull/3158

View file

@ -1,9 +0,0 @@
Bugfix: Don't create invalid snapshots when `backup` is interrupted
When canceling a backup run at a certain moment it was possible that
restic created a snapshot with an invalid "null" tree. This caused
`check` and other operations to fail. The `backup` command now properly
handles interruptions and never saves a snapshot when interrupted.
https://github.com/restic/restic/issues/3151
https://github.com/restic/restic/pull/3164

View file

@ -1,9 +0,0 @@
Bugfix: Improve error handling in the `restore` command
The `restore` command used to not print errors while downloading file contents
from the repository. It also incorrectly exited with a zero error code even
when there were errors during the restore process. This has all been fixed and
`restore` now returns with a non-zero exit code when there's an error.
https://github.com/restic/restic/issues/3166
https://github.com/restic/restic/pull/3207

View file

@ -1,8 +0,0 @@
Enhancement: Add release binaries for MIPS architectures
We've added a few new architectures for Linux to the release binaries: `mips`,
`mipsle`, `mips64`, and `mip64le`. MIPS is mostly used for low-end embedded
systems.
https://github.com/restic/restic/issues/3191
https://github.com/restic/restic/pull/3208

View file

@ -1,11 +0,0 @@
Bugfix: Correct statistics for overlapping backup sources
A user reported that restic's statistics and progress information during backup
was not correctly calculated when the backup sources (files/dirs to save)
overlap. For example, consider a directory `foo` which contains (among others)
a file `foo/bar`. When `restic backup foo foo/bar` was run, restic counted the
size of the file `foo/bar` twice, so the completeness percentage as well as the
number of files was wrong. This is now corrected.
https://github.com/restic/restic/issues/3232
https://github.com/restic/restic/pull/3243

View file

@ -1,12 +0,0 @@
Enhancement: Back up mountpoints as empty directories
When the `--one-file-system` option is specified to `restic backup`, it
ignores all file systems mounted below one of the target directories. This
means that when a snapshot is restored, users needed to manually recreate
the mountpoint directories.
Restic now backs up mountpoints as empty directories and therefore implements
the same approach as `tar`.
https://github.com/restic/restic/issues/909
https://github.com/restic/restic/pull/3119

View file

@ -1,6 +0,0 @@
Enhancement: Add several more error checks
We've added a lot more error checks in places where errors were previously
ignored (as hinted by the static analysis program `errcheck` via `golangci-lint`).
https://github.com/restic/restic/pull/3250

View file

@ -1,29 +0,0 @@
Enhancement: Improve `prune` performance and make it more customizable
The `prune` command is now much faster. This is especially the case for remote
repositories or repositories with not much data to remove. Also the memory
usage of the `prune` command is now reduced.
Restic used to rebuild the index from scratch after pruning. This could lead
to missing packs in the index in some cases for eventually consistent backends
such as e.g. AWS S3. This behavior is now changed and the index rebuilding
uses the information already known by `prune`.
By default, the `prune` command no longer removes all unused data. This
behavior can be fine-tuned by new options, like the acceptable amount of
unused space or the maximum size of data to reorganize. For more details,
please see https://restic.readthedocs.io/en/stable/060_forget.html .
Moreover, `prune` now accepts the `--dry-run` option and also running
`forget --dry-run --prune` will show what `prune` would do.
This enhancement also fixes several open issues, e.g.:
- https://github.com/restic/restic/issues/1140
- https://github.com/restic/restic/issues/1599
- https://github.com/restic/restic/issues/1985
- https://github.com/restic/restic/issues/2112
- https://github.com/restic/restic/issues/2227
- https://github.com/restic/restic/issues/2305
https://github.com/restic/restic/pull/2718
https://github.com/restic/restic/pull/2842

View file

@ -1,27 +0,0 @@
Enhancement: Add option to let `backup` trust mtime without checking ctime
The `backup` command used to require that both `ctime` and `mtime` of a file
matched with a previously backed up version to determine that the file was
unchanged. In other words, if either `ctime` or `mtime` of the file had
changed, it would be considered changed and restic would read the file's
content again to back up the relevant (changed) parts of it.
The new option `--ignore-ctime` makes restic look at `mtime` only, such that
`ctime` changes for a file does not cause restic to read the file's contents
again.
The check for both `ctime` and `mtime` was introduced in restic 0.9.6 to make
backups more reliable in the face of programs that reset `mtime` (some Unix
archivers do that), but it turned out to often be expensive because it made
restic read file contents even if only the metadata (owner, permissions) of
a file had changed. The new `--ignore-ctime` option lets the user restore the
0.9.5 behavior when needed. The existing `--ignore-inode` option already
turned off this behavior, but also removed a different check.
Please note that changes in files' metadata are still recorded, regardless of
the command line options provided to the backup command.
https://github.com/restic/restic/issues/2495
https://github.com/restic/restic/issues/2558
https://github.com/restic/restic/issues/2819
https://github.com/restic/restic/pull/2823

View file

@ -1,8 +0,0 @@
Enhancement: Speed up the repacking step of the `prune` command
The repack step of the `prune` command, which moves still used file parts into
new pack files such that the old ones can be garbage collected later on, now
processes multiple pack files in parallel. This is especially beneficial for
high latency backends or when using a fast network connection.
https://github.com/restic/restic/pull/2941

View file

@ -1,11 +0,0 @@
Enhancement: Speed up the `rebuild-index` command
We've optimized the `rebuild-index` command. Now, existing index entries are used
to minimize the number of pack files that must be read. This speeds up the index
rebuild a lot.
Additionally, the option `--read-all-packs` has been added, implementing the
previous behavior.
https://github.com/restic/restic/issue/2547
https://github.com/restic/restic/pull/3006

View file

@ -1,13 +0,0 @@
Bugfix: Fix sporadic stream reset between rclone and restic
Sometimes when using restic with the `rclone` backend, an error message
similar to the following would be printed:
Didn't finish writing GET request (wrote 0/xxx): http2: stream closed
It was found that this was caused by restic closing the connection to rclone
to soon when downloading data. A workaround has been added which waits for
the end of the download before closing the connection.
https://github.com/restic/restic/pull/3014
https://github.com/rclone/rclone/issues/2598

View file

@ -1,23 +0,0 @@
Enhancement: Add more checks for index and pack files in the `check` command
The `check` command run with the `--read-data` or `--read-data-subset` options
used to only verify only the pack file content - it did not check if the blobs
within the pack are correctly contained in the index.
A check for the latter is now in place, which can print the following error:
Blob ID is not contained in index or position is incorrect
Another test is also added, which compares pack file sizes computed from the
index and the pack header with the actual file size. This test is able to
detect truncated pack files.
If the index is not correct, it can be rebuilt by using the `rebuild-index`
command.
Having added these tests, `restic check` is now able to detect non-existing
blobs which are wrongly referenced in the index. This situation could have
lead to missing data.
https://github.com/restic/restic/pull/3048
https://github.com/restic/restic/pull/3082

View file

@ -1,8 +0,0 @@
Enhancement: Make the `dump` command support `zip` format
Previously, restic could dump the contents of a whole folder structure only
in the `tar` format. The `dump` command now has a new flag to change output
format to `zip`. Just pass `--archive zip` as an option to `restic dump`.
https://github.com/restic/restic/pull/2433
https://github.com/restic/restic/pull/3081

View file

@ -1,6 +0,0 @@
Enhancement: Reduce memory usage of `check` command
The `check` command now requires less memory if it is run without the
`--check-unused` option.
https://github.com/restic/restic/pull/3099

View file

@ -1,10 +0,0 @@
Enhancement: Parallelize scan of snapshot content in `copy` and `prune`
The `copy` and `prune` commands used to traverse the directories of
snapshots one by one to find used data. This snapshot traversal is
now parallelized which can speed up this step several times.
In addition the `check` command now reports how many snapshots have
already been processed.
https://github.com/restic/restic/pull/3106

View file

@ -1,13 +0,0 @@
Enhancement: Parallelize reading of locks and snapshots
Restic used to read snapshots sequentially. For repositories containing
many snapshots this slowed down commands which have to read all snapshots.
Now the reading of snapshots is parallelized. This speeds up for example
`prune`, `backup` and other commands that search for snapshots with certain
properties or which have to find the `latest` snapshot.
The speed up also applies to locks stored in the backup repository.
https://github.com/restic/restic/pull/3130
https://github.com/restic/restic/pull/3174

View file

@ -1,8 +0,0 @@
Bugfix: Do not hang until foregrounded when completed in background
On Linux, when running in the background restic failed to stop the terminal
output of the `backup` command after it had completed. This caused restic to
hang until moved to the foreground. This has now been fixed.
https://github.com/restic/restic/pull/3152
https://forum.restic.net/t/restic-alpine-container-cron-hangs-epoll-pwait/3334

View file

@ -1,7 +0,0 @@
Bugfix: Improve error handling in `gs` backend
The `gs` backend did not notice when the last step of completing a
file upload failed. Under rare circumstances, this could cause
missing files in the backup repository. This has now been fixed.
https://github.com/restic/restic/pull/3249

View file

@ -1,8 +0,0 @@
Enhancement: Enable HTTP/2 for backend connections
Go's HTTP library usually automatically chooses between HTTP/1.x and HTTP/2
depending on what the server supports. But for compatibility this mechanism
is disabled if DialContext is used (which is the case for restic). This change
allows restic's HTTP client to negotiate HTTP/2 if supported by the server.
https://github.com/restic/restic/pull/3254

View file

@ -1,13 +0,0 @@
Bugfix: Improve error handling for rclone and REST backend over HTTP2
When retrieving data from the rclone / REST backend while also using HTTP2
restic did not detect when no data was returned at all. This could cause
for example the `check` command to report the following error:
Pack ID does not match, want [...], got e3b0c442
This has been fixed by correctly detecting and retrying the incomplete download.
https://github.com/restic/restic/issues/2742
https://github.com/restic/restic/pull/3453
https://forum.restic.net/t/http2-stream-closed-connection-reset-context-canceled/3743/10

View file

@ -1,6 +0,0 @@
Enhancement: Add release binaries for s390x architecture on Linux
We've added release binaries for Linux using the s390x architecture.
https://github.com/restic/restic/issues/2780
https://github.com/restic/restic/pull/3452

View file

@ -1,11 +0,0 @@
Bugfix: Fix terminal output redirection for PowerShell
When redirecting the output of restic using PowerShell on Windows, the
output contained terminal escape characters. This has been fixed by
properly detecting the terminal type.
In addition, the mintty terminal now shows progress output for the backup
command.
https://github.com/restic/restic/issues/3111
https://github.com/restic/restic/pull/3325

View file

@ -1,9 +0,0 @@
Bugfix: Treat an empty password as a fatal error for repository init
When attempting to initialize a new repository, if an empty password was
supplied, the repository would be created but the init command would return
an error with a stack trace. Now, if an empty password is provided, it is
treated as a fatal error, and no repository is created.
https://github.com/restic/restic/issues/3214
https://github.com/restic/restic/pull/3283

View file

@ -1,8 +0,0 @@
Change: Empty files now have size of 0 in `ls --json` output
The `ls --json` command used to omit the sizes of empty files in its
output. It now reports a size of zero explicitly for regular files,
while omitting the size field for all other types.
https://github.com/restic/restic/issues/3247
https://github.com/restic/restic/pull/3257

View file

@ -1,11 +0,0 @@
Bugfix: `copy` failed to copy snapshots in rare cases
The `copy` command could in rare cases fail with the error message `SaveTree(...)
returned unexpected id ...`. This has been fixed.
On Linux/BSDs, the error could be caused by backing up symlinks with non-UTF-8
target paths. Note that, due to limitations in the repository format, these are
not stored properly and should be avoided if possible.
https://github.com/restic/restic/issues/3267
https://github.com/restic/restic/pull/3310

View file

@ -1,11 +0,0 @@
Bugfix: `backup --quiet` no longer prints status information
A regression in the latest restic version caused the output of `backup --quiet`
to contain large amounts of backup progress information when run using an
interactive terminal. This is fixed now.
A workaround for this bug is to run restic as follows:
`restic backup --quiet [..] | cat -`.
https://github.com/restic/restic/issues/3184
https://github.com/restic/restic/pull/3186

View file

@ -1,14 +0,0 @@
Enhancement: Add `--repository-file2` option to `init` and `copy` command
The `init` and `copy` command can now be used with the `--repository-file2`
option or the `$RESTIC_REPOSITORY_FILE2` environment variable.
These to options are in addition to the `--repo2` flag and allow you to read
the destination repository from a file.
Using both `--repository-file` and `--repo2` options resulted in an error for
the `copy` or `init` command. The handling of this combination of options has
been fixed. A workaround for this issue is to only use `--repo` or `-r` and
`--repo2` for `init` or `copy`.
https://github.com/restic/restic/issues/3293
https://github.com/restic/restic/pull/3294

View file

@ -1,7 +0,0 @@
Bugfix: Fix crash of `check --read-data-subset=x%` run for an empty repository
The command `restic check --read-data-subset=x%` crashed when run for an empty
repository. This has been fixed.
https://github.com/restic/restic/issues/3296
https://github.com/restic/restic/pull/3309

View file

@ -1,8 +0,0 @@
Bugfix: Fix `fdopendir: not a directory` error for local backend
The `check`, `list packs`, `prune` and `rebuild-index` commands failed
for the local backend when the `data` folder in the repository contained
files. This has been fixed.
https://github.com/restic/restic/issues/3302
https://github.com/restic/restic/pull/3308

View file

@ -1,5 +0,0 @@
Enhancement: Add auto-completion support for fish
The `generate` command now supports fish auto completion.
https://github.com/restic/restic/pull/3312

View file

@ -1,8 +0,0 @@
Bugfix: Print `created new cache` message only on a terminal
The message `created new cache` was printed even when the output wasn't a
terminal. That broke piping `restic dump` output to tar or zip if cache
directory didn't exist. The message is now only printed on a terminal.
https://github.com/restic/restic/issues/3334
https://github.com/restic/restic/pull/3343

View file

@ -1,8 +0,0 @@
Enhancement: SFTP backend now checks for disk space
Backing up over SFTP previously spewed multiple generic "failure" messages
when the remote disk was full. It now checks for disk space before writing
a file and fails immediately with a "no space left on device" message.
https://github.com/restic/restic/issues/3336
https://github.com/restic/restic/pull/3345

View file

@ -1,6 +0,0 @@
Enhancement: Add release binaries for Apple Silicon
We've added release binaries for macOS on Apple Silicon (M1).
https://github.com/restic/restic/issues/3377
https://github.com/restic/restic/pull/3394

View file

@ -1,7 +0,0 @@
Bugfix: Fix crash of `backup --exclude='**'`
The exclude filter `**`, which excludes all files, caused restic to crash. This
has been corrected.
https://github.com/restic/restic/issues/3380
https://github.com/restic/restic/pull/3393

View file

@ -1,20 +0,0 @@
Enhancement: Add `--keep-within-hourly` option to restic forget
The `forget` command allowed keeping a given number of hourly
backups or to keep all backups within a given interval, but it
was not possible to specify keeping hourly backups within a given
interval.
The new `--keep-within-hourly` option now offers this functionality.
Similar options for daily/weekly/monthly/yearly are also implemented,
the new options are:
--keep-within-hourly <1y2m3d4h>
--keep-within-daily <1y2m3d4h>
--keep-within-weekly <1y2m3d4h>
--keep-within-monthly <1y2m3d4h>
--keep-within-yearly <1y2m3d4h>
https://github.com/restic/restic/issues/3414
https://github.com/restic/restic/pull/3416
https://forum.restic.net/t/forget-policy/4014/11

View file

@ -1,9 +0,0 @@
Enhancement: Support filtering and specifying untagged snapshots
It was previously not possible to specify an empty tag with the `--tag` and
`--keep-tag` options. This has now been fixed, such that `--tag ''` and
`--keep-tag ''` now matches snapshots without tags. This allows e.g. the
`snapshots` and `forget` commands to only operate on untagged snapshots.
https://github.com/restic/restic/issues/3456
https://github.com/restic/restic/pull/3457

View file

@ -1,9 +0,0 @@
Enhancement: Allow specifying limit of `snapshots` list
The `--last` option allowed limiting the output of the `snapshots`
command to the latest snapshot for each host. The new `--latest n`
option allows limiting the output to the latest `n` snapshots.
This change deprecates the option `--last` in favour of `--latest 1`.
https://github.com/restic/restic/pull/3167

View file

@ -1,6 +0,0 @@
Bugfix: Fix possibly missing backup summary of JSON output in case of error
When using `--json` output it happened from time to time that the summary
output was missing in case an error occurred. This has been fixed.
https://github.com/restic/restic/pull/3305

View file

@ -1,7 +0,0 @@
Enhancement: Optimize read performance of mount command
Reading large files in a mounted repository may be up to five times faster.
This improvement primarily applies to repositories stored at a backend that can
be accessed with low latency, like e.g. the local backend.
https://github.com/restic/restic/pull/3426

View file

@ -1,13 +0,0 @@
Enhancement: `find --pack` fallback to index if data file is missing
When investigating a repository with missing data files, it might be useful to
determine affected snapshots before running `rebuild-index`. Previously,
`find --pack pack-id` returned no data as it required accessing the data file.
Now, if the necessary data is still available in the repository index, it gets
retrieved from there.
The command now also supports looking up multiple pack files in a single `find`
run.
https://github.com/restic/restic/pull/3427
https://forum.restic.net/t/missing-packs-not-found/2600

View file

@ -1,8 +0,0 @@
Bugfix: Correctly handle download errors during `restore`
Due to a regression in restic 0.12.0, the `restore` command in some cases did
not retry download errors and only printed a warning. This has been fixed by
retrying incomplete data downloads.
https://github.com/restic/restic/issues/3439
https://github.com/restic/restic/pull/3449

View file

@ -1,10 +0,0 @@
Bugfix: Never lock repository for `list locks`
The `list locks` command previously locked to the repository by default. This
had the problem that it wouldn't work for an exclusively locked repository and
that the command would also display its own lock file which can be confusing.
Now, the `list locks` command never locks the repository.
https://github.com/restic/restic/issues/1106
https://github.com/restic/restic/pull/3665

View file

@ -1,16 +0,0 @@
Enhancement: Add `--dry-run`/`-n` option to `backup` command
Testing exclude filters and other configuration options was error prone as
wrong filters could cause files to be uploaded unintentionally. It was also
not possible to estimate beforehand how much data would be uploaded.
The `backup` command now has a `--dry-run`/`-n` option, which performs all the
normal steps of a backup without actually writing anything to the repository.
Passing -vv will log information about files that would be added, allowing for
verification of source and exclusion options before running the real backup.
https://github.com/restic/restic/issues/1542
https://github.com/restic/restic/pull/2308
https://github.com/restic/restic/pull/3210
https://github.com/restic/restic/pull/3300

View file

@ -1,15 +0,0 @@
Enhancement: Add upload checksum for Azure, GS, S3 and Swift backends
Previously only the B2 and partially the Swift backends verified the integrity
of uploaded (encrypted) files. The verification works by informing the backend
about the expected hash of the uploaded file. The backend then verifies the
upload and thereby rules out any data corruption during upload.
We have now added upload checksums for the Azure, GS, S3 and Swift backends,
which besides integrity checking for uploads also means that restic can now be
used to store backups in S3 buckets which have Object Lock enabled.
https://github.com/restic/restic/issues/2202
https://github.com/restic/restic/issues/2700
https://github.com/restic/restic/issues/3023
https://github.com/restic/restic/pull/3246

View file

@ -1,29 +0,0 @@
Enhancement: Support negative include/exclude patterns
If a pattern starts with an exclamation mark and it matches a file that was
previously matched by a regular pattern, the match is cancelled. Notably,
this can be used with `--exclude-file` to cancel the exclusion of some files.
It works similarly to `.gitignore`, with the same limitation; Once a directory
is excluded, it is not possible to include files inside the directory.
Example of use as an exclude pattern for the `backup` command:
$HOME/**/*
!$HOME/Documents
!$HOME/code
!$HOME/.emacs.d
!$HOME/games
# [...]
node_modules
*~
*.o
*.lo
*.pyc
# [...]
$HOME/code/linux/*
!$HOME/code/linux/.git
# [...]
https://github.com/restic/restic/issues/233
https://github.com/restic/restic/pull/2311

View file

@ -1,13 +0,0 @@
Bugfix: Make cache crash-resistant and usable by multiple concurrent processes
The restic cache directory (`RESTIC_CACHE_DIR`) could end up in a broken state
in the event of restic (or the OS) crashing. This is now less likely to occur
as files are downloaded to a temporary location before being moved to their
proper location.
This also allows multiple concurrent restic processes to operate on a single
repository without conflicts. Previously, concurrent operations could cause
segfaults because the processes saw each other's partially downloaded files.
https://github.com/restic/restic/issues/2345
https://github.com/restic/restic/pull/2838

View file

@ -1,10 +0,0 @@
Enhancement: Add warning for S3 if partial credentials are provided
Previously restic did not notify about incomplete credentials when using the
S3 backend, instead just reporting access denied.
Restic now checks that both the AWS key ID and secret environment variables are
set before connecting to the remote server, and reports an error if not.
https://github.com/restic/restic/issues/2388
https://github.com/restic/restic/pull/3532

View file

@ -1,11 +0,0 @@
Bugfix: Improve error handling of repository locking
Previously, when the lock refresh failed to delete the old lock file, it forgot
about the newly created one. Instead it continued trying to delete the old
(usually no longer existing) lock file and thus over time lots of lock files
accumulated. This has now been fixed.
https://github.com/restic/restic/issues/2452
https://github.com/restic/restic/issues/2473
https://github.com/restic/restic/issues/2562
https://github.com/restic/restic/pull/3512

View file

@ -1,8 +0,0 @@
Enhancement: Support JSON output and quiet mode for the `diff` command
The `diff` command now supports outputting machine-readable output in JSON
format. To enable this, pass the `--json` option to the command. To only print
the summary and suppress detailed output, pass the `--quiet` option.
https://github.com/restic/restic/issues/2508
https://github.com/restic/restic/pull/3592

View file

@ -1,8 +0,0 @@
Enhancement: Add flag to disable TLS verification for self-signed certificates
There is now an `--insecure-tls` global option in restic, which disables TLS
verification for self-signed certificates in order to support some development
workflows.
https://github.com/restic/restic/issues/2656
https://github.com/restic/restic/pull/2657

View file

@ -1,8 +0,0 @@
Bugfix: Don't print progress for `backup --json --quiet`
Unlike the text output, the `--json` output format still printed progress
information even in `--quiet` mode. This has now been fixed by always
disabling the progress output in quiet mode.
https://github.com/restic/restic/issues/2738
https://github.com/restic/restic/pull/3264

View file

@ -1,9 +0,0 @@
Enhancement: Atomic uploads for the SFTP backend
The SFTP backend did not upload files atomically. An interrupted upload could
leave an incomplete file behind which could prevent restic from accessing the
repository. This has now been fixed and uploads in the SFTP backend are done
atomically.
https://github.com/restic/restic/issues/3003
https://github.com/restic/restic/pull/3524

View file

@ -1,6 +0,0 @@
Enhancement: Add xattr (extended attributes) support for Solaris
Restic now supports xattr for the Solaris operating system.
https://github.com/restic/restic/issues/3127
https://github.com/restic/restic/pull/3628

View file

@ -1,8 +0,0 @@
Bugfix: Make `check` command honor `RESTIC_CACHE_DIR` environment variable
Previously, the `check` command didn't honor the `RESTIC_CACHE_DIR` environment
variable, which caused problems in certain system/usage configurations. This
has now been fixed.
https://github.com/restic/restic/issues/3382
https://github.com/restic/restic/pull/3474

View file

@ -1,10 +0,0 @@
Enhancement: Skip lock creation on `forget` if `--no-lock` and `--dry-run`
Restic used to silently ignore the `--no-lock` option of the `forget` command.
It now skips creation of lock file in case both `--dry-run` and `--no-lock`
are specified. If `--no-lock` option is specified without `--dry-run`, restic
prints a warning message to stderr.
https://github.com/restic/restic/issues/3464
https://github.com/restic/restic/pull/3623

View file

@ -1,8 +0,0 @@
Enhancement: Support random subset by size in `check --read-data-subset`
The `--read-data-subset` option of the `check` command now supports a third way
of specifying the subset to check, namely `nS` where `n` is a size in bytes with
suffix `S` as k/K, m/M, g/G or t/T.
https://github.com/restic/restic/issues/3490
https://github.com/restic/restic/pull/3548

View file

@ -1,8 +0,0 @@
Bugfix: Make `copy` command honor `--no-lock` for source repository
The `copy` command previously did not respect the `--no-lock` option for the
source repository, causing failures with read-only storage backends. This has
now been fixed such that the option is now respected.
https://github.com/restic/restic/issues/3518
https://github.com/restic/restic/pull/3589

View file

@ -1,9 +0,0 @@
Enhancement: Improve handling of temporary B2 delete errors
Deleting files on B2 could sometimes fail temporarily, which required restic to
retry the delete operation. In some cases the file was deleted nevertheless,
causing the retries and ultimately the restic command to fail. This has now been
fixed.
https://github.com/restic/restic/issues/3541
https://github.com/restic/restic/pull/3544

View file

@ -1,8 +0,0 @@
Enhancement: Add file mode in symbolic notation to `ls --json`
The `ls --json` command now provides the file mode in symbolic notation (using
the `permissions` key), aligned with `find --json`.
https://github.com/restic/restic/issues/3542
https://github.com/restic/restic/pull/3573
https://forum.restic.net/t/restic-ls-understanding-file-mode-with-json/4371

View file

@ -1,10 +0,0 @@
Bugfix: Fix hang with Backblaze B2 on SSL certificate authority error
Previously, if a request failed with an SSL unknown certificate authority
error, the B2 backend retried indefinitely and restic would appear to hang.
This has now been fixed and restic instead fails with an error message.
https://github.com/restic/restic/issues/3556
https://github.com/restic/restic/issues/2355
https://github.com/restic/restic/pull/3571

View file

@ -1,15 +0,0 @@
Bugfix: Fix rclone backend prematurely exiting when receiving SIGINT on Windows
Previously, pressing Ctrl+C in a Windows console where restic was running with
rclone as the backend would cause rclone to exit prematurely due to getting a
`SIGINT` signal at the same time as restic. Restic would then wait for a long
time for time with "unexpected EOF" and "rclone stdio connection already closed"
errors.
This has now been fixed by restic starting the rclone process detached from the
console restic runs in (similar to starting processes in a new process group on
Linux), which enables restic to gracefully clean up rclone (which now never gets
the `SIGINT`).
https://github.com/restic/restic/issues/3601
https://github.com/restic/restic/pull/3602

View file

@ -1,14 +0,0 @@
Change: Ignore parent snapshot for `backup --stdin`
Restic uses a parent snapshot to speed up directory scanning when performing
backups, but this only wasted time and memory when the backup source is stdin
(using the `--stdin` option of the `backup` command), since no directory scanning
is performed in this case.
Snapshots made with `backup --stdin` no longer have a parent snapshot, which allows
restic to skip some startup operations and saves a bit of resources.
The `--parent` option is still available for `backup --stdin`, but is now ignored.
https://github.com/restic/restic/issues/3641
https://github.com/restic/restic/pull/3645

View file

@ -1,8 +0,0 @@
Bugfix: The `mount` command now reports symlinks sizes
Symlinks used to have size zero in restic mountpoints, confusing some
third-party tools. They now have a size equal to the byte length of their
target path, as required by POSIX.
https://github.com/restic/restic/issues/3667
https://github.com/restic/restic/pull/3668

Some files were not shown because too many files have changed in this diff Show more