Compare commits

...
Sign in to create a new pull request.

33 commits

Author SHA1 Message Date
Nick Craig-Wood
17c21aed5a Version v1.49.5 2019-10-05 12:07:20 +01:00
Nick Craig-Wood
75cd733d3e build: fix macOS build after brew changes 2019-10-05 12:04:11 +01:00
Nick Craig-Wood
b5ea6af6e4 build: revert back to go1.12 for the v1.49.x builds
The go1.13 build has had various problems reported so revert back to
go1.12 for the stable branch.

See: https://forum.rclone.org/t/1-49-4-plex-internal-errors-on-google-drive/12108
Fixes #3578
2019-10-02 13:53:52 +01:00
Nick Craig-Wood
d8729441db build: use the release builds not master of nfpm and github-release
Fixes #3580
2019-10-02 13:33:27 +01:00
Nick Craig-Wood
5ac39c2176 bin/get-github-release: support tar.bz2 files 2019-10-02 13:33:22 +01:00
Nick Craig-Wood
8aae04208b Version v1.49.4 2019-09-29 17:33:45 +01:00
Richard Patel
d9bdd0575e cmd/rcd: Address ZipSlip vulnerability
Don't create files outside of target
directory while unzipping.

Fixes #3529 reported by Nico Waisman at Semmle Security Team
2019-09-29 11:15:14 +01:00
Nick Craig-Wood
b3cafe8f06 s3: fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier
- Read the storage class for each object
- Implement SetTier/GetTier
- Check the storage class on the **object** before using SetModTime

This updates the fix in 1a2fb52 so that SetModTime works when you are
using objects which have been migrated to GLACIER but you aren't using
GLACIER as a storage class.

Fixes #3522
2019-09-29 10:54:37 +01:00
Nick Craig-Wood
a7a4666ddd oauthutil: fix security problem when running with two users on the same machine
Before this change two users could run `rclone config` for the same
backend on the same machine at the same time.

User A would get as far as starting the web server.  User B would then
fail to start the webserver, but it would open the browser on the
/auth URL which would redirect the user to the login.  This would then
cause user B to authenticate to user A's rclone.

This changes fixes the problem in two ways.

Firstly it passes the state to the /auth call before redirecting and
checks it there, erroring with a 403 error if it doesn't match.  This
would have fixed the problem on its own.

Secondly it delays the opening of the web browser until after the auth
webserver has started which prevents the user entering the credentials
if another auth server is running.

Fixes #3573
2019-09-29 10:53:24 +01:00
Nick Craig-Wood
54d409a7dd ftp: fix listing of an empty root returning: error dir not found
Before this change if rclone listed an empty root directory then it
would return an error dir not found.

After this change we assume the root directory exists and don't
attempt to check it which was failing before.

See: https://forum.rclone.org/t/ftp-empty-directory-yields-directory-not-found-error/12069/
2019-09-29 10:53:11 +01:00
Nick Craig-Wood
9054542be5 build: make VERSION file be master of the last release - fixes #3570
Prior to this beta releases would appear to be older than the point
release, eg v1.49.0-096-gc41812fc which was released after v1.49.3 and
contains all the patches from v1.49.3.
2019-09-29 10:52:47 +01:00
Nick Craig-Wood
4a8a8578a5 build: replace Circle CI build and make GitHub actions the default CI 2019-09-29 10:51:08 +01:00
Nick Craig-Wood
04da57fc68 build: remove Appveyor, Circle CI, Travis and Pkgr builds 2019-09-29 10:50:48 +01:00
Nick Craig-Wood
3aeb6a5a4c build: remove azure pipelines build 2019-09-29 10:49:33 +01:00
Nick Craig-Wood
c5d2da9a77 build: build rclone with github actions 2019-09-29 10:48:31 +01:00
Nick Craig-Wood
c89261bd99 accounting: fix file handle leak on errors - fixes #3547
In 53a1a0e3ef we introduced a problem where if there was an
error on the file being transferred then the file was re-opened and
the old one wasn't closed.

This was partially fixed in bfbddab46b however this didn't
address the case of the old file being closed.

This is now fixed by
- marking the file as open again in UpdateReader
- moving the stopping the accounting machinery to a new method Done
2019-09-19 16:21:20 +01:00
Nick Craig-Wood
1bdab29eab Version v1.49.3 2019-09-15 16:42:10 +01:00
Nick Craig-Wood
f77027e6b7 fs/accounting: Fix "file already closed" on transfer retries
This was caused by the recent reworking of the accounting interface.
The Transfer object was recycling the Accounting object without
resetting the stream.

See: https://forum.rclone.org/t/error-file-already-closed/11469/
See: https://forum.rclone.org/t/rclone-b2-sync-post-error-method-not-supported/11718/
2019-09-13 18:37:01 +01:00
Aleksandar Jankovic
f73d0eb920 accounting: fix total duration calculation
Fixes: #3498
2019-09-12 12:33:57 +01:00
Nick Craig-Wood
f1a9d821e4 Version v1.49.2 2019-09-08 16:48:54 +01:00
Nick Craig-Wood
5fe78936d5 test_all: write index.json and add branch, commit and Go version to report 2019-09-08 11:38:18 +01:00
Nick Craig-Wood
4f3eee8d65 build: make sure we add version info to test_all build 2019-09-08 11:38:11 +01:00
Nick Craig-Wood
f2c05bc239 operations: fix -u/--update with google photos / files of unknown size
Before this change if -u/--update was in effect we compared the size
of the files to see if the transfer should go ahead.  This was
comparing -1 with an actual size so the transfer always proceeded.

After this change we use the existing `sizeDiffers` function which
does the correct comparison with -1 for files of unknown length.

See: https://forum.rclone.org/t/sync-with-google-photos-to-local-drive-will-result-in-recoping/11605
2019-09-06 10:11:59 +01:00
Nick Craig-Wood
b463032901 accounting: fix locking in Transfer to avoid deadlock with --progress
Before this change, using -P occasionally deadlocked on the Transfer
mutex when Transfer.Done() was called with a non nil error and the
StatsInfo mutex since they mutually call each other.

This was fixed by making sure that the Transfer mutex is always
released before calling any StatsInfo methods.

This improves on: 6f87267b34

Fixes #3505
2019-09-06 10:10:53 +01:00
Nick Craig-Wood
358decb933 rc: fix docs for config/create /update /password 2019-09-03 08:33:56 +01:00
Nick Craig-Wood
cefa2df3b2 docs: add info on how to build and use the docker images 2019-09-02 14:31:19 +01:00
Alfonso Montero
52efb7e6d0 Add Docker workflow support #3460
* Use a multi-stage build to reduce final image size.
* Run 'quicktest' make target before building.
* Built binary won't run on Alpine unless statically linked.
2019-09-02 14:31:10 +01:00
Nick Craig-Wood
01fa6835c7 gcs: fix need for elevated permissions on SetModTime - fixes #3493
Before this change we used PATCH on the object to update the metadata.

Apparently this requires the "full_control" scope which Google were
unhappy with in their oauth review.

This changes it to update the metadata by copying the object ontop of
itself (which is the way s3 works).  This can be done with normal
permissions.
2019-09-02 12:04:45 +01:00
Cnly
8adf22e294 docs: fix template argument for mktemp in install.sh 2019-09-02 12:04:33 +01:00
Nick Craig-Wood
45f7c687e2 Version v1.49.1 2019-08-28 17:51:23 +01:00
Nick Craig-Wood
a05dd6fc27 config: Fix generated passwords being stored as empty password - Fixes #3492 2019-08-28 14:24:18 +01:00
Nick Craig-Wood
642cb03121 googlephotos,onedrive: fix crash on error response - fixes #3491
This fixes a crash on the google photos backend when an error is
returned from the rest.Call function.

This turned out to be a mis-understanding of the rest docs so
- improved rest.Call docs
- fixed mis-understanding in google photos backend
- fixed similar mis-understading in onedrive backend
2019-08-28 14:24:08 +01:00
Chaitanya
da4dfdc3ec rcd: Added missing parameter for web-gui info logs. 2019-08-28 14:24:04 +01:00
111 changed files with 1387 additions and 767 deletions

View file

@ -1,49 +0,0 @@
version: "{build}"
os: Windows Server 2012 R2
clone_folder: c:\gopath\src\github.com\rclone\rclone
cache:
- '%LocalAppData%\go-build'
environment:
GOPATH: C:\gopath
CPATH: C:\Program Files (x86)\WinFsp\inc\fuse
ORIGPATH: '%PATH%'
NOCCPATH: C:\MinGW\bin;%GOPATH%\bin;%PATH%
PATHCC64: C:\mingw-w64\x86_64-6.3.0-posix-seh-rt_v5-rev1\mingw64\bin;%NOCCPATH%
PATHCC32: C:\mingw-w64\i686-6.3.0-posix-dwarf-rt_v5-rev1\mingw32\bin;%NOCCPATH%
PATH: '%PATHCC64%'
RCLONE_CONFIG_PASS:
secure: sq9CPBbwaeKJv+yd24U44neORYPQVy6jsjnQptC+5yk=
install:
- choco install winfsp -y
- choco install zip -y
- copy c:\MinGW\bin\mingw32-make.exe c:\MinGW\bin\make.exe
build_script:
- echo %PATH%
- echo %GOPATH%
- go version
- go env
- go install
- go build
- make log_since_last_release > %TEMP%\git-log.txt
- make version > %TEMP%\version
- set /p RCLONE_VERSION=<%TEMP%\version
- set PATH=%PATHCC32%
- go run bin/cross-compile.go -release beta-latest -git-log %TEMP%\git-log.txt -include "^windows/386" -cgo -tags cmount %RCLONE_VERSION%
- set PATH=%PATHCC64%
- go run bin/cross-compile.go -release beta-latest -git-log %TEMP%\git-log.txt -include "^windows/amd64" -cgo -no-clean -tags cmount %RCLONE_VERSION%
test_script:
- make GOTAGS=cmount quicktest
artifacts:
- path: rclone.exe
- path: build/*-v*.zip
deploy_script:
- IF "%APPVEYOR_REPO_NAME%" == "rclone/rclone" IF "%APPVEYOR_PULL_REQUEST_NUMBER%" == "" make appveyor_upload

View file

@ -1,50 +0,0 @@
---
version: 2
jobs:
build:
machine: true
working_directory: ~/.go_workspace/src/github.com/rclone/rclone
steps:
- checkout
- run:
name: Cross-compile rclone
command: |
docker pull rclone/xgo-cgofuse
go get -v github.com/karalabe/xgo
xgo \
--image=rclone/xgo-cgofuse \
--targets=darwin/386,darwin/amd64,linux/386,linux/amd64,windows/386,windows/amd64 \
-tags cmount \
.
xgo \
--targets=android/*,ios/* \
.
- run:
name: Prepare artifacts
command: |
mkdir -p /tmp/rclone.dist
cp -R rclone-* /tmp/rclone.dist
mkdir build
cp -R rclone-* build/
- run:
name: Build rclone
command: |
go version
go build
- run:
name: Upload artifacts
command: |
if [[ $CIRCLE_PULL_REQUEST != "" ]]; then
make circleci_upload
fi
- store_artifacts:
path: /tmp/rclone.dist

243
.github/workflows/build.yml vendored Normal file
View file

@ -0,0 +1,243 @@
---
# Github Actions build for rclone
# -*- compile-command: "yamllint -f parsable build.yml" -*-
name: build
# Trigger the workflow on push or pull request
on:
push:
branches:
- '*'
tags:
- '*'
pull_request:
jobs:
build:
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
job_name: ['linux', 'mac', 'windows_amd64', 'windows_386', 'other_os', 'modules_race', 'go1.10', 'go1.11']
include:
- job_name: linux
os: ubuntu-latest
go: '1.12.x'
modules: 'off'
gotags: cmount
build_flags: '-include "^linux/"'
check: true
quicktest: true
deploy: true
- job_name: mac
os: macOS-latest
go: '1.12.x'
modules: 'off'
gotags: '' # cmount doesn't work on osx travis for some reason
build_flags: '-include "^darwin/amd64" -cgo'
quicktest: true
racequicktest: true
deploy: true
- job_name: windows_amd64
os: windows-latest
go: '1.12.x'
modules: 'off'
gotags: cmount
build_flags: '-include "^windows/amd64" -cgo'
quicktest: true
racequicktest: true
deploy: true
- job_name: windows_386
os: windows-latest
go: '1.12.x'
modules: 'off'
gotags: cmount
goarch: '386'
cgo: '1'
build_flags: '-include "^windows/386" -cgo'
quicktest: true
deploy: true
- job_name: other_os
os: ubuntu-latest
go: '1.12.x'
modules: 'off'
build_flags: '-exclude "^(windows/|darwin/amd64|linux/)"'
compile_all: true
deploy: true
- job_name: modules_race
os: ubuntu-latest
go: '1.12.x'
modules: 'on'
quicktest: true
racequicktest: true
- job_name: go1.10
os: ubuntu-latest
go: '1.10.x'
modules: 'off'
quicktest: true
- job_name: go1.11
os: ubuntu-latest
go: '1.11.x'
modules: 'off'
quicktest: true
name: ${{ matrix.job_name }}
runs-on: ${{ matrix.os }}
steps:
- name: Checkout
uses: actions/checkout@master
with:
path: ./src/github.com/${{ github.repository }}
- name: Install Go
uses: actions/setup-go@v1
with:
go-version: ${{ matrix.go }}
- name: Set environment variables
shell: bash
run: |
echo '::set-env name=GOPATH::${{ runner.workspace }}'
echo '::add-path::${{ runner.workspace }}/bin'
echo '::set-env name=GO111MODULE::${{ matrix.modules }}'
echo '::set-env name=GOTAGS::${{ matrix.gotags }}'
echo '::set-env name=BUILD_FLAGS::${{ matrix.build_flags }}'
if [[ "${{ matrix.goarch }}" != "" ]]; then echo '::set-env name=GOARCH::${{ matrix.goarch }}' ; fi
if [[ "${{ matrix.cgo }}" != "" ]]; then echo '::set-env name=CGO_ENABLED::${{ matrix.cgo }}' ; fi
- name: Install Libraries on Linux
shell: bash
run: |
sudo modprobe fuse
sudo chmod 666 /dev/fuse
sudo chown root:$USER /etc/fuse.conf
sudo apt-get install fuse libfuse-dev rpm pkg-config
if: matrix.os == 'ubuntu-latest'
- name: Install Libraries on macOS
shell: bash
run: |
brew update
brew cask install osxfuse
if: matrix.os == 'macOS-latest'
- name: Install Libraries on Windows
shell: powershell
run: |
$ProgressPreference = 'SilentlyContinue'
choco install -y winfsp zip
Write-Host "::set-env name=CPATH::C:\Program Files\WinFsp\inc\fuse;C:\Program Files (x86)\WinFsp\inc\fuse"
if ($env:GOARCH -eq "386") {
choco install -y mingw --forcex86 --force
Write-Host "::add-path::C:\\ProgramData\\chocolatey\\lib\\mingw\\tools\\install\\mingw32\\bin"
}
# Copy mingw32-make.exe to make.exe so the same command line
# can be used on Windows as on macOS and Linux
$path = (get-command mingw32-make.exe).Path
Copy-Item -Path $path -Destination (Join-Path (Split-Path -Path $path) 'make.exe')
if: matrix.os == 'windows-latest'
- name: Print Go version and environment
shell: bash
run: |
printf "Using go at: $(which go)\n"
printf "Go version: $(go version)\n"
printf "\n\nGo environment:\n\n"
go env
printf "\n\nRclone environment:\n\n"
make vars
printf "\n\nSystem environment:\n\n"
env
- name: Run tests
shell: bash
run: |
make
make quicktest
if: matrix.quicktest
- name: Race test
shell: bash
run: |
make racequicktest
if: matrix.racequicktest
- name: Code quality test
shell: bash
run: |
make build_dep
make check
if: matrix.check
- name: Compile all architectures test
shell: bash
run: |
make
make compile_all
if: matrix.compile_all
- name: Deploy built binaries
shell: bash
run: |
if [[ "${{ matrix.os }}" == "ubuntu-latest" ]]; then make release_dep ; fi
make travis_beta
env:
RCLONE_CONFIG_PASS: ${{ secrets.RCLONE_CONFIG_PASS }}
# working-directory: '$(modulePath)'
if: matrix.deploy && github.head_ref == ''
xgo:
timeout-minutes: 60
name: "xgo cross compile"
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@master
with:
path: ./src/github.com/${{ github.repository }}
- name: Set environment variables
shell: bash
run: |
echo '::set-env name=GOPATH::${{ runner.workspace }}'
echo '::add-path::${{ runner.workspace }}/bin'
- name: Cross-compile rclone
run: |
docker pull billziss/xgo-cgofuse
go get -v github.com/karalabe/xgo
xgo \
-image=billziss/xgo-cgofuse \
-targets=darwin/386,darwin/amd64,linux/386,linux/amd64,windows/386,windows/amd64 \
-tags cmount \
-dest build \
.
xgo \
-image=billziss/xgo-cgofuse \
-targets=android/*,ios/* \
-dest build \
.
- name: Build rclone
run: |
docker pull golang
docker run --rm -v "$PWD":/usr/src/rclone -w /usr/src/rclone golang go build -mod=vendor -v
- name: Upload artifacts
run: |
make circleci_upload
env:
RCLONE_CONFIG_PASS: ${{ secrets.RCLONE_CONFIG_PASS }}

View file

@ -1,2 +0,0 @@
default_dependencies: false
cli: rclone

View file

@ -1,128 +0,0 @@
---
language: go
sudo: required
dist: xenial
os:
- linux
go_import_path: github.com/rclone/rclone
before_install:
- git fetch --unshallow --tags
- |
if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
sudo modprobe fuse
sudo chmod 666 /dev/fuse
sudo chown root:$USER /etc/fuse.conf
fi
if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
brew update
brew tap caskroom/cask
brew cask install osxfuse
fi
if [[ "$TRAVIS_OS_NAME" == "windows" ]]; then
choco install -y winfsp zip make
cd ../.. # fix crlf in git checkout
mv $TRAVIS_REPO_SLUG _old
git config --global core.autocrlf false
git clone _old $TRAVIS_REPO_SLUG
cd $TRAVIS_REPO_SLUG
fi
install:
- make vars
env:
global:
- GOTAGS=cmount
- GOMAXPROCS=8 # workaround for cmd/mount tests locking up - see #3154
- GO111MODULE=off
- GITHUB_USER=ncw
- secure: gU8gCV9R8Kv/Gn0SmCP37edpfIbPoSvsub48GK7qxJdTU628H0KOMiZW/T0gtV5d67XJZ4eKnhJYlxwwxgSgfejO32Rh5GlYEKT/FuVoH0BD72dM1GDFLSrUiUYOdoHvf/BKIFA3dJFT4lk2ASy4Zh7SEoXHG6goBlqUpYx8hVA=
- secure: Uaiveq+/rvQjO03GzvQZV2J6pZfedoFuhdXrLVhhHSeP4ZBca0olw7xaqkabUyP3LkVYXMDSX8EbyeuQT1jfEe5wp5sBdfaDtuYW6heFyjiHIIIbVyBfGXon6db4ETBjOaX/Xt8uktrgNge6qFlj+kpnmpFGxf0jmDLw1zgg7tk=
addons:
apt:
packages:
- fuse
- libfuse-dev
- rpm
- pkg-config
cache:
directories:
- $HOME/.cache/go-build
matrix:
allow_failures:
- go: tip
include:
- go: 1.9.x
script:
- make quicktest
- go: 1.10.x
script:
- make quicktest
- go: 1.11.x
script:
- make quicktest
- go: 1.12.x
name: Linux
env:
- GOTAGS=cmount
- BUILD_FLAGS='-include "^linux/"'
- DEPLOY=true
script:
- make build_dep
- make check
- make quicktest
- go: 1.12.x
name: Go Modules / Race
env:
- GO111MODULE=on
- GOPROXY=https://proxy.golang.org
script:
- make quicktest
- make racequicktest
- go: 1.12.x
name: Other OS
env:
- DEPLOY=true
- BUILD_FLAGS='-exclude "^(windows|darwin|linux)/"'
script:
- make
- make compile_all
- go: 1.12.x
name: macOS
os: osx
env:
- GOTAGS= # cmount doesn't work on osx travis for some reason
- BUILD_FLAGS='-include "^darwin/" -cgo'
- DEPLOY=true
cache:
directories:
- $HOME/Library/Caches/go-build
script:
- make
- make quicktest
- make racequicktest
# - os: windows
# name: Windows
# go: 1.12.x
# env:
# - GOTAGS=cmount
# - CPATH='C:\Program Files (x86)\WinFsp\inc\fuse'
# - BUILD_FLAGS='-include "^windows/amd64" -cgo' # 386 doesn't build yet
# #filter_secrets: false # works around a problem with secrets under windows
# cache:
# directories:
# - ${LocalAppData}/go-build
# script:
# - make
# - make quicktest
# - make racequicktest
- go: tip
script:
- make quicktest
deploy:
provider: script
script: make travis_beta
skip_cleanup: true
on:
repo: rclone/rclone
all_branches: true
condition: $TRAVIS_PULL_REQUEST == false && $DEPLOY == true

21
Dockerfile Normal file
View file

@ -0,0 +1,21 @@
FROM golang AS builder
COPY . /go/src/github.com/rclone/rclone/
WORKDIR /go/src/github.com/rclone/rclone/
RUN make quicktest
RUN \
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
make
RUN ./rclone version
# Begin final image
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /go/src/github.com/rclone/rclone/rclone .
ENTRYPOINT [ "./rclone" ]

102
MANUAL.html generated
View file

@ -17,7 +17,7 @@
<header>
<h1 class="title">rclone(1) User Manual</h1>
<p class="author">Nick Craig-Wood</p>
<p class="date">Aug 26, 2019</p>
<p class="date">Oct 05, 2019</p>
</header>
<h1 id="rclone---rsync-for-cloud-storage">Rclone - rsync for cloud storage</h1>
<p>Rclone is a command line program to sync files and directories to and from:</p>
@ -134,6 +134,20 @@ sudo mv rclone /usr/local/bin/</code></pre>
<pre><code>cd .. &amp;&amp; rm -rf rclone-*-osx-amd64 rclone-current-osx-amd64.zip</code></pre>
<p>Run <code>rclone config</code> to setup. See <a href="https://rclone.org/docs/">rclone config docs</a> for more details.</p>
<pre><code>rclone config</code></pre>
<h2 id="install-with-docker">Install with docker</h2>
<p>The rclone maintains a <a href="https://hub.docker.com/r/rclone/rclone">docker image for rclone</a>. These images are autobuilt by docker hub from the rclone source based on a minimal Alpine linux image.</p>
<p>The <code>:latest</code> tag will always point to the latest stable release. You can use the <code>:beta</code> tag to get the latest build from master. You can also use version tags, eg <code>:1.49.1</code>, <code>:1.49</code> or <code>:1</code>.</p>
<pre><code>$ docker pull rclone/rclone:latest
latest: Pulling from rclone/rclone
Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
...
$ docker run --rm rclone/rclone:latest version
rclone v1.49.1
- os/arch: linux/amd64
- go version: go1.12.9</code></pre>
<p>You will probably want to mount rclones config file directory or file from the host, or configure rclone with environment variables.</p>
<p>Eg to share your local config with the container</p>
<pre><code>docker run -v ~/.config/rclone:/root/.config/rclone rclone/rclone:latest listremotes</code></pre>
<h2 id="install-from-source">Install from source</h2>
<p>Make sure you have at least <a href="https://golang.org/">Go</a> 1.7 installed. <a href="https://golang.org/dl/">Download go</a> if necessary. The latest release is recommended. Then</p>
<pre><code>git clone https://github.com/rclone/rclone.git
@ -3301,12 +3315,16 @@ rclone rc cache/expire remote=/ withData=true</code></pre>
<p>This takes the following parameters</p>
<ul>
<li>name - name of remote</li>
<li>parameters - a map of { “key”: “value” } pairs</li>
<li>type - type of the new remote</li>
</ul>
<p>See the <a href="https://rclone.org/commands/rclone_config_create/">config create command</a> command for more information on the above.</p>
<p>Authentication is required for this call.</p>
<h3 id="configdelete-delete-a-remote-in-the-config-file.-configdelete">config/delete: Delete a remote in the config file. {#config/delete}</h3>
<p>Parameters: - name - name of remote to delete</p>
<p>Parameters:</p>
<ul>
<li>name - name of remote to delete</li>
</ul>
<p>See the <a href="https://rclone.org/commands/rclone_config_delete/">config delete command</a> command for more information on the above.</p>
<p>Authentication is required for this call.</p>
<h3 id="configdump-dumps-the-config-file.-configdump">config/dump: Dumps the config file. {#config/dump}</h3>
@ -3326,6 +3344,7 @@ rclone rc cache/expire remote=/ withData=true</code></pre>
<p>This takes the following parameters</p>
<ul>
<li>name - name of remote</li>
<li>parameters - a map of { “key”: “value” } pairs</li>
</ul>
<p>See the <a href="https://rclone.org/commands/rclone_config_password/">config password command</a> command for more information on the above.</p>
<p>Authentication is required for this call.</p>
@ -3337,6 +3356,7 @@ rclone rc cache/expire remote=/ withData=true</code></pre>
<p>This takes the following parameters</p>
<ul>
<li>name - name of remote</li>
<li>parameters - a map of { “key”: “value” } pairs</li>
</ul>
<p>See the <a href="https://rclone.org/commands/rclone_config_update/">config update command</a> command for more information on the above.</p>
<p>Authentication is required for this call.</p>
@ -4590,7 +4610,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default &quot;rclone/v1.49.0&quot;)
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default &quot;rclone/v1.49.5&quot;)
-v, --verbose count Print lots more stuff (repeat for more)</code></pre>
<h2 id="backend-flags">Backend Flags</h2>
<p>These flags are available for every command. They control the backends and may be set in the config file.</p>
@ -12288,6 +12308,78 @@ $ tree /tmp/b
</ul>
<!--- autogenerated options stop -->
<h1 id="changelog">Changelog</h1>
<h2 id="v1.49.5---2019-10-05">v1.49.5 - 2019-10-05</h2>
<ul>
<li>Bug Fixes
<ul>
<li>Revert back to go1.12.x for the v1.49.x builds as go1.13.x was causing issues (Nick Craig-Wood)</li>
<li>Fix rpm packages by using master builds of nfpm (Nick Craig-Wood)</li>
<li>Fix macOS build after brew changes (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.49.4---2019-09-29">v1.49.4 - 2019-09-29</h2>
<ul>
<li>Bug Fixes
<ul>
<li>cmd/rcd: Address ZipSlip vulnerability (Richard Patel)</li>
<li>accounting: Fix file handle leak on errors (Nick Craig-Wood)</li>
<li>oauthutil: Fix security problem when running with two users on the same machine (Nick Craig-Wood)</li>
</ul></li>
<li>FTP
<ul>
<li>Fix listing of an empty root returning: error dir not found (Nick Craig-Wood)</li>
</ul></li>
<li>S3
<ul>
<li>Fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.49.3---2019-09-15">v1.49.3 - 2019-09-15</h2>
<ul>
<li>Bug Fixes
<ul>
<li>accounting
<ul>
<li>Fix total duration calculation (Aleksandar Jankovic)</li>
<li>Fix “file already closed” on transfer retries (Nick Craig-Wood)</li>
</ul></li>
</ul></li>
</ul>
<h2 id="v1.49.2---2019-09-08">v1.49.2 - 2019-09-08</h2>
<ul>
<li>New Features
<ul>
<li>build: Add Docker workflow support (Alfonso Montero)</li>
</ul></li>
<li>Bug Fixes
<ul>
<li>accounting: Fix locking in Transfer to avoid deadlock with progress (Nick Craig-Wood)</li>
<li>docs: Fix template argument for mktemp in install.sh (Cnly)</li>
<li>operations: Fix -u/update with google photos / files of unknown size (Nick Craig-Wood)</li>
<li>rc: Fix docs for config/create /update /password (Nick Craig-Wood)</li>
</ul></li>
<li>Google Cloud Storage
<ul>
<li>Fix need for elevated permissions on SetModTime (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.49.1---2019-08-28">v1.49.1 - 2019-08-28</h2>
<p>Point release to fix config bug and google photos backend.</p>
<ul>
<li>Bug Fixes
<ul>
<li>config: Fix generated passwords being stored as empty password (Nick Craig-Wood)</li>
<li>rcd: Added missing parameter for web-gui info logs. (Chaitanya)</li>
</ul></li>
<li>Googlephotos
<ul>
<li>Fix crash on error response (Nick Craig-Wood)</li>
</ul></li>
<li>Onedrive
<ul>
<li>Fix crash on error response (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.49.0---2019-08-26">v1.49.0 - 2019-08-26</h2>
<ul>
<li>New backends
@ -12302,8 +12394,10 @@ $ tree /tmp/b
<li>Experimental <a href="https://rclone.org/gui/">web GUI</a> (Chaitanya Bankanhal)</li>
<li>Implement <code>--compare-dest</code> &amp; <code>--copy-dest</code> (yparitcher)</li>
<li>Implement <code>--suffix</code> without <code>--backup-dir</code> for backup to current dir (yparitcher)</li>
<li><code>config reconnect</code> to re-login (re-run the oauth login) for the backend. (Nick Craig-Wood)</li>
<li><code>config userinfo</code> to discover which user you are logged in as. (Nick Craig-Wood)</li>
<li><code>config disconnect</code> to disconnect you (log out) from the backend. (Nick Craig-Wood)</li>
<li>Add <code>--use-json-log</code> for JSON logging (justinalin)</li>
<li>Add <code>config reconnect</code>, <code>config userinfo</code> and <code>config disconnect</code> subcommands. (Nick Craig-Wood)</li>
<li>Add context propagation to rclone (Aleksandar Jankovic)</li>
<li>Reworking internal statistics interfaces so they work with rc jobs (Aleksandar Jankovic)</li>
<li>Add Higher units for ETA (AbelThar)</li>

91
MANUAL.md generated
View file

@ -1,6 +1,6 @@
% rclone(1) User Manual
% Nick Craig-Wood
% Aug 26, 2019
% Oct 05, 2019
# Rclone - rsync for cloud storage
@ -151,6 +151,36 @@ Run `rclone config` to setup. See [rclone config docs](https://rclone.org/docs/)
rclone config
## Install with docker ##
The rclone maintains a [docker image for rclone](https://hub.docker.com/r/rclone/rclone).
These images are autobuilt by docker hub from the rclone source based
on a minimal Alpine linux image.
The `:latest` tag will always point to the latest stable release. You
can use the `:beta` tag to get the latest build from master. You can
also use version tags, eg `:1.49.1`, `:1.49` or `:1`.
```
$ docker pull rclone/rclone:latest
latest: Pulling from rclone/rclone
Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
...
$ docker run --rm rclone/rclone:latest version
rclone v1.49.1
- os/arch: linux/amd64
- go version: go1.12.9
```
You will probably want to mount rclone's config file directory or file
from the host, or configure rclone with environment variables.
Eg to share your local config with the container
```
docker run -v ~/.config/rclone:/root/.config/rclone rclone/rclone:latest listremotes
```
## Install from source ##
Make sure you have at least [Go](https://golang.org/) 1.7
@ -7010,6 +7040,7 @@ Show statistics for the cache remote.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
- type - type of the new remote
@ -7020,6 +7051,7 @@ Authentication is required for this call.
### config/delete: Delete a remote in the config file. {#config/delete}
Parameters:
- name - name of remote to delete
See the [config delete command](https://rclone.org/commands/rclone_config_delete/) command for more information on the above.
@ -7060,6 +7092,7 @@ Authentication is required for this call.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
See the [config password command](https://rclone.org/commands/rclone_config_password/) command for more information on the above.
@ -7080,6 +7113,7 @@ Authentication is required for this call.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
See the [config update command](https://rclone.org/commands/rclone_config_update/) command for more information on the above.
@ -8245,7 +8279,7 @@ These flags are available for every command.
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.0")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.5")
-v, --verbose count Print lots more stuff (repeat for more)
```
@ -18466,6 +18500,55 @@ to override the default choice.
# Changelog
## v1.49.5 - 2019-10-05
* Bug Fixes
* Revert back to go1.12.x for the v1.49.x builds as go1.13.x was causing issues (Nick Craig-Wood)
* Fix rpm packages by using master builds of nfpm (Nick Craig-Wood)
* Fix macOS build after brew changes (Nick Craig-Wood)
## v1.49.4 - 2019-09-29
* Bug Fixes
* cmd/rcd: Address ZipSlip vulnerability (Richard Patel)
* accounting: Fix file handle leak on errors (Nick Craig-Wood)
* oauthutil: Fix security problem when running with two users on the same machine (Nick Craig-Wood)
* FTP
* Fix listing of an empty root returning: error dir not found (Nick Craig-Wood)
* S3
* Fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier (Nick Craig-Wood)
## v1.49.3 - 2019-09-15
* Bug Fixes
* accounting
* Fix total duration calculation (Aleksandar Jankovic)
* Fix "file already closed" on transfer retries (Nick Craig-Wood)
## v1.49.2 - 2019-09-08
* New Features
* build: Add Docker workflow support (Alfonso Montero)
* Bug Fixes
* accounting: Fix locking in Transfer to avoid deadlock with --progress (Nick Craig-Wood)
* docs: Fix template argument for mktemp in install.sh (Cnly)
* operations: Fix -u/--update with google photos / files of unknown size (Nick Craig-Wood)
* rc: Fix docs for config/create /update /password (Nick Craig-Wood)
* Google Cloud Storage
* Fix need for elevated permissions on SetModTime (Nick Craig-Wood)
## v1.49.1 - 2019-08-28
Point release to fix config bug and google photos backend.
* Bug Fixes
* config: Fix generated passwords being stored as empty password (Nick Craig-Wood)
* rcd: Added missing parameter for web-gui info logs. (Chaitanya)
* Googlephotos
* Fix crash on error response (Nick Craig-Wood)
* Onedrive
* Fix crash on error response (Nick Craig-Wood)
## v1.49.0 - 2019-08-26
* New backends
@ -18477,8 +18560,10 @@ to override the default choice.
* Experimental [web GUI](https://rclone.org/gui/) (Chaitanya Bankanhal)
* Implement `--compare-dest` & `--copy-dest` (yparitcher)
* Implement `--suffix` without `--backup-dir` for backup to current dir (yparitcher)
* `config reconnect` to re-login (re-run the oauth login) for the backend. (Nick Craig-Wood)
* `config userinfo` to discover which user you are logged in as. (Nick Craig-Wood)
* `config disconnect` to disconnect you (log out) from the backend. (Nick Craig-Wood)
* Add `--use-json-log` for JSON logging (justinalin)
* Add `config reconnect`, `config userinfo` and `config disconnect` subcommands. (Nick Craig-Wood)
* Add context propagation to rclone (Aleksandar Jankovic)
* Reworking internal statistics interfaces so they work with rc jobs (Aleksandar Jankovic)
* Add Higher units for ETA (AbelThar)

111
MANUAL.txt generated
View file

@ -1,6 +1,6 @@
rclone(1) User Manual
Nick Craig-Wood
Aug 26, 2019
Oct 05, 2019
@ -164,6 +164,33 @@ Run rclone config to setup. See rclone config docs for more details.
rclone config
Install with docker
The rclone maintains a docker image for rclone. These images are
autobuilt by docker hub from the rclone source based on a minimal Alpine
linux image.
The :latest tag will always point to the latest stable release. You can
use the :beta tag to get the latest build from master. You can also use
version tags, eg :1.49.1, :1.49 or :1.
$ docker pull rclone/rclone:latest
latest: Pulling from rclone/rclone
Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
...
$ docker run --rm rclone/rclone:latest version
rclone v1.49.1
- os/arch: linux/amd64
- go version: go1.12.9
You will probably want to mount rclones config file directory or file
from the host, or configure rclone with environment variables.
Eg to share your local config with the container
docker run -v ~/.config/rclone:/root/.config/rclone rclone/rclone:latest listremotes
Install from source
Make sure you have at least Go 1.7 installed. Download go if necessary.
@ -6650,6 +6677,7 @@ config/create: create the config for a remote. {#config/create}
This takes the following parameters
- name - name of remote
- parameters - a map of { “key”: “value” } pairs
- type - type of the new remote
See the config create command command for more information on the above.
@ -6658,7 +6686,9 @@ Authentication is required for this call.
config/delete: Delete a remote in the config file. {#config/delete}
Parameters: - name - name of remote to delete
Parameters:
- name - name of remote to delete
See the config delete command command for more information on the above.
@ -6695,6 +6725,7 @@ config/password: password the config for a remote. {#config/password}
This takes the following parameters
- name - name of remote
- parameters - a map of { “key”: “value” } pairs
See the config password command command for more information on the
above.
@ -6715,6 +6746,7 @@ config/update: update the config for a remote. {#config/update}
This takes the following parameters
- name - name of remote
- parameters - a map of { “key”: “value” } pairs
See the config update command command for more information on the above.
@ -7824,7 +7856,7 @@ These flags are available for every command.
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.0")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.5")
-v, --verbose count Print lots more stuff (repeat for more)
@ -17923,6 +17955,71 @@ override the default choice.
CHANGELOG
v1.49.5 - 2019-10-05
- Bug Fixes
- Revert back to go1.12.x for the v1.49.x builds as go1.13.x was
causing issues (Nick Craig-Wood)
- Fix rpm packages by using master builds of nfpm (Nick
Craig-Wood)
- Fix macOS build after brew changes (Nick Craig-Wood)
v1.49.4 - 2019-09-29
- Bug Fixes
- cmd/rcd: Address ZipSlip vulnerability (Richard Patel)
- accounting: Fix file handle leak on errors (Nick Craig-Wood)
- oauthutil: Fix security problem when running with two users on
the same machine (Nick Craig-Wood)
- FTP
- Fix listing of an empty root returning: error dir not found
(Nick Craig-Wood)
- S3
- Fix SetModTime on GLACIER/ARCHIVE objects and implement set/get
tier (Nick Craig-Wood)
v1.49.3 - 2019-09-15
- Bug Fixes
- accounting
- Fix total duration calculation (Aleksandar Jankovic)
- Fix “file already closed” on transfer retries (Nick
Craig-Wood)
v1.49.2 - 2019-09-08
- New Features
- build: Add Docker workflow support (Alfonso Montero)
- Bug Fixes
- accounting: Fix locking in Transfer to avoid deadlock with
progress (Nick Craig-Wood)
- docs: Fix template argument for mktemp in install.sh (Cnly)
- operations: Fix -u/update with google photos / files of unknown
size (Nick Craig-Wood)
- rc: Fix docs for config/create /update /password (Nick
Craig-Wood)
- Google Cloud Storage
- Fix need for elevated permissions on SetModTime (Nick
Craig-Wood)
v1.49.1 - 2019-08-28
Point release to fix config bug and google photos backend.
- Bug Fixes
- config: Fix generated passwords being stored as empty password
(Nick Craig-Wood)
- rcd: Added missing parameter for web-gui info logs. (Chaitanya)
- Googlephotos
- Fix crash on error response (Nick Craig-Wood)
- Onedrive
- Fix crash on error response (Nick Craig-Wood)
v1.49.0 - 2019-08-26
- New backends
@ -17935,9 +18032,13 @@ v1.49.0 - 2019-08-26
- Implement --compare-dest & --copy-dest (yparitcher)
- Implement --suffix without --backup-dir for backup to current
dir (yparitcher)
- config reconnect to re-login (re-run the oauth login) for the
backend. (Nick Craig-Wood)
- config userinfo to discover which user you are logged in as.
(Nick Craig-Wood)
- config disconnect to disconnect you (log out) from the backend.
(Nick Craig-Wood)
- Add --use-json-log for JSON logging (justinalin)
- Add config reconnect, config userinfo and config disconnect
subcommands. (Nick Craig-Wood)
- Add context propagation to rclone (Aleksandar Jankovic)
- Reworking internal statistics interfaces so they work with rc
jobs (Aleksandar Jankovic)

View file

@ -1,18 +1,29 @@
SHELL = bash
BRANCH := $(or $(APPVEYOR_REPO_BRANCH),$(TRAVIS_BRANCH),$(BUILD_SOURCEBRANCHNAME),$(shell git rev-parse --abbrev-ref HEAD))
# Branch we are working on
BRANCH := $(or $(APPVEYOR_REPO_BRANCH),$(TRAVIS_BRANCH),$(BUILD_SOURCEBRANCHNAME),$(lastword $(subst /, ,$(GITHUB_REF))),$(shell git rev-parse --abbrev-ref HEAD))
# Tag of the current commit, if any. If this is not "" then we are building a release
RELEASE_TAG := $(shell git tag -l --points-at HEAD)
# Version of last release (may not be on this branch)
VERSION := $(shell cat VERSION)
# Last tag on this branch
LAST_TAG := $(shell git describe --tags --abbrev=0)
ifeq ($(BRANCH),$(LAST_TAG))
# If we are working on a release, override branch to master
ifdef RELEASE_TAG
BRANCH := master
endif
TAG_BRANCH := -$(BRANCH)
BRANCH_PATH := branch/
# If building HEAD or master then unset TAG_BRANCH and BRANCH_PATH
ifeq ($(subst HEAD,,$(subst master,,$(BRANCH))),)
TAG_BRANCH :=
BRANCH_PATH :=
endif
TAG := $(shell echo $$(git describe --abbrev=8 --tags | sed 's/-\([0-9]\)-/-00\1-/; s/-\([0-9][0-9]\)-/-0\1-/'))$(TAG_BRANCH)
NEW_TAG := $(shell echo $(LAST_TAG) | perl -lpe 's/v//; $$_ += 0.01; $$_ = sprintf("v%.2f.0", $$_)')
ifneq ($(TAG),$(LAST_TAG))
# Make version suffix -DDD-gCCCCCCCC (D=commits since last relase, C=Commit) or blank
VERSION_SUFFIX := $(shell git describe --abbrev=8 --tags | perl -lpe 's/^v\d+\.\d+\.\d+//; s/^-(\d+)/"-".sprintf("%03d",$$1)/e;')
# TAG is current version + number of commits since last release + branch
TAG := $(VERSION)$(VERSION_SUFFIX)$(TAG_BRANCH)
NEXT_VERSION := $(shell echo $(VERSION) | perl -lpe 's/v//; $$_ += 0.01; $$_ = sprintf("v%.2f.0", $$_)')
ifndef RELEASE_TAG
TAG := $(TAG)-beta
endif
GO_VERSION := $(shell go version)
@ -30,19 +41,22 @@ BUILDTAGS=-tags "$(GOTAGS)"
LINTTAGS=--build-tags "$(GOTAGS)"
endif
.PHONY: rclone vars version
.PHONY: rclone test_all vars version
rclone:
touch fs/version.go
go install -v --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS)
cp -av `go env GOPATH`/bin/rclone .
go build -v --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS)
mkdir -p `go env GOPATH`/bin/
cp -av rclone`go env GOEXE` `go env GOPATH`/bin/
test_all:
go install --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS) github.com/rclone/rclone/fstest/test_all
vars:
@echo SHELL="'$(SHELL)'"
@echo BRANCH="'$(BRANCH)'"
@echo TAG="'$(TAG)'"
@echo LAST_TAG="'$(LAST_TAG)'"
@echo NEW_TAG="'$(NEW_TAG)'"
@echo VERSION="'$(VERSION)'"
@echo NEXT_VERSION="'$(NEXT_VERSION)'"
@echo GO_VERSION="'$(GO_VERSION)'"
@echo BETA_URL="'$(BETA_URL)'"
@ -50,8 +64,7 @@ version:
@echo '$(TAG)'
# Full suite of integration tests
test: rclone
go install --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS) github.com/rclone/rclone/fstest/test_all
test: rclone test_all
-test_all 2>&1 | tee test_all.log
@echo "Written logs in test_all.log"
@ -74,8 +87,8 @@ build_dep:
# Get the release dependencies
release_dep:
go get -u github.com/goreleaser/nfpm/...
go get -u github.com/aktau/github-release
go run bin/get-github-release.go -extract nfpm goreleaser/nfpm 'nfpm_.*_Linux_x86_64.tar.gz'
go run bin/get-github-release.go -extract github-release aktau/github-release 'linux-amd64-github-release.tar.bz2'
# Update dependencies
update:
@ -190,24 +203,25 @@ serve: website
cd docs && hugo server -v -w
tag: doc
@echo "Old tag is $(LAST_TAG)"
@echo "New tag is $(NEW_TAG)"
echo -e "package fs\n\n// Version of rclone\nvar Version = \"$(NEW_TAG)\"\n" | gofmt > fs/version.go
echo -n "$(NEW_TAG)" > docs/layouts/partials/version.html
git tag -s -m "Version $(NEW_TAG)" $(NEW_TAG)
bin/make_changelog.py $(LAST_TAG) $(NEW_TAG) > docs/content/changelog.md.new
@echo "Old tag is $(VERSION)"
@echo "New tag is $(NEXT_VERSION)"
echo -e "package fs\n\n// Version of rclone\nvar Version = \"$(NEXT_VERSION)\"\n" | gofmt > fs/version.go
echo -n "$(NEXT_VERSION)" > docs/layouts/partials/version.html
echo "$(NEXT_VERSION)" > VERSION
git tag -s -m "Version $(NEXT_VERSION)" $(NEXT_VERSION)
bin/make_changelog.py $(LAST_TAG) $(NEXT_VERSION) > docs/content/changelog.md.new
mv docs/content/changelog.md.new docs/content/changelog.md
@echo "Edit the new changelog in docs/content/changelog.md"
@echo "Then commit all the changes"
@echo git commit -m \"Version $(NEW_TAG)\" -a -v
@echo git commit -m \"Version $(NEXT_VERSION)\" -a -v
@echo "And finally run make retag before make cross etc"
retag:
git tag -f -s -m "Version $(LAST_TAG)" $(LAST_TAG)
git tag -f -s -m "Version $(VERSION)" $(VERSION)
startdev:
echo -e "package fs\n\n// Version of rclone\nvar Version = \"$(LAST_TAG)-DEV\"\n" | gofmt > fs/version.go
git commit -m "Start $(LAST_TAG)-DEV development" fs/version.go
echo -e "package fs\n\n// Version of rclone\nvar Version = \"$(VERSION)-DEV\"\n" | gofmt > fs/version.go
git commit -m "Start $(VERSION)-DEV development" fs/version.go
winzip:
zip -9 rclone-$(TAG).zip rclone.exe

View file

@ -48,24 +48,56 @@ Can be fixed with
* GO111MODULE=on go mod vendor
Making a point release. If rclone needs a point release due to some
horrendous bug, then
* git branch v1.XX v1.XX-fixes
## Making a point release
If rclone needs a point release due to some horrendous bug:
First make the release branch. If this is a second point release then
this will be done already.
* BASE_TAG=v1.XX # eg v1.49
* NEW_TAG=${BASE_TAG}.Y # eg v1.49.1
* echo $BASE_TAG $NEW_TAG # v1.49 v1.49.1
* git branch ${BASE_TAG} ${BASE_TAG}-fixes
Now
* git co ${BASE_TAG}-fixes
* git cherry-pick any fixes
* Test (see above)
* make NEW_TAG=v1.XX.1 tag
* edit docs/content/changelog.md
* make TAG=v1.43.1 doc
* git commit -a -v -m "Version v1.XX.1"
* git tag -d -v1.XX.1
* git tag -s -m "Version v1.XX.1" v1.XX.1
* git push --tags -u origin v1.XX-fixes
* make BRANCH_PATH= TAG=v1.43.1 fetch_binaries
* make TAG=v1.43.1 tarball
* make TAG=v1.43.1 sign_upload
* make TAG=v1.43.1 check_sign
* make TAG=v1.43.1 upload
* make TAG=v1.43.1 upload_website
* make TAG=v1.43.1 upload_github
* NB this overwrites the current beta so after the release, rebuild the last travis build
* make TAG=${NEW_TAG} doc
* git commit -a -v -m "Version ${NEW_TAG}"
* git tag -d ${NEW_TAG}
* git tag -s -m "Version ${NEW_TAG}" ${NEW_TAG}
* git push --tags -u origin ${BASE_TAG}-fixes
* Wait for builds to complete
* make BRANCH_PATH= TAG=${NEW_TAG} fetch_binaries
* make TAG=${NEW_TAG} tarball
* make TAG=${NEW_TAG} sign_upload
* make TAG=${NEW_TAG} check_sign
* make TAG=${NEW_TAG} upload
* make TAG=${NEW_TAG} upload_website
* make TAG=${NEW_TAG} upload_github
* NB this overwrites the current beta so we need to do this
* git co master
* make LAST_TAG=${NEW_TAG} startdev
* # cherry pick the changes to the changelog and VERSION
* git checkout ${BASE_TAG}-fixes VERSION docs/content/changelog.md
* git commit --amend
* git push
* Announce!
## Making a manual build of docker
The rclone docker image should autobuild on docker hub. If it doesn't
or needs to be updated then rebuild like this.
```
docker build -t rclone/rclone:1.49.1 -t rclone/rclone:1.49 -t rclone/rclone:1 -t rclone/rclone:latest .
docker push rclone/rclone:1.49.1
docker push rclone/rclone:1.49
docker push rclone/rclone:1
docker push rclone/rclone:latest
```

1
VERSION Normal file
View file

@ -0,0 +1 @@
v1.49.5

View file

@ -1,239 +0,0 @@
---
# Azure pipelines build for rclone
# Parts stolen shamelessly from all round the Internet, especially Caddy
# -*- compile-command: "yamllint -f parsable azure-pipelines.yml" -*-
trigger:
branches:
include:
- '*'
tags:
include:
- '*'
variables:
GOROOT: $(gorootDir)/go
GOPATH: $(system.defaultWorkingDirectory)/gopath
GOCACHE: $(system.defaultWorkingDirectory)/gocache
GOBIN: $(GOPATH)/bin
modulePath: '$(GOPATH)/src/github.com/$(build.repository.name)'
GO111MODULE: 'off'
GOTAGS: cmount
GO_LATEST: false
CPATH: ''
GO_INSTALL_ARCH: amd64
strategy:
matrix:
linux:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: latest
GOTAGS: cmount
BUILD_FLAGS: '-include "^linux/"'
MAKE_CHECK: true
MAKE_QUICKTEST: true
DEPLOY: true
mac:
imageName: macos-10.13
gorootDir: /usr/local
GO_VERSION: latest
GOTAGS: "" # cmount doesn't work on osx travis for some reason
BUILD_FLAGS: '-include "^darwin/" -cgo'
MAKE_QUICKTEST: true
MAKE_RACEQUICKTEST: true
DEPLOY: true
windows_amd64:
imageName: windows-2019
gorootDir: C:\
GO_VERSION: latest
BUILD_FLAGS: '-include "^windows/amd64" -cgo'
MAKE_QUICKTEST: true
DEPLOY: true
windows_386:
imageName: windows-2019
gorootDir: C:\
GO_VERSION: latest
GO_INSTALL_ARCH: 386
BUILD_FLAGS: '-include "^windows/386" -cgo'
MAKE_QUICKTEST: true
DEPLOY: true
other_os:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: latest
BUILD_FLAGS: '-exclude "^(windows|darwin|linux)/"'
MAKE_COMPILE_ALL: true
DEPLOY: true
modules_race:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: latest
GO111MODULE: on
GOPROXY: https://proxy.golang.org
MAKE_QUICKTEST: true
MAKE_RACEQUICKTEST: true
go1.9:
imageName: ubuntu-16.04
gorootDir: /usr/local
GOCACHE: '' # build caching only came in go1.10
GO_VERSION: go1.9.7
MAKE_QUICKTEST: true
go1.10:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: go1.10.8
MAKE_QUICKTEST: true
go1.11:
imageName: ubuntu-16.04
gorootDir: /usr/local
GO_VERSION: go1.11.12
MAKE_QUICKTEST: true
pool:
vmImage: $(imageName)
steps:
- bash: |
latestGo=$(curl "https://golang.org/VERSION?m=text")
echo "##vso[task.setvariable variable=GO_VERSION]$latestGo"
echo "##vso[task.setvariable variable=GO_LATEST]true"
echo "Latest Go version: $latestGo"
condition: eq( variables['GO_VERSION'], 'latest' )
continueOnError: false
displayName: "Get latest Go version"
- bash: |
sudo rm -f $(which go)
echo '##vso[task.prependpath]$(GOBIN)'
echo '##vso[task.prependpath]$(GOROOT)/bin'
mkdir -p '$(modulePath)'
shopt -s extglob
shopt -s dotglob
mv !(gopath) '$(modulePath)'
continueOnError: false
displayName: Remove old Go, set GOBIN/GOROOT, and move project into GOPATH
- task: CacheBeta@0
inputs:
key: go-build-cache | "$(Agent.JobName)"
path: $(GOCACHE)
continueOnError: true
displayName: Cache go build
condition: ne( variables['GOCACHE'], '' )
# Install Libraries (varies by platform)
- bash: |
sudo modprobe fuse
sudo chmod 666 /dev/fuse
sudo chown root:$USER /etc/fuse.conf
sudo apt-get install fuse libfuse-dev rpm pkg-config
condition: eq( variables['Agent.OS'], 'Linux' )
continueOnError: false
displayName: Install Libraries on Linux
- bash: |
brew update
brew tap caskroom/cask
brew cask install osxfuse
condition: eq( variables['Agent.OS'], 'Darwin' )
continueOnError: false
displayName: Install Libraries on macOS
- powershell: |
$ProgressPreference = 'SilentlyContinue'
choco install -y winfsp zip
Write-Host "##vso[task.setvariable variable=CPATH]C:\Program Files\WinFsp\inc\fuse;C:\Program Files (x86)\WinFsp\inc\fuse"
if ($env:GO_INSTALL_ARCH -eq "386") {
choco install -y mingw --forcex86 --force
Write-Host "##vso[task.prependpath]C:\\ProgramData\\chocolatey\\lib\\mingw\\tools\\install\\mingw32\\bin"
}
# Copy mingw32-make.exe to make.exe so the same command line
# can be used on Windows as on macOS and Linux
$path = (get-command mingw32-make.exe).Path
Copy-Item -Path $path -Destination (Join-Path (Split-Path -Path $path) 'make.exe')
condition: eq( variables['Agent.OS'], 'Windows_NT' )
continueOnError: false
displayName: Install Libraries on Windows
# Install Go (this varies by platform)
- bash: |
wget "https://dl.google.com/go/$(GO_VERSION).linux-$(GO_INSTALL_ARCH).tar.gz"
sudo mkdir $(gorootDir)
sudo chown ${USER}:${USER} $(gorootDir)
tar -C $(gorootDir) -xzf "$(GO_VERSION).linux-$(GO_INSTALL_ARCH).tar.gz"
condition: eq( variables['Agent.OS'], 'Linux' )
continueOnError: false
displayName: Install Go on Linux
- bash: |
wget "https://dl.google.com/go/$(GO_VERSION).darwin-$(GO_INSTALL_ARCH).tar.gz"
sudo tar -C $(gorootDir) -xzf "$(GO_VERSION).darwin-$(GO_INSTALL_ARCH).tar.gz"
condition: eq( variables['Agent.OS'], 'Darwin' )
continueOnError: false
displayName: Install Go on macOS
- powershell: |
$ProgressPreference = 'SilentlyContinue'
Write-Host "Downloading Go $(GO_VERSION) for $(GO_INSTALL_ARCH)"
(New-Object System.Net.WebClient).DownloadFile("https://dl.google.com/go/$(GO_VERSION).windows-$(GO_INSTALL_ARCH).zip", "$(GO_VERSION).windows-$(GO_INSTALL_ARCH).zip")
Write-Host "Extracting Go"
Expand-Archive "$(GO_VERSION).windows-$(GO_INSTALL_ARCH).zip" -DestinationPath "$(gorootDir)"
condition: eq( variables['Agent.OS'], 'Windows_NT' )
continueOnError: false
displayName: Install Go on Windows
# Display environment for debugging
- bash: |
printf "Using go at: $(which go)\n"
printf "Go version: $(go version)\n"
printf "\n\nGo environment:\n\n"
go env
printf "\n\nRclone environment:\n\n"
make vars
printf "\n\nSystem environment:\n\n"
env
workingDirectory: '$(modulePath)'
displayName: Print Go version and environment
# Run Tests
- bash: |
make
make quicktest
workingDirectory: '$(modulePath)'
displayName: Run tests
condition: eq( variables['MAKE_QUICKTEST'], 'true' )
- bash: |
make racequicktest
workingDirectory: '$(modulePath)'
displayName: Race test
condition: eq( variables['MAKE_RACEQUICKTEST'], 'true' )
- bash: |
make build_dep
make check
workingDirectory: '$(modulePath)'
displayName: Code quality test
condition: eq( variables['MAKE_CHECK'], 'true' )
- bash: |
make
make compile_all
workingDirectory: '$(modulePath)'
displayName: Compile all architectures test
condition: eq( variables['MAKE_COMPILE_ALL'], 'true' )
- bash: |
make travis_beta
env:
RCLONE_CONFIG_PASS: $(RCLONE_CONFIG_PASS)
BETA_SUBDIR: 'azure_pipelines' # FIXME remove when removing travis/appveyor
workingDirectory: '$(modulePath)'
displayName: Deploy built binaries
condition: and( eq( variables['DEPLOY'], 'true' ), ne( variables['Build.Reason'], 'PullRequest' ) )

View file

@ -299,6 +299,14 @@ func translateErrorDir(err error) error {
func (f *Fs) findItem(remote string) (entry *ftp.Entry, err error) {
// defer fs.Trace(remote, "")("o=%v, err=%v", &o, &err)
fullPath := path.Join(f.root, remote)
if fullPath == "" || fullPath == "." || fullPath == "/" {
// if root, assume exists and synthesize an entry
return &ftp.Entry{
Name: "",
Type: ftp.EntryTypeFolder,
Time: time.Now(),
}, nil
}
dir := path.Dir(fullPath)
base := path.Base(fullPath)
@ -366,7 +374,7 @@ func (f *Fs) dirExists(remote string) (exists bool, err error) {
// This should return ErrDirNotFound if the directory isn't
// found.
func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
// defer fs.Trace(dir, "curlevel=%d", curlevel)("")
// defer log.Trace(dir, "dir=%q", dir)("entries=%v, err=%v", &entries, &err)
c, err := f.getFtpConnection()
if err != nil {
return nil, errors.Wrap(err, "list")

View file

@ -824,7 +824,11 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
var newObject *storage.Object
err = f.pacer.Call(func() (bool, error) {
newObject, err = f.svc.Objects.Copy(srcBucket, srcPath, dstBucket, dstPath, nil).Do()
copyObject := f.svc.Objects.Copy(srcBucket, srcPath, dstBucket, dstPath, nil)
if !f.opt.BucketPolicyOnly {
copyObject.DestinationPredefinedAcl(f.opt.ObjectACL)
}
newObject, err = copyObject.Do()
return shouldRetry(err)
})
if err != nil {
@ -907,15 +911,9 @@ func (o *Object) setMetaData(info *storage.Object) {
}
}
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
func (o *Object) readMetaData() (err error) {
if !o.modTime.IsZero() {
return nil
}
// readObjectInfo reads the definition for an object
func (o *Object) readObjectInfo() (object *storage.Object, err error) {
bucket, bucketPath := o.split()
var object *storage.Object
err = o.fs.pacer.Call(func() (bool, error) {
object, err = o.fs.svc.Objects.Get(bucket, bucketPath).Do()
return shouldRetry(err)
@ -923,9 +921,23 @@ func (o *Object) readMetaData() (err error) {
if err != nil {
if gErr, ok := err.(*googleapi.Error); ok {
if gErr.Code == http.StatusNotFound {
return fs.ErrorObjectNotFound
return nil, fs.ErrorObjectNotFound
}
}
return nil, err
}
return object, nil
}
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
func (o *Object) readMetaData() (err error) {
if !o.modTime.IsZero() {
return nil
}
object, err := o.readObjectInfo()
if err != nil {
return err
}
o.setMetaData(object)
@ -954,16 +966,27 @@ func metadataFromModTime(modTime time.Time) map[string]string {
// SetModTime sets the modification time of the local fs object
func (o *Object) SetModTime(ctx context.Context, modTime time.Time) (err error) {
// This only adds metadata so will perserve other metadata
bucket, bucketPath := o.split()
object := storage.Object{
Bucket: bucket,
Name: bucketPath,
Metadata: metadataFromModTime(modTime),
// read the complete existing object first
object, err := o.readObjectInfo()
if err != nil {
return err
}
// Add the mtime to the existing metadata
mtime := modTime.Format(timeFormatOut)
if object.Metadata == nil {
object.Metadata = make(map[string]string, 1)
}
object.Metadata[metaMtime] = mtime
// Copy the object to itself to update the metadata
// Using PATCH requires too many permissions
bucket, bucketPath := o.split()
var newObject *storage.Object
err = o.fs.pacer.Call(func() (bool, error) {
newObject, err = o.fs.svc.Objects.Patch(bucket, bucketPath, &object).Do()
copyObject := o.fs.svc.Objects.Copy(bucket, bucketPath, bucket, bucketPath, object)
if !o.fs.opt.BucketPolicyOnly {
copyObject.DestinationPredefinedAcl(o.fs.opt.ObjectACL)
}
newObject, err = copyObject.Do()
return shouldRetry(err)
})
if err != nil {

View file

@ -956,7 +956,6 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
if err != nil {
_ = resp.Body.Close()
return shouldRetry(resp, err)
}
token, err = rest.ReadBody(resp)

View file

@ -1464,22 +1464,24 @@ func (o *Object) uploadFragment(url string, start int64, totalSize int64, chunk
}
// var response api.UploadFragmentResponse
var resp *http.Response
var body []byte
err = o.fs.pacer.Call(func() (bool, error) {
_, _ = chunk.Seek(0, io.SeekStart)
resp, err = o.fs.srv.Call(&opts)
if resp != nil {
defer fs.CheckClose(resp.Body, &err)
if err != nil {
return shouldRetry(resp, err)
}
body, err = rest.ReadBody(resp)
if err != nil {
return shouldRetry(resp, err)
}
retry, err := shouldRetry(resp, err)
if !retry && resp != nil {
if resp.StatusCode == 200 || resp.StatusCode == 201 {
// we are done :)
// read the item
info = &api.Item{}
return false, json.NewDecoder(resp.Body).Decode(info)
return false, json.Unmarshal(body, info)
}
}
return retry, err
return false, nil
})
return info, err
}

View file

@ -818,6 +818,7 @@ type Object struct {
lastModified time.Time // Last modified
meta map[string]*string // The object metadata if known - may be nil
mimeType string // MimeType of object - may be ""
storageClass string // eg GLACIER
}
// ------------------------------------------------------------
@ -1089,6 +1090,8 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
WriteMimeType: true,
BucketBased: true,
BucketBasedRootOK: true,
SetTier: true,
GetTier: true,
}).Fill(f)
if f.rootBucket != "" && f.rootDirectory != "" {
// Check to see if the object exists
@ -1132,6 +1135,7 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *s3.Obje
}
o.etag = aws.StringValue(info.ETag)
o.bytes = aws.Int64Value(info.Size)
o.storageClass = aws.StringValue(info.StorageClass)
} else {
err := o.readMetaData(ctx) // reads info and meta, returning an error
if err != nil {
@ -1550,6 +1554,31 @@ func pathEscape(s string) string {
return strings.Replace(rest.URLPathEscape(s), "+", "%2B", -1)
}
// copy does a server side copy
//
// It adds the boiler plate to the req passed in and calls the s3
// method
func (f *Fs) copy(ctx context.Context, req *s3.CopyObjectInput, dstBucket, dstPath, srcBucket, srcPath string) error {
req.Bucket = &dstBucket
req.ACL = &f.opt.ACL
req.Key = &dstPath
source := pathEscape(path.Join(srcBucket, srcPath))
req.CopySource = &source
if f.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &f.opt.ServerSideEncryption
}
if f.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &f.opt.SSEKMSKeyID
}
if req.StorageClass == nil && f.opt.StorageClass != "" {
req.StorageClass = &f.opt.StorageClass
}
return f.pacer.Call(func() (bool, error) {
_, err := f.c.CopyObjectWithContext(ctx, req)
return f.shouldRetry(err)
})
}
// Copy src to this remote using server side copy operations.
//
// This is stored with the remote path given
@ -1571,27 +1600,10 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
return nil, fs.ErrorCantCopy
}
srcBucket, srcPath := srcObj.split()
source := pathEscape(path.Join(srcBucket, srcPath))
req := s3.CopyObjectInput{
Bucket: &dstBucket,
ACL: &f.opt.ACL,
Key: &dstPath,
CopySource: &source,
MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
}
if f.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &f.opt.ServerSideEncryption
}
if f.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &f.opt.SSEKMSKeyID
}
if f.opt.StorageClass != "" {
req.StorageClass = &f.opt.StorageClass
}
err = f.pacer.Call(func() (bool, error) {
_, err = f.c.CopyObjectWithContext(ctx, &req)
return f.shouldRetry(err)
})
err = f.copy(ctx, &req, dstBucket, dstPath, srcBucket, srcPath)
if err != nil {
return nil, err
}
@ -1691,6 +1703,7 @@ func (o *Object) readMetaData(ctx context.Context) (err error) {
o.etag = aws.StringValue(resp.ETag)
o.bytes = size
o.meta = resp.Metadata
o.storageClass = aws.StringValue(resp.StorageClass)
if resp.LastModified == nil {
fs.Logf(o, "Failed to read last modified from HEAD: %v", err)
o.lastModified = time.Now()
@ -1741,39 +1754,19 @@ func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
return nil
}
// Guess the content type
mimeType := fs.MimeType(ctx, o)
// Can't update metadata here, so return this error to force a recopy
if o.storageClass == "GLACIER" || o.storageClass == "DEEP_ARCHIVE" {
return fs.ErrorCantSetModTime
}
// Copy the object to itself to update the metadata
bucket, bucketPath := o.split()
sourceKey := path.Join(bucket, bucketPath)
directive := s3.MetadataDirectiveReplace // replace metadata with that passed in
req := s3.CopyObjectInput{
Bucket: &bucket,
ACL: &o.fs.opt.ACL,
Key: &bucketPath,
ContentType: &mimeType,
CopySource: aws.String(pathEscape(sourceKey)),
ContentType: aws.String(fs.MimeType(ctx, o)), // Guess the content type
Metadata: o.meta,
MetadataDirective: &directive,
MetadataDirective: aws.String(s3.MetadataDirectiveReplace), // replace metadata with that passed in
}
if o.fs.opt.ServerSideEncryption != "" {
req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
}
if o.fs.opt.SSEKMSKeyID != "" {
req.SSEKMSKeyId = &o.fs.opt.SSEKMSKeyID
}
if o.fs.opt.StorageClass == "GLACIER" || o.fs.opt.StorageClass == "DEEP_ARCHIVE" {
return fs.ErrorCantSetModTime
}
if o.fs.opt.StorageClass != "" {
req.StorageClass = &o.fs.opt.StorageClass
}
err = o.fs.pacer.Call(func() (bool, error) {
_, err := o.fs.c.CopyObjectWithContext(ctx, &req)
return o.fs.shouldRetry(err)
})
return err
return o.fs.copy(ctx, &req, bucket, bucketPath, bucket, bucketPath)
}
// Storable raturns a boolean indicating if this object is storable
@ -1998,6 +1991,31 @@ func (o *Object) MimeType(ctx context.Context) string {
return o.mimeType
}
// SetTier performs changing storage class
func (o *Object) SetTier(tier string) (err error) {
ctx := context.TODO()
tier = strings.ToUpper(tier)
bucket, bucketPath := o.split()
req := s3.CopyObjectInput{
MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
StorageClass: aws.String(tier),
}
err = o.fs.copy(ctx, &req, bucket, bucketPath, bucket, bucketPath)
if err != nil {
return err
}
o.storageClass = tier
return err
}
// GetTier returns storage class as string
func (o *Object) GetTier() string {
if o.storageClass == "" {
return "STANDARD"
}
return o.storageClass
}
// Check the interfaces are satisfied
var (
_ fs.Fs = &Fs{}
@ -2006,4 +2024,6 @@ var (
_ fs.ListRer = &Fs{}
_ fs.Object = &Object{}
_ fs.MimeTyper = &Object{}
_ fs.GetTierer = &Object{}
_ fs.SetTierer = &Object{}
)

View file

@ -13,6 +13,7 @@ func TestIntegration(t *testing.T) {
fstests.Run(t, &fstests.Opt{
RemoteName: "TestS3:",
NilObject: (*Object)(nil),
TiersToTest: []string{"STANDARD", "STANDARD_IA"},
ChunkedUpload: fstests.ChunkedUploadConfig{
MinChunkSize: minChunkSize,
},

View file

@ -9,6 +9,7 @@ package main
import (
"archive/tar"
"compress/bzip2"
"compress/gzip"
"encoding/json"
"flag"
@ -349,6 +350,8 @@ func untar(srcFile, fileName, extractDir string) {
log.Fatalf("Couldn't open gzip: %v", err)
}
in = gzf
} else if srcExt == ".bz2" {
in = bzip2.NewReader(f)
}
tarReader := tar.NewReader(in)

View file

@ -3,12 +3,14 @@ package rcd
import (
"archive/zip"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/rclone/rclone/cmd"
@ -61,7 +63,7 @@ See the [rc documentation](/rc/) for more info on the rc flags.
}
if rcflags.Opt.HTTPOptions.BasicUser == "" {
rcflags.Opt.HTTPOptions.BasicUser = "gui"
fs.Infof("Using default username: %s \n", rcflags.Opt.HTTPOptions.BasicUser)
fs.Infof(nil, "Using default username: %s \n", rcflags.Opt.HTTPOptions.BasicUser)
}
if rcflags.Opt.HTTPOptions.BasicPass == "" {
randomPass, err := random.Password(128)
@ -69,7 +71,7 @@ See the [rc documentation](/rc/) for more info on the rc flags.
log.Fatalf("Failed to make password: %v", err)
}
rcflags.Opt.HTTPOptions.BasicPass = randomPass
fs.Infof("No password specified. Using random password: %s \n", randomPass)
fs.Infof(nil, "No password specified. Using random password: %s \n", randomPass)
}
rcflags.Opt.Serve = true
}
@ -179,6 +181,8 @@ func downloadFile(filepath string, url string) error {
// unzip is a helper function to unzip a file specified in src to path dest
func unzip(src, dest string) (err error) {
dest = filepath.Clean(dest) + string(os.PathSeparator)
r, err := zip.OpenReader(src)
if err != nil {
return err
@ -191,14 +195,18 @@ func unzip(src, dest string) (err error) {
// Closure to address file descriptors issue with all the deferred .Close() methods
extractAndWriteFile := func(f *zip.File) error {
path := filepath.Join(dest, f.Name)
// Check for Zip Slip: https://github.com/rclone/rclone/issues/3529
if !strings.HasPrefix(path, dest) {
return fmt.Errorf("%s: illegal file path", path)
}
rc, err := f.Open()
if err != nil {
return err
}
defer fs.CheckClose(rc, &err)
path := filepath.Join(dest, f.Name)
if f.FileInfo().IsDir() {
if err := os.MkdirAll(path, 0755); err != nil {
return err

View file

@ -1,11 +1,60 @@
---
title: "Documentation"
description: "Rclone Changelog"
date: "2019-08-26"
date: "2019-10-05"
---
# Changelog
## v1.49.5 - 2019-10-05
* Bug Fixes
* Revert back to go1.12.x for the v1.49.x builds as go1.13.x was causing issues (Nick Craig-Wood)
* Fix rpm packages by using master builds of nfpm (Nick Craig-Wood)
* Fix macOS build after brew changes (Nick Craig-Wood)
## v1.49.4 - 2019-09-29
* Bug Fixes
* cmd/rcd: Address ZipSlip vulnerability (Richard Patel)
* accounting: Fix file handle leak on errors (Nick Craig-Wood)
* oauthutil: Fix security problem when running with two users on the same machine (Nick Craig-Wood)
* FTP
* Fix listing of an empty root returning: error dir not found (Nick Craig-Wood)
* S3
* Fix SetModTime on GLACIER/ARCHIVE objects and implement set/get tier (Nick Craig-Wood)
## v1.49.3 - 2019-09-15
* Bug Fixes
* accounting
* Fix total duration calculation (Aleksandar Jankovic)
* Fix "file already closed" on transfer retries (Nick Craig-Wood)
## v1.49.2 - 2019-09-08
* New Features
* build: Add Docker workflow support (Alfonso Montero)
* Bug Fixes
* accounting: Fix locking in Transfer to avoid deadlock with --progress (Nick Craig-Wood)
* docs: Fix template argument for mktemp in install.sh (Cnly)
* operations: Fix -u/--update with google photos / files of unknown size (Nick Craig-Wood)
* rc: Fix docs for config/create /update /password (Nick Craig-Wood)
* Google Cloud Storage
* Fix need for elevated permissions on SetModTime (Nick Craig-Wood)
## v1.49.1 - 2019-08-28
Point release to fix config bug and google photos backend.
* Bug Fixes
* config: Fix generated passwords being stored as empty password (Nick Craig-Wood)
* rcd: Added missing parameter for web-gui info logs. (Chaitanya)
* Googlephotos
* Fix crash on error response (Nick Craig-Wood)
* Onedrive
* Fix crash on error response (Nick Craig-Wood)
## v1.49.0 - 2019-08-26
* New backends

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone"
slug: rclone
url: /commands/rclone/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone about"
slug: rclone_about
url: /commands/rclone_about/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone authorize"
slug: rclone_authorize
url: /commands/rclone_authorize/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone cachestats"
slug: rclone_cachestats
url: /commands/rclone_cachestats/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone cat"
slug: rclone_cat
url: /commands/rclone_cat/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone check"
slug: rclone_check
url: /commands/rclone_check/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone cleanup"
slug: rclone_cleanup
url: /commands/rclone_cleanup/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config"
slug: rclone_config
url: /commands/rclone_config/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config create"
slug: rclone_config_create
url: /commands/rclone_config_create/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config delete"
slug: rclone_config_delete
url: /commands/rclone_config_delete/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config disconnect"
slug: rclone_config_disconnect
url: /commands/rclone_config_disconnect/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config dump"
slug: rclone_config_dump
url: /commands/rclone_config_dump/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config edit"
slug: rclone_config_edit
url: /commands/rclone_config_edit/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config file"
slug: rclone_config_file
url: /commands/rclone_config_file/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config password"
slug: rclone_config_password
url: /commands/rclone_config_password/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config providers"
slug: rclone_config_providers
url: /commands/rclone_config_providers/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config reconnect"
slug: rclone_config_reconnect
url: /commands/rclone_config_reconnect/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config show"
slug: rclone_config_show
url: /commands/rclone_config_show/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config update"
slug: rclone_config_update
url: /commands/rclone_config_update/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone config userinfo"
slug: rclone_config_userinfo
url: /commands/rclone_config_userinfo/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone copy"
slug: rclone_copy
url: /commands/rclone_copy/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone copyto"
slug: rclone_copyto
url: /commands/rclone_copyto/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone copyurl"
slug: rclone_copyurl
url: /commands/rclone_copyurl/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone cryptcheck"
slug: rclone_cryptcheck
url: /commands/rclone_cryptcheck/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone cryptdecode"
slug: rclone_cryptdecode
url: /commands/rclone_cryptdecode/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone dbhashsum"
slug: rclone_dbhashsum
url: /commands/rclone_dbhashsum/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone dedupe"
slug: rclone_dedupe
url: /commands/rclone_dedupe/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone delete"
slug: rclone_delete
url: /commands/rclone_delete/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone deletefile"
slug: rclone_deletefile
url: /commands/rclone_deletefile/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone genautocomplete"
slug: rclone_genautocomplete
url: /commands/rclone_genautocomplete/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone genautocomplete bash"
slug: rclone_genautocomplete_bash
url: /commands/rclone_genautocomplete_bash/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone genautocomplete zsh"
slug: rclone_genautocomplete_zsh
url: /commands/rclone_genautocomplete_zsh/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone gendocs"
slug: rclone_gendocs
url: /commands/rclone_gendocs/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone hashsum"
slug: rclone_hashsum
url: /commands/rclone_hashsum/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone link"
slug: rclone_link
url: /commands/rclone_link/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone listremotes"
slug: rclone_listremotes
url: /commands/rclone_listremotes/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone ls"
slug: rclone_ls
url: /commands/rclone_ls/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone lsd"
slug: rclone_lsd
url: /commands/rclone_lsd/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone lsf"
slug: rclone_lsf
url: /commands/rclone_lsf/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone lsjson"
slug: rclone_lsjson
url: /commands/rclone_lsjson/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone lsl"
slug: rclone_lsl
url: /commands/rclone_lsl/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone md5sum"
slug: rclone_md5sum
url: /commands/rclone_md5sum/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone mkdir"
slug: rclone_mkdir
url: /commands/rclone_mkdir/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone mount"
slug: rclone_mount
url: /commands/rclone_mount/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone move"
slug: rclone_move
url: /commands/rclone_move/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone moveto"
slug: rclone_moveto
url: /commands/rclone_moveto/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone ncdu"
slug: rclone_ncdu
url: /commands/rclone_ncdu/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone obscure"
slug: rclone_obscure
url: /commands/rclone_obscure/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone purge"
slug: rclone_purge
url: /commands/rclone_purge/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone rc"
slug: rclone_rc
url: /commands/rclone_rc/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone rcat"
slug: rclone_rcat
url: /commands/rclone_rcat/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone rcd"
slug: rclone_rcd
url: /commands/rclone_rcd/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone rmdir"
slug: rclone_rmdir
url: /commands/rclone_rmdir/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone rmdirs"
slug: rclone_rmdirs
url: /commands/rclone_rmdirs/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone serve"
slug: rclone_serve
url: /commands/rclone_serve/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone serve dlna"
slug: rclone_serve_dlna
url: /commands/rclone_serve_dlna/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone serve ftp"
slug: rclone_serve_ftp
url: /commands/rclone_serve_ftp/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone serve http"
slug: rclone_serve_http
url: /commands/rclone_serve_http/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone serve restic"
slug: rclone_serve_restic
url: /commands/rclone_serve_restic/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone serve sftp"
slug: rclone_serve_sftp
url: /commands/rclone_serve_sftp/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone serve webdav"
slug: rclone_serve_webdav
url: /commands/rclone_serve_webdav/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone settier"
slug: rclone_settier
url: /commands/rclone_settier/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone sha1sum"
slug: rclone_sha1sum
url: /commands/rclone_sha1sum/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone size"
slug: rclone_size
url: /commands/rclone_size/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone sync"
slug: rclone_sync
url: /commands/rclone_sync/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone touch"
slug: rclone_touch
url: /commands/rclone_touch/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone tree"
slug: rclone_tree
url: /commands/rclone_tree/

View file

@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-10-05T12:07:00+01:00
title: "rclone version"
slug: rclone_version
url: /commands/rclone_version/

View file

@ -1,7 +1,7 @@
---
title: "Global Flags"
description: "Rclone Global Flags"
date: "2019-08-26T15:19:45+01:00"
date: "2019-10-05T12:07:00+01:00"
---
# Global Flags
@ -127,7 +127,7 @@ These flags are available for every command.
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.0")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.5")
-v, --verbose count Print lots more stuff (repeat for more)
```

View file

@ -81,6 +81,36 @@ Run `rclone config` to setup. See [rclone config docs](/docs/) for more details.
rclone config
## Install with docker ##
The rclone maintains a [docker image for rclone](https://hub.docker.com/r/rclone/rclone).
These images are autobuilt by docker hub from the rclone source based
on a minimal Alpine linux image.
The `:latest` tag will always point to the latest stable release. You
can use the `:beta` tag to get the latest build from master. You can
also use version tags, eg `:1.49.1`, `:1.49` or `:1`.
```
$ docker pull rclone/rclone:latest
latest: Pulling from rclone/rclone
Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
...
$ docker run --rm rclone/rclone:latest version
rclone v1.49.1
- os/arch: linux/amd64
- go version: go1.12.9
```
You will probably want to mount rclone's config file directory or file
from the host, or configure rclone with environment variables.
Eg to share your local config with the container
```
docker run -v ~/.config/rclone:/root/.config/rclone rclone/rclone:latest listremotes
```
## Install from source ##
Make sure you have at least [Go](https://golang.org/) 1.7

View file

@ -25,7 +25,7 @@ fi
#create tmp directory and move to it with macOS compatibility fallback
tmp_dir=`mktemp -d 2>/dev/null || mktemp -d -t 'rclone-install'`; cd $tmp_dir
tmp_dir=`mktemp -d 2>/dev/null || mktemp -d -t 'rclone-install.XXXXXXXXXX'`; cd $tmp_dir
#make sure unzip tool is available and choose one to work with

View file

@ -308,6 +308,7 @@ Show statistics for the cache remote.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
- type - type of the new remote
@ -318,6 +319,7 @@ Authentication is required for this call.
### config/delete: Delete a remote in the config file. {#config/delete}
Parameters:
- name - name of remote to delete
See the [config delete command](/commands/rclone_config_delete/) command for more information on the above.
@ -358,6 +360,7 @@ Authentication is required for this call.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
See the [config password command](/commands/rclone_config_password/) command for more information on the above.
@ -378,6 +381,7 @@ Authentication is required for this call.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
See the [config update command](/commands/rclone_config_update/) command for more information on the above.

View file

@ -1 +1 @@
v1.49.0
v1.49.5

View file

@ -118,11 +118,16 @@ func (acc *Account) StopBuffering() {
// async buffer (if any) and re-adding it
func (acc *Account) UpdateReader(in io.ReadCloser) {
acc.mu.Lock()
if acc.withBuf {
acc.StopBuffering()
}
acc.in = in
acc.close = in
acc.origIn = in
acc.closed = false
if acc.withBuf {
acc.WithBuffer()
}
acc.mu.Unlock()
}
@ -217,14 +222,20 @@ func (acc *Account) Close() error {
return nil
}
acc.closed = true
close(acc.exit)
acc.stats.inProgress.clear(acc.name)
if acc.close == nil {
return nil
}
return acc.close.Close()
}
// Done with accounting - must be called to free accounting goroutine
func (acc *Account) Done() {
acc.mu.Lock()
defer acc.mu.Unlock()
close(acc.exit)
acc.stats.inProgress.clear(acc.name)
}
// progress returns bytes read as well as the size.
// Size can be <= 0 if the size is unknown.
func (acc *Account) progress() (bytes, size int64) {

View file

@ -32,6 +32,8 @@ func TestNewAccountSizeName(t *testing.T) {
assert.Equal(t, acc, stats.inProgress.get("test"))
err := acc.Close()
assert.NoError(t, err)
assert.Equal(t, acc, stats.inProgress.get("test"))
acc.Done()
assert.Nil(t, stats.inProgress.get("test"))
}
@ -55,18 +57,31 @@ func TestAccountWithBuffer(t *testing.T) {
}
func TestAccountGetUpdateReader(t *testing.T) {
test := func(doClose bool) func(t *testing.T) {
return func(t *testing.T) {
in := ioutil.NopCloser(bytes.NewBuffer([]byte{1}))
stats := NewStats()
acc := newAccountSizeName(stats, in, 1, "test")
assert.Equal(t, in, acc.GetReader())
assert.Equal(t, acc, stats.inProgress.get("test"))
if doClose {
// close the account before swapping it out
require.NoError(t, acc.Close())
}
in2 := ioutil.NopCloser(bytes.NewBuffer([]byte{1}))
acc.UpdateReader(in2)
assert.Equal(t, in2, acc.GetReader())
assert.Equal(t, acc, stats.inProgress.get("test"))
assert.NoError(t, acc.Close())
}
}
t.Run("NoClose", test(false))
t.Run("Close", test(true))
}
func TestAccountRead(t *testing.T) {

View file

@ -138,7 +138,7 @@ func (s *StatsInfo) totalDuration() time.Duration {
var total time.Duration
var i, j = 0, 1
for i < len(timeRanges) {
if j < len(timeRanges)-1 {
if j < len(timeRanges) {
if timeRanges[j].start.Before(timeRanges[i].end) {
if timeRanges[i].end.Before(timeRanges[j].end) {
timeRanges[i].end = timeRanges[j].end

View file

@ -130,10 +130,40 @@ func TestStatsError(t *testing.T) {
}
func TestStatsTotalDuration(t *testing.T) {
time1 := time.Now().Add(-40 * time.Second)
startTime := time.Now()
time1 := startTime.Add(-40 * time.Second)
time2 := time1.Add(10 * time.Second)
time3 := time2.Add(10 * time.Second)
time4 := time3.Add(10 * time.Second)
t.Run("Single completed transfer", func(t *testing.T) {
s := NewStats()
s.AddTransfer(&Transfer{
startedAt: time1,
completedAt: time2,
})
s.mu.Lock()
total := s.totalDuration()
s.mu.Unlock()
assert.Equal(t, 10*time.Second, total)
})
t.Run("Single uncompleted transfer", func(t *testing.T) {
s := NewStats()
s.AddTransfer(&Transfer{
startedAt: time1,
})
s.mu.Lock()
total := s.totalDuration()
s.mu.Unlock()
assert.Equal(t, time.Since(time1)/time.Second, total/time.Second)
})
t.Run("Overlapping without ending", func(t *testing.T) {
s := NewStats()
s.AddTransfer(&Transfer{
startedAt: time2,
@ -161,21 +191,29 @@ func TestStatsTotalDuration(t *testing.T) {
total := s.totalDuration()
s.mu.Unlock()
assert.True(t, 30*time.Second < total && total < 31*time.Second, total)
}
assert.Equal(t, time.Duration(30), total/time.Second)
})
func TestStatsTotalDuration2(t *testing.T) {
time1 := time.Now().Add(-40 * time.Second)
time2 := time1.Add(10 * time.Second)
t.Run("Mixed completed and uncompleted transfers", func(t *testing.T) {
s := NewStats()
s.AddTransfer(&Transfer{
startedAt: time1,
completedAt: time2,
})
s.AddTransfer(&Transfer{
startedAt: time2,
})
s.AddTransfer(&Transfer{
startedAt: time3,
})
s.AddTransfer(&Transfer{
startedAt: time3,
})
s.mu.Lock()
total := s.totalDuration()
s.mu.Unlock()
assert.Equal(t, 10*time.Second, total)
assert.Equal(t, startTime.Sub(time1)/time.Second, total/time.Second)
})
}

View file

@ -48,6 +48,10 @@ type Transfer struct {
checking bool
// Protects all below
//
// NB to avoid deadlocks we must release this lock before
// calling any methods on Transfer.stats. This is because
// StatsInfo calls back into Transfer.
mu sync.RWMutex
acc *Account
err error
@ -79,20 +83,29 @@ func newTransferRemoteSize(stats *StatsInfo, remote string, size int64, checking
// Done ends the transfer.
// Must be called after transfer is finished to run proper cleanups.
func (tr *Transfer) Done(err error) {
tr.mu.Lock()
if err != nil {
tr.stats.Error(err)
tr.mu.Lock()
tr.err = err
tr.mu.Unlock()
}
if tr.acc != nil {
if err := tr.acc.Close(); err != nil {
tr.mu.RLock()
acc := tr.acc
tr.mu.RUnlock()
if acc != nil {
// Close the file if it is still open
if err := acc.Close(); err != nil {
fs.LogLevelPrintf(fs.Config.StatsLogLevel, nil, "can't close account: %+v\n", err)
}
// Signal done with accounting
acc.Done()
}
tr.mu.Lock()
tr.completedAt = time.Now()
tr.mu.Unlock()
if tr.checking {
@ -107,6 +120,8 @@ func (tr *Transfer) Account(in io.ReadCloser) *Account {
tr.mu.Lock()
if tr.acc == nil {
tr.acc = newAccountSizeName(tr.stats, in, tr.size, tr.remote)
} else {
tr.acc.UpdateReader(in)
}
tr.mu.Unlock()
return tr.acc

Some files were not shown because too many files have changed in this diff Show more