Commit graph

803 commits

Author SHA1 Message Date
yuwaMSFT2
0bb696c5bf add test for manifest handler where storage driver fails
Signed-off-by: Yu Wang (UC) <yuwa@microsoft.com>
2017-01-18 11:39:45 -08:00
Yu Wang (UC)
2e94c22fe4 issue#2135 image pull returns 404 on manifest request if there is storage error
When get manifest, the handler will try to retrieve it from storage driver. When storage driver is cloud storage, it can fail due to various reasons even if the manifest exists
(like 500, 503, etc. from storage server). Currently manifest handler blindly return 404 which can be confusing to user.

This change will return 404 if the manifest blob doesn't exist, and return 500 UnknownError for all other errors (consistent with the behavior of other handlers).

Signed-off-by: Yu Wang (UC) <yuwa@microsoft.com>
2017-01-11 12:34:11 -08:00
Stephen J Day
c91a68ca05
registy/handlers: move images to manifests
Once upon a time, we referred to manifests and images interchangably.
That simple past is no more. As we grow, we update our nomenclature and
so follows our code.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-10 16:09:15 -08:00
Ahmet Alp Balkan
0a1ce58e2c
azure: revendor + remove hacky solution in is404
Removing the temporary workaround in is404() method by re-vendoring
the azure-sdk-for-go.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2017-01-09 17:22:28 -08:00
Derek McGowan
2bc4a9459c Merge pull request #2121 from nwt/delete-action
Change DELETE action from "*" to "delete"
2017-01-09 12:34:23 -08:00
Stephen Day
7dba427612 Merge pull request #2111 from stevvooe/use-digest-package
digest: use new go-digest package
2017-01-06 16:25:57 -08:00
Noah Treuhaft
a33af0587b Add test for auth token with "*" action
Test that an auth token with the "*" action is allowed any action on its
resource.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2017-01-06 16:08:32 -08:00
Stephen J Day
532ec9f036
digest: migrate to opencontainers/go-digest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-01-06 15:42:03 -08:00
Derek McGowan
1d6c1a6468
Update logrus vendor
closes #2125

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2017-01-05 11:40:18 -08:00
Noah Treuhaft
ccb839e0e3 Change DELETE action from "*" to "delete"
With token authentication, requiring the "*" action for DELETE requests
makes it impossible to administratively lock a repository against pushes
and pulls but still allow deletion.  This change adds a new "delete"
action for DELETE requests to make that possible.

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2017-01-03 12:27:12 -08:00
yixi zhang
8e915d69f4 Use app.driver.Stat for registry health check
`app.driver.List` on `"/"` is very expensive if registry contains significant amount of images. And the result isn't used anyways.
In most (if not all) storage drivers, `Stat` has a cheaper implementation, so use it instead to achieve the same goal.

Signed-off-by: yixi zhang <yixi@memsql.com>
2016-12-21 17:12:43 -08:00
Stephen J Day
1040c824ba
registry/handlers: fix incorrect use of Digest.Verifiers
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 18:33:26 -08:00
Derek McGowan
729b8c5b91 Merge pull request #2105 from stevvooe/algorithm-own-file
digest: cleanup digester and verifier creation
2016-12-15 16:55:37 -08:00
Stephen J Day
9159833265
digest: remove error return from Digest.Verifier
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 16:43:59 -08:00
Stephen J Day
e37baed88e
digest: cleanup digester and verifier creation
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 15:17:21 -08:00
Stephen J Day
01dfa0fcb9
digest: use digest.Parse over ParseDigest
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-12-15 15:07:42 -08:00
Derek McGowan
8867e8fac3
Update schema2 builder to take media type
Modify manifest builder so it can be used to build
manifests with different configuration media types.
Rename config media type const to image config.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2016-12-14 16:17:20 -08:00
Derek McGowan
844b92879f Merge pull request #2095 from dmcgowan/enforce-class-backward-compatibility
Enforce image is treated as empty class
2016-12-07 14:28:13 -08:00
Derek McGowan
15dc1296af Merge pull request #2088 from ahmetalpbalkan/pr-upstream-azure-race-fix
azure: fix race condition in PutContent()
2016-12-06 14:07:53 -08:00
Derek McGowan
426d0734d4
Enforce image is treated as empty class
Enforces backwards compatibility with older authorization servers
without requiring the client to know about the compatibility
requirements.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-12-05 15:10:29 -08:00
Antonio Murdaca
0fb25dd094
registry/handles/app: always append default urls regexps
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-12-05 20:28:51 +01:00
Derek McGowan
314144ac0b Merge pull request #2070 from kdada/master
filter listResponse.Contents in OSS driver.List()
2016-12-05 10:23:46 -08:00
Ahmet Alp Balkan
78d0660319
azure: fix race condition in PutContent()
See #2077 for background.

The PR #1438 which was not reviewed by azure folks basically introduced
a race condition around uploads to the same blob by multiple clients
concurrently as it used the "writer" type for PutContent(), introduced in #1438.
This does chunked upload of blobs using "AppendBlob" type, which was not atomic.

Usage of "writer" type and thus AppendBlobs on metadata files is currently not
concurrency-safe and generally, they are not the right type of blob for the job.

This patch fixes PutContent() to use the atomic upload operation that works
for uploads smaller than 64 MB and creates blobs with "BlockBlob" type. To be
backwards compatible, we query the type of the blob first and if it is not
a "BlockBlob" we delete the blob first before doing an atomic PUT. This
creates a small inconsistency/race window "only once". Once the blob is made
"BlockBlob", it is overwritten with a single PUT atomicallly next time.

Therefore, going forward, PutContent() will be producing BlockBlobs and it
will silently migrate the AppendBlobs introduced in #1438 to BlockBlobs with
this patch.

Tested with existing code side by side, both registries with and without this
patch work fine without breaking each other. So this should be good from a
backwards/forward compatiblity perspective, with a cost of doing an extra
HEAD checking the blob type.

Fixes #2077.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-30 12:40:43 -08:00
Derek McGowan
e02278f22a
Update registry server to support repository class
Use whitelist of allowed repository classes to enforce.
By default all repository classes are allowed.

Add authorized resources to context after authorization.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-21 16:36:36 -08:00
Derek McGowan
01509db714
Add class to repository scope
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-21 13:32:12 -08:00
Kira
4accc8f2be filter listResponse.Contents in driver.List()
Signed-off-by: Kira <me@imkira.com>
2016-11-17 10:38:56 +08:00
Derek McGowan
8b71d39c34
Remove newlines from end of error strings
Golint now checks for new lines at the end of go error strings,
remove these unneeded new lines.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-14 14:03:01 -08:00
Richard Scothern
6e62b39842 Merge pull request #2036 from pyr/fix/sort-v2-headers
v2 signer: correctly sort headers
2016-11-10 15:31:24 -08:00
Richard Scothern
4d65dd513e Merge pull request #2038 from spacexnice/master
fix: oss driver would get connection reset by peer when upload large image layer.
2016-11-10 14:44:32 -08:00
Derek McGowan
051801f1d0
Update oauth errors to use api errors
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-09 19:29:18 -08:00
Derek McGowan
16396a7a80
Add OAuth error for client
Allow clients to handle errors being set in the WWW-Authenticate
rather than in the body. The WWW-Authenticate errors give a
more precise error describing what is needed to authorize
with the server.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-09 15:27:40 -08:00
Derek McGowan
a1a73884f9
Refactor authorization challenges to its own package
Split challenges into its own package. Avoids possible
import cycle with challenges from client.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-11-09 15:27:40 -08:00
yaoyao.xyy
a4a227e351 oss native large file copy consume too much time which will eventually lead to client timeout because of no data transmit throughout native copy. change maxCopySize to 128MB, ensure only sm all medium size file use oss native copy to avoid connection reset by peer. And fix Move function with CopyLargeFileInParallel to optimize oss upload copy
Signed-off-by: yaoyao.xyy <yaoyao.xyy@alibaba-inc.com>
2016-11-08 12:14:13 +08:00
Derek McGowan
a2611c7520 Merge pull request #2027 from ahmetalpbalkan/pr-azure-memleak2
Update vendored azure-sdk-for-go
2016-11-04 10:08:40 -07:00
Ahmet Alp Balkan
2ab25288a2
Update vendored azure-sdk-for-go
Updating to a recent version of Azure Storage SDK to be
able to patch some memory leaks through configurable HTTP client
changes which were made possible by recent patches to it.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-03 13:24:57 -07:00
Derek McGowan
fbe6e8d212 Merge pull request #2008 from miminar/honor-x-forwarded-port
Honor X-Forwarded-Port header
2016-11-02 12:56:17 -07:00
Pierre-Yves Ritschard
f1cf7de788 fixup! v2 signer: correctly sort headers
Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:07:02 +01:00
Pierre-Yves Ritschard
775cc6d632 v2 signer: correctly sort headers
The current code determines the header order for the
"string-to-sign" payload by sorting on the concatenation
of headers and values, whereas it should only happen on the
key.

During multipart uploads, since `x-amz-copy-source-range` and
`x-amz-copy-source` headers are present, V2 signatures fail to
validate since header order is swapped.

This patch reverts to the expected behavior.

Signed-off-by: Pierre-Yves Ritschard <pyr@spootnik.org>
2016-11-02 17:01:34 +01:00
Michal Minář
1b43e1e30d
Honor X-Forwarded-Port and Forwarded headers
Prefer non-standard headers like X-Forwarded-Proto, X-Forwarded-Host and
X-Forwarded-Port over the standard Forwarded header to maintain
backwards compatibility.

If a port is not specified neither in Host nor in forwarded headers but
it is specified just with X-Forwarded-Port, use its value in base urls
for redirects.

Forwarded header is defined in rfc7239.

X-Forwarded-Port is a non-standard header. Here's a description copied
from "HTTP Headers and Elastic Load Balancing" of AWS ELB docs:

> The X-Forwarded-Port request header helps you identify the port that
> an HTTP or HTTPS load balancer uses to connect to the client.

Signed-off-by: Michal Minář <miminar@redhat.com>
2016-11-02 16:49:42 +01:00
Ahmet Alp Balkan
a994f35657
driver/swift: Fix go vet warning
Driver was passing connections by copying. Storing
`swift.Connection` as pointer to fix the warnings.

Ref: #2030.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-31 11:41:53 -07:00
Ahmet Alp Balkan
6d2a0bafcd
storagedriver/azure: close leaking response body
In GetContent() we read the bytes from a blob but do not close
the underlying response body.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-28 15:13:22 -07:00
Antonio Murdaca
16f086a0ec
registry/client: set Accept: identity header when getting layers
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-10-25 10:33:11 +02:00
Stephen J Day
c9aaff00f8
manifest: references should cover all children
To allow generic manifest walking, we define an interface method of
`References` that returns the referenced items in the manifest. The
current implementation does not return the config target from schema2,
making this useless for most applications.

The garbage collector has been modified to show the utility of this
correctly formed `References` method. We may be able to make more
generic traversal methods with this, as well.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-10-18 11:43:33 -07:00
Richard Scothern
96f63cbe7f Merge pull request #2010 from mbentley/add-new-s3-region
Added new us-east-2 region for S3
2016-10-18 10:25:30 -07:00
Richard Scothern
5bd2cfaddd Merge pull request #2000 from jessfraz/1950-comment-on-non-dryrun-as-well
storage/garbagecollect: add emit comments for when dryRun is false
2016-10-18 10:00:04 -07:00
Matt Bentley
3857f50825
Added new us-east-2 region for S3
Signed-off-by: Matt Bentley <mbentley@mbentley.net>
2016-10-18 12:30:34 -04:00
Richard Scothern
d0cdc4802b Merge pull request #2002 from lnr0626/1996-instance-roles-with-regionendpoint
Allow using ec2 roles when specifying region endpoint
2016-10-17 13:50:02 -07:00
Richard Scothern
a621a86cb4 Fix aliyun OSS Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-10-17 09:43:15 -07:00
Noah Treuhaft
12e73f01d2 Fix s3-goamz Delete method's notion of subpaths
Deleting "/a" was deleting "/a/b" but also "/ab".

Signed-off-by: Noah Treuhaft <noah.treuhaft@docker.com>
2016-10-17 09:43:15 -07:00
Richard Scothern
bd91493195 Satisfy the latest go lint rules
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-10-14 17:03:08 -07:00