Commit graph

667 commits

Author SHA1 Message Date
Stephen J Day
2aa09ff9a8 registry/storage: more efficient path compare in catalog
Previous component-wise path comparison is recursive and generates a
large amount of garbage. This more efficient version simply replaces the
path comparison with the zero-value to sort before everything. We do
this by replacing the byte-wise comparison that swaps a single character
inline for the separator comparison, such that separators sort first.

The resulting implementation provides component-wise path comparison
with no cost incurred for allocation or stack frame.

Direction of the comparison is also reversed to match Go style.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2016-08-26 11:08:59 -07:00
Edgar Lee
fdc51bb1f2 Stop ErrFinishedWalk from escaping from Repositories walk
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-26 11:08:41 -07:00
Sebastien Coavoux
0567fa3c2a Fix: Compare path properly when list repository in catalog. #1854
Signed-off-by: Sebastien Coavoux <alignak@pyseb.cx>
2016-08-26 11:07:41 -07:00
Edgar Lee
22a59f2512 Refactor errVal named parameter for catalog repositories to err
Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-26 11:07:32 -07:00
Edgar Lee
734caef0f4 Fix storage drivers dropping non EOF errors when listing repositories
This fixes errors other than io.EOF from being dropped when a storage driver
lists repositories. For example, filesystem driver may point to a missing
directory and errors, which then gets subsequently dropped.

Signed-off-by: Edgar Lee <edgar.lee@docker.com>
2016-08-26 11:07:24 -07:00
Richard Scothern
edd7cb5249 Merge pull request #1739 from cezarsa/master
[Swift] Expose EndpointType parameter in driver
2016-06-15 10:33:48 -07:00
Richard Scothern
ccfa25cf00 If resumable digest support is disabled, detct this when closing the blobwriter
and allow the close to continue.  Also update the name of the function.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-13 17:35:06 -07:00
Richard Scothern
1fc752c718 Merge pull request #1706 from aibaars/registry-size-close
Blobwriter: call BlobWriter.Size after BlobWriter.Close
2016-06-13 16:29:35 -07:00
Cezar Sa Espinola
7f72092940
Expose EndpointType parameter in swift storage driver
Signed-off-by: Cezar Sa Espinola <cezarsa@gmail.com>
2016-06-13 19:28:45 -03:00
Richard Scothern
fb106e167a Merge pull request #1465 from dmcgowan/token-server-oauth
Integration token server supporting oauth
2016-06-13 15:01:06 -07:00
Richard Scothern
4e08e7d1d6 Merge pull request #1775 from dmcgowan/get-content-digest
Add option to get content digest from manifest get
2016-06-13 14:59:05 -07:00
Richard Scothern
1bf52359f1 Merge pull request #1782 from tianon/accept-lists
Update "Accept" header parsing for list values
2016-06-13 14:33:46 -07:00
Aaron Lehmann
9198d642ba Merge pull request #1779 from dmcgowan/letsencrypt-support
Let's Encrypt support
2016-06-13 10:48:55 -10:00
Derek McGowan
be2ed961aa Add support for Let's Encrypt
Add configuration and certificate manager to use letsencrypt

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-06-13 11:30:42 -07:00
Tianon Gravi
8907f7d189 Update "Accept" header parsing for list values
In Go's header parsing, the same header multiple times results in multiple entries in the `r.Header[...]` slice, but Go does no further parsing beyond that (and in https://golang.org/cl/4528086 it was determined that until/unless the stdlib itself needs it, Go will not do so).

The consequence here for parsing of `Accept:` headers is that we support the way Go outputs headers, but not all language HTTP libraries have a facility to output multiple headers instead of a single list header.

This change ensures that the following (valid) header blocks all parse to the same result for the purposes of what is being tested here:

```
Accept: a/b
Accept: b/c
Accept: d/e
```

```
Accept: a/b; q=0.5, b/c
Accept: d/e
```

```
Accept: a/b; q=0.1, b/c; q=0.2, d/e; q=0.8
```

Signed-off-by: Andrew "Tianon" Page <admwiggin@gmail.com>
2016-06-10 16:52:27 -07:00
Derek McGowan
125f4ff7d7 Add option to get content digest from manifest get
The client may need the content digest to delete a manifest using the digest used by the registry.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-06-08 17:02:29 -07:00
Richard Scothern
75882f079c Merge pull request #1774 from RichardScothern/catalog-clarifcation
Clarify API documentation around catalog fetch behavior
2016-06-08 12:59:09 -07:00
Richard Scothern
352924cd85 Clarify API documentation around catalog fetch behavior
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-08 10:39:17 -07:00
Richard Scothern
79102ecf85 Merge pull request #1772 from runcom/go1.6const
registry: use const for status code 429
2016-06-08 10:22:57 -07:00
Antonio Murdaca
4901f1b278 registry: use const for status code 429
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-06-08 10:19:15 +02:00
allencloud
db90724ab0 fix typos
Signed-off-by: allencloud <allen.sun@daocloud.io>
2016-06-02 23:03:27 +08:00
Richard Scothern
df2184c810 Merge pull request #1627 from luckyraul/swift_auth_url
Swift auth version param
2016-06-01 11:23:23 -07:00
Richard Scothern
641f102967 Merge pull request #1687 from RichardScothern/signature-store
Remove signature store from registry.
2016-05-31 09:09:39 -07:00
Richard Scothern
0c15ab6952 Remove signature store from registry. Return a generated signature for manifest
pull.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-27 13:19:26 -07:00
Richard Scothern
a5f72a6032 Merge pull request #1688 from tt/pass-through-known-errors
Pass through known errors
2016-05-27 11:35:34 -07:00
Richard Scothern
596ca8b86a Merge pull request #1744 from tonyhb/add-config-to-middleware
Pass in `app` as context to apply{N}Middleware
2016-05-25 14:08:09 -07:00
Tony Holdstock-Brown
166c4a957f Pass in app as context to apply{N}Middleware
This lets us access registry config within middleware for additional
configuration of whatever it is that you're overriding.

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-24 11:08:57 -07:00
Richard Scothern
32e9779faf Merge pull request #1734 from legionus/blob-access-controller
Add support for  blobAccessController middleware
2016-05-24 10:34:36 -07:00
John Starks
f0052b8434 Add support for layers from foreign sources
This will be used to support downloading Windows base layers from
Microsoft URLs.

Signed-off-by: John Starks <jostarks@microsoft.com>
2016-05-20 12:31:10 -07:00
Alexey Gladkov
f97eca5ad6 Add support for blobAccessController middleware
Signed-off-by: Michal Minar <miminar@redhat.com>
Signed-off-by: Alexey Gladkov <gladkov.alexey@gmail.com>
2016-05-19 14:02:15 +02:00
Richard Scothern
4f2ee029a2 Add 'us-gov-west-1' to the valid region list.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-09 16:38:16 +01:00
Arthur Baars
eca581cf36 StorageDriver: GCS: allow Cancel on a closed FileWriter
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
0490ff450b Blobwriter: call BlobWriter.Size after BlobWriter.Close
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
1d782c38f2 StorageDriver: Test case for #1698
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Richard Scothern
c047d34b22 Merge pull request #1695 from tonyhb/add-regulator-to-filesystem
Add regulator to filesystem
2016-05-04 10:05:51 -07:00
Tony Holdstock-Brown
c9c62380ff Don't wrap thead limits when using a negative int
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 16:03:44 -07:00
Antonio Murdaca
f60f275c29 registry: do not use http.StatusTooManyRequests
go1.5 doesn't export http.StatusTooManyRequests while
go1.6 does. Fix this by hardcoding the status code for now.

Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-05-03 21:24:45 +02:00
Antonio Murdaca
a264f9ae29 registry: type too many requests error
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-05-03 18:41:54 +02:00
Tony Holdstock-Brown
33c448f147 Implement regulator in filesystem driver
This commit refactors base.regulator into the 2.4 interfaces and adds a
filesystem configuration option `maxthreads` to configure the regulator.

By default `maxthreads` is set to 100. This means the FS driver is
limited to 100 concurrent blocking file operations. Any subsequent
operations will block in Go until previous filesystem operations
complete.

This ensures that the registry can never open thousands of simultaneous
threads from os filesystem operations.

Note that `maxthreads` can never be less than 25.

Add test case covering parsable string maxthreads

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 09:33:22 -07:00
Richard Scothern
5d08dfa70c Merge pull request #1650 from majewsky/swift/wait-for-dlo-segments
[Swift] wait for DLO segments to show up when Close()ing the writer
2016-05-02 13:41:26 -07:00
Derek McGowan
a1d7463d67 Merge pull request #1669 from RichardScothern/close-after-commit
Clean uploads
2016-04-29 16:18:29 -07:00
Troels Thomsen
3730470b64 Pass through known errors
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2016-04-29 23:34:24 +02:00
Richard Scothern
ba927007b0 Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error
Move GC into storage package and add tests
2016-04-28 14:09:58 -07:00
Richard Scothern
a7dda2ce93 Merge pull request #1665 from andrewhsu/middleware-redirect
add middleware storage driver for redirect
2016-04-27 15:05:52 -07:00
Richard Scothern
d2e29acce0 When a blob upload is committed prevent writing out hashstate in the
subsequent close.

When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 14:42:00 -07:00
Richard Scothern
69ba30dc03 Add a test with a missing _manifests directory
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Richard Scothern
ea492aca1a Move garbage collect code into storage package
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Tony Holdstock-Brown
a5aaae1f06 Ensure GC continues marking if _manifests is nonexistent
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-27 13:33:36 -07:00
Serge Dubrouski
32193bdcf0 Fix wording for dry-run flag in useage message for garbage collector.
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-04-26 19:44:23 -06:00
Josh Hawn
e4dd3359cc Regulate filesystem driver to max of 100 calls
It's easily possible for a flood of requests to trigger thousands of
concurrent file accesses on the storage driver. Each file I/O call creates
a new OS thread that is not reaped by the Golang runtime. By limiting it
to only 100 at a time we can effectively bound the number of OS threads
in use by the storage driver.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-26 14:44:13 -07:00