Richard Scothern
352924cd85
Clarify API documentation around catalog fetch behavior
...
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-06-08 10:39:17 -07:00
Richard Scothern
79102ecf85
Merge pull request #1772 from runcom/go1.6const
...
registry: use const for status code 429
2016-06-08 10:22:57 -07:00
Antonio Murdaca
4901f1b278
registry: use const for status code 429
...
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-06-08 10:19:15 +02:00
allencloud
db90724ab0
fix typos
...
Signed-off-by: allencloud <allen.sun@daocloud.io>
2016-06-02 23:03:27 +08:00
Richard Scothern
df2184c810
Merge pull request #1627 from luckyraul/swift_auth_url
...
Swift auth version param
2016-06-01 11:23:23 -07:00
Richard Scothern
641f102967
Merge pull request #1687 from RichardScothern/signature-store
...
Remove signature store from registry.
2016-05-31 09:09:39 -07:00
Richard Scothern
0c15ab6952
Remove signature store from registry. Return a generated signature for manifest
...
pull.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-27 13:19:26 -07:00
Richard Scothern
a5f72a6032
Merge pull request #1688 from tt/pass-through-known-errors
...
Pass through known errors
2016-05-27 11:35:34 -07:00
Richard Scothern
596ca8b86a
Merge pull request #1744 from tonyhb/add-config-to-middleware
...
Pass in `app` as context to apply{N}Middleware
2016-05-25 14:08:09 -07:00
Tony Holdstock-Brown
166c4a957f
Pass in app
as context to apply{N}Middleware
...
This lets us access registry config within middleware for additional
configuration of whatever it is that you're overriding.
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-24 11:08:57 -07:00
Richard Scothern
32e9779faf
Merge pull request #1734 from legionus/blob-access-controller
...
Add support for blobAccessController middleware
2016-05-24 10:34:36 -07:00
John Starks
f0052b8434
Add support for layers from foreign sources
...
This will be used to support downloading Windows base layers from
Microsoft URLs.
Signed-off-by: John Starks <jostarks@microsoft.com>
2016-05-20 12:31:10 -07:00
Alexey Gladkov
f97eca5ad6
Add support for blobAccessController middleware
...
Signed-off-by: Michal Minar <miminar@redhat.com>
Signed-off-by: Alexey Gladkov <gladkov.alexey@gmail.com>
2016-05-19 14:02:15 +02:00
Richard Scothern
4f2ee029a2
Add 'us-gov-west-1' to the valid region list.
...
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-09 16:38:16 +01:00
Arthur Baars
eca581cf36
StorageDriver: GCS: allow Cancel on a closed FileWriter
...
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
0490ff450b
Blobwriter: call BlobWriter.Size after BlobWriter.Close
...
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Arthur Baars
1d782c38f2
StorageDriver: Test case for #1698
...
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-06 13:04:30 +01:00
Richard Scothern
c047d34b22
Merge pull request #1695 from tonyhb/add-regulator-to-filesystem
...
Add regulator to filesystem
2016-05-04 10:05:51 -07:00
Tony Holdstock-Brown
c9c62380ff
Don't wrap thead limits when using a negative int
...
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 16:03:44 -07:00
Antonio Murdaca
f60f275c29
registry: do not use http.StatusTooManyRequests
...
go1.5 doesn't export http.StatusTooManyRequests while
go1.6 does. Fix this by hardcoding the status code for now.
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-05-03 21:24:45 +02:00
Antonio Murdaca
a264f9ae29
registry: type too many requests error
...
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-05-03 18:41:54 +02:00
Tony Holdstock-Brown
33c448f147
Implement regulator in filesystem driver
...
This commit refactors base.regulator into the 2.4 interfaces and adds a
filesystem configuration option `maxthreads` to configure the regulator.
By default `maxthreads` is set to 100. This means the FS driver is
limited to 100 concurrent blocking file operations. Any subsequent
operations will block in Go until previous filesystem operations
complete.
This ensures that the registry can never open thousands of simultaneous
threads from os filesystem operations.
Note that `maxthreads` can never be less than 25.
Add test case covering parsable string maxthreads
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 09:33:22 -07:00
Richard Scothern
5d08dfa70c
Merge pull request #1650 from majewsky/swift/wait-for-dlo-segments
...
[Swift] wait for DLO segments to show up when Close()ing the writer
2016-05-02 13:41:26 -07:00
Derek McGowan
a1d7463d67
Merge pull request #1669 from RichardScothern/close-after-commit
...
Clean uploads
2016-04-29 16:18:29 -07:00
Troels Thomsen
3730470b64
Pass through known errors
...
Signed-off-by: Troels Thomsen <troels@thomsen.io>
2016-04-29 23:34:24 +02:00
Richard Scothern
ba927007b0
Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error
...
Move GC into storage package and add tests
2016-04-28 14:09:58 -07:00
Richard Scothern
a7dda2ce93
Merge pull request #1665 from andrewhsu/middleware-redirect
...
add middleware storage driver for redirect
2016-04-27 15:05:52 -07:00
Richard Scothern
d2e29acce0
When a blob upload is committed prevent writing out hashstate in the
...
subsequent close.
When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 14:42:00 -07:00
Richard Scothern
69ba30dc03
Add a test with a missing _manifests directory
...
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Richard Scothern
ea492aca1a
Move garbage collect code into storage package
...
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Tony Holdstock-Brown
a5aaae1f06
Ensure GC continues marking if _manifests is nonexistent
...
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-27 13:33:36 -07:00
Serge Dubrouski
32193bdcf0
Fix wording for dry-run flag in useage message for garbage collector.
...
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-04-26 19:44:23 -06:00
Josh Hawn
e4dd3359cc
Regulate filesystem driver to max of 100 calls
...
It's easily possible for a flood of requests to trigger thousands of
concurrent file accesses on the storage driver. Each file I/O call creates
a new OS thread that is not reaped by the Golang runtime. By limiting it
to only 100 at a time we can effectively bound the number of OS threads
in use by the storage driver.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-26 14:44:13 -07:00
Andrew Hsu
09a9b0cf90
separate the go/non-go imports and reorder
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-26 14:33:54 -07:00
Richard Scothern
c83afea0c9
Merge pull request #1660 from jhaohai/cn-north-1-fix
...
Add cn-north-1 to valid check
2016-04-25 16:07:54 -07:00
Richard Scothern
ef32134592
Merge pull request #1666 from sergeyfd/master
...
Add blobWrtiter.Close() call into blobWriter.Commit()
2016-04-25 16:02:48 -07:00
Andrew Hsu
c4df027d41
modify redirect test to include port
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:46 -07:00
Andrew Hsu
80248c3d3a
scheme and host mandatory in baseurl
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:25 -07:00
Andrew Hsu
059bc5f5ef
separate the go/non-go imports and reorder
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:03 -07:00
Anis Elleuch
987faca8a6
Sorting completed parts by part number for a better accordance with the S3 spec
...
Signed-off-by: Anis Elleuch <vadmeste@gmail.com>
2016-04-23 22:36:04 +01:00
Serge Dubrouski
21f38a74e6
Add blobWrtiter.Close() call into blobWriter.Commit()
...
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-04-22 19:23:17 -06:00
Andrew Hsu
4b217ccbf5
add middleware storage driver for redirect
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-21 16:02:52 -07:00
jhaohai
f76c622d8c
add cn-north-1 to valid check
...
Signed-off-by: jhaohai <jhaohai@foxmail.com>
2016-04-21 11:51:34 +08:00
Stefan Majewsky
9a67520af7
wait for DLO segments to show up when Close()ing the writer
...
Not just when Commit()ing the result. This fixes some errors I observed
when the layer (i.e. the DLO) is Stat()ed immediately after closing,
and reports the wrong file size because the container listing is not
yet up-to-date.
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-04-19 14:31:49 +02:00
Nikita Tarasov
b4f060599a
docs + fix test
...
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-17 20:05:51 +03:00
Nikita Tarasov
b51607f9f0
fix test
...
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-17 19:52:40 +03:00
Nikita
bcb7989fca
test
...
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-13 18:49:38 +03:00
Nikita
e6f8d7c28e
Update swift.go
...
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-13 18:49:25 +03:00
Richard Scothern
4c119524f1
Merge pull request #1604 from ArdaXi/custom-s3-skip-region-check
...
Only check validity of S3 region if not using custom endpoint
2016-04-08 15:38:24 -07:00
Richard Scothern
437eeeda44
Merge pull request #1605 from majewsky/swift/deal-with-outdated-container-listings
...
registry/storage/swift: detect and fix outdated container listings
2016-04-08 15:38:06 -07:00
Stefan Majewsky
67321cb622
detect outdated container listings during Stat() and getAllSegments()
...
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-04-06 15:21:27 +02:00
Arien Holthuizen
dbb6e28da2
Only check validity of S3 region if not using custom endpoint
...
Signed-off-by: Arien Holthuizen <aholthuizen@schubergphilis.com>
2016-04-06 13:38:09 +02:00
Tony Holdstock-Brown
25c5efdef9
Ensure we log io.Copy errors and bytes copied/total in uploads
...
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-04 17:21:36 -07:00
Richard Scothern
04ff3c0359
Merge pull request #1578 from majewsky/do-not-swallow-error
...
don't swallow errors in Swift driver's GetContent()
2016-03-30 17:46:27 -07:00
Richard Scothern
eb0b7f0173
Update the gc documentation.
...
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:57:21 -07:00
Richard Scothern
4324b70c50
Fix signature handling with GC.
...
If a schema 1 manifest is uploaded with the `disablesignaturestore` option set
to true, then no signatures will exist. Handle this case.
If a schema 1 manifest is pushed, deleted, garbage collected and pushed again, the
repository will contain signature links from the first version, but the blobs will
not exist. Disable the signature store in the garbage-collect command so
signatures are not fetched.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-30 16:10:27 -07:00
Richard Scothern
80b310ca44
Add a --dry-run flag. If enabled this will print the mark and sweep process
...
with removing any files.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:10:27 -07:00
Aaron Lehmann
267e268775
garbagecollect: Clean up errors
...
- Clean up error messages
- Add a missing error check on the result of blobService.Enumerate.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-30 11:35:24 -07:00
Stefan Majewsky
4c9bed2507
don't swallow errors in Swift driver's GetContent()
...
In 326c3a9c49
, which was only intended to
be a refactoring commit, the behavior of this block subtly changed so
that unknown types of errors would be swallowed instead of propagated.
I noticed this while investigating an error similar to #1539 aka
docker/docker#21290 . It appears that during GetContent() for a
hashstate, the Swift proxy produces an error. Since this error was
silently swallowed, an empty []byte is used to restart the hash, then
producing the digest of the empty string instead of the layer's digest.
This PR will not fix the issue, but it should make the actual error more
visible by propagating it into `blobWriter#resumeDigest' and
'blobWriter#validateBlob', respectively.
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-03-30 16:11:29 +02:00
Stephen Day
0eefe17730
Merge pull request #1546 from arschles/doc-storage
...
Add documentation for how to register new StorageDrivers
2016-03-29 15:58:51 -07:00
Aaron Schlesinger
f4bdc6287a
Remove the example
...
Instead, direct users to the one in the factory package
Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:28 -07:00
Aaron Schlesinger
204ad474e4
Add documentation for how to register new StorageDrivers
...
This commit adds context-specific documentation on StorageDriver,
StorageDriverFactory, and the factory’s Register func, explaining how
the internal registration mechanism should be used.
This documentation follows from the thread starting at
https://github.com/deis/builder/pull/262/files#r56720200 .
cc/ @stevvooe
Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:19 -07:00
Tony Holdstock-Brown
31d5dd5175
utulize config log format within gc
...
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-03-24 14:29:43 -07:00
Richard Scothern
87a997249d
Merge pull request #1522 from RichardScothern/tag-events
...
Send tag events to notification listeners
2016-03-23 16:41:29 -07:00
Stephen Day
9e690c7fa2
Merge pull request #1491 from RichardScothern/relative-url
...
Return relative URLs
2016-03-23 16:19:39 -07:00
Richard Scothern
bc9c820e4b
Enable URLs returned from the registry to be configured as relative.
...
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-23 15:14:32 -07:00
Richard Scothern
afe2bdd1c5
Propogate tag as a functional argument into the notification system to attach
...
tags to manifest push and pull event notifications.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-23 14:57:52 -07:00
姜继忠
92a9ee34d4
fix manifest revision search, closes #1535
...
Signed-off-by: 姜继忠 <jizhong.jiangjz@alibaba-inc.com>
2016-03-19 19:11:25 +08:00
Antonio Murdaca
ea148448a2
registry: client: auth: type errors
...
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-03-18 09:34:50 +01:00
Richard Scothern
ec6ac0c05e
Merge pull request #1531 from aaronlehmann/empty-errors-slice
...
Don't return empty errcode.Errors slices
2016-03-17 10:46:51 -07:00
Antonio Murdaca
75b5a1b1f6
registry: client: repository: close response body
...
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-03-16 19:53:17 +01:00
Richard Scothern
2c635d1f7e
Merge pull request #1532 from RichardScothern/azure-error-types
...
Update missing blob error checking with latest Azure API
2016-03-15 10:08:36 -07:00
Aaron Lehmann
3a2231fe39
Include status code in UnexpectedHTTPResponseError
...
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-15 09:03:56 -07:00
Richard Scothern
a5d372535e
Merge pull request #1502 from RichardScothern/auth-url-endpoints
...
URL parse auth endpoints to normalize hostname to lowercase.
2016-03-14 16:28:24 -07:00
Richard Scothern
772e6f4057
Update missing blob error checking with latest Azure API
...
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-14 15:59:03 -07:00
Aaron Lehmann
cd6482ecb8
Don't return empty errcode.Errors slices
...
If this slice ends up empty after parsing the HTTP response body, it
means the body is not well-formed. We've probably encountered an error
message produced by something that uses a different JSON schema, or
an error that just happens to validate as JSON.
An empty errcode.Errors slice is not a very useful thing to return,
since its Error() output is just `<nil>`. Detect this case, and instend
return an UnexpectedHTTPResponseError.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-14 10:27:01 -07:00
Matt Duch
fcb247dfce
registry/storage/driver/s3-aws kms support
...
Signed-off-by: Matt Duch <matt@learnmetrics.com>
2016-03-11 17:19:01 -06:00
Brian Bland
c03b5fc5ee
Merge pull request #1438 from BrianBland/newStorageDriverWriter
...
Adds new StorageDriver.FileWriter interface
2016-03-11 15:06:07 -08:00
Brian Bland
5967d33342
Removes ceph rados driver in favor of Swift API gateway support
...
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-10 16:49:08 -08:00
Brian Bland
ab49f85db8
Merge pull request #1512 from kmala/master
...
Added support to specifiy custom endpoint
2016-03-10 13:48:59 -08:00
Keerthan Mala
2be1b4ef4f
Added support to specifiy custom endpoint
...
Signed-off-by: Keerthan Reddy Mala <keerthan.mala@gmail.com>
2016-03-09 16:12:20 -07:00
Derek McGowan
a59448784b
Add client ID to token fetch to GET endpoint
...
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-09 12:44:55 -08:00
Brian Bland
7fd1db9312
Updates Swift driver to support new storagedriver.FileWriter interface
...
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:38:39 -08:00
Li Yi
a9bf7a2aae
Support FileWriter interface for OSS storage driver
...
Change-Id: Ie5533ad85f944800499ca1040fd67bf1378815e0
Signed-off-by: Li Yi <denverdino@gmail.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
307504713f
Storagedriver: GCS: add chunksize parameter
...
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
7162cb19c6
Storagedriver: GCS: implement resumable uploads
...
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:39 -08:00
Arthur Baars
666273d9f6
StorageDriver: Testsuite: call Close before getting Size
...
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-08 16:38:38 -08:00
Brian Bland
ff03381d49
Adds new storagedriver.FileWriter interface
...
Updates registry storage code to use this for better resumable writes.
Implements this interface for the following drivers:
+ Inmemory
+ Filesystem
+ S3
+ Azure
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:37:44 -08:00
Aaron Lehmann
87e34bd307
Fix two misspellings in source code comments
...
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-08 15:13:24 -08:00
Richard Scothern
8bd613aa11
URL parse auth endpoints to normalize hostname to lowercase.
...
It is possible for a middlebox to lowercase the URL at somepoint causing a
lookup in the auth challenges table to fail. Lowercase hostname before
using as keys to challenge map.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-07 14:08:21 -08:00
Brian Bland
302cc39937
[driver/s3aws] Update s3aws driver parameter parsing to match s3goamz
...
Mirrors changes from #1414 into the newer driver
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-07 11:52:34 -08:00
Derek McGowan
a8aa2f78de
Fix oauth cross repository push
...
Cross repository push tokens were not being cached and could not be used,
now any returned token will be used and the caching is hidden in the getToken function.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 15:13:27 -08:00
Derek McGowan
efd7ea4624
Add offline token option
...
Login needs to add an offline token flag to ensure a refresh token is returned by the token endpoint.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 14:32:51 -08:00
Derek McGowan
1126e32234
Add post token implementation
...
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 13:53:06 -08:00
Derek McGowan
2ef7a872de
Add options struct to initialize handler
...
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 11:32:48 -08:00
Derek McGowan
c9880e6b05
Fetch token by credentials and refresh token
...
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-03-04 00:34:17 -08:00
Richard Scothern
62d46ef07d
Merge pull request #1388 from aibaars/gcs-simplify-move
...
StorageDriver: GCS: remove support for directory Moves
2016-03-03 10:20:51 -08:00
Arthur Baars
1593c2413f
StorageDriver: GCS: remove support for directory Moves
...
The Move operation is only used to move uploaded blobs
to their final destination. There is no point in implementing
Move on "folders". Apart from simplifying the code, this also
saves an HTTP request.
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-03 13:28:13 +00:00
Richard Scothern
c8dff1bb57
Merge pull request #1364 from aaronlehmann/content-type-in-api-docs
...
Add information about manifest content types to API spec
2016-03-01 13:53:49 -08:00
Richard Scothern
5cbac00dd4
Merge pull request #1386 from andrewnguyen/garbage_collect_pick_from_1050
...
garbage collection
2016-02-29 15:01:48 -08:00