TrueCloudLab
/
distribution
13
0
Fork 3
Code Issues 2 Pull Requests Packages Projects Releases 1 Wiki Activity
2,097 Commits
2 Branches
59 Tags
134 MiB
Commit Graph

700 Commits (bba5a0d05c27328d7a75a16e8082f9b16b9721f0)

Author SHA1 Message Date
Richard Scothern 352924cd85 Clarify API documentation around catalog fetch behavior 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-06-08 10:39:17 -07:00
Richard Scothern 79102ecf85 Merge pull request #1772 from runcom/go1.6const 
registry: use const for status code 429
 2016-06-08 10:22:57 -07:00
Antonio Murdaca 4901f1b278 registry: use const for status code 429 
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
 2016-06-08 10:19:15 +02:00
allencloud db90724ab0 fix typos 
Signed-off-by: allencloud <allen.sun@daocloud.io>
 2016-06-02 23:03:27 +08:00
Richard Scothern df2184c810 Merge pull request #1627 from luckyraul/swift_auth_url 
Swift auth version param
 2016-06-01 11:23:23 -07:00
Richard Scothern 641f102967 Merge pull request #1687 from RichardScothern/signature-store 
Remove signature store from registry.
 2016-05-31 09:09:39 -07:00
Richard Scothern 0c15ab6952 Remove signature store from registry. Return a generated signature for manifest 
pull.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-05-27 13:19:26 -07:00
Richard Scothern a5f72a6032 Merge pull request #1688 from tt/pass-through-known-errors 
Pass through known errors
 2016-05-27 11:35:34 -07:00
Richard Scothern 596ca8b86a Merge pull request #1744 from tonyhb/add-config-to-middleware 
Pass in `app` as context to apply{N}Middleware
 2016-05-25 14:08:09 -07:00
Tony Holdstock-Brown 166c4a957f Pass in `app` as context to apply{N}Middleware 
This lets us access registry config within middleware for additional
configuration of whatever it is that you're overriding.

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-05-24 11:08:57 -07:00
Richard Scothern 32e9779faf Merge pull request #1734 from legionus/blob-access-controller 
Add support for  blobAccessController middleware
 2016-05-24 10:34:36 -07:00
John Starks f0052b8434 Add support for layers from foreign sources 
This will be used to support downloading Windows base layers from
Microsoft URLs.

Signed-off-by: John Starks <jostarks@microsoft.com>
 2016-05-20 12:31:10 -07:00
Alexey Gladkov f97eca5ad6 Add support for blobAccessController middleware 
Signed-off-by: Michal Minar <miminar@redhat.com>
Signed-off-by: Alexey Gladkov <gladkov.alexey@gmail.com>
 2016-05-19 14:02:15 +02:00
Richard Scothern 4f2ee029a2 Add 'us-gov-west-1' to the valid region list. 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-05-09 16:38:16 +01:00
Arthur Baars eca581cf36 StorageDriver: GCS: allow Cancel on a closed FileWriter 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-05-06 13:04:30 +01:00
Arthur Baars 0490ff450b Blobwriter: call BlobWriter.Size after BlobWriter.Close 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-05-06 13:04:30 +01:00
Arthur Baars 1d782c38f2 StorageDriver: Test case for #1698 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-05-06 13:04:30 +01:00
Richard Scothern c047d34b22 Merge pull request #1695 from tonyhb/add-regulator-to-filesystem 
Add regulator to filesystem
 2016-05-04 10:05:51 -07:00
Tony Holdstock-Brown c9c62380ff Don't wrap thead limits when using a negative int 
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-05-03 16:03:44 -07:00
Antonio Murdaca f60f275c29 registry: do not use http.StatusTooManyRequests 
go1.5 doesn't export http.StatusTooManyRequests while
go1.6 does. Fix this by hardcoding the status code for now.

Signed-off-by: Antonio Murdaca <runcom@redhat.com>
 2016-05-03 21:24:45 +02:00
Antonio Murdaca a264f9ae29 registry: type too many requests error 
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
 2016-05-03 18:41:54 +02:00
Tony Holdstock-Brown 33c448f147 Implement regulator in filesystem driver 
This commit refactors base.regulator into the 2.4 interfaces and adds a
filesystem configuration option `maxthreads` to configure the regulator.

By default `maxthreads` is set to 100. This means the FS driver is
limited to 100 concurrent blocking file operations. Any subsequent
operations will block in Go until previous filesystem operations
complete.

This ensures that the registry can never open thousands of simultaneous
threads from os filesystem operations.

Note that `maxthreads` can never be less than 25.

Add test case covering parsable string maxthreads

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-05-03 09:33:22 -07:00
Richard Scothern 5d08dfa70c Merge pull request #1650 from majewsky/swift/wait-for-dlo-segments 
[Swift] wait for DLO segments to show up when Close()ing the writer
 2016-05-02 13:41:26 -07:00
Derek McGowan a1d7463d67 Merge pull request #1669 from RichardScothern/close-after-commit 
Clean uploads
 2016-04-29 16:18:29 -07:00
Troels Thomsen 3730470b64 Pass through known errors 
Signed-off-by: Troels Thomsen <troels@thomsen.io>
 2016-04-29 23:34:24 +02:00
Richard Scothern ba927007b0 Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error 
Move GC into storage package and add tests
 2016-04-28 14:09:58 -07:00
Richard Scothern a7dda2ce93 Merge pull request #1665 from andrewhsu/middleware-redirect 
add middleware storage driver for redirect
 2016-04-27 15:05:52 -07:00
Richard Scothern d2e29acce0 When a blob upload is committed prevent writing out hashstate in the 
subsequent close.

When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-04-27 14:42:00 -07:00
Richard Scothern 69ba30dc03 Add a test with a missing _manifests directory 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-04-27 13:34:25 -07:00
Richard Scothern ea492aca1a Move garbage collect code into storage package 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-04-27 13:34:25 -07:00
Tony Holdstock-Brown a5aaae1f06 Ensure GC continues marking if _manifests is nonexistent 
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-04-27 13:33:36 -07:00
Serge Dubrouski 32193bdcf0 Fix wording for dry-run flag in useage message for garbage collector. 
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
 2016-04-26 19:44:23 -06:00
Josh Hawn e4dd3359cc Regulate filesystem driver to max of 100 calls 
It's easily possible for a flood of requests to trigger thousands of
concurrent file accesses on the storage driver. Each file I/O call creates
a new OS thread that is not reaped by the Golang runtime. By limiting it
to only 100 at a time we can effectively bound the number of OS threads
in use by the storage driver.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-04-26 14:44:13 -07:00
Andrew Hsu 09a9b0cf90 separate the go/non-go imports and reorder 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-26 14:33:54 -07:00
Richard Scothern c83afea0c9 Merge pull request #1660 from jhaohai/cn-north-1-fix 
Add cn-north-1 to valid check
 2016-04-25 16:07:54 -07:00
Richard Scothern ef32134592 Merge pull request #1666 from sergeyfd/master 
Add blobWrtiter.Close() call into blobWriter.Commit()
 2016-04-25 16:02:48 -07:00
Andrew Hsu c4df027d41 modify redirect test to include port 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-25 11:52:46 -07:00
Andrew Hsu 80248c3d3a scheme and host mandatory in baseurl 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-25 11:52:25 -07:00
Andrew Hsu 059bc5f5ef separate the go/non-go imports and reorder 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-25 11:52:03 -07:00
Anis Elleuch 987faca8a6 Sorting completed parts by part number for a better accordance with the S3 spec 
Signed-off-by: Anis Elleuch <vadmeste@gmail.com>
 2016-04-23 22:36:04 +01:00
Serge Dubrouski 21f38a74e6 Add blobWrtiter.Close() call into blobWriter.Commit() 
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
 2016-04-22 19:23:17 -06:00
Andrew Hsu 4b217ccbf5 add middleware storage driver for redirect 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-21 16:02:52 -07:00
jhaohai f76c622d8c add cn-north-1 to valid check 
Signed-off-by: jhaohai <jhaohai@foxmail.com>
 2016-04-21 11:51:34 +08:00
Stefan Majewsky 9a67520af7 wait for DLO segments to show up when Close()ing the writer 
Not just when Commit()ing the result. This fixes some errors I observed
when the layer (i.e. the DLO) is Stat()ed immediately after closing,
and reports the wrong file size because the container listing is not
yet up-to-date.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
 2016-04-19 14:31:49 +02:00
Nikita Tarasov b4f060599a docs + fix test 
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
 2016-04-17 20:05:51 +03:00
Nikita Tarasov b51607f9f0 fix test 
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>

Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
 2016-04-17 19:52:40 +03:00
Nikita bcb7989fca test 
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
 2016-04-13 18:49:38 +03:00
Nikita e6f8d7c28e Update swift.go 
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
 2016-04-13 18:49:25 +03:00
Richard Scothern 4c119524f1 Merge pull request #1604 from ArdaXi/custom-s3-skip-region-check 
Only check validity of S3 region if not using custom endpoint
 2016-04-08 15:38:24 -07:00
Richard Scothern 437eeeda44 Merge pull request #1605 from majewsky/swift/deal-with-outdated-container-listings 
registry/storage/swift: detect and fix outdated container listings
 2016-04-08 15:38:06 -07:00
Stefan Majewsky 67321cb622 detect outdated container listings during Stat() and getAllSegments() 
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
 2016-04-06 15:21:27 +02:00
Arien Holthuizen dbb6e28da2 Only check validity of S3 region if not using custom endpoint 
Signed-off-by: Arien Holthuizen <aholthuizen@schubergphilis.com>
 2016-04-06 13:38:09 +02:00
Tony Holdstock-Brown 25c5efdef9 Ensure we log io.Copy errors and bytes copied/total in uploads 
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-04-04 17:21:36 -07:00
Richard Scothern 04ff3c0359 Merge pull request #1578 from majewsky/do-not-swallow-error 
don't swallow errors in Swift driver's GetContent()
 2016-03-30 17:46:27 -07:00
Richard Scothern eb0b7f0173 Update the gc documentation. 
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
 2016-03-30 16:57:21 -07:00
Richard Scothern 4324b70c50 Fix signature handling with GC. 
If a schema 1 manifest is uploaded with the `disablesignaturestore` option set
to true, then no signatures will exist.  Handle this case.

If a schema 1 manifest is pushed, deleted, garbage collected and pushed again, the
repository will contain signature links from the first version, but the blobs will
not exist.  Disable the signature store in the garbage-collect command so
signatures are not fetched.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-03-30 16:10:27 -07:00
Richard Scothern 80b310ca44 Add a --dry-run flag. If enabled this will print the mark and sweep process 
with removing any files.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
 2016-03-30 16:10:27 -07:00
Aaron Lehmann 267e268775 garbagecollect: Clean up errors 
- Clean up error messages

- Add a missing error check on the result of blobService.Enumerate.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
 2016-03-30 11:35:24 -07:00
Stefan Majewsky 4c9bed2507 don't swallow errors in Swift driver's GetContent() 
In 326c3a9c49, which was only intended to
be a refactoring commit, the behavior of this block subtly changed so
that unknown types of errors would be swallowed instead of propagated.

I noticed this while investigating an error similar to #1539 aka
docker/docker#21290. It appears that during GetContent() for a
hashstate, the Swift proxy produces an error. Since this error was
silently swallowed, an empty []byte is used to restart the hash, then
producing the digest of the empty string instead of the layer's digest.

This PR will not fix the issue, but it should make the actual error more
visible by propagating it into `blobWriter#resumeDigest' and
'blobWriter#validateBlob', respectively.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
 2016-03-30 16:11:29 +02:00
Stephen Day 0eefe17730 Merge pull request #1546 from arschles/doc-storage 
Add documentation for how to register new StorageDrivers
 2016-03-29 15:58:51 -07:00
Aaron Schlesinger f4bdc6287a Remove the example 
Instead, direct users to the one in the factory package

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
 2016-03-29 14:42:28 -07:00
Aaron Schlesinger 204ad474e4 Add documentation for how to register new StorageDrivers 
This commit adds context-specific documentation on StorageDriver,
StorageDriverFactory, and the factory’s Register func, explaining how
the internal registration mechanism should be used.

This documentation follows from the thread starting at
https://github.com/deis/builder/pull/262/files#r56720200.

cc/ @stevvooe

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
 2016-03-29 14:42:19 -07:00
Tony Holdstock-Brown 31d5dd5175 utulize config log format within gc 
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-03-24 14:29:43 -07:00
Richard Scothern 87a997249d Merge pull request #1522 from RichardScothern/tag-events 
Send tag events to notification listeners
 2016-03-23 16:41:29 -07:00
Stephen Day 9e690c7fa2 Merge pull request #1491 from RichardScothern/relative-url 
Return relative URLs
 2016-03-23 16:19:39 -07:00
Richard Scothern bc9c820e4b Enable URLs returned from the registry to be configured as relative. 
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
 2016-03-23 15:14:32 -07:00
Richard Scothern afe2bdd1c5 Propogate tag as a functional argument into the notification system to attach 
tags to manifest push and pull event notifications.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
 2016-03-23 14:57:52 -07:00
姜继忠 92a9ee34d4 fix manifest revision search, closes #1535 
Signed-off-by: 姜继忠 <jizhong.jiangjz@alibaba-inc.com>
 2016-03-19 19:11:25 +08:00
Antonio Murdaca ea148448a2 registry: client: auth: type errors 
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
 2016-03-18 09:34:50 +01:00
Richard Scothern ec6ac0c05e Merge pull request #1531 from aaronlehmann/empty-errors-slice 
Don't return empty errcode.Errors slices
 2016-03-17 10:46:51 -07:00
Antonio Murdaca 75b5a1b1f6 registry: client: repository: close response body 
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
 2016-03-16 19:53:17 +01:00
Richard Scothern 2c635d1f7e Merge pull request #1532 from RichardScothern/azure-error-types 
Update missing blob error checking with latest Azure API
 2016-03-15 10:08:36 -07:00
Aaron Lehmann 3a2231fe39 Include status code in UnexpectedHTTPResponseError 
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
 2016-03-15 09:03:56 -07:00
Richard Scothern a5d372535e Merge pull request #1502 from RichardScothern/auth-url-endpoints 
URL parse auth endpoints to normalize hostname to lowercase.
 2016-03-14 16:28:24 -07:00
Richard Scothern 772e6f4057 Update missing blob error checking with latest Azure API 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-03-14 15:59:03 -07:00
Aaron Lehmann cd6482ecb8 Don't return empty errcode.Errors slices 
If this slice ends up empty after parsing the HTTP response body, it
means the body is not well-formed. We've probably encountered an error
message produced by something that uses a different JSON schema, or
an error that just happens to validate as JSON.

An empty errcode.Errors slice is not a very useful thing to return,
since its Error() output is just `<nil>`. Detect this case, and instend
return an UnexpectedHTTPResponseError.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
 2016-03-14 10:27:01 -07:00
Matt Duch fcb247dfce registry/storage/driver/s3-aws kms support 
Signed-off-by: Matt Duch <matt@learnmetrics.com>
 2016-03-11 17:19:01 -06:00
Brian Bland c03b5fc5ee Merge pull request #1438 from BrianBland/newStorageDriverWriter 
Adds new StorageDriver.FileWriter interface
 2016-03-11 15:06:07 -08:00
Brian Bland 5967d33342 Removes ceph rados driver in favor of Swift API gateway support 
Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-03-10 16:49:08 -08:00
Brian Bland ab49f85db8 Merge pull request #1512 from kmala/master 
Added support to specifiy custom endpoint
 2016-03-10 13:48:59 -08:00
Keerthan Mala 2be1b4ef4f Added support to specifiy custom endpoint 
Signed-off-by: Keerthan Reddy Mala <keerthan.mala@gmail.com>
 2016-03-09 16:12:20 -07:00
Derek McGowan a59448784b Add client ID to token fetch to GET endpoint 
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
 2016-03-09 12:44:55 -08:00
Brian Bland 7fd1db9312 Updates Swift driver to support new storagedriver.FileWriter interface 
Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-03-08 16:38:39 -08:00
Li Yi a9bf7a2aae Support FileWriter interface for OSS storage driver 
Change-Id: Ie5533ad85f944800499ca1040fd67bf1378815e0
Signed-off-by: Li Yi <denverdino@gmail.com>
 2016-03-08 16:38:39 -08:00
Arthur Baars 307504713f Storagedriver: GCS: add chunksize parameter 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-03-08 16:38:39 -08:00
Arthur Baars 7162cb19c6 Storagedriver: GCS: implement resumable uploads 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-03-08 16:38:39 -08:00
Arthur Baars 666273d9f6 StorageDriver: Testsuite: call Close before getting Size 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-03-08 16:38:38 -08:00
Brian Bland ff03381d49 Adds new storagedriver.FileWriter interface 
Updates registry storage code to use this for better resumable writes.
Implements this interface for the following drivers:
 + Inmemory
 + Filesystem
 + S3
 + Azure

Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-03-08 16:37:44 -08:00
Aaron Lehmann 87e34bd307 Fix two misspellings in source code comments 
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
 2016-03-08 15:13:24 -08:00
Richard Scothern 8bd613aa11 URL parse auth endpoints to normalize hostname to lowercase. 
It is possible for a middlebox to lowercase the URL at somepoint causing a
lookup in the auth challenges table to fail.  Lowercase hostname before
using as keys to challenge map.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
 2016-03-07 14:08:21 -08:00
Brian Bland 302cc39937 [driver/s3aws] Update s3aws driver parameter parsing to match s3goamz 
Mirrors changes from #1414 into the newer driver

Signed-off-by: Brian Bland <brian.bland@docker.com>
 2016-03-07 11:52:34 -08:00
Derek McGowan a8aa2f78de Fix oauth cross repository push 
Cross repository push tokens were not being cached and could not be used,
now any returned token will be used and the caching is hidden in the getToken function.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
 2016-03-04 15:13:27 -08:00
Derek McGowan efd7ea4624 Add offline token option 
Login needs to add an offline token flag to ensure a refresh token is returned by the token endpoint.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
 2016-03-04 14:32:51 -08:00
Derek McGowan 1126e32234 Add post token implementation 
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
 2016-03-04 13:53:06 -08:00
Derek McGowan 2ef7a872de Add options struct to initialize handler 
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
 2016-03-04 11:32:48 -08:00
Derek McGowan c9880e6b05 Fetch token by credentials and refresh token 
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
 2016-03-04 00:34:17 -08:00
Richard Scothern 62d46ef07d Merge pull request #1388 from aibaars/gcs-simplify-move 
StorageDriver: GCS: remove support for directory Moves
 2016-03-03 10:20:51 -08:00
Arthur Baars 1593c2413f StorageDriver: GCS: remove support for directory Moves 
The Move operation is only used to move uploaded blobs
to their final destination. There is no point in implementing
Move on "folders". Apart from simplifying the code, this also
saves an HTTP request.

Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-03-03 13:28:13 +00:00
Richard Scothern c8dff1bb57 Merge pull request #1364 from aaronlehmann/content-type-in-api-docs 
Add information about manifest content types to API spec
 2016-03-01 13:53:49 -08:00
Richard Scothern 5cbac00dd4 Merge pull request #1386 from andrewnguyen/garbage_collect_pick_from_1050 
garbage collection
 2016-02-29 15:01:48 -08:00