Derek McGowan
a1d7463d67
Merge pull request #1669 from RichardScothern/close-after-commit
...
Clean uploads
2016-04-29 16:18:29 -07:00
Richard Scothern
ba927007b0
Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error
...
Move GC into storage package and add tests
2016-04-28 14:09:58 -07:00
Richard Scothern
a7dda2ce93
Merge pull request #1665 from andrewhsu/middleware-redirect
...
add middleware storage driver for redirect
2016-04-27 15:05:52 -07:00
Richard Scothern
d2e29acce0
When a blob upload is committed prevent writing out hashstate in the
...
subsequent close.
When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 14:42:00 -07:00
Richard Scothern
69ba30dc03
Add a test with a missing _manifests directory
...
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Richard Scothern
ea492aca1a
Move garbage collect code into storage package
...
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Tony Holdstock-Brown
a5aaae1f06
Ensure GC continues marking if _manifests is nonexistent
...
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-27 13:33:36 -07:00
Serge Dubrouski
32193bdcf0
Fix wording for dry-run flag in useage message for garbage collector.
...
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-04-26 19:44:23 -06:00
Andrew Hsu
09a9b0cf90
separate the go/non-go imports and reorder
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-26 14:33:54 -07:00
Richard Scothern
c83afea0c9
Merge pull request #1660 from jhaohai/cn-north-1-fix
...
Add cn-north-1 to valid check
2016-04-25 16:07:54 -07:00
Richard Scothern
ef32134592
Merge pull request #1666 from sergeyfd/master
...
Add blobWrtiter.Close() call into blobWriter.Commit()
2016-04-25 16:02:48 -07:00
Andrew Hsu
c4df027d41
modify redirect test to include port
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:46 -07:00
Andrew Hsu
80248c3d3a
scheme and host mandatory in baseurl
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:25 -07:00
Andrew Hsu
059bc5f5ef
separate the go/non-go imports and reorder
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:03 -07:00
Anis Elleuch
987faca8a6
Sorting completed parts by part number for a better accordance with the S3 spec
...
Signed-off-by: Anis Elleuch <vadmeste@gmail.com>
2016-04-23 22:36:04 +01:00
Serge Dubrouski
21f38a74e6
Add blobWrtiter.Close() call into blobWriter.Commit()
...
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-04-22 19:23:17 -06:00
Andrew Hsu
4b217ccbf5
add middleware storage driver for redirect
...
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-21 16:02:52 -07:00
jhaohai
f76c622d8c
add cn-north-1 to valid check
...
Signed-off-by: jhaohai <jhaohai@foxmail.com>
2016-04-21 11:51:34 +08:00
Richard Scothern
4c119524f1
Merge pull request #1604 from ArdaXi/custom-s3-skip-region-check
...
Only check validity of S3 region if not using custom endpoint
2016-04-08 15:38:24 -07:00
Richard Scothern
437eeeda44
Merge pull request #1605 from majewsky/swift/deal-with-outdated-container-listings
...
registry/storage/swift: detect and fix outdated container listings
2016-04-08 15:38:06 -07:00
Stefan Majewsky
67321cb622
detect outdated container listings during Stat() and getAllSegments()
...
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-04-06 15:21:27 +02:00
Arien Holthuizen
dbb6e28da2
Only check validity of S3 region if not using custom endpoint
...
Signed-off-by: Arien Holthuizen <aholthuizen@schubergphilis.com>
2016-04-06 13:38:09 +02:00
Tony Holdstock-Brown
25c5efdef9
Ensure we log io.Copy errors and bytes copied/total in uploads
...
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-04 17:21:36 -07:00
Richard Scothern
04ff3c0359
Merge pull request #1578 from majewsky/do-not-swallow-error
...
don't swallow errors in Swift driver's GetContent()
2016-03-30 17:46:27 -07:00
Richard Scothern
eb0b7f0173
Update the gc documentation.
...
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:57:21 -07:00
Richard Scothern
4324b70c50
Fix signature handling with GC.
...
If a schema 1 manifest is uploaded with the `disablesignaturestore` option set
to true, then no signatures will exist. Handle this case.
If a schema 1 manifest is pushed, deleted, garbage collected and pushed again, the
repository will contain signature links from the first version, but the blobs will
not exist. Disable the signature store in the garbage-collect command so
signatures are not fetched.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-30 16:10:27 -07:00
Richard Scothern
80b310ca44
Add a --dry-run flag. If enabled this will print the mark and sweep process
...
with removing any files.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:10:27 -07:00
Aaron Lehmann
267e268775
garbagecollect: Clean up errors
...
- Clean up error messages
- Add a missing error check on the result of blobService.Enumerate.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-30 11:35:24 -07:00
Stefan Majewsky
4c9bed2507
don't swallow errors in Swift driver's GetContent()
...
In 326c3a9c49
, which was only intended to
be a refactoring commit, the behavior of this block subtly changed so
that unknown types of errors would be swallowed instead of propagated.
I noticed this while investigating an error similar to #1539 aka
docker/docker#21290 . It appears that during GetContent() for a
hashstate, the Swift proxy produces an error. Since this error was
silently swallowed, an empty []byte is used to restart the hash, then
producing the digest of the empty string instead of the layer's digest.
This PR will not fix the issue, but it should make the actual error more
visible by propagating it into `blobWriter#resumeDigest' and
'blobWriter#validateBlob', respectively.
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-03-30 16:11:29 +02:00
Stephen Day
0eefe17730
Merge pull request #1546 from arschles/doc-storage
...
Add documentation for how to register new StorageDrivers
2016-03-29 15:58:51 -07:00
Aaron Schlesinger
f4bdc6287a
Remove the example
...
Instead, direct users to the one in the factory package
Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:28 -07:00
Aaron Schlesinger
204ad474e4
Add documentation for how to register new StorageDrivers
...
This commit adds context-specific documentation on StorageDriver,
StorageDriverFactory, and the factory’s Register func, explaining how
the internal registration mechanism should be used.
This documentation follows from the thread starting at
https://github.com/deis/builder/pull/262/files#r56720200 .
cc/ @stevvooe
Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:19 -07:00
Tony Holdstock-Brown
31d5dd5175
utulize config log format within gc
...
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-03-24 14:29:43 -07:00
Richard Scothern
87a997249d
Merge pull request #1522 from RichardScothern/tag-events
...
Send tag events to notification listeners
2016-03-23 16:41:29 -07:00
Stephen Day
9e690c7fa2
Merge pull request #1491 from RichardScothern/relative-url
...
Return relative URLs
2016-03-23 16:19:39 -07:00
Richard Scothern
bc9c820e4b
Enable URLs returned from the registry to be configured as relative.
...
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-23 15:14:32 -07:00
Richard Scothern
afe2bdd1c5
Propogate tag as a functional argument into the notification system to attach
...
tags to manifest push and pull event notifications.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-23 14:57:52 -07:00
姜继忠
92a9ee34d4
fix manifest revision search, closes #1535
...
Signed-off-by: 姜继忠 <jizhong.jiangjz@alibaba-inc.com>
2016-03-19 19:11:25 +08:00
Antonio Murdaca
ea148448a2
registry: client: auth: type errors
...
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-03-18 09:34:50 +01:00
Richard Scothern
ec6ac0c05e
Merge pull request #1531 from aaronlehmann/empty-errors-slice
...
Don't return empty errcode.Errors slices
2016-03-17 10:46:51 -07:00
Antonio Murdaca
75b5a1b1f6
registry: client: repository: close response body
...
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
2016-03-16 19:53:17 +01:00
Richard Scothern
2c635d1f7e
Merge pull request #1532 from RichardScothern/azure-error-types
...
Update missing blob error checking with latest Azure API
2016-03-15 10:08:36 -07:00
Aaron Lehmann
3a2231fe39
Include status code in UnexpectedHTTPResponseError
...
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-15 09:03:56 -07:00
Richard Scothern
a5d372535e
Merge pull request #1502 from RichardScothern/auth-url-endpoints
...
URL parse auth endpoints to normalize hostname to lowercase.
2016-03-14 16:28:24 -07:00
Richard Scothern
772e6f4057
Update missing blob error checking with latest Azure API
...
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-14 15:59:03 -07:00
Aaron Lehmann
cd6482ecb8
Don't return empty errcode.Errors slices
...
If this slice ends up empty after parsing the HTTP response body, it
means the body is not well-formed. We've probably encountered an error
message produced by something that uses a different JSON schema, or
an error that just happens to validate as JSON.
An empty errcode.Errors slice is not a very useful thing to return,
since its Error() output is just `<nil>`. Detect this case, and instend
return an UnexpectedHTTPResponseError.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-14 10:27:01 -07:00
Matt Duch
fcb247dfce
registry/storage/driver/s3-aws kms support
...
Signed-off-by: Matt Duch <matt@learnmetrics.com>
2016-03-11 17:19:01 -06:00
Brian Bland
c03b5fc5ee
Merge pull request #1438 from BrianBland/newStorageDriverWriter
...
Adds new StorageDriver.FileWriter interface
2016-03-11 15:06:07 -08:00
Brian Bland
5967d33342
Removes ceph rados driver in favor of Swift API gateway support
...
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-10 16:49:08 -08:00
Brian Bland
ab49f85db8
Merge pull request #1512 from kmala/master
...
Added support to specifiy custom endpoint
2016-03-10 13:48:59 -08:00