Commit graph

1881 commits

Author SHA1 Message Date
Richard Scothern
03efb43768 Merge pull request #1713 from aibaars/gcs-fix
GCS: FileWriter.Size: return offset + buffer size for Writers that are not closed
2016-05-10 11:32:14 +01:00
Arthur Baars
431e46a7f9 GCS: FileWriter.Size: return offset + buffer size for Writers that are not closed
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-09 15:40:49 +01:00
Richard Scothern
1e0f3b7b64 Merge pull request #1703 from aibaars/gcs-fix
GCS: FileWriter.Size: include number of buffered bytes if the FileWriter is not closed
2016-05-05 12:09:13 -07:00
Arthur Baars
93e3aa9b21 GCS: FileWriter.Size: include number of buffered bytes if the FileWriter is not closed
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-05-05 11:07:45 +01:00
Aaron Lehmann
93d76247f2 Preserve author information in schema1 manifests
When we push a schema1 manifest, we encode history information from the
image JSON into v1Compatibility strings for the respective layers. The
"author" field was not being set in these v1Compatibility strings, so if
a parent layer had an author set, it would not be preserved after
pushing through a schema1 manifest and repulling, so the image ID would
change after the pull. This change preserves the authorship information
for parent layers so that the image ID does not change.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-05-04 10:35:49 -07:00
Richard Scothern
ba672e8b69 When a blob upload is committed prevent writing out hashstate in the
subsequent close.

When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-04 10:35:37 -07:00
Richard Scothern
96230decb9 Add a test with a missing _manifests directory
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-04 10:35:26 -07:00
Richard Scothern
c0d3813f86 Move garbage collect code into storage package
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-04 10:35:18 -07:00
Tony Holdstock-Brown
011b7e493b Ensure GC continues marking if _manifests is nonexistent
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-04 10:35:04 -07:00
Serge Dubrouski
0a1fcf9712 Fix wording for dry-run flag in useage message for garbage collector.
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-05-04 10:31:44 -07:00
Anis Elleuch
ed02e88075 Sorting completed parts by part number for a better accordance with the S3 spec
Signed-off-by: Anis Elleuch <vadmeste@gmail.com>
2016-05-04 10:30:37 -07:00
Serge Dubrouski
fd5a404996 Add blobWrtiter.Close() call into blobWriter.Commit()
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-05-04 10:30:29 -07:00
jhaohai
3f538cac90 add cn-north-1 to valid check
Signed-off-by: jhaohai <jhaohai@foxmail.com>
2016-05-04 10:30:18 -07:00
Stefan Majewsky
3330cc567e wait for DLO segments to show up when Close()ing the writer
Not just when Commit()ing the result. This fixes some errors I observed
when the layer (i.e. the DLO) is Stat()ed immediately after closing,
and reports the wrong file size because the container listing is not
yet up-to-date.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-05-04 10:30:08 -07:00
Aaron Lehmann
775d0968cc Use correct media type for config blob in schema2 manifest
The schema2 manifest builder fills in this part of the manifest based on
the descriptor it gets back from BlobIngester's Put method. It passes
the correct media type to Put, but Put ends up replacing this value with
application/octet-stream in its return value.

This commit works around the issue in the manifest builder. Arguably Put
should not be changing the media type in its return value, but this
commit is a targeted fix to keep it very low-risk for possible inclusion
in Docker 1.11.

Fixes #1621 (but maybe we should open a separate issue for the media
type behavior in the distribution client, and the unnecessary stat).

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-05-04 10:28:54 -07:00
Arien Holthuizen
64a9727f11 Only check validity of S3 region if not using custom endpoint
Signed-off-by: Arien Holthuizen <aholthuizen@schubergphilis.com>
2016-05-04 10:28:31 -07:00
Tony Holdstock-Brown
dafb59f4ab Ensure we log io.Copy errors and bytes copied/total in uploads
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-04 10:28:12 -07:00
Richard Scothern
3f7fa41272 Update AUTHORS 2016-04-13 10:27:36 -07:00
Richard Scothern
04ff3c0359 Merge pull request #1578 from majewsky/do-not-swallow-error
don't swallow errors in Swift driver's GetContent()
2016-03-30 17:46:27 -07:00
Richard Scothern
13f72ef7dc Merge pull request #1560 from RichardScothern/gc
Fix signature handling with GC.
2016-03-30 17:46:04 -07:00
Richard Scothern
eb0b7f0173 Update the gc documentation.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:57:21 -07:00
Richard Scothern
4324b70c50 Fix signature handling with GC.
If a schema 1 manifest is uploaded with the `disablesignaturestore` option set
to true, then no signatures will exist.  Handle this case.

If a schema 1 manifest is pushed, deleted, garbage collected and pushed again, the
repository will contain signature links from the first version, but the blobs will
not exist.  Disable the signature store in the garbage-collect command so
signatures are not fetched.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-30 16:10:27 -07:00
Richard Scothern
80b310ca44 Add a --dry-run flag. If enabled this will print the mark and sweep process
with removing any files.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:10:27 -07:00
Richard Scothern
ee8450ff13 Merge pull request #1581 from RichardScothern/docs
Manifest delete documentation enhancement
2016-03-30 16:05:57 -07:00
Richard Scothern
e5aa9c8e56 Merge pull request #1577 from xiekeyang/context
type judgment in context
2016-03-30 14:29:48 -07:00
Richard Scothern
791865af71 Merge pull request #1582 from aaronlehmann/garbagecollect-errors
garbagecollect: Clean up errors
2016-03-30 14:29:27 -07:00
Aaron Lehmann
267e268775 garbagecollect: Clean up errors
- Clean up error messages

- Add a missing error check on the result of blobService.Enumerate.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-30 11:35:24 -07:00
Richard Scothern
9b5caf64cf Add a note to the delete manifest section to aid getting the correct
digest to delete manifests.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-30 10:59:26 -07:00
Stefan Majewsky
4c9bed2507 don't swallow errors in Swift driver's GetContent()
In 326c3a9c49, which was only intended to
be a refactoring commit, the behavior of this block subtly changed so
that unknown types of errors would be swallowed instead of propagated.

I noticed this while investigating an error similar to #1539 aka
docker/docker#21290. It appears that during GetContent() for a
hashstate, the Swift proxy produces an error. Since this error was
silently swallowed, an empty []byte is used to restart the hash, then
producing the digest of the empty string instead of the layer's digest.

This PR will not fix the issue, but it should make the actual error more
visible by propagating it into `blobWriter#resumeDigest' and
'blobWriter#validateBlob', respectively.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-03-30 16:11:29 +02:00
xiekeyang
e388517d21 type judgment in context
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2016-03-30 17:35:48 +08:00
Stephen Day
0eefe17730 Merge pull request #1546 from arschles/doc-storage
Add documentation for how to register new StorageDrivers
2016-03-29 15:58:51 -07:00
Aaron Schlesinger
f4bdc6287a Remove the example
Instead, direct users to the one in the factory package

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:28 -07:00
Aaron Schlesinger
204ad474e4 Add documentation for how to register new StorageDrivers
This commit adds context-specific documentation on StorageDriver,
StorageDriverFactory, and the factory’s Register func, explaining how
the internal registration mechanism should be used.

This documentation follows from the thread starting at
https://github.com/deis/builder/pull/262/files#r56720200.

cc/ @stevvooe

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:19 -07:00
Richard Scothern
19ad02e6ac Merge pull request #1559 from moxiegirl/fix-925
Update the Duration value for Cloudfront
2016-03-29 10:30:19 -07:00
Mary Anthony
6018bdf58a Updating example
Signed-off-by: Mary Anthony <mary@docker.com>
2016-03-29 09:53:13 -07:00
Mary Anthony
188206eede Fixes and closes #925
Seconds to minutes as per code
Correction per Derek
Clarifying failure case

Signed-off-by: Mary Anthony <mary@docker.com>

Clarifying failure case

Signed-off-by: Mary Anthony <mary@docker.com>
2016-03-29 06:17:29 -07:00
Olivier Gambier
60b9f1756d Merge pull request #1570 from RichardScothern/version
Rev base version to 2.4.0
2016-03-28 13:56:24 -07:00
Richard Scothern
1ea9a3638d Rev base version to 2.4.0
Now that we've tagged a release candidate, we can now say all builds are
"2.4.0+unknown" if the makefile is not used.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-28 13:54:28 -07:00
Richard Scothern
72ec74834b Merge pull request #1569 from GameScripting/patch-1
added heading to get more focus into the headers that need to be set
2016-03-28 10:34:48 -07:00
Daniel Huhn
321e0420bf added heading to get more focus into the headers that need to be set
Signed-off-by: Daniel Huhn <daniel@danielhuhn.de>
2016-03-28 18:07:03 +02:00
Aaron Lehmann
f4b6fc8d68 Merge pull request #1557 from tonyhb/gc-logging
Utulize config log format within gc
2016-03-24 15:53:26 -07:00
Tony Holdstock-Brown
31d5dd5175 utulize config log format within gc
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-03-24 14:29:43 -07:00
Olivier Gambier
687fc1e164 Merge pull request #1558 from RichardScothern/AUTHORS
Update AUTHORS
2016-03-24 13:07:41 -07:00
Richard Scothern
ba60866139 Update AUTHORS
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-24 13:03:46 -07:00
Richard Scothern
f2d128e2b0 Merge pull request #1493 from thaJeztah/fix-storage-driver-links
Fix broken links in configuration
2016-03-24 10:57:58 -07:00
Richard Scothern
9cd193d395 Merge pull request #1556 from dmp42/6.fix-codecov
Fix codecov following vendor
2016-03-24 10:57:07 -07:00
Olivier Gambier
37c2b9e3f1 Fix codecov following vendor
Signed-off-by: Olivier Gambier <olivier@docker.com>
2016-03-24 10:30:14 -07:00
Sebastiaan van Stijn
a92c93d770 Fix broken links in configuration
HTML links are not converted by HUGO, so will work
on GitHub, but not in the online documentation.

Converted the HTML table (and links) to Markdown
to fix broken links.

Also added a header for the table, because none
was present.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2016-03-24 10:28:36 -07:00
Richard Scothern
b787035d8b Merge pull request #1545 from jonjohnsonjr/typos
Fix typos in docs/spec/api.md
2016-03-23 16:41:44 -07:00
Richard Scothern
87a997249d Merge pull request #1522 from RichardScothern/tag-events
Send tag events to notification listeners
2016-03-23 16:41:29 -07:00