Commit graph

331 commits

Author SHA1 Message Date
Richard Scothern
df2184c810 Merge pull request #1627 from luckyraul/swift_auth_url
Swift auth version param
2016-06-01 11:23:23 -07:00
Richard Scothern
0c15ab6952 Remove signature store from registry. Return a generated signature for manifest
pull.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-27 13:19:26 -07:00
Richard Scothern
32e9779faf Merge pull request #1734 from legionus/blob-access-controller
Add support for  blobAccessController middleware
2016-05-24 10:34:36 -07:00
John Starks
f0052b8434 Add support for layers from foreign sources
This will be used to support downloading Windows base layers from
Microsoft URLs.

Signed-off-by: John Starks <jostarks@microsoft.com>
2016-05-20 12:31:10 -07:00
Alexey Gladkov
f97eca5ad6 Add support for blobAccessController middleware
Signed-off-by: Michal Minar <miminar@redhat.com>
Signed-off-by: Alexey Gladkov <gladkov.alexey@gmail.com>
2016-05-19 14:02:15 +02:00
Richard Scothern
4f2ee029a2 Add 'us-gov-west-1' to the valid region list.
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-05-09 16:38:16 +01:00
Richard Scothern
c047d34b22 Merge pull request #1695 from tonyhb/add-regulator-to-filesystem
Add regulator to filesystem
2016-05-04 10:05:51 -07:00
Tony Holdstock-Brown
c9c62380ff Don't wrap thead limits when using a negative int
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 16:03:44 -07:00
Tony Holdstock-Brown
33c448f147 Implement regulator in filesystem driver
This commit refactors base.regulator into the 2.4 interfaces and adds a
filesystem configuration option `maxthreads` to configure the regulator.

By default `maxthreads` is set to 100. This means the FS driver is
limited to 100 concurrent blocking file operations. Any subsequent
operations will block in Go until previous filesystem operations
complete.

This ensures that the registry can never open thousands of simultaneous
threads from os filesystem operations.

Note that `maxthreads` can never be less than 25.

Add test case covering parsable string maxthreads

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-05-03 09:33:22 -07:00
Richard Scothern
5d08dfa70c Merge pull request #1650 from majewsky/swift/wait-for-dlo-segments
[Swift] wait for DLO segments to show up when Close()ing the writer
2016-05-02 13:41:26 -07:00
Derek McGowan
a1d7463d67 Merge pull request #1669 from RichardScothern/close-after-commit
Clean uploads
2016-04-29 16:18:29 -07:00
Richard Scothern
ba927007b0 Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error
Move GC into storage package and add tests
2016-04-28 14:09:58 -07:00
Richard Scothern
a7dda2ce93 Merge pull request #1665 from andrewhsu/middleware-redirect
add middleware storage driver for redirect
2016-04-27 15:05:52 -07:00
Richard Scothern
d2e29acce0 When a blob upload is committed prevent writing out hashstate in the
subsequent close.

When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 14:42:00 -07:00
Richard Scothern
69ba30dc03 Add a test with a missing _manifests directory
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Richard Scothern
ea492aca1a Move garbage collect code into storage package
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-04-27 13:34:25 -07:00
Josh Hawn
e4dd3359cc Regulate filesystem driver to max of 100 calls
It's easily possible for a flood of requests to trigger thousands of
concurrent file accesses on the storage driver. Each file I/O call creates
a new OS thread that is not reaped by the Golang runtime. By limiting it
to only 100 at a time we can effectively bound the number of OS threads
in use by the storage driver.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
2016-04-26 14:44:13 -07:00
Andrew Hsu
09a9b0cf90 separate the go/non-go imports and reorder
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-26 14:33:54 -07:00
Richard Scothern
c83afea0c9 Merge pull request #1660 from jhaohai/cn-north-1-fix
Add cn-north-1 to valid check
2016-04-25 16:07:54 -07:00
Richard Scothern
ef32134592 Merge pull request #1666 from sergeyfd/master
Add blobWrtiter.Close() call into blobWriter.Commit()
2016-04-25 16:02:48 -07:00
Andrew Hsu
c4df027d41 modify redirect test to include port
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:46 -07:00
Andrew Hsu
80248c3d3a scheme and host mandatory in baseurl
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:25 -07:00
Andrew Hsu
059bc5f5ef separate the go/non-go imports and reorder
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-25 11:52:03 -07:00
Anis Elleuch
987faca8a6 Sorting completed parts by part number for a better accordance with the S3 spec
Signed-off-by: Anis Elleuch <vadmeste@gmail.com>
2016-04-23 22:36:04 +01:00
Serge Dubrouski
21f38a74e6 Add blobWrtiter.Close() call into blobWriter.Commit()
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
2016-04-22 19:23:17 -06:00
Andrew Hsu
4b217ccbf5 add middleware storage driver for redirect
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
2016-04-21 16:02:52 -07:00
jhaohai
f76c622d8c add cn-north-1 to valid check
Signed-off-by: jhaohai <jhaohai@foxmail.com>
2016-04-21 11:51:34 +08:00
Stefan Majewsky
9a67520af7 wait for DLO segments to show up when Close()ing the writer
Not just when Commit()ing the result. This fixes some errors I observed
when the layer (i.e. the DLO) is Stat()ed immediately after closing,
and reports the wrong file size because the container listing is not
yet up-to-date.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-04-19 14:31:49 +02:00
Nikita Tarasov
b4f060599a docs + fix test
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-17 20:05:51 +03:00
Nikita Tarasov
b51607f9f0 fix test
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>

Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-17 19:52:40 +03:00
Nikita
bcb7989fca test
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-13 18:49:38 +03:00
Nikita
e6f8d7c28e Update swift.go
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
2016-04-13 18:49:25 +03:00
Richard Scothern
4c119524f1 Merge pull request #1604 from ArdaXi/custom-s3-skip-region-check
Only check validity of S3 region if not using custom endpoint
2016-04-08 15:38:24 -07:00
Stefan Majewsky
67321cb622 detect outdated container listings during Stat() and getAllSegments()
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-04-06 15:21:27 +02:00
Arien Holthuizen
dbb6e28da2 Only check validity of S3 region if not using custom endpoint
Signed-off-by: Arien Holthuizen <aholthuizen@schubergphilis.com>
2016-04-06 13:38:09 +02:00
Richard Scothern
04ff3c0359 Merge pull request #1578 from majewsky/do-not-swallow-error
don't swallow errors in Swift driver's GetContent()
2016-03-30 17:46:27 -07:00
Richard Scothern
4324b70c50 Fix signature handling with GC.
If a schema 1 manifest is uploaded with the `disablesignaturestore` option set
to true, then no signatures will exist.  Handle this case.

If a schema 1 manifest is pushed, deleted, garbage collected and pushed again, the
repository will contain signature links from the first version, but the blobs will
not exist.  Disable the signature store in the garbage-collect command so
signatures are not fetched.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-30 16:10:27 -07:00
Richard Scothern
80b310ca44 Add a --dry-run flag. If enabled this will print the mark and sweep process
with removing any files.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-03-30 16:10:27 -07:00
Stefan Majewsky
4c9bed2507 don't swallow errors in Swift driver's GetContent()
In 326c3a9c49, which was only intended to
be a refactoring commit, the behavior of this block subtly changed so
that unknown types of errors would be swallowed instead of propagated.

I noticed this while investigating an error similar to #1539 aka
docker/docker#21290. It appears that during GetContent() for a
hashstate, the Swift proxy produces an error. Since this error was
silently swallowed, an empty []byte is used to restart the hash, then
producing the digest of the empty string instead of the layer's digest.

This PR will not fix the issue, but it should make the actual error more
visible by propagating it into `blobWriter#resumeDigest' and
'blobWriter#validateBlob', respectively.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
2016-03-30 16:11:29 +02:00
Aaron Schlesinger
f4bdc6287a Remove the example
Instead, direct users to the one in the factory package

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:28 -07:00
Aaron Schlesinger
204ad474e4 Add documentation for how to register new StorageDrivers
This commit adds context-specific documentation on StorageDriver,
StorageDriverFactory, and the factory’s Register func, explaining how
the internal registration mechanism should be used.

This documentation follows from the thread starting at
https://github.com/deis/builder/pull/262/files#r56720200.

cc/ @stevvooe

Signed-off-by: Aaron Schlesinger <aschlesinger@deis.com>
2016-03-29 14:42:19 -07:00
姜继忠
92a9ee34d4 fix manifest revision search, closes #1535
Signed-off-by: 姜继忠 <jizhong.jiangjz@alibaba-inc.com>
2016-03-19 19:11:25 +08:00
Richard Scothern
2c635d1f7e Merge pull request #1532 from RichardScothern/azure-error-types
Update missing blob error checking with latest Azure API
2016-03-15 10:08:36 -07:00
Richard Scothern
772e6f4057 Update missing blob error checking with latest Azure API
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-14 15:59:03 -07:00
Matt Duch
fcb247dfce registry/storage/driver/s3-aws kms support
Signed-off-by: Matt Duch <matt@learnmetrics.com>
2016-03-11 17:19:01 -06:00
Brian Bland
c03b5fc5ee Merge pull request #1438 from BrianBland/newStorageDriverWriter
Adds new StorageDriver.FileWriter interface
2016-03-11 15:06:07 -08:00
Brian Bland
5967d33342 Removes ceph rados driver in favor of Swift API gateway support
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-10 16:49:08 -08:00
Keerthan Mala
2be1b4ef4f Added support to specifiy custom endpoint
Signed-off-by: Keerthan Reddy Mala <keerthan.mala@gmail.com>
2016-03-09 16:12:20 -07:00
Brian Bland
7fd1db9312 Updates Swift driver to support new storagedriver.FileWriter interface
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:38:39 -08:00
Li Yi
a9bf7a2aae Support FileWriter interface for OSS storage driver
Change-Id: Ie5533ad85f944800499ca1040fd67bf1378815e0
Signed-off-by: Li Yi <denverdino@gmail.com>
2016-03-08 16:38:39 -08:00