Commit graph

278 commits

Author SHA1 Message Date
Brian Bland
ff03381d49 Adds new storagedriver.FileWriter interface
Updates registry storage code to use this for better resumable writes.
Implements this interface for the following drivers:
 + Inmemory
 + Filesystem
 + S3
 + Azure

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:37:44 -08:00
Aaron Lehmann
87e34bd307 Fix two misspellings in source code comments
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-08 15:13:24 -08:00
Brian Bland
302cc39937 [driver/s3aws] Update s3aws driver parameter parsing to match s3goamz
Mirrors changes from #1414 into the newer driver

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-07 11:52:34 -08:00
Richard Scothern
62d46ef07d Merge pull request #1388 from aibaars/gcs-simplify-move
StorageDriver: GCS: remove support for directory Moves
2016-03-03 10:20:51 -08:00
Arthur Baars
1593c2413f StorageDriver: GCS: remove support for directory Moves
The Move operation is only used to move uploaded blobs
to their final destination. There is no point in implementing
Move on "folders". Apart from simplifying the code, this also
saves an HTTP request.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-03 13:28:13 +00:00
Andrew T Nguyen
feab4aafbc Implements garbage collection subcommand
- Includes a change in the command to run the registry. The registry
  server itself is now started up as a subcommand.
- Includes changes to the high level interfaces to support enumeration
  of various registry objects.

Signed-off-by: Andrew T Nguyen <andrew.nguyen@docker.com>
2016-02-29 14:15:21 -08:00
Stefan Weil
615c6dfced Fix some typos in comments and strings
All of them were found and fixed by codespell.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2016-02-23 22:33:38 +01:00
xiekeyang
d58188c4d7 compare error output in tagstore unit test
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
2016-02-22 17:34:22 +08:00
Anton Tiurin
a048a4c8d5 Fix description of StorageDriver.WriteStream
Offset can be more than CurrentSize as long as this case is checked
by DriverSuite.testContinueStreamAppend.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2016-02-17 13:57:20 +03:00
Brian Bland
c770f28f04 [driver/s3aws] Fix TestStorageClass
Fixes bug in TestStorageClass for s3aws driver where the "standard" file
was checked for reduced-redundnancy storage.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-16 17:50:55 -08:00
Brian Bland
b9907c4853 Merge pull request #1385 from BrianBland/s3UseAWSLibrary
[driver/s3] Use aws/aws-sdk-go instead of goamz for s3 driver and cloudfront
2016-02-16 11:19:15 -08:00
liuchang0812
0758653b42 fix gofmt
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:42:09 +08:00
liuchang0812
dbba9cf299 closes #1461, enhance log message of oss driver
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:17:09 +08:00
Brian Bland
1319be2cae Adds new s3 driver using aws-sdk-go instead of goamz
Keeps old s3 driver, renames to s3goamz, registers new s3 driver as both
"s3" and "s3aws"

Changes cloudfront middleware to use aws-sdk-go

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-12 15:59:26 -08:00
Aaron Lehmann
aa80478b64 Typo fixes in comments
Correct spelling of words in source code comments.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:26:29 -08:00
Aaron Lehmann
a3213ff331 Merge pull request #1420 from dmcgowan/configurable-trust-key
Add option to disable signatures
2016-02-10 16:15:59 -08:00
Derek McGowan
b34e571bff Add option to disable signatures
Add option for specifying trust key for signing schema1 manifests.
Since schema1 signature key identifiers are not verified anywhere and deprecated, storing signatures is no longer a requirement.
Furthermore in schema2 there is no signature, requiring the registry to already add signatures to generated schema1 manifests.

Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2016-02-10 15:20:39 -08:00
Brian Bland
52b4a328dd Improves flexibility of configuration handling for S3 driver
Treats nil parameters the same as unprovided parameters (fixes issues
where certain parameters are printed to "<nil>").
Accepts "true" and "false" string values for boolean parameters.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-09 17:27:16 -08:00
Richard Scothern
4a1d36c22d Correct type for repo reference
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-02-04 17:32:55 -08:00
Richard Scothern
579981b979 Merge pull request #1350 from aibaars/storage-filewriter-pointer
Storage: remove bufferedFileWriter (dead code)
2016-02-04 16:47:52 -08:00
Richard Scothern
2cc6ccbded Merge pull request #1401 from BrianBland/s3StorageClass
Adds "storageclass" configuration parameter for S3 driver.
2016-02-04 16:44:13 -08:00
Richard Scothern
b737de4856 Merge pull request #1408 from aaronlehmann/repository-interface
Rename Name method of Repository to Named
2016-02-04 16:43:35 -08:00
Aaron Lehmann
b0989446eb Rename Name method of Repository to Named
This makes code that gets the name as a string read like
repo.Named().Name() instead of repo.Name().Name().

Requested in
https://github.com/docker/docker/pull/19887#discussion_r51479753

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-04 09:55:36 -08:00
yuzou
44c0b9ae87 read the actual number of bytes according to the initial size.
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-02-04 16:14:35 +08:00
Brian Bland
695bac7b0f Adds test for S3 storage class configuration option
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:55 -08:00
Brian Bland
4688d1adc2 Adds "storageclass" configuration parameter for S3 driver.
Defaults to STANDARD, also supports REDUCED_REDUNDANCY.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:19 -08:00
Aaron Lehmann
2fc586d2a4 Merge pull request #1381 from BrianBland/s3CustomUAString
Adds custom registry User-Agent header to s3 HTTP requests
2016-02-01 15:40:36 -08:00
Brian Bland
2dc1af12a1 Adds custom registry User-Agent header to s3 HTTP requests
Uses docker/goamz instead of AdRoll/goamz

Adds a registry UA string param to the storage parameters when
constructing the storage driver for the registry App.
This could be used by other storage drivers as well

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 13:43:46 -08:00
Richard Scothern
584c9b517c Correct test digest lengths and enable all unit tests
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-26 17:02:46 -08:00
Arthur Baars
3d67dae525 Storage: remove bufferedFileWriter (dead code)
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:08 +00:00
Arthur Baars
26e7268a86 Storage: blobwriter.Write/Seek test case
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-23 10:30:07 +00:00
Aaron Lehmann
4441333912 Use reference package internally
Most places in the registry were using string types to refer to
repository names. This changes them to use reference.Named, so the type
system can enforce validation of the naming rules.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-22 14:47:05 -08:00
Arthur Baars
2a4345ca4b StorageDriver: GCS: retry all api calls
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Arthur Baars
6ee339464c StorageDriver: Test suite: improve cleanup
Verify that the file(s) have been deleted after calling Delete,
and retry if this is not the case. Furthermore, report the error
if a Delete operation fails.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Richard Scothern
fc7ee6d789 Merge pull request #1355 from hopkings2008/master
In testsuites.go, enlarge the size of randomBytes to 128M to fix the …
2016-01-18 12:38:26 -08:00
Arthur Baars
19d7c9e051 StorageDriver GCS: try google.DefaultTokenSource first
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-15 11:47:47 +00:00
yuzou
cc82b0d48e In testsuites.go, enlarge the size of randomBytes to 128M to fix the crash of running TestConcurrentStreamReads
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-01-15 17:22:43 +08:00
Richard Scothern
7378e21678 Merge pull request #1332 from RichardScothern/gcs-params
Change the parameters to the GCS drivers to allow CircleCI testing.
2016-01-14 15:50:09 -08:00
Brian Bland
8c5a6c13c0 Splits up blob create options definitions to be package-specific
Redefines privately in both storage and client packages

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-14 10:47:33 -08:00
Richard Scothern
81dbebc7d3 Change the parameters to the GCS drivers to allow CircleCI testing.
Remove the requirement of file system access to run GCS unit tests.  Deconstruct
the input parameters to take the private key and email which can be specified on
the build system via environment variables.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-13 18:00:59 -08:00
Brian Bland
ce88d8a6f4 Adds functional options arguments to the Blobs Create method
Removes the Mount operation and instead implements this behavior as part
of Create a From option is provided, which in turn returns a rich
ErrBlobMounted indicating that a blob upload session was not initiated,
but instead the blob was mounted from another repository

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-13 16:42:59 -08:00
Richard Scothern
3a1220de01 Merge pull request #1319 from RichardScothern/update-tags
Remove tags referencing deleted manifests.
2016-01-11 11:33:22 -08:00
Richard Scothern
93d9070c8b Merge pull request #1269 from BrianBland/crossRepositoryPush
Adds cross-repository blob mounting behavior
2016-01-08 14:37:00 -08:00
Brian Bland
5df21570a7 Adds cross-repository blob mounting behavior
Extends blob upload POST endpoint to support mount and from query
parameters as described in #634

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-01-08 13:53:18 -08:00
Richard Scothern
cc0b6b86ea Merge pull request #1276 from denverdino/oss-test
Support large layer for OSS driver
2016-01-08 13:26:09 -08:00
Aaron Lehmann
6d17423a6d Move MediaType into manifest.Versioned
This makes content type sniffing cleaner. The document just needs to be
decoded into a manifest.Versioned structure. It's no longer a two-step
process.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann
9c416f0e94 Add support for manifest list ("fat manifest")
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:27 -08:00
Aaron Lehmann
befd4d6e3c Factor out schema-specific portions of manifestStore
Create signedManifestHandler and schema2ManifestHandler. Use these to
unmarshal and put the respective types of manifests from manifestStore.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-01-07 15:26:26 -08:00
Richard Scothern
9f83bfc071 Merge pull request #1321 from aibaars/gcs-fix-retry
GCS storage driver: fix retry function
2016-01-06 12:00:12 -08:00
Arthur Baars
bb1e9e0a82 GCS Storagedriver: fix test failure caused by #1187
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-06 18:36:28 +00:00