TrueCloudLab/distribution
13
0
Fork 3
Code Issues 3 Pull requests Projects Releases 2 Packages Wiki Activity Actions
773 commits 2 branches 61 tags 129 MiB
Commit graph

773 commits

This branch
This branch
All branches
Author SHA1 Message Date
Arthur Baars
af00617b99 Blobwriter: call BlobWriter.Size after BlobWriter.Close 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-05-06 13:04:30 +01:00
Arthur Baars
ddec546466 StorageDriver: Test case for #1698 
Signed-off-by: Arthur Baars <arthur@semmle.com>
 2016-05-06 13:04:30 +01:00
Richard Scothern
c8592da977 Merge pull request #1695 from tonyhb/add-regulator-to-filesystem 
Add regulator to filesystem
 2016-05-04 10:05:51 -07:00
Richard Scothern
e334dbd5bb Merge pull request #1696 from runcom/go1.6-lol 
registry: do not use http.StatusTooManyRequests
 2016-05-04 09:45:18 -07:00
Tony Holdstock-Brown
1e05d81a71 Don't wrap thead limits when using a negative int 
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-05-03 16:03:44 -07:00
Antonio Murdaca
db274d3c00 registry: do not use http.StatusTooManyRequests 
go1.5 doesn't export http.StatusTooManyRequests while
go1.6 does. Fix this by hardcoding the status code for now.

Signed-off-by: Antonio Murdaca <runcom@redhat.com>
 2016-05-03 21:24:45 +02:00
Aaron Lehmann
262f93c7c5 Merge pull request #1693 from runcom/too-many-requests 
registry: type too many requests error
 2016-05-03 10:16:21 -07:00
Antonio Murdaca
8762c800f1 registry: type too many requests error 
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
 2016-05-03 18:41:54 +02:00
Tony Holdstock-Brown
cbae4dd7bf Implement regulator in filesystem driver 
This commit refactors base.regulator into the 2.4 interfaces and adds a
filesystem configuration option `maxthreads` to configure the regulator.

By default `maxthreads` is set to 100. This means the FS driver is
limited to 100 concurrent blocking file operations. Any subsequent
operations will block in Go until previous filesystem operations
complete.

This ensures that the registry can never open thousands of simultaneous
threads from os filesystem operations.

Note that `maxthreads` can never be less than 25.

Add test case covering parsable string maxthreads

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-05-03 09:33:22 -07:00
Richard Scothern
03abc6ebae Merge pull request #1650 from majewsky/swift/wait-for-dlo-segments 
[Swift] wait for DLO segments to show up when Close()ing the writer
 2016-05-02 13:41:26 -07:00
Derek McGowan
32585845b7 Merge pull request #1669 from RichardScothern/close-after-commit 
Clean uploads
 2016-04-29 16:18:29 -07:00
Troels Thomsen
28be207bc0 Pass through known errors 
Signed-off-by: Troels Thomsen <troels@thomsen.io>
 2016-04-29 23:34:24 +02:00
Richard Scothern
c0dcad3a48 Merge pull request #1677 from RichardScothern/tonyhb-fix-s3-gc-error 
Move GC into storage package and add tests
 2016-04-28 14:09:58 -07:00
Richard Scothern
ea4f065672 Merge pull request #1665 from andrewhsu/middleware-redirect 
add middleware storage driver for redirect
 2016-04-27 15:05:52 -07:00
Richard Scothern
2a2577d7b1 When a blob upload is committed prevent writing out hashstate in the 
subsequent close.

When a blob upload is cancelled close the blobwriter before removing
upload state to ensure old hashstates don't persist.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-04-27 14:42:00 -07:00
Richard Scothern
63d28d3b81 Add a test with a missing _manifests directory 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-04-27 13:34:25 -07:00
Richard Scothern
3a034b477e Move garbage collect code into storage package 
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-04-27 13:34:25 -07:00
Tony Holdstock-Brown
898fdb48a1 Ensure GC continues marking if _manifests is nonexistent 
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-04-27 13:33:36 -07:00
Richard Scothern
cd4f752e49 Merge pull request #1675 from sergeyfd/master 
Fix wording for dry-run flag in usage message for garbage collector.
 2016-04-27 10:08:53 -07:00
Serge Dubrouski
8775da93d6 Fix wording for dry-run flag in useage message for garbage collector. 
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
 2016-04-26 19:44:23 -06:00
Josh Hawn
a88088a59d Regulate filesystem driver to max of 100 calls 
It's easily possible for a flood of requests to trigger thousands of
concurrent file accesses on the storage driver. Each file I/O call creates
a new OS thread that is not reaped by the Golang runtime. By limiting it
to only 100 at a time we can effectively bound the number of OS threads
in use by the storage driver.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-04-26 14:44:13 -07:00
Andrew Hsu
54edbdfee6 separate the go/non-go imports and reorder 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-26 14:33:54 -07:00
Richard Scothern
51972682c7 Merge pull request #1660 from jhaohai/cn-north-1-fix 
Add cn-north-1 to valid check
 2016-04-25 16:07:54 -07:00
Richard Scothern
f91dc3ca51 Merge pull request #1666 from sergeyfd/master 
Add blobWrtiter.Close() call into blobWriter.Commit()
 2016-04-25 16:02:48 -07:00
Richard Scothern
e6ad3a227a Merge pull request #1670 from vadmeste/fix_parts_sorting 
s3 driver: Sorting completed parts by part number for a better accordance with S3 spec
 2016-04-25 16:02:17 -07:00
Andrew Hsu
3336cc13e4 modify redirect test to include port 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-25 11:52:46 -07:00
Andrew Hsu
fba2e3a206 scheme and host mandatory in baseurl 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-25 11:52:25 -07:00
Andrew Hsu
cec7248bd1 separate the go/non-go imports and reorder 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-25 11:52:03 -07:00
Anis Elleuch
d11a979591 Sorting completed parts by part number for a better accordance with the S3 spec 
Signed-off-by: Anis Elleuch <vadmeste@gmail.com>
 2016-04-23 22:36:04 +01:00
Serge Dubrouski
6615b77a09 Add blobWrtiter.Close() call into blobWriter.Commit() 
Signed-off-by: Serge Dubrouski <sergeyfd@gmail.com>
 2016-04-22 19:23:17 -06:00
Andrew Hsu
a691d82aee add middleware storage driver for redirect 
Signed-off-by: Andrew Hsu <andrewhsu@acm.org> (github: andrewhsu)
 2016-04-21 16:02:52 -07:00
jhaohai
fdb0fb77df add cn-north-1 to valid check 
Signed-off-by: jhaohai <jhaohai@foxmail.com>
 2016-04-21 11:51:34 +08:00
Stefan Majewsky
ea5abc9935 wait for DLO segments to show up when Close()ing the writer 
Not just when Commit()ing the result. This fixes some errors I observed
when the layer (i.e. the DLO) is Stat()ed immediately after closing,
and reports the wrong file size because the container listing is not
yet up-to-date.

Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
 2016-04-19 14:31:49 +02:00
Nikita Tarasov
346bfed907 docs + fix test 
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
 2016-04-17 20:05:51 +03:00
Nikita Tarasov
007af250b4 fix test 
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>

Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
 2016-04-17 19:52:40 +03:00
Nikita
b55719daaa test 
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
 2016-04-13 18:49:38 +03:00
Nikita
63fe2d1429 Update swift.go 
Signed-off-by: Nikita Tarasov <nikita@mygento.ru>
 2016-04-13 18:49:25 +03:00
Richard Scothern
406d03bdb0 Merge pull request #1604 from ArdaXi/custom-s3-skip-region-check 
Only check validity of S3 region if not using custom endpoint
 2016-04-08 15:38:24 -07:00
Richard Scothern
ede3c5033e Merge pull request #1605 from majewsky/swift/deal-with-outdated-container-listings 
registry/storage/swift: detect and fix outdated container listings
 2016-04-08 15:38:06 -07:00
Stefan Majewsky
84aa48b56c detect outdated container listings during Stat() and getAllSegments() 
Signed-off-by: Stefan Majewsky <stefan.majewsky@sap.com>
 2016-04-06 15:21:27 +02:00
Arien Holthuizen
c655241209 Only check validity of S3 region if not using custom endpoint 
Signed-off-by: Arien Holthuizen <aholthuizen@schubergphilis.com>
 2016-04-06 13:38:09 +02:00
Richard Scothern
4e45a3a903 Merge pull request #1597 from tonyhb/log-upload-copy-errors 
Ensure we log io.Copy errors and bytes copied/total in uploads
 2016-04-05 09:46:12 -07:00
Tony Holdstock-Brown
86ca50dfe5 Ensure we log io.Copy errors and bytes copied/total in uploads 
Signed-off-by: Tony Holdstock-Brown <tony@docker.com>
 2016-04-04 17:21:36 -07:00
Richard Scothern
2a3d48fb82 Merge pull request #1578 from majewsky/do-not-swallow-error 
don't swallow errors in Swift driver's GetContent()
 2016-03-30 17:46:27 -07:00
Richard Scothern
bf182a4d3f Merge pull request #1560 from RichardScothern/gc 
Fix signature handling with GC.
 2016-03-30 17:46:04 -07:00
Richard Scothern
3d4b652b58 Update the gc documentation. 
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
 2016-03-30 16:57:21 -07:00
Richard Scothern
31ece3d3b6 Fix signature handling with GC. 
If a schema 1 manifest is uploaded with the `disablesignaturestore` option set
to true, then no signatures will exist.  Handle this case.

If a schema 1 manifest is pushed, deleted, garbage collected and pushed again, the
repository will contain signature links from the first version, but the blobs will
not exist.  Disable the signature store in the garbage-collect command so
signatures are not fetched.

Signed-off-by: Richard Scothern <richard.scothern@docker.com>
 2016-03-30 16:10:27 -07:00
Richard Scothern
15e3ffb3f2 Add a --dry-run flag. If enabled this will print the mark and sweep process 
with removing any files.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
 2016-03-30 16:10:27 -07:00
Richard Scothern
2f2875e8db Merge pull request #1582 from aaronlehmann/garbagecollect-errors 
garbagecollect: Clean up errors
 2016-03-30 14:29:27 -07:00
Aaron Lehmann
59ef6d2d40 garbagecollect: Clean up errors 
- Clean up error messages

- Add a missing error check on the result of blobService.Enumerate.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
 2016-03-30 11:35:24 -07:00