Commit graph

20 commits

Author SHA1 Message Date
Manish Tomar
3c64ff10bb Fix gometalint errors
Signed-off-by: Manish Tomar <manish.tomar@docker.com>
Signed-off-by: wang yan <wangyan@vmware.com>
2021-03-23 21:03:10 +08:00
Derek McGowan
db0a4ec1c8
Enable static checks
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-08-06 14:49:11 -07:00
Yu Wang
62797237b9 closes #2496 and #2552
Update Azure SDK with release v16.2.1
Update Azure autorest SDK with release v10.8.1

Signed-off-by: Yu Wang <yuwa@microsoft.com>
2018-06-08 16:53:02 -07:00
Sargun Dhillon
32ac467992 Introduce Walk Method Per Storage Driver
Move the Walk types into registry/storage/driver, and add a Walk method to each
storage driver. Although this is yet another API to implement, there is a fall
back implementation that relies on List and Stat. For some filesystems this is
very slow.

Also, this WalkDir Method conforms better do a traditional WalkDir (a la filepath).

This change is in preparation for refactoring.

Signed-off-by: Sargun Dhillon <sargun@sargun.me>
2018-01-07 22:45:17 -08:00
Stephen J Day
9c88801a12
context: remove definition of Context
Back in the before time, the best practices surrounding usage of Context
weren't quite worked out. We defined our own type to make usage easier.
As this packaged was used elsewhere, it make it more and more
challenging to integrate with the forked `Context` type. Now that it is
available in the standard library, we can just use that one directly.

To make usage more consistent, we now use `dcontext` when referring to
the distribution context package.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2017-08-11 15:53:31 -07:00
Yu Wang (UC)
ac05d143d8 closes #2224: re-vendor the latest Azure Storage SDK for better performance
Signed-off-by: Yu Wang <yuwa@microsoft.com>
2017-04-14 14:20:18 -07:00
Ahmet Alp Balkan
0a1ce58e2c
azure: revendor + remove hacky solution in is404
Removing the temporary workaround in is404() method by re-vendoring
the azure-sdk-for-go.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2017-01-09 17:22:28 -08:00
Ahmet Alp Balkan
78d0660319
azure: fix race condition in PutContent()
See #2077 for background.

The PR #1438 which was not reviewed by azure folks basically introduced
a race condition around uploads to the same blob by multiple clients
concurrently as it used the "writer" type for PutContent(), introduced in #1438.
This does chunked upload of blobs using "AppendBlob" type, which was not atomic.

Usage of "writer" type and thus AppendBlobs on metadata files is currently not
concurrency-safe and generally, they are not the right type of blob for the job.

This patch fixes PutContent() to use the atomic upload operation that works
for uploads smaller than 64 MB and creates blobs with "BlockBlob" type. To be
backwards compatible, we query the type of the blob first and if it is not
a "BlockBlob" we delete the blob first before doing an atomic PUT. This
creates a small inconsistency/race window "only once". Once the blob is made
"BlockBlob", it is overwritten with a single PUT atomicallly next time.

Therefore, going forward, PutContent() will be producing BlockBlobs and it
will silently migrate the AppendBlobs introduced in #1438 to BlockBlobs with
this patch.

Tested with existing code side by side, both registries with and without this
patch work fine without breaking each other. So this should be good from a
backwards/forward compatiblity perspective, with a cost of doing an extra
HEAD checking the blob type.

Fixes #2077.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-30 12:40:43 -08:00
Derek McGowan
a2611c7520 Merge pull request #2027 from ahmetalpbalkan/pr-azure-memleak2
Update vendored azure-sdk-for-go
2016-11-04 10:08:40 -07:00
Ahmet Alp Balkan
2ab25288a2
Update vendored azure-sdk-for-go
Updating to a recent version of Azure Storage SDK to be
able to patch some memory leaks through configurable HTTP client
changes which were made possible by recent patches to it.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-11-03 13:24:57 -07:00
Ahmet Alp Balkan
6d2a0bafcd
storagedriver/azure: close leaking response body
In GetContent() we read the bytes from a blob but do not close
the underlying response body.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2016-10-28 15:13:22 -07:00
Richard Scothern
772e6f4057 Update missing blob error checking with latest Azure API
Signed-off-by: Richard Scothern <richard.scothern@docker.com>
2016-03-14 15:59:03 -07:00
Brian Bland
ff03381d49 Adds new storagedriver.FileWriter interface
Updates registry storage code to use this for better resumable writes.
Implements this interface for the following drivers:
 + Inmemory
 + Filesystem
 + S3
 + Azure

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-08 16:37:44 -08:00
Ahmet Alp Balkan
daa22cacba storage/driver/azure: Update vendored Azure SDK
This change refreshes the updated version of Azure SDK
for Go that has the latest changes.

I manually vendored the new SDK (github.com/Azure/azure-sdk-for-go)
and I removed `management/` `core/` packages manually simply because
they're not used here and they have a fork of `net/http` and `crypto/tls`
for a particular reason. It was introducing a 44k SLOC change otherwise...

This also undoes the `include_azure` flag (actually Steven removed the
driver from imports but forgot to add the build flag apparently, so the
flag wasn't really including azure. 😄 ). This also must be obsolete
now.

Fixes #620, #175.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-06-16 17:13:44 -07:00
Richard
ae216e365a Make Storage Driver API calls context aware.
- Change driver interface to take a context as its first argument
     - Make newFileReader take a context as its first argument
     - Make newFileWriter take a context as its first argument
     - Make blobstore exists and delete take a context as a first argument
     - Pass the layerreader's context to the storage layer
     - Pass the app's context to purgeuploads
     - Store the app's context into the blobstore (was previously null)
     - Pass the trace'd context to the storage drivers

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2015-04-27 15:58:58 -07:00
Stephen J Day
9e146437e4 Require storage drivers to report their name
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-04-22 17:30:01 -07:00
Stephen Day
02b8c7b8cf Merge pull request #294 from fkautz/pr_out_updating_msopentech_azure_sdk_for_go_to_latest_master
Updating MSOpenTech/azure-sdk-for-go to latest master
2015-03-24 13:27:19 -07:00
Frederick F. Kautz IV
11db8185bc Updating MSOpenTech/azure-sdk-for-go to latest master
Signed-off-by: Frederick F. Kautz IV <fkautz@alumni.cmu.edu>
2015-03-23 21:59:21 -07:00
Ahmet Alp Balkan
ce9f500a31 storage/driver/azure: Allow non-default realms
This enables Azure storage driver to be used with non-default
cloud endpoints like Azure China or Azure Government that does
not use `.blob.core.windows.net` FQDN suffix.

Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
2015-03-23 20:41:16 -07:00
Stephen J Day
65b0d73cb7 Move storagedriver package to registry/storage/driver
This change is slightly more complex than previous package maves in that the
package name changed. To address this, we simply always reference the package
driver as storagedriver to avoid compatbility issues with existing code. While
unfortunate, this can be cleaned up over time.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-11 12:43:04 -08:00
Renamed from storagedriver/azure/azure.go (Browse further)