Commit graph

935 commits

Author SHA1 Message Date
Stephen Day
05600271d7 Merge pull request #149 from AndreyKostov/s3-change-default-signature
storagedriver/s3: Change the default signature to v2 auth
2015-02-06 15:59:35 -08:00
Stephen J Day
d513d07e5f Add common targets to Makefile
A nice set of common default targets have been added to the Makefile to make
running tests and building binaries more straightforward. Running the Makefile
before committing should make build failures less common.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-06 15:39:11 -08:00
Stephen Day
7028457ef4 Merge pull request #150 from BrianBland/layerUploadReadFrom
Makes LayerUpload an io.ReaderFrom
2015-02-05 18:00:12 -08:00
Andrey Kostov
97be787339 Change the default signature to v2 auth
The reason we prefer v2 is that it performs better since it does not
hash the payloads.
2015-02-05 17:41:04 -08:00
Brian Bland
0bc84606cc Makes LayerUpload an io.ReaderFrom
This allows wrappers of LayerUpload to implement io.ReadFrom, which
prevents io.Copy on LayerUpload implementations from using repeated 32kB
Writes.

This has a huge performance implication, especially for s3/azure storage
drivers.
2015-02-05 17:37:52 -08:00
Stephen Day
9b9ea55797 Merge pull request #140 from jlhawn/137_auth_context
Use context for auth access controllers
2015-02-04 19:25:10 -08:00
Josh Hawn
2c3d738a05 Use context for auth access controllers
The auth package has been updated to use "golang.org/x/net/context" for
passing information between the application and the auth backend.

AccessControllers should now set a "auth.user" context value to a AuthUser
struct containing a single "Name" field for now with possible, optional, values
in the future.

The "silly" auth backend always sets the name to "silly", while the "token" auth
backend will set the name to match the "subject" claim of the JWT.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-02-04 19:17:33 -08:00
Stephen Day
15df84768d Merge pull request #145 from AndreyKostov/s3-test-fix
Skip s3 tests if no region is specified
2015-02-04 12:00:22 -08:00
Andrey Kostov
85768b7638 Skip s3 tests if no region is specified 2015-02-04 11:52:59 -08:00
Stephen Day
1a1d87801a Merge pull request #139 from stevvooe/storagedriver-base
Move shared checks to base storage driver
2015-02-04 10:58:01 -08:00
Stephen Day
d7c62e06f4 Merge pull request #141 from stevvooe/put-manifest-http-status
Manifest PUT should return 202 Accepted status
2015-02-04 10:57:17 -08:00
Stephen J Day
f5c01af13f Manifest PUT should return 202 Accepted status
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 18:27:40 -08:00
Stephen J Day
f265a29f24 Move shared checks to base storage driver
This changeset moves common checks from driver implementations into base
package. The Base type can be embedded in a driver implementation, ensuring
that incoming arguments are checked for validity.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 17:05:50 -08:00
Stephen Day
c1c7d3dabf Merge pull request #113 from stevvooe/event-sink-endpoint
Webhook notification support in registry webapp
2015-02-03 16:25:03 -08:00
Stephen Day
624fb9b565 Merge pull request #78 from ahmetalpbalkan/TestPutContentMultipleTimes
Add TestPutContentMultipleTimes to storage driver suite
2015-02-03 13:35:55 -08:00
Stephen J Day
080e329cb1 Separate request data from actor in Event
To clarify the role of actor, the request data that initiates an event has been
separated. The ActorRecord is pared down to just the username. This eliminates
confusion about where event related data should be added.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:32:37 -08:00
Stephen J Day
0a29b59e14 Webhook notification support in registry webapp
Endpoints are now created at applications startup time, using notification
configuration. The instances are then added to a Broadcaster instance, which
becomes the main event sink for the application. At request time, an event
bridge is configured to listen to repository method calls. The actor and source
of the eventBridge are created from the requeest context and application,
respectively. The result is notifications are dispatched with calls to the
context's Repository instance and are queued to each endpoint via the
broadcaster.

This commit also adds the concept of a RequestID and App.InstanceID. The
request id uniquely identifies each request and the InstanceID uniquely
identifies a run of the registry. These identifiers can be used in the future
to correlate log messages with generated events to support rich debugging.

The fields of the app were slightly reorganized for clarity and a few horrid
util functions have been removed.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:32:37 -08:00
Stephen J Day
499382dd0b Add debug server to support pprof and expvar
If configured, a debug http server will be started to serve default registered
endpoints, such as pprof and expvar. The endpoint should be secured carefully
and not available to external traffic. It is disabled by default but the
development config has been modified to make it available on localhost.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:32:37 -08:00
Stephen J Day
e5de2594ad Remove decorator package
After implementing notifications end to end, it was found that decorating
repositories was more straightforward that previously thought. It's unfortunate
to can this package, but it led to the techniques employed in
storage/notifications/listeners.go. The ultimate result turned out much better.
2015-02-03 13:30:20 -08:00
Stephen J Day
9f0c8d6616 Implement notification endpoint webhook dispatch
This changeset implements webhook notification endpoints for dispatching
registry events. Repository instances can be decorated by a listener that
converts calls into context-aware events, using a bridge. Events generated in
the bridge are written to a sink. Implementations of sink include a broadcast
and endpoint sink which can be used to configure event dispatch. Endpoints
represent a webhook notification target, with queueing and retries built in.
They can be added to a Broadcaster, which is a simple sink that writes a block
of events to several sinks, to provide a complete dispatch mechanism.

The main caveat to the current approach is that all unsent notifications are
inmemory. Best effort is made to ensure that notifications are not dropped, to
the point where queues may back up on faulty endpoints. If the endpoint is
fixed, the events will be retried and all messages will go through.

Internally, this functionality is all made up of Sink objects. The queuing
functionality is implemented with an eventQueue sink and retries are
implemented with retryingSink. Replacing the inmemory queuing with something
persistent should be as simple as replacing broadcaster with a remote queue and
that sets up the sinks to be local workers listening to that remote queue.

Metrics are kept for each endpoint and exported via expvar. This may not be a
permanent appraoch but should provide enough information for troubleshooting
notification problems.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:30:20 -08:00
Stephen J Day
14fb80d6c3 Add payload and signatures method to SignedManifest
To provide easier access to digestible content, the paylaod has been made
accessible on the signed manifest type. This hides the specifics of the
interaction with libtrust with the caveat that signatures may be parsed twice.

We'll have to have a future look at the interface for manifest as we may be
making problematic architectural decisions. We'll visit this after the initial
release.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:30:20 -08:00
Stephen J Day
af7eb42793 Event notification message definition
This commit defines the message format used to notify external parties of
activity within a registry instance. The event includes information about which
action was taken on which registry object, including what user created the
action and which instance generated the event.

Message instances can be sent throughout an application or transmitted
externally. An envelope format along with a custom media type is defined along
with tests to detect changes to the wire format.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-03 13:30:20 -08:00
Olivier Gambier
092dadde6d Merge pull request #121 from stevvooe/address-layer-upload-errors
Address server errors received during layer upload
2015-02-03 11:48:34 -08:00
Stephen Day
d91e4bc34d Merge pull request #130 from stevvooe/path-names-escaped
Prefix non-name path components
2015-02-03 11:24:45 -08:00
Stephen Day
64cdd3e024 Merge pull request #132 from stevvooe/add-tls-support
Add native TLS support to registry
2015-02-02 14:53:19 -08:00
Stephen Day
dd5e24b6f2 Merge pull request #131 from noxiouz/fix_range_header_parsing
[Client] Fix error in parsing of 'Range' header.
2015-02-02 14:52:57 -08:00
Stephen J Day
43b36970f5 Prefix non-name path components
To address the possibility of confusing registry name components with
repository paths, path components that abut user provided repository names are
escaped with a prefixed underscore. This works because repository name
components are no allowed to start with underscores. The requirements on
backend driver path names have been relaxed greatly to support this use case.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 14:47:24 -08:00
Stephen J Day
9f5916b2c4 Add native TLS support to registry
This changeset provides simple tls support for a registry instance. Simply
providing a cert and key file are enough to get a tls registry running. If the
certs are trusted by the client, tls can be used throughout the push and pull
process.

If more complex TLS options are required, it is recommend that a proxy be used.
Contributions will be accepted to add more features, if necessary.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 14:38:47 -08:00
Stephen Day
6cd2945b5c Merge pull request #125 from jlhawn/better_url_builder
Improve URL Builders
2015-02-02 13:41:01 -08:00
Anton Tiurin
9c0519c4ed [Client] Fix error in parsing of 'Range' header.
* Result of regexp.FindStringSubmatch must be checked to be not nil.
Otherwise it leads to `index out of range`.
* Range header regexp is compiled only once to speedup (5x) the header parsing.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-02-03 00:34:38 +03:00
Stephen Day
e492579718 Merge pull request #129 from noxiouz/fix_missing_mutex
[InMemory] Add missing mutex.RLock/RUnlock in List
2015-02-02 13:15:25 -08:00
Stephen J Day
0270bec916 Handle empty blob files more appropriately
Several API tests were added to ensure correct acceptance of zero-size and
empty tar files. This led to several changes in the storage backend around the
guarantees of remote file reading, which backs the layer and layer upload type.

In support of these changes, zero-length and empty checks have been added to
the digest package. These provide a sanity check against upstream tarsum
changes. The fileReader has been modified to be more robust when reading and
seeking on zero-length or non-existent files. The file no longer needs to exist
for the reader to be created. Seeks can now move beyond the end of the file,
causing reads to issue an io.EOF. This eliminates errors during certain race
conditions for reading files which should be detected by stat calls. As a part
of this, a few error types were factored out and the read buffer size was
increased to something more reasonable.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-02-02 13:01:49 -08:00
Anton Tiurin
cea15b22e3 [InMemory] Add missing mutex.RLock/RUnlock in List to protect internal map
of dir struct.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-02-02 23:37:25 +03:00
Stephen Day
2bacac0606 Merge pull request #127 from BrianBland/goamz-godep
Updates goamz dependency from crowdmob->AdRoll
2015-02-02 12:06:29 -08:00
Brian Bland
fb71af75c8 Updates goamz dependency from crowdmob->AdRoll
Also includes goamz PR #331 for s3 v4 auth + IAM role support
2015-02-02 11:03:20 -08:00
Josh Hawn
f801b9a7bd Improve URL Builders
Handles an issue where mux.Route does not set the desired scheme
when building a url and always uses `http`.

Now uses X-Forwarded-Proto when creating a URLBuilder from a request.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
2015-02-02 10:38:14 -08:00
Olivier Gambier
c41141fbd3 Merge pull request #114 from docker/stevvooe-patch-1
Add Stephen Day to maintainers file
2015-01-30 16:08:03 -08:00
Stephen Day
fce5115336 Merge pull request #120 from stevvooe/add-version-reporting
Add version reporting to registry binary
2015-01-30 11:46:32 -08:00
Stephen J Day
9abefef2a3 Quote argument in Makefile clean target
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-30 11:13:45 -08:00
Stephen Day
9e21347d75 Merge pull request #115 from dmcgowan/single-name-repo
Allow single name component repository names
2015-01-29 22:04:09 -08:00
Stephen J Day
097fce3bb2 Address server errors received during layer upload
This changeset addresses intermittent internal server errors encountered during
pushes.  The root cause has been isolated to layers that result in identical,
empty filesystems but may have some path declarations (imaginge "./"),
resulting in different tarsums. The main error message reported during these
upload problems was a 500 error, which was not correct.  Further investigation
showed the errors to be rooted in digest verification when finishing uploads.

Inspection of the surrounding code also identified a few issues. PutLayerChunk
was slightly refactered into PutLayerUploadComplete. Helper methods were
avoided to make handler less confusing. This simplification leveraged an
earlier change in the spec that moved non-complete chunk uploads to the PATCH
method. Simple logging was also added in the unknown error case that should
help to avoid mysterious 500 errors in the future.

At the same time, the glaring omission of a proper layer upload cancel method
was rectified. This has been added in this change so it is not missed in the
future.

In the future, we may want to refactor the handler code to be more
straightforward, hopefully letting us avoid these problems in the future.

Added test cases that reproduce these errors and drove these changes include
the following:

1. Push a layer with an empty body results in invalid blob upload.
2. Push a layer with a different tarsum (in this case, empty tar)
3. Deleting a layer upload works.
4. Getting status on a deleted layer upload returns 404.

Common functionality was grouped into shared functions to remove repitition.
The API tests will still require future love.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-29 21:26:35 -08:00
Stephen J Day
f926a93778 Report layer upload as unavialable when data missing
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-29 20:45:19 -08:00
Stephen J Day
c080c40030 Remove erroneous error code on layer upload delete
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-29 20:44:22 -08:00
Stephen J Day
b75455d2bd Move version printing to version package
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-29 18:32:16 -08:00
Stephen J Day
8f70e298bf Update Dockerfile to use Makefile to build binary
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-29 18:32:16 -08:00
Stephen J Day
d0abfe0b92 Report version in registry binary
We've added support to the registry command to report the current version of
the distribution package. The version package is generated with a shell script
that gets the latest tag and add "+unknown". This allows builds from "go get"
and "go install" to have a rough version number. Generated periodically, it
will provide a decent indication of what code built the binary. For more
accurate versioning, one can build with the "binaries" make target. Linker
flags are used to replace the version string with the actual current tag at
build time.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-29 18:32:01 -08:00
Stephen J Day
c4406baf8a Update AUTHORS file
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-01-29 15:30:53 -08:00
Derek McGowan
94309badec Allow single name component repository names
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
2015-01-28 14:51:02 -08:00
Stephen Day
26b01842e3 Add Stephen Day to maintainers file 2015-01-28 10:31:54 -08:00
Stephen Day
1650088629 Merge pull request #77 from ahmetalpbalkan/TestStatCall-fix
Allow modtime to be a few seconds off on TestStatCall
2015-01-27 17:07:28 -08:00