Since we fixed the rgw redirect code, it turns out that boto actually
follows on them. We just want to check if redirect is sent.
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
AWS S3 has two behaviors for recreating a bucket depending if you use
the us-standard or another region:
>>> bucket = conn.create_bucket('gaul-default', location=Location.DEFAULT)
>>> bucket = conn.create_bucket('gaul-default', location=Location.DEFAULT)
>>> bucket = conn.create_bucket('gaul-uswest', location=Location.USWest)
>>> bucket = conn.create_bucket('gaul-uswest', location=Location.USWest)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 499, in create_bucket
response.status, response.reason, body)
boto.exception.S3CreateError: S3CreateError: 409 Conflict
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>gaul-uswest</BucketName><RequestId>24B6DC3170365CD7</RequestId><HostId>hUynMTyqc9WZFxAJ2RFK6P7BqmmeHHlMl9xL2NOy56xBUnOZCAlHqGvtMeGeAfVs</HostId></Error>
Additional discussion:
https://issues.apache.org/jira/browse/JCLOUDS-334
This reverts commit c82649b635.
The suite does not support duality in behaviors (e.g.
US Standard vs. Regional behavior) so we adhere to US
Standard only.
Signed-off-by: Alfredo Deza <alfredo@deza.pe>
Test was calling _multipart_upload() with extra useless param, this
broke when we added a new param to the function.
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Previously test_object_create_bad_contentlength_mismatch_above failed
with:
Traceback (most recent call last):
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/gaul/work/s3-tests/s3tests/functional/test_headers.py", line 343, in test_object_create_bad_contentlength_mismatch_above
e = assert_raises(boto.exception.S3ResponseError, key.set_contents_from_string, content)
File "/home/gaul/work/s3-tests/s3tests/functional/utils.py", line 11, in assert_raises
callableObj(*args, **kwargs)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1379, in set_contents_from_string
encrypt_key=encrypt_key)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1246, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 725, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 914, in _send_file_internal
query_args=query_args
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 658, in make_request
retry_handler=retry_handler
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1048, in make_request
retry_handler=retry_handler)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1004, in _mexe
raise BotoServerError(response.status, response.reason, body)
BotoServerError: BotoServerError: 400 Bad Request
AWS S3 has two behaviors for recreating a bucket depending if you use
the us-standard or another region:
>>> bucket = conn.create_bucket('gaul-default', location=Location.DEFAULT)
>>> bucket = conn.create_bucket('gaul-default', location=Location.DEFAULT)
>>> bucket = conn.create_bucket('gaul-uswest', location=Location.USWest)
>>> bucket = conn.create_bucket('gaul-uswest', location=Location.USWest)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 499, in create_bucket
response.status, response.reason, body)
boto.exception.S3CreateError: S3CreateError: 409 Conflict
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>gaul-uswest</BucketName><RequestId>24B6DC3170365CD7</RequestId><HostId>hUynMTyqc9WZFxAJ2RFK6P7BqmmeHHlMl9xL2NOy56xBUnOZCAlHqGvtMeGeAfVs</HostId></Error>
Additional discussion:
https://issues.apache.org/jira/browse/JCLOUDS-334
Since other classes need to do syncs,
let's move it to the utils file.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Nuke all buckets on the master, sync, and then
nuke any remaining buckets on non-master regions.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
The multi-region tests were not being diligent
about cleaning up their buckets / keys
and making sure that those deletes were synced.
This was causing the nuke_prefixed_buckets()
method to run into errors.
This patch adds more cleanup code and
validation of that cleanup within
the pertinent tests.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Changes to actually use the
'default_region' setting in the
S3TEST_CONF file.
Previously, the first connection set
was assumed to be the default.
This means that the parse order
of the S3TEST_CONF file was dictating
how the tests behaved, resulting in
failures in some cases.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
An error code path was missing the traceback
object and was causing a key error when
that path was executed.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Add an optional parameter to trigger deterministic
file name creation (for separate write/read tasks).
Also, change the behavior when zero writers
are specified to actually generate no data.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Can now configure sync agent rest address in order to force
a sync operation. Another option is to set a waiting time for
meta sync.
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Now creating a connection per region for each user, can access
master and secondaries, and set a default region.
No longer using a specific region per user, as it doesn't
make sense.
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
can now create a region-specific configuration:
[region foo]
api_name = ...
host = ...
port = ...
and set that region to be used for specific connection:
[s3 main]
region = foo
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Use the rewind=True argument when uploading objects to make
realistic.py's use of upload_objects compatible with boto>=2.4.1, which
will try to seek to the end of the fp if rewind is not True.
Some proxies/load balancers may sanitize input and may remove
headers that are not RFC 2616 complaint. This allows several
tests to be skipped if you are putting one of these products
in front of an S3 compatible API.
- should fail when not trying to change metadata
- should succeed when changing metadata; also verify metadata
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Annotated S3 functional and error-detection test cases
with resource/method/operation/assertion properties to
enable automatic test case inventory generation.
Signed-off-by: Mark Kampe <mark.kampe@dreamhost.com>
Makes the random file generator not generate more data
than the size of the file. Previously it would generate
1 MB of data even if the file was much smaller.
Add calling_format switch with options ordinary, subdomain, and vhost to
s3tests/functional/__init__.py and /s3tests/common.py to allow the use
of OrdinaryCallingFormat, SubdomainCallingFormat, and VHostCallingFormat
in the config files for the nosetests and the tools using the common
parser.
It turns out that the e-mail comments were legitimate. The reason the
tests were passing was because the specific users now had e-mail
addresses, but this not the default case.
This decreases the run time significantly.
Unfortunately, there seem to be cleanup-related bugs
in parallel runs for now. It's not safe to actually
run with --processes=N >1 yet, but at least the right
magic incantation is now recorded.
This way, its unit tests are separate from s3tests/functional,
and s3tests/functional is the thing that actually talks to an
S3 implementation over the wire.
To actually run the fuzzer, use ./virtualenv/bin/s3tests-fuzz-headers
Overriding num-retries to 0 sometimes causes boto to fail when the server
doesn't respond fast enough. Removing the override shouldn't cause any
problems with the fuzzer.
- Change random seed generator to always spit out ints between 0 and 100,000
- Fix seeds so randomly generated seeds are *actually* the same as those seeds
read from a file.
- Tweaks to decision graph
Remaining Bugs:
- Single } encountered in format string
- _mexe complains about "BadStatusLine"
Tv looked over the fuzzer and had some idiomatic and design suggestions.
Adds several tests and reworks how expansion happens in addition to idiom
changes.
Random can sometimes include } or { which will confuse the string formatter.
Formatter escapes those values when they're doubled: }}, {{ but this
required some slight hacking to the expander.
Helper class to catch sentinal keys and turn them into random values. This
will be used to generate garbage data when expanding a decision.
Also add unit tests for expand_decision and assemble_decision
This test effectively tells the server that this request will contain no
data and any data in the body should be interpreted to be part of the
next request. By luck, we close the tcp connection beforehand so we
don't see this additional data. However, the issues caused here would
not be at the rgw/dho level.
This avoids a SAX/boto interaction bug that is not of much interest,
yet still seems "evil" enough that it fulfills the original intent
of the test.
Add a link to the relevant bug report.
The existing test_object_create_bad_md5_invalid is not valid
base64, and hence triggers a different code path. This time,
submit valid base64, but have the result not match the expected
md5sum.
Depending on the version of boto, different methods are necessary for
manipulating headers. We now support 2.0rc1 and 2.0 by sub-classing
boto.s3.connection.S3Connection and overloading
boto.connection.HTTPRequest.authorize, respectfully.
If a client specifies a range when requesting an object the server should
respond with 206 Partial Content when answering a request to get an object,
not 200 OK. It should also respond with only those bytes requested.
This commit introduces a test, test_ranged_request_response_code, that
checks those criteria.
Simpler, and all the benchmarks are pretty much as pseudorandom,
in the end; that's not a relevant characteristic. The mix of read
and write operations is.
Re-run ./bootstrap to get the s3tests-test-readwrite script
regenerated to refer to the new module name.
Now generates a list of filenames to use duing the setup, and passes this
list to the workers rather than having the workers list the bucket contents
or generate file names themselves.
Added a "warmup" stage that first writes out all the files so that content
exists for each filename.
Added setting the random seed as a command line parameter and uses this seed
for both generating the file names and the file content generator.
When the owner has no grants on their bucket, they should not be able to
write to it, but still read and write the ACL. Amazon allows users to
read the bucket, and our behavior is consistent with this.
Parent waits for the duration anyway, just make it unconditionally
kill the workers. They already had timeouts aborting them; they may
be in the middle of an operation, but we really don't care, time's up.
This also avoids oddities with the queue where there may be items
after the StopIteration.
This also avoids workers potentially not exiting within the 1-second
timeout the old code had.
This reverts commit fb8f110e69.
realistic.files was meant to work just fine with short files. Commit
91c4f88860 has a fix for the verifying
side, to support short files.
Conflicts:
s3tests/realistic.py
Given a tuple of tuples, construct several files() generators, and
yield from those randomly. Randomness is weighted based on the number
of files remaining in each group.
context is a generic Bunch that we can store items in to be shared
across modules, without invoking the wrath of the globals -- yuck!
get_next_key will generate a sequentially numbered key for us to use
fill_pools will take a variable list of gevent.pool.Pool objects and
top them all off
The file verifier will now correctly verify small files (smaller than
the md5 digest size), as long as the data that is there matches what
the digest would be.
Accessing a non-constant global from another module is tricky,
as the import binds to the original object. Rebinding the name
in the original module does not change it in importers. Use a
getter as a quick workaround.
