retry bucket removal after setting the bucket permissions if we're
getting pemission denied. This will make sure the bucket gets removed.
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
This commit allows test_bucket_head test_bucket_head,
test_bucket_head_extended, test_multipart_upload, and
test_abort_multipart_upload to succeed against AWS-S3. Also use
public buckets for head tests since they do not require private
buckets.
Signed-off-by: Andrew Gaul <andrew@gaul.org>
Since we fixed the rgw redirect code, it turns out that boto actually
follows on them. We just want to check if redirect is sent.
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
s3tests: add regression test for bug #10311 - rgw: Object is deleted automatically if conditional PUT returns 412
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Since we fixed the rgw redirect code, it turns out that boto actually
follows on them. We just want to check if redirect is sent.
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
AWS S3 has two behaviors for recreating a bucket depending if you use
the us-standard or another region:
>>> bucket = conn.create_bucket('gaul-default', location=Location.DEFAULT)
>>> bucket = conn.create_bucket('gaul-default', location=Location.DEFAULT)
>>> bucket = conn.create_bucket('gaul-uswest', location=Location.USWest)
>>> bucket = conn.create_bucket('gaul-uswest', location=Location.USWest)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 499, in create_bucket
response.status, response.reason, body)
boto.exception.S3CreateError: S3CreateError: 409 Conflict
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>gaul-uswest</BucketName><RequestId>24B6DC3170365CD7</RequestId><HostId>hUynMTyqc9WZFxAJ2RFK6P7BqmmeHHlMl9xL2NOy56xBUnOZCAlHqGvtMeGeAfVs</HostId></Error>
Additional discussion:
https://issues.apache.org/jira/browse/JCLOUDS-334
This reverts commit c82649b635.
The suite does not support duality in behaviors (e.g.
US Standard vs. Regional behavior) so we adhere to US
Standard only.
Signed-off-by: Alfredo Deza <alfredo@deza.pe>
Test was calling _multipart_upload() with extra useless param, this
broke when we added a new param to the function.
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Previously test_object_create_bad_contentlength_mismatch_above failed
with:
Traceback (most recent call last):
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/gaul/work/s3-tests/s3tests/functional/test_headers.py", line 343, in test_object_create_bad_contentlength_mismatch_above
e = assert_raises(boto.exception.S3ResponseError, key.set_contents_from_string, content)
File "/home/gaul/work/s3-tests/s3tests/functional/utils.py", line 11, in assert_raises
callableObj(*args, **kwargs)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1379, in set_contents_from_string
encrypt_key=encrypt_key)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1246, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 725, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 914, in _send_file_internal
query_args=query_args
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 658, in make_request
retry_handler=retry_handler
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1048, in make_request
retry_handler=retry_handler)
File "/home/gaul/work/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1004, in _mexe
raise BotoServerError(response.status, response.reason, body)
BotoServerError: BotoServerError: 400 Bad Request
AWS S3 has two behaviors for recreating a bucket depending if you use
the us-standard or another region:
>>> bucket = conn.create_bucket('gaul-default', location=Location.DEFAULT)
>>> bucket = conn.create_bucket('gaul-default', location=Location.DEFAULT)
>>> bucket = conn.create_bucket('gaul-uswest', location=Location.USWest)
>>> bucket = conn.create_bucket('gaul-uswest', location=Location.USWest)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 499, in create_bucket
response.status, response.reason, body)
boto.exception.S3CreateError: S3CreateError: 409 Conflict
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>BucketAlreadyOwnedByYou</Code><Message>Your previous request to create the named bucket succeeded and you already own it.</Message><BucketName>gaul-uswest</BucketName><RequestId>24B6DC3170365CD7</RequestId><HostId>hUynMTyqc9WZFxAJ2RFK6P7BqmmeHHlMl9xL2NOy56xBUnOZCAlHqGvtMeGeAfVs</HostId></Error>
Additional discussion:
https://issues.apache.org/jira/browse/JCLOUDS-334
Since other classes need to do syncs,
let's move it to the utils file.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Nuke all buckets on the master, sync, and then
nuke any remaining buckets on non-master regions.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
The multi-region tests were not being diligent
about cleaning up their buckets / keys
and making sure that those deletes were synced.
This was causing the nuke_prefixed_buckets()
method to run into errors.
This patch adds more cleanup code and
validation of that cleanup within
the pertinent tests.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Changes to actually use the
'default_region' setting in the
S3TEST_CONF file.
Previously, the first connection set
was assumed to be the default.
This means that the parse order
of the S3TEST_CONF file was dictating
how the tests behaved, resulting in
failures in some cases.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
An error code path was missing the traceback
object and was causing a key error when
that path was executed.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Add an optional parameter to trigger deterministic
file name creation (for separate write/read tasks).
Also, change the behavior when zero writers
are specified to actually generate no data.
Signed-off-by: Joe Buck <jbbuck@gmail.com>
Reviewed-by: Josh Durgin <josh.durgin@inktank.com>
Can now configure sync agent rest address in order to force
a sync operation. Another option is to set a waiting time for
meta sync.
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Now creating a connection per region for each user, can access
master and secondaries, and set a default region.
No longer using a specific region per user, as it doesn't
make sense.
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
can now create a region-specific configuration:
[region foo]
api_name = ...
host = ...
port = ...
and set that region to be used for specific connection:
[s3 main]
region = foo
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Use the rewind=True argument when uploading objects to make
realistic.py's use of upload_objects compatible with boto>=2.4.1, which
will try to seek to the end of the fp if rewind is not True.
Some proxies/load balancers may sanitize input and may remove
headers that are not RFC 2616 complaint. This allows several
tests to be skipped if you are putting one of these products
in front of an S3 compatible API.
- should fail when not trying to change metadata
- should succeed when changing metadata; also verify metadata
Signed-off-by: Yehuda Sadeh <yehuda@inktank.com>
Annotated S3 functional and error-detection test cases
with resource/method/operation/assertion properties to
enable automatic test case inventory generation.
Signed-off-by: Mark Kampe <mark.kampe@dreamhost.com>
Makes the random file generator not generate more data
than the size of the file. Previously it would generate
1 MB of data even if the file was much smaller.
Add calling_format switch with options ordinary, subdomain, and vhost to
s3tests/functional/__init__.py and /s3tests/common.py to allow the use
of OrdinaryCallingFormat, SubdomainCallingFormat, and VHostCallingFormat
in the config files for the nosetests and the tools using the common
parser.
It turns out that the e-mail comments were legitimate. The reason the
tests were passing was because the specific users now had e-mail
addresses, but this not the default case.
This decreases the run time significantly.
Unfortunately, there seem to be cleanup-related bugs
in parallel runs for now. It's not safe to actually
run with --processes=N >1 yet, but at least the right
magic incantation is now recorded.
This way, its unit tests are separate from s3tests/functional,
and s3tests/functional is the thing that actually talks to an
S3 implementation over the wire.
To actually run the fuzzer, use ./virtualenv/bin/s3tests-fuzz-headers
Overriding num-retries to 0 sometimes causes boto to fail when the server
doesn't respond fast enough. Removing the override shouldn't cause any
problems with the fuzzer.
- Change random seed generator to always spit out ints between 0 and 100,000
- Fix seeds so randomly generated seeds are *actually* the same as those seeds
read from a file.
- Tweaks to decision graph
Remaining Bugs:
- Single } encountered in format string
- _mexe complains about "BadStatusLine"
Tv looked over the fuzzer and had some idiomatic and design suggestions.
Adds several tests and reworks how expansion happens in addition to idiom
changes.
Random can sometimes include } or { which will confuse the string formatter.
Formatter escapes those values when they're doubled: }}, {{ but this
required some slight hacking to the expander.
Helper class to catch sentinal keys and turn them into random values. This
will be used to generate garbage data when expanding a decision.
Also add unit tests for expand_decision and assemble_decision
