Merge pull request #591 from cbodley/wip-68292

s3: test GetObject with PartNumber and SSE-C encryption

.gitignore

@@ -10,5 +10,6 @@
 
 /*.egg-info
 /virtualenv
+/venv
 
 config.yaml

README.rst

@@ -2,90 +2,101 @@
  S3 compatibility tests
 ========================
 
-This is a set of completely unofficial Amazon AWS S3 compatibility
+This is a set of unofficial Amazon AWS S3 compatibility
-tests, that will hopefully be useful to people implementing software
+tests, that can be useful to people implementing software
-that exposes an S3-like API.
+that exposes an S3-like API. The tests use the Boto2 and Boto3 libraries.
 
-The tests only cover the REST interface.
+The tests use the Tox tool. To get started, ensure you have the ``tox``
+software installed; e.g. on Debian/Ubuntu::
 
-TODO: test direct HTTP downloads, like a web browser would do.
+	sudo apt-get install tox
-
-The tests use the Boto library, so any e.g. HTTP-level differences
-that Boto papers over, the tests will not be able to discover. Raw
-HTTP tests may be added later.
-
-The tests use the Nose test framework. To get started, ensure you have
-the ``virtualenv`` software installed; e.g. on Debian/Ubuntu::
-
-	sudo apt-get install python-virtualenv
-
-and then run::
-
-	./bootstrap
 
 You will need to create a configuration file with the location of the
-service and two different credentials, something like this::
+service and two different credentials. A sample configuration file named
+``s3tests.conf.SAMPLE`` has been provided in this repo. This file can be
+used to run the s3 tests on a Ceph cluster started with vstart.
 
-	[DEFAULT]
+Once you have that file copied and edited, you can run the tests with::
-	## this section is just used as default for all the "s3 *"
-        ## sections, you can place these variables also directly there
 
-	## replace with e.g. "localhost" to run against local software
+	S3TEST_CONF=your.conf tox
-	host = s3.amazonaws.com
 
-	## uncomment the port to use something other than 80
+You can specify which directory of tests to run::
-	# port = 8080
 
-	## say "no" to disable TLS
+	S3TEST_CONF=your.conf tox -- s3tests_boto3/functional
-	is_secure = yes
 
-	[fixtures]
+You can specify which file of tests to run::
-	## all the buckets created will start with this prefix;
-	## {random} will be filled with random characters to pad
-	## the prefix to 30 characters long, and avoid collisions
-	bucket prefix = YOURNAMEHERE-{random}-
 
-	[s3 main]
+	S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_s3.py
-	## the tests assume two accounts are defined, "main" and "alt".
 
-	## user_id is a 64-character hexstring
+You can specify which test to run::
-	user_id = 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
 
-	## display name typically looks more like a unix login, "jdoe" etc
+	S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_s3.py::test_bucket_list_empty
-	display_name = youruseridhere
-
-	## replace these with your access keys
-	access_key = ABCDEFGHIJKLMNOPQRST
-	secret_key = abcdefghijklmnopqrstuvwxyzabcdefghijklmn
-
-	[s3 alt]
-	## another user account, used for ACL-related tests
-	user_id = 56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
-	display_name = john.doe
-	## the "alt" user needs to have email set, too
-	email = john.doe@example.com
-	access_key = NOPQRSTUVWXYZABCDEFG
-	secret_key = nopqrstuvwxyzabcdefghijklmnabcdefghijklm
-
-Once you have that, you can run the tests with::
-
-	S3TEST_CONF=your.conf ./virtualenv/bin/nosetests
-
-You can specify what test(s) to run::
-
-	S3TEST_CONF=your.conf ./virtualenv/bin/nosetests s3tests.functional.test_s3:test_bucket_list_empty
 
 Some tests have attributes set based on their current reliability and
 things like AWS not enforcing their spec stricly. You can filter tests
 based on their attributes::
 
-	S3TEST_CONF=aws.conf ./virtualenv/bin/nosetests -a '!fails_on_aws'
+	S3TEST_CONF=aws.conf tox -- -m 'not fails_on_aws'
 
+Most of the tests have both Boto3 and Boto2 versions. Tests written in
+Boto2 are in the ``s3tests`` directory. Tests written in Boto3 are
+located in the ``s3test_boto3`` directory.
 
-TODO
+You can run only the boto3 tests with::
-====
 
-- We should assume read-after-write consistency, and make the tests
+	S3TEST_CONF=your.conf tox -- s3tests_boto3/functional
-  actually request such a location.
+
+========================
+ STS compatibility tests
+========================
+
+This section contains some basic tests for the AssumeRole, GetSessionToken and AssumeRoleWithWebIdentity API's. The test file is located under ``s3tests_boto3/functional``.
+
+To run the STS tests, the vstart cluster should be started with the following parameter (in addition to any parameters already used with it)::
+
+        vstart.sh -o rgw_sts_key=abcdefghijklmnop -o rgw_s3_auth_use_sts=true
+
+Note that the ``rgw_sts_key`` can be set to anything that is 128 bits in length.
+After the cluster is up the following command should be executed::
+
+      radosgw-admin caps add --tenant=testx --uid="9876543210abcdef0123456789abcdef0123456789abcdef0123456789abcdef" --caps="roles=*"
+
+You can run only the sts tests (all the three API's) with::
+
+        S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_sts.py
+
+You can filter tests based on the attributes. There is a attribute named ``test_of_sts`` to run AssumeRole and GetSessionToken tests and ``webidentity_test`` to run the AssumeRoleWithWebIdentity tests. If you want to execute only ``test_of_sts`` tests you can apply that filter as below::
+
+        S3TEST_CONF=your.conf tox -- -m test_of_sts s3tests_boto3/functional/test_sts.py
+
+For running ``webidentity_test`` you'll need have Keycloak running.
+
+In order to run any STS test you'll need to add "iam" section to the config file. For further reference on how your config file should look check ``s3tests.conf.SAMPLE``.
+
+========================
+ IAM policy tests
+========================
+
+This is a set of IAM policy tests.
+This section covers tests for user policies such as Put, Get, List, Delete, user policies with s3 actions, conflicting user policies etc
+These tests uses Boto3 libraries. Tests are written in the ``s3test_boto3`` directory.
+
+These iam policy tests uses two users with profile name "iam" and "s3 alt" as mentioned in s3tests.conf.SAMPLE.
+If Ceph cluster is started with vstart, then above two users will get created as part of vstart with same access key, secrete key etc as mentioned in s3tests.conf.SAMPLE.
+Out of those two users, "iam" user is with capabilities --caps=user-policy=* and "s3 alt" user is without capabilities.
+Adding above capabilities to "iam" user is also taken care by vstart (If Ceph cluster is started with vstart).
+
+To run these tests, create configuration file with section "iam" and "s3 alt" refer s3tests.conf.SAMPLE.
+Once you have that configuration file copied and edited, you can run all the tests with::
+
+	S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_iam.py
+
+You can also specify specific test to run::
+
+	S3TEST_CONF=your.conf tox s3tests_boto3/functional/test_iam.py::test_put_user_policy
+
+Some tests have attributes set such as "fails_on_rgw".
+You can filter tests based on their attributes::
+
+	S3TEST_CONF=your.conf tox -- s3tests_boto3/functional/test_iam.py -m 'not fails_on_rgw'
 
-  http://aws.amazon.com/s3/faqs/#What_data_consistency_model_does_Amazon_S3_employ

bootstrap

@@ -1,51 +0,0 @@
-#!/bin/sh
-set -e
-
-if [ -f /etc/debian_version ]; then
-    for package in python-pip python-virtualenv python-dev libevent-dev; do
-        if [ "$(dpkg --status -- $package 2>/dev/null|sed -n 's/^Status: //p')" != "install ok installed" ]; then
-            # add a space after old values
-            missing="${missing:+$missing }$package"
-        fi
-    done
-    if [ -n "$missing" ]; then
-        echo "$0: missing required DEB packages. Installing via sudo." 1>&2
-        sudo apt-get -y install $missing
-    fi
-elif [ -f /etc/fedora-release ]; then
-    for package in python-pip python2-virtualenv python-devel libevent-devel; do
-        if [ "$(rpm -qa $package 2>/dev/null)" == "" ]; then
-            missing="${missing:+$missing }$package"
-        fi
-    done
-    if [ -n "$missing" ]; then
-        echo "$0: missing required RPM packages. Installing via sudo." 1>&2
-        sudo yum -y install $missing
-    fi
-elif [ -f /etc/redhat-release ]; then
-    for package in python2-pip python-virtualenv python-devel libevent-devel; do
-        if [ "$(rpm -qa $package 2>/dev/null)" == "" ]; then
-            missing="${missing:+$missing }$package"
-        fi
-    done
-    if [ -n "$missing" ]; then
-        echo "$0: missing required RPM packages. Installing via sudo." 1>&2
-        sudo yum -y install $missing
-    fi
-fi
-
-virtualenv --no-site-packages --distribute virtualenv
-
-# avoid pip bugs
-./virtualenv/bin/pip install --upgrade pip
-
-# slightly old version of setuptools; newer fails w/ requests 0.14.0
-./virtualenv/bin/pip install setuptools==32.3.1
-
-./virtualenv/bin/pip install -r requirements.txt
-
-# forbid setuptools from using the network because it'll try to use
-# easy_install, and we really wanted pip; next line will fail if pip
-# requirements.txt does not match setup.py requirements -- sucky but
-# good enough for now
-./virtualenv/bin/python setup.py develop

config.yaml.SAMPLE

@@ -1,82 +0,0 @@
-fixtures:
-## All the buckets created will start with this prefix;
-## {random} will be filled with random characters to pad
-## the prefix to 30 characters long, and avoid collisions
-  bucket prefix: YOURNAMEHERE-{random}-
-
-file_generation:
-  groups:
-## File generation works by creating N groups of files. Each group of
-## files is defined by three elements: number of files, avg(filesize),
-## and stddev(filesize) -- in that order.
-    - [1, 2, 3]
-    - [4, 5, 6]
-
-## Config for the readwrite tool.
-## The readwrite tool concurrently reads and writes to files in a
-## single bucket for a set duration.
-## Note: the readwrite tool does not need the s3.alt connection info.
-## only s3.main is used.
-readwrite:
-## The number of reader and writer worker threads. This sets how many
-## files will be read and written concurrently.
-  readers: 2
-  writers: 2
-## The duration to run in seconds. Doesn't count setup/warmup time
-  duration: 15
-
-  files:
-## The number of files to use. This number of files is created during the
-## "warmup" phase. After the warmup, readers will randomly pick a file to
-## read, and writers will randomly pick a file to overwrite
-    num: 3
-## The file size to use, in KB
-    size: 1024
-## The stddev for the file size, in KB
-    stddev: 0
-
-s3:
-## This section contains all the connection information
-
-  defaults:
-## This section contains the defaults for all of the other connections
-## below. You can also place these variables directly there.
-
-## Replace with e.g. "localhost" to run against local software
-    host: s3.amazonaws.com
-
-## Uncomment the port to use soemthing other than 80
-#    port: 8080
-
-## Say "no" to disable TLS.
-    is_secure: yes
-
-## The tests assume two accounts are defined, "main" and "alt". You
-## may add other connections to be instantianted as well, however
-## any additional ones will not be used unless your tests use them.
-
-  main:
-
-## The User ID that the S3 provider gives you. For AWS, this is
-## typically a 64-char hexstring.
-    user_id: AWS_USER_ID
-
-## Display name typically looks more like a unix login, "jdoe" etc
-    display_name: AWS_DISPLAY_NAME
-
-## The email for this account.
-    email: AWS_EMAIL
-
-## Replace these with your access keys.
-    access_key: AWS_ACCESS_KEY
-    secret_key: AWS_SECRET_KEY
-
-  alt:
-## Another user accout, used for ACL-related tests.
-
-    user_id: AWS_USER_ID
-    display_name: AWS_DISPLAY_NAME
-    email: AWS_EMAIL
-    access_key: AWS_ACCESS_KEY
-    secret_key: AWS_SECRET_KEY
-

pytest.ini

@@ -0,0 +1,51 @@
+[pytest]
+markers =
+    abac_test
+    appendobject
+    auth_aws2
+    auth_aws4
+    auth_common
+    bucket_policy
+    bucket_encryption
+    checksum
+    cloud_transition
+    encryption
+    fails_on_aws
+    fails_on_dbstore
+    fails_on_dho
+    fails_on_mod_proxy_fcgi
+    fails_on_rgw
+    fails_on_s3
+    fails_with_subdomain
+    group
+    group_policy
+    iam_account
+    iam_cross_account
+    iam_role
+    iam_tenant
+    iam_user
+    lifecycle
+    lifecycle_expiration
+    lifecycle_transition
+    list_objects_v2
+    object_lock
+    role_policy
+    session_policy
+    s3select
+    s3website
+    s3website_routing_rules
+    s3website_redirect_location
+    sns
+    sse_s3
+    storage_class
+    tagging
+    test_of_sts
+    token_claims_trust_policy_test
+    token_principal_tag_role_policy_test
+    token_request_tag_trust_policy_test
+    token_resource_tags_test
+    token_role_tags_test
+    token_tag_keys_test
+    user_policy
+    versioning
+    webidentity_test

request_decision_graph.yml

@@ -1,569 +0,0 @@
-#
-# FUZZ testing uses a probabalistic grammar to generate
-# pseudo-random requests which will be sent to a server
-# over long periods of time, with the goal of turning up 
-# garbage-input and buffer-overflow sensitivities.
-#
-# Each state ...
-#       generates/chooses contents for variables
-#       chooses a next state (from a weighted set of options)
-#
-# A terminal state is one from which there are no successors,
-# at which point a message is generated (from the variables)
-# and sent to the server.  
-#
-# The test program doesn't actually know (or care) what 
-# response should be returned ... since the goal is to
-# crash the server.
-#
-start:
-    set:
-        garbage:
-            - '{random 10-3000 printable}'
-            - '{random 10-1000 binary}'
-        garbage_no_whitespace:
-            - '{random 10-3000 printable_no_whitespace}'
-            - '{random 10-1000 binary_no_whitespace}'
-        acl_header:
-            - 'private'
-            - 'public-read'
-            - 'public-read-write'
-            - 'authenticated-read'
-            - 'bucket-owner-read'
-            - 'bucket-owner-full-control'
-            - '{random 3000 letters}'
-            - '{random 100-1000 binary_no_whitespace}'
-    choices:
-        - bucket
-        - object
-
-bucket:
-    set:
-        urlpath: '/{bucket}'
-    choices:
-        - 13 bucket_get
-        - 8 bucket_put
-        - 5 bucket_delete
-        - bucket_garbage_method
-
-bucket_garbage_method:
-    set:
-        method:
-            - '{random 1-100 printable}'
-            - '{random 10-100 binary}'
-        bucket:
-            - '{bucket_readable}'
-            - '{bucket_not_readable}'
-            - '{bucket_writable}'
-            - '{bucket_not_writable}'
-            - '2 {garbage_no_whitespace}'
-    choices:
-        - bucket_get_simple
-        - bucket_get_filtered
-        - bucket_get_uploads
-        - bucket_put_create
-        - bucket_put_versioning
-        - bucket_put_simple
-
-bucket_delete:
-    set:
-        method: DELETE
-        bucket:
-            - '{bucket_writable}'
-            - '{bucket_not_writable}'
-            - '2 {garbage_no_whitespace}'
-        query:
-            - null
-            - policy
-            - website
-            - '2 {garbage_no_whitespace}'
-    choices: []
-
-bucket_get:
-    set:
-        method: GET
-        bucket:
-            - '{bucket_readable}'
-            - '{bucket_not_readable}'
-            - '2 {garbage_no_whitespace}'
-    choices:
-        - 11 bucket_get_simple
-        - bucket_get_filtered
-        - bucket_get_uploads
-
-bucket_get_simple:
-    set:
-        query:
-            - acl
-            - policy
-            - location
-            - logging
-            - notification
-            - versions
-            - requestPayment
-            - versioning
-            - website
-            - '2 {garbage_no_whitespace}'
-    choices: []
-
-bucket_get_uploads:
-    set:
-        delimiter:
-            - null
-            - '3 delimiter={garbage_no_whitespace}'
-        prefix:
-            - null
-            - '3 prefix={garbage_no_whitespace}'
-        key_marker:
-            - null
-            - 'key-marker={object_readable}'
-            - 'key-marker={object_not_readable}'
-            - 'key-marker={invalid_key}'
-            - 'key-marker={random 100-1000 printable_no_whitespace}'
-        max_uploads:
-            - null
-            - 'max-uploads={random 1-5 binary_no_whitespace}'
-            - 'max-uploads={random 1-1000 digits}'
-        upload_id_marker:
-            - null
-            - '3 upload-id-marker={random 0-1000 printable_no_whitespace}'
-        query:
-            - 'uploads'
-            - 'uploads&{delimiter}&{prefix}'
-            - 'uploads&{max_uploads}&{key_marker}&{upload_id_marker}'
-            - '2 {garbage_no_whitespace}'
-    choices: []
-
-bucket_get_filtered:
-    set:
-        delimiter:
-            - 'delimiter={garbage_no_whitespace}'
-        prefix:
-            - 'prefix={garbage_no_whitespace}'
-        marker:
-            - 'marker={object_readable}'
-            - 'marker={object_not_readable}'
-            - 'marker={invalid_key}'
-            - 'marker={random 100-1000 printable_no_whitespace}'
-        max_keys:
-            - 'max-keys={random 1-5 binary_no_whitespace}'
-            - 'max-keys={random 1-1000 digits}'
-        query:
-            - null
-            - '{delimiter}&{prefix}'
-            - '{max-keys}&{marker}'
-            - '2 {garbage_no_whitespace}'
-    choices: []
-
-bucket_put:
-    set:
-        bucket:
-            - '{bucket_writable}'
-            - '{bucket_not_writable}'
-            - '2 {garbage_no_whitespace}'
-        method: PUT
-    choices:
-        - bucket_put_simple
-        - bucket_put_create
-        - bucket_put_versioning
-
-bucket_put_create:
-    set:
-        body:
-            - '2 {garbage}'
-            - '<CreateBucketConfiguration><LocationConstraint>{random 2-10 binary}</LocationConstraint></CreateBucketConfiguration>'
-    headers:
-        - ['0-5', 'x-amz-acl', '{acl_header}']
-    choices: []
-
-bucket_put_versioning:
-    set:
-        body:
-            - '{garbage}'
-            - '4 <VersioningConfiguration>{versioning_status}{mfa_delete_body}</VersioningConfiguration>'
-        mfa_delete_body:
-            - null
-            - '<Status>{random 2-10 binary}</Status>'
-            - '<Status>{random 2000-3000 printable}</Status>'
-        versioning_status:
-            - null
-            - '<MfaDelete>{random 2-10 binary}</MfaDelete>'
-            - '<MfaDelete>{random 2000-3000 printable}</MfaDelete>'
-        mfa_header:
-            - '{random 10-1000 printable_no_whitespace} {random 10-1000 printable_no_whitespace}'
-    headers:
-        - ['0-1', 'x-amz-mfa', '{mfa_header}']
-    choices: []
-
-bucket_put_simple:
-    set:
-        body:
-            - '{acl_body}'
-            - '{policy_body}'
-            - '{logging_body}'
-            - '{notification_body}'
-            - '{request_payment_body}'
-            - '{website_body}'
-        acl_body:
-            - null
-            - '<AccessControlPolicy>{owner}{acl}</AccessControlPolicy>'
-        owner:
-            - null
-            - '7 <Owner>{id}{display_name}</Owner>'
-        id:
-            - null
-            - '<ID>{random 10-200 binary}</ID>'
-            - '<ID>{random 1000-3000 printable}</ID>'
-        display_name:
-            - null
-            - '2 <DisplayName>{random 10-200 binary}</DisplayName>'
-            - '2 <DisplayName>{random 1000-3000 printable}</DisplayName>'
-            - '2 <DisplayName>{random 10-300 letters}@{random 10-300 letters}.{random 2-4 letters}</DisplayName>'
-        acl:
-            - null
-            - '10 <AccessControlList><Grant>{grantee}{permission}</Grant></AccessControlList>'
-        grantee:
-            - null
-            - '7 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">{id}{display_name}</Grantee>'
-        permission:
-            - null
-            - '7 <Permission>{permission_value}</Permission>'
-        permission_value:
-            - '2 {garbage}'
-            - FULL_CONTROL
-            - WRITE
-            - WRITE_ACP
-            - READ
-            - READ_ACP
-        policy_body:
-            - null
-            - '2 {garbage}'
-        logging_body:
-            - null
-            - '<BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01" />'
-            - '<BucketLoggingStatus xmlns="http://doc.s3.amazonaws.com/2006-03-01"><LoggingEnabled>{bucket}{target_prefix}{target_grants}</LoggingEnabled></BucketLoggingStatus>'
-        target_prefix:
-            - null
-            - '<TargetPrefix>{random 10-1000 printable}</TargetPrefix>'
-            - '<TargetPrefix>{random 10-1000 binary}</TargetPrefix>'
-        target_grants:
-            - null
-            - '10 <TargetGrants><Grant>{grantee}{permission}</Grant></TargetGrants>'
-        notification_body:
-            - null
-            - '<NotificationConfiguration />'
-            - '2 <NotificationConfiguration><TopicConfiguration>{topic}{event}</TopicConfiguration></NotificationConfiguration>'
-        topic:
-            - null
-            - '2 <Topic>{garbage}</Topic>'
-        event:
-            - null
-            - '<Event>s3:ReducedRedundancyLostObject</Event>'
-            - '2 <Event>{garbage}</Event>'
-        request_payment_body:
-            - null
-            - '<RequestPaymentConfiguration xlmns="http://s3.amazonaws.com/doc/2006-03-01/"><Payer>{payer}</Payer></RequestPaymentConfiguration>'
-        payer:
-            - Requester
-            - BucketOwner
-            - '2 {garbage}'
-        website_body:
-            - null
-            - '<WebsiteConfiguration>{index_doc}{error_doc}{routing_rules}</WebsiteConfiguration>'
-            - '<WebsiteConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">{index_doc}{error_doc}{routing_rules}</WebsiteConfiguration>'
-        index_doc:
-            - null
-            - '<IndexDocument>{filename}</IndexDocument>'
-            - '<IndexDocument><Suffix>{filename}</Suffix></IndexDocument>'
-        filename:
-            - null
-            - '2 {garbage}'
-            - '{random 2-10 printable}.html'
-            - '{random 100-1000 printable}.html'
-            - '{random 100-1000 printable_no_whitespace}.html'
-        error_doc:
-            - null
-            - '<ErrorDocument>{filename}</ErrorDocument>'
-            - '<ErrorDocument><Key>{filename}</Key></ErrorDocument>'
-        routing_rules:
-            - null
-            - ['0-10', '<RoutingRules>{routing_rules_content}</RoutingRules>']
-        routing_rules_content:
-            - null
-            - ['0-1000', '<RoutingRule>{routing_rule}</RoutingRule>']
-        routing_rule:
-            - null
-            - ['0-2', '{routing_rule_condition}{routing_rule_redirect}']
-        routing_rule_condition:
-            - null
-            - ['0-10', '<Condition>{KeyPrefixEquals}{HttpErrorCodeReturnedEquals}</Condition>']
-        KeyPrefixEquals:
-            - null
-            - ['0-2', '<KeyPrefixEquals>{filename}</KeyPrefixEquals>']
-        HttpErrorCodeReturnedEquals:
-            - null
-            - ['0-2', '<HttpErrorCodeReturnedEquals>{HttpErrorCode}</HttpErrorCodeReturnedEquals>']
-        HttpErrorCode:
-            - null
-            - '2 {garbage}'
-            - '{random 1-10 digits}'
-            - '{random 1-100 printable}'
-        routing_rule_redirect:
-            - null
-            - '{protocol}{hostname}{ReplaceKeyPrefixWith}{ReplaceKeyWith}{HttpRedirectCode}'
-        protocol:
-            - null
-            - '<Protocol>http</Protocol>'
-            - '<Protocol>https</Protocol>'
-            - ['1-5', '<Protocol>{garbage}</Protocol>']
-            - ['1-5', '<Protocol>{filename}</Protocol>']
-        hostname:
-            - null
-            - ['1-5', '<HostHame>{hostname_val}</HostHame>']
-            - ['1-5', '<HostHame>{garbage}</HostHame>']
-        hostname_val:
-            - null
-            - '{random 1-255 printable_no_whitespace}'
-            - '{random 1-255 printable}'
-            - '{random 1-255 punctuation}'
-            - '{random 1-255 whitespace}'
-            - '{garbage}'
-        ReplaceKeyPrefixWith:
-            - null
-            - ['1-5', '<ReplaceKeyPrefixWith>{filename}</ReplaceKeyPrefixWith>']
-        HttpRedirectCode:
-            - null
-            - ['1-5', '<HttpRedirectCode>{random 1-10 digits}</HttpRedirectCode>']
-            - ['1-5', '<HttpRedirectCode>{random 1-100 printable}</HttpRedirectCode>']
-            - ['1-5', '<HttpRedirectCode>{filename}</HttpRedirectCode>']
-
-    choices: []
-
-object:
-    set:
-        urlpath: '/{bucket}/{object}'
-
-        range_header:
-            - null
-            - 'bytes={random 1-2 digits}-{random 1-4 digits}'
-            - 'bytes={random 1-1000 binary_no_whitespace}'
-        if_modified_since_header:
-            - null
-            - '2 {garbage_no_whitespace}'
-        if_match_header:
-            - null
-            - '2 {garbage_no_whitespace}'
-        if_none_match_header:
-            - null
-            - '2 {garbage_no_whitespace}'
-    choices:
-        - object_delete
-        - object_get
-        - object_put
-        - object_head
-        - object_garbage_method
-
-object_garbage_method:
-    set:
-        method:
-            - '{random 1-100 printable}'
-            - '{random 10-100 binary}'
-        bucket:
-            - '{bucket_readable}'
-            - '{bucket_not_readable}'
-            - '{bucket_writable}'
-            - '{bucket_not_writable}'
-            - '2 {garbage_no_whitespace}'
-        object:
-            - '{object_readable}'
-            - '{object_not_readable}'
-            - '{object_writable}'
-            - '{object_not_writable}'
-            - '2 {garbage_no_whitespace}'
-    choices:
-        - object_get_query
-        - object_get_head_simple
-
-object_delete:
-    set:
-        method: DELETE
-        bucket:
-            - '5 {bucket_writable}'
-            - '{bucket_not_writable}'
-            - '{garbage_no_whitespace}'
-        object:
-            - '{object_writable}'
-            - '{object_not_writable}'
-            - '2 {garbage_no_whitespace}'
-    choices: []
-
-object_get:
-    set:
-        method: GET
-        bucket:
-            - '5 {bucket_readable}'
-            - '{bucket_not_readable}'
-            - '{garbage_no_whitespace}'
-        object:
-            - '{object_readable}'
-            - '{object_not_readable}'
-            - '{garbage_no_whitespace}'
-    choices:
-        - 5 object_get_head_simple
-        - 2 object_get_query
-
-object_get_query:
-    set:
-        query: 
-            - 'torrent'
-            - 'acl'
-    choices: []
-
-object_get_head_simple:
-    set: {}
-    headers:
-        - ['0-1', 'range', '{range_header}']
-        - ['0-1', 'if-modified-since', '{if_modified_since_header}']
-        - ['0-1', 'if-unmodified-since', '{if_modified_since_header}']
-        - ['0-1', 'if-match', '{if_match_header}']
-        - ['0-1', 'if-none-match', '{if_none_match_header}']
-    choices: []
-
-object_head:
-    set:
-        method: HEAD
-        bucket:
-            - '5 {bucket_readable}'
-            - '{bucket_not_readable}'
-            - '{garbage_no_whitespace}'
-        object:
-            - '{object_readable}'
-            - '{object_not_readable}'
-            - '{garbage_no_whitespace}'
-    choices:
-        - object_get_head_simple
-
-object_put:
-    set:
-        method: PUT
-        bucket:
-            - '5 {bucket_writable}'
-            - '{bucket_not_writable}'
-            - '{garbage_no_whitespace}'
-        object:
-            - '{object_writable}'
-            - '{object_not_writable}'
-            - '{garbage_no_whitespace}'
-        cache_control:
-            - null
-            - '{garbage_no_whitespace}'
-            - 'no-cache'
-        content_disposition:
-            - null
-            - '{garbage_no_whitespace}'
-        content_encoding:
-            - null
-            - '{garbage_no_whitespace}'
-        content_length:
-            - '{random 1-20 digits}'
-            - '{garbage_no_whitespace}'
-        content_md5:
-            - null
-            - '{garbage_no_whitespace}'
-        content_type:
-            - null
-            - 'binary/octet-stream'
-            - '{garbage_no_whitespace}'
-        expect:
-            - null
-            - '100-continue'
-            - '{garbage_no_whitespace}'
-        expires:
-            - null
-            - '{random 1-10000000 digits}'
-            - '{garbage_no_whitespace}'
-        meta_key:
-            - null
-            - 'foo'
-            - '{garbage_no_whitespace}'
-        meta_value:
-            - null
-            - '{garbage_no_whitespace}'
-    choices:
-        - object_put_simple
-        - object_put_acl
-        - object_put_copy
-
-object_put_simple:
-    set: {}
-    headers:
-        - ['0-1', 'cache-control', '{cache_control}']
-        - ['0-1', 'content-disposition', '{content_disposition}']
-        - ['0-1', 'content-encoding', '{content_encoding}']
-        - ['0-1', 'content-length', '{content_length}']
-        - ['0-1', 'content-md5', '{content_md5}']
-        - ['0-1', 'content-type', '{content_type}']
-        - ['0-1', 'expect', '{expect}']
-        - ['0-1', 'expires', '{expires}']
-        - ['0-1', 'x-amz-acl', '{acl_header}']
-        - ['0-6', 'x-amz-meta-{meta_key}', '{meta_value}']
-    choices: []
-
-object_put_acl:
-    set:
-        query: 'acl'
-        body:
-            - null
-            - '2 {garbage}'
-            - '<AccessControlPolicy>{owner}{acl}</AccessControlPolicy>'
-        owner:
-            - null
-            - '7 <Owner>{id}{display_name}</Owner>'
-        id:
-            - null
-            - '<ID>{random 10-200 binary}</ID>'
-            - '<ID>{random 1000-3000 printable}</ID>'
-        display_name:
-            - null
-            - '2 <DisplayName>{random 10-200 binary}</DisplayName>'
-            - '2 <DisplayName>{random 1000-3000 printable}</DisplayName>'
-            - '2 <DisplayName>{random 10-300 letters}@{random 10-300 letters}.{random 2-4 letters}</DisplayName>'
-        acl:
-            - null
-            - '10 <AccessControlList><Grant>{grantee}{permission}</Grant></AccessControlList>'
-        grantee:
-            - null
-            - '7 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">{id}{display_name}</Grantee>'
-        permission:
-            - null
-            - '7 <Permission>{permission_value}</Permission>'
-        permission_value:
-            - '2 {garbage}'
-            - FULL_CONTROL
-            - WRITE
-            - WRITE_ACP
-            - READ
-            - READ_ACP
-    headers:
-        - ['0-1', 'cache-control', '{cache_control}']
-        - ['0-1', 'content-disposition', '{content_disposition}']
-        - ['0-1', 'content-encoding', '{content_encoding}']
-        - ['0-1', 'content-length', '{content_length}']
-        - ['0-1', 'content-md5', '{content_md5}']
-        - ['0-1', 'content-type', '{content_type}']
-        - ['0-1', 'expect', '{expect}']
-        - ['0-1', 'expires', '{expires}']
-        - ['0-1', 'x-amz-acl', '{acl_header}']
-    choices: []
-
-object_put_copy:
-    set: {}
-    headers:
-        - ['1-1', 'x-amz-copy-source', '{source_object}']
-        - ['0-1', 'x-amz-acl', '{acl_header}']
-        - ['0-1', 'x-amz-metadata-directive', '{metadata_directive}']
-        - ['0-1', 'x-amz-copy-source-if-match', '{if_match_header}']
-        - ['0-1', 'x-amz-copy-source-if-none-match', '{if_none_match_header}']
-        - ['0-1', 'x-amz-copy-source-if-modified-since', '{if_modified_since_header}']
-        - ['0-1', 'x-amz-copy-source-if-unmodified-since', '{if_modified_since_header}']
-    choices: []

requirements.txt

@@ -1,11 +1,15 @@
 PyYAML
-nose >=1.0.0
 boto >=2.6.0
-bunch >=1.0.0
+boto3 >=1.0.0
+# botocore-1.28 broke v2 signatures, see https://tracker.ceph.com/issues/58059
+botocore <1.28.0
+munch >=2.0.0
 # 0.14 switches to libev, that means bootstrap needs to change too
 gevent >=1.0
 isodate >=0.4.4
-requests ==0.14.0
+requests >=2.23.0
-pytz >=2011k
+pytz
-ordereddict
 httplib2
+lxml
+pytest
+tox

s3tests.conf.SAMPLE

@@ -0,0 +1,171 @@
+[DEFAULT]
+## this section is just used for host, port and bucket_prefix
+
+# host set for rgw in vstart.sh
+host = localhost
+
+# port set for rgw in vstart.sh
+port = 8000
+
+## say "False" to disable TLS
+is_secure = False
+
+## say "False" to disable SSL Verify
+ssl_verify = False
+
+[fixtures]
+## all the buckets created will start with this prefix;
+## {random} will be filled with random characters to pad
+## the prefix to 30 characters long, and avoid collisions
+bucket prefix = yournamehere-{random}-
+
+# all the iam account resources (users, roles, etc) created
+# will start with this name prefix
+iam name prefix = s3-tests-
+
+# all the iam account resources (users, roles, etc) created
+# will start with this path prefix
+iam path prefix = /s3-tests/
+
+[s3 main]
+# main display_name set in vstart.sh
+display_name = M. Tester
+
+# main user_idname set in vstart.sh
+user_id = testid
+
+# main email set in vstart.sh
+email = tester@ceph.com
+
+# zonegroup api_name for bucket location
+api_name = default
+
+## main AWS access key
+access_key = 0555b35654ad1656d804
+
+## main AWS secret key
+secret_key = h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
+
+## replace with key id obtained when secret is created, or delete if KMS not tested
+#kms_keyid = 01234567-89ab-cdef-0123-456789abcdef
+
+## Storage classes
+#storage_classes = "LUKEWARM, FROZEN"
+
+## Lifecycle debug interval (default: 10)
+#lc_debug_interval = 20
+
+[s3 alt]
+# alt display_name set in vstart.sh
+display_name = john.doe
+## alt email set in vstart.sh
+email = john.doe@example.com
+
+# alt user_id set in vstart.sh
+user_id = 56789abcdef0123456789abcdef0123456789abcdef0123456789abcdef01234
+
+# alt AWS access key set in vstart.sh
+access_key = NOPQRSTUVWXYZABCDEFG
+
+# alt AWS secret key set in vstart.sh
+secret_key = nopqrstuvwxyzabcdefghijklmnabcdefghijklm
+
+#[s3 cloud]
+## to run the testcases with "cloud_transition" attribute.
+## Note: the waiting time may have to tweaked depending on
+## the I/O latency to the cloud endpoint.
+
+## host set for cloud endpoint
+# host = localhost
+
+## port set for cloud endpoint
+# port = 8001
+
+## say "False" to disable TLS
+# is_secure = False
+
+## cloud endpoint credentials
+# access_key = 0555b35654ad1656d804
+# secret_key = h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==
+
+## storage class configured as cloud tier on local rgw server
+# cloud_storage_class = CLOUDTIER
+
+## Below are optional -
+
+## Above configured cloud storage class config options
+# retain_head_object = false
+# target_storage_class = Target_SC
+# target_path = cloud-bucket
+
+## another regular storage class to test multiple transition rules,
+# storage_class = S1
+
+[s3 tenant]
+# tenant display_name set in vstart.sh
+display_name = testx$tenanteduser
+
+# tenant user_id set in vstart.sh
+user_id = 9876543210abcdef0123456789abcdef0123456789abcdef0123456789abcdef
+
+# tenant AWS secret key set in vstart.sh
+access_key = HIJKLMNOPQRSTUVWXYZA
+
+# tenant AWS secret key set in vstart.sh
+secret_key = opqrstuvwxyzabcdefghijklmnopqrstuvwxyzab
+
+# tenant email set in vstart.sh
+email = tenanteduser@example.com
+
+# tenant name
+tenant = testx
+
+#following section needs to be added for all sts-tests
+[iam]
+#used for iam operations in sts-tests
+#email from vstart.sh
+email = s3@example.com
+
+#user_id from vstart.sh
+user_id = 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
+
+#access_key from vstart.sh
+access_key = ABCDEFGHIJKLMNOPQRST
+
+#secret_key vstart.sh
+secret_key = abcdefghijklmnopqrstuvwxyzabcdefghijklmn
+
+#display_name from vstart.sh
+display_name = youruseridhere
+
+# iam account root user for iam_account tests
+[iam root]
+access_key = AAAAAAAAAAAAAAAAAAaa
+secret_key = aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
+user_id = RGW11111111111111111
+email = account1@ceph.com
+
+# iam account root user in a different account than [iam root]
+[iam alt root]
+access_key = BBBBBBBBBBBBBBBBBBbb
+secret_key = bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
+user_id = RGW22222222222222222
+email = account2@ceph.com
+
+#following section needs to be added when you want to run Assume Role With Webidentity test
+[webidentity]
+#used for assume role with web identity test in sts-tests
+#all parameters will be obtained from ceph/qa/tasks/keycloak.py
+token=<access_token>
+
+aud=<obtained after introspecting token>
+
+sub=<obtained after introspecting token>
+
+azp=<obtained after introspecting token>
+
+user_token=<access token for a user, with attribute Department=[Engineering, Marketing>]
+
+thumbprint=<obtained from x509 certificate>
+
+KC_REALM=<name of the realm>

s3tests/analysis/rwstats.py

@@ -1,142 +0,0 @@
-#!/usr/bin/python
-import sys
-import os
-import yaml
-import optparse
-
-NANOSECONDS = int(1e9)
-
-# Output stats in a format similar to siege
-# see http://www.joedog.org/index/siege-home
-OUTPUT_FORMAT = """Stats for type: [{type}]
-Transactions:            {trans:>11} hits
-Availability:            {avail:>11.2f} %
-Elapsed time:            {elapsed:>11.2f} secs
-Data transferred:        {data:>11.2f} MB
-Response time:           {resp_time:>11.2f} secs
-Transaction rate:        {trans_rate:>11.2f} trans/sec
-Throughput:              {data_rate:>11.2f} MB/sec
-Concurrency:             {conc:>11.2f}
-Successful transactions: {trans_success:>11}
-Failed transactions:     {trans_fail:>11}
-Longest transaction:     {trans_long:>11.2f}
-Shortest transaction:    {trans_short:>11.2f}
-"""
-
-def parse_options():
-    usage = "usage: %prog [options]"
-    parser = optparse.OptionParser(usage=usage)
-    parser.add_option(
-        "-f", "--file", dest="input", metavar="FILE",
-        help="Name of input YAML file. Default uses sys.stdin")
-    parser.add_option(
-        "-v", "--verbose", dest="verbose", action="store_true",
-        help="Enable verbose output")
-
-    (options, args) = parser.parse_args()
-
-    if not options.input and os.isatty(sys.stdin.fileno()):
-        parser.error("option -f required if no data is provided "
-                     "in stdin")
-
-    return (options, args)
-
-def main():
-    (options, args) = parse_options()
-
-    total     = {}
-    durations = {}
-    min_time  = {}
-    max_time  = {}
-    errors    = {}
-    success   = {}
-
-    calculate_stats(options, total, durations, min_time, max_time, errors,
-                    success)
-    print_results(total, durations, min_time, max_time, errors, success)
-
-def calculate_stats(options, total, durations, min_time, max_time, errors,
-                    success):
-    print 'Calculating statistics...'
-    
-    f = sys.stdin
-    if options.input:
-        f = file(options.input, 'r')
-
-    for item in yaml.safe_load_all(f):
-        type_ = item.get('type')
-        if type_ not in ('r', 'w'):
-            continue # ignore any invalid items
-
-        if 'error' in item:
-            errors[type_] = errors.get(type_, 0) + 1
-            continue # skip rest of analysis for this item
-        else:
-            success[type_] = success.get(type_, 0) + 1
-
-        # parse the item
-        data_size = item['chunks'][-1][0]
-        duration = item['duration']
-        start = item['start']
-        end = start + duration / float(NANOSECONDS)
-
-        if options.verbose:
-            print "[{type}] POSIX time: {start:>18.2f} - {end:<18.2f} " \
-                  "{data:>11.2f} KB".format(
-                type=type_,
-                start=start,
-                end=end,
-                data=data_size / 1024.0, # convert to KB
-                )
-
-        # update time boundaries
-        prev = min_time.setdefault(type_, start)
-        if start < prev:
-            min_time[type_] = start
-        prev = max_time.setdefault(type_, end)
-        if end > prev:
-            max_time[type_] = end
-
-        # save the duration
-        if type_ not in durations:
-            durations[type_] = []
-        durations[type_].append(duration)
-
-        # add to running totals
-        total[type_] = total.get(type_, 0) + data_size
-
-def print_results(total, durations, min_time, max_time, errors, success):
-    for type_ in total.keys():
-        trans_success = success.get(type_, 0)
-        trans_fail    = errors.get(type_, 0)
-        trans         = trans_success + trans_fail
-        avail         = trans_success * 100.0 / trans
-        elapsed       = max_time[type_] - min_time[type_]
-        data          = total[type_] / 1024.0 / 1024.0 # convert to MB
-        resp_time     = sum(durations[type_]) / float(NANOSECONDS) / \
-                        len(durations[type_])
-        trans_rate    = trans / elapsed
-        data_rate     = data / elapsed
-        conc          = trans_rate * resp_time
-        trans_long    = max(durations[type_]) / float(NANOSECONDS)
-        trans_short   = min(durations[type_]) / float(NANOSECONDS)
-
-        print OUTPUT_FORMAT.format(
-            type=type_,
-            trans_success=trans_success,
-            trans_fail=trans_fail,
-            trans=trans,
-            avail=avail,
-            elapsed=elapsed,
-            data=data,
-            resp_time=resp_time,
-            trans_rate=trans_rate,
-            data_rate=data_rate,
-            conc=conc,
-            trans_long=trans_long,
-            trans_short=trans_short,
-            )
-
-if __name__ == '__main__':
-    main()
-

s3tests/common.py

@@ -1,13 +1,18 @@
 import boto.s3.connection
-import bunch
+import munch
 import itertools
 import os
 import random
 import string
 import yaml
+import re
+from lxml import etree
 
-s3 = bunch.Bunch()
+from doctest import Example
-config = bunch.Bunch()
+from lxml.doctestcompare import LXMLOutputChecker
+
+s3 = munch.Munch()
+config = munch.Munch()
 prefix = ''
 
 bucket_counter = itertools.count(1)
@@ -46,10 +51,10 @@ def nuke_bucket(bucket):
         while deleted_cnt:
             deleted_cnt = 0
             for key in bucket.list():
-                print 'Cleaning bucket {bucket} key {key}'.format(
+                print('Cleaning bucket {bucket} key {key}'.format(
                     bucket=bucket,
                     key=key,
-                    )
+                    ))
                 key.set_canned_acl('private')
                 key.delete()
                 deleted_cnt += 1
@@ -62,26 +67,26 @@ def nuke_bucket(bucket):
             and e.body == ''):
             e.error_code = 'AccessDenied'
         if e.error_code != 'AccessDenied':
-            print 'GOT UNWANTED ERROR', e.error_code
+            print('GOT UNWANTED ERROR', e.error_code)
             raise
         # seems like we're not the owner of the bucket; ignore
         pass
 
 def nuke_prefixed_buckets():
-    for name, conn in s3.items():
+    for name, conn in list(s3.items()):
-        print 'Cleaning buckets from connection {name}'.format(name=name)
+        print('Cleaning buckets from connection {name}'.format(name=name))
         for bucket in conn.get_all_buckets():
             if bucket.name.startswith(prefix):
-                print 'Cleaning bucket {bucket}'.format(bucket=bucket)
+                print('Cleaning bucket {bucket}'.format(bucket=bucket))
                 nuke_bucket(bucket)
 
-    print 'Done with cleanup of test buckets.'
+    print('Done with cleanup of test buckets.')
 
 def read_config(fp):
-    config = bunch.Bunch()
+    config = munch.Munch()
     g = yaml.safe_load_all(fp)
     for new in g:
-        config.update(bunch.bunchify(new))
+        config.update(munch.Munchify(new))
     return config
 
 def connect(conf):
@@ -92,7 +97,7 @@ def connect(conf):
         access_key='aws_access_key_id',
         secret_key='aws_secret_access_key',
         )
-    kwargs = dict((mapping[k],v) for (k,v) in conf.iteritems() if k in mapping)
+    kwargs = dict((mapping[k],v) for (k,v) in conf.items() if k in mapping)
     #process calling_format argument
     calling_formats = dict(
         ordinary=boto.s3.connection.OrdinaryCallingFormat(),
@@ -100,7 +105,7 @@ def connect(conf):
         vhost=boto.s3.connection.VHostCallingFormat(),
         )
     kwargs['calling_format'] = calling_formats['ordinary']
-    if conf.has_key('calling_format'):
+    if 'calling_format' in conf:
         raw_calling_format = conf['calling_format']
         try:
             kwargs['calling_format'] = calling_formats[raw_calling_format]
@@ -141,7 +146,7 @@ def setup():
         raise RuntimeError("Empty Prefix! Aborting!")
 
     defaults = config.s3.defaults
-    for section in config.s3.keys():
+    for section in list(config.s3.keys()):
         if section == 'defaults':
             continue
 
@@ -181,3 +186,117 @@ def get_new_bucket(connection=None):
 
 def teardown():
     nuke_prefixed_buckets()
+
+def with_setup_kwargs(setup, teardown=None):
+    """Decorator to add setup and/or teardown methods to a test function::
+
+      @with_setup_args(setup, teardown)
+      def test_something():
+          " ... "
+
+    The setup function should return (kwargs) which will be passed to
+    test function, and teardown function.
+
+    Note that `with_setup_kwargs` is useful *only* for test functions, not for test
+    methods or inside of TestCase subclasses.
+    """
+    def decorate(func):
+        kwargs = {}
+
+        def test_wrapped(*args, **kwargs2):
+            k2 = kwargs.copy()
+            k2.update(kwargs2)
+            k2['testname'] = func.__name__
+            func(*args, **k2)
+
+        test_wrapped.__name__ = func.__name__
+
+        def setup_wrapped():
+            k = setup()
+            kwargs.update(k)
+            if hasattr(func, 'setup'):
+                func.setup()
+        test_wrapped.setup = setup_wrapped
+
+        if teardown:
+            def teardown_wrapped():
+                if hasattr(func, 'teardown'):
+                    func.teardown()
+                teardown(**kwargs)
+
+            test_wrapped.teardown = teardown_wrapped
+        else:
+            if hasattr(func, 'teardown'):
+                test_wrapped.teardown = func.teardown()
+        return test_wrapped
+    return decorate
+
+# Demo case for the above, when you run test_gen():
+# _test_gen will run twice,
+# with the following stderr printing
+# setup_func {'b': 2}
+# testcase ('1',) {'b': 2, 'testname': '_test_gen'}
+# teardown_func {'b': 2}
+# setup_func {'b': 2}
+# testcase () {'b': 2, 'testname': '_test_gen'}
+# teardown_func {'b': 2}
+# 
+#def setup_func():
+#    kwargs = {'b': 2}
+#    print("setup_func", kwargs, file=sys.stderr)
+#    return kwargs
+#
+#def teardown_func(**kwargs):
+#    print("teardown_func", kwargs, file=sys.stderr)
+#
+#@with_setup_kwargs(setup=setup_func, teardown=teardown_func)
+#def _test_gen(*args, **kwargs):
+#    print("testcase", args, kwargs, file=sys.stderr)
+#
+#def test_gen():
+#    yield _test_gen, '1'
+#    yield _test_gen
+
+def trim_xml(xml_str):
+    p = etree.XMLParser(encoding="utf-8", remove_blank_text=True)
+    xml_str = bytes(xml_str, "utf-8")
+    elem = etree.XML(xml_str, parser=p)
+    return etree.tostring(elem, encoding="unicode")
+
+def normalize_xml(xml, pretty_print=True):
+    if xml is None:
+        return xml
+
+    root = etree.fromstring(xml.encode(encoding='ascii'))
+
+    for element in root.iter('*'):
+        if element.text is not None and not element.text.strip():
+            element.text = None
+        if element.text is not None:
+            element.text = element.text.strip().replace("\n", "").replace("\r", "")
+        if element.tail is not None and not element.tail.strip():
+            element.tail = None
+        if element.tail is not None:
+            element.tail = element.tail.strip().replace("\n", "").replace("\r", "")
+
+    # Sort the elements
+    for parent in root.xpath('//*[./*]'): # Search for parent elements
+          parent[:] = sorted(parent,key=lambda x: x.tag)
+
+    xmlstr = etree.tostring(root, encoding="unicode", pretty_print=pretty_print)
+    # there are two different DTD URIs
+    xmlstr = re.sub(r'xmlns="[^"]+"', 'xmlns="s3"', xmlstr)
+    xmlstr = re.sub(r'xmlns=\'[^\']+\'', 'xmlns="s3"', xmlstr)
+    for uri in ['http://doc.s3.amazonaws.com/doc/2006-03-01/', 'http://s3.amazonaws.com/doc/2006-03-01/']:
+        xmlstr = xmlstr.replace(uri, 'URI-DTD')
+    #xmlstr = re.sub(r'>\s+', '>', xmlstr, count=0, flags=re.MULTILINE)
+    return xmlstr
+
+def assert_xml_equal(got, want):
+    assert want is not None, 'Wanted XML cannot be None'
+    if got is None:
+        raise AssertionError('Got input to validate was None')
+    checker = LXMLOutputChecker()
+    if not checker.check_output(want, got, 0):
+        message = checker.output_difference(Example("", want), got, 0)
+        raise AssertionError(message)

s3tests/functional/AnonymousAuth.py

@@ -1,5 +0,0 @@
-from boto.auth_handler import AuthHandler
-
-class AnonymousAuthHandler(AuthHandler):
-    def add_auth(self, http_request, **kwargs):
-        return # Nothing to do for anonymous access!

s3tests/functional/__init__.py

@@ -1,17 +1,21 @@
-import ConfigParser
+import sys
+import configparser
 import boto.exception
 import boto.s3.connection
-import bunch
+import munch
 import itertools
 import os
 import random
 import string
+import pytest
+from http.client import HTTPConnection, HTTPSConnection
+from urllib.parse import urlparse
 
 from .utils import region_sync_meta
 
-s3 = bunch.Bunch()
+s3 = munch.Munch()
-config = bunch.Bunch()
+config = munch.Munch()
-targets = bunch.Bunch()
+targets = munch.Munch()
 
 # this will be assigned by setup()
 prefix = None
@@ -55,17 +59,17 @@ def choose_bucket_prefix(template, max_len=30):
 
 
 def nuke_prefixed_buckets_on_conn(prefix, name, conn):
-    print 'Cleaning buckets from connection {name} prefix {prefix!r}.'.format(
+    print('Cleaning buckets from connection {name} prefix {prefix!r}.'.format(
         name=name,
         prefix=prefix,
-        )
+        ))
 
     for bucket in conn.get_all_buckets():
-        print 'prefix=',prefix
+        print('prefix=',prefix)
         if bucket.name.startswith(prefix):
-            print 'Cleaning bucket {bucket}'.format(bucket=bucket)
+            print('Cleaning bucket {bucket}'.format(bucket=bucket))
             success = False
-            for i in xrange(2):
+            for i in range(2):
                 try:
                     try:
                         iterator = iter(bucket.list_versions())
@@ -81,17 +85,23 @@ def nuke_prefixed_buckets_on_conn(prefix, name, conn):
                             raise e
                         keys = bucket.list();
                     for key in keys:
-                        print 'Cleaning bucket {bucket} key {key}'.format(
+                        print('Cleaning bucket {bucket} key {key}'.format(
                             bucket=bucket,
                             key=key,
-                            )
+                            ))
                         # key.set_canned_acl('private')
                         bucket.delete_key(key.name, version_id = key.version_id)
+                    try:
                         bucket.delete()
+                    except boto.exception.S3ResponseError as e:
+                        # if DELETE times out, the retry may see NoSuchBucket
+                        if e.error_code != 'NoSuchBucket':
+                            raise e
+                        pass
                     success = True
                 except boto.exception.S3ResponseError as e:
                     if e.error_code != 'AccessDenied':
-                        print 'GOT UNWANTED ERROR', e.error_code
+                        print('GOT UNWANTED ERROR', e.error_code)
                         raise
                     # seems like we don't have permissions set appropriately, we'll
                     # modify permissions and retry
@@ -106,27 +116,27 @@ def nuke_prefixed_buckets_on_conn(prefix, name, conn):
 def nuke_prefixed_buckets(prefix):
     # If no regions are specified, use the simple method
     if targets.main.master == None:
-        for name, conn in s3.items():
+        for name, conn in list(s3.items()):
-            print 'Deleting buckets on {name}'.format(name=name)
+            print('Deleting buckets on {name}'.format(name=name))
             nuke_prefixed_buckets_on_conn(prefix, name, conn)
     else: 
 		    # First, delete all buckets on the master connection 
-		    for name, conn in s3.items():
+		    for name, conn in list(s3.items()):
 		        if conn == targets.main.master.connection:
-		            print 'Deleting buckets on {name} (master)'.format(name=name)
+		            print('Deleting buckets on {name} (master)'.format(name=name))
 		            nuke_prefixed_buckets_on_conn(prefix, name, conn)
 		
 		    # Then sync to propagate deletes to secondaries
 		    region_sync_meta(targets.main, targets.main.master.connection)
-		    print 'region-sync in nuke_prefixed_buckets'
+		    print('region-sync in nuke_prefixed_buckets')
 		
 		    # Now delete remaining buckets on any other connection 
-		    for name, conn in s3.items():
+		    for name, conn in list(s3.items()):
 		        if conn != targets.main.master.connection:
-		            print 'Deleting buckets on {name} (non-master)'.format(name=name)
+		            print('Deleting buckets on {name} (non-master)'.format(name=name))
 		            nuke_prefixed_buckets_on_conn(prefix, name, conn)
 
-    print 'Done with cleanup of test buckets.'
+    print('Done with cleanup of test buckets.')
 
 class TargetConfig:
     def __init__(self, cfg, section):
@@ -139,46 +149,46 @@ class TargetConfig:
         self.sync_meta_wait = 0
         try:
             self.api_name = cfg.get(section, 'api_name')
-        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+        except (configparser.NoSectionError, configparser.NoOptionError):
             pass
         try:
             self.port = cfg.getint(section, 'port')
-        except ConfigParser.NoOptionError:
+        except configparser.NoOptionError:
             pass
         try:
             self.host=cfg.get(section, 'host')
-        except ConfigParser.NoOptionError:
+        except configparser.NoOptionError:
             raise RuntimeError(
                 'host not specified for section {s}'.format(s=section)
                 )
         try:
             self.is_master=cfg.getboolean(section, 'is_master')
-        except ConfigParser.NoOptionError:
+        except configparser.NoOptionError:
             pass
 
         try:
             self.is_secure=cfg.getboolean(section, 'is_secure')
-        except ConfigParser.NoOptionError:
+        except configparser.NoOptionError:
             pass
 
         try:
             raw_calling_format = cfg.get(section, 'calling_format')
-        except ConfigParser.NoOptionError:
+        except configparser.NoOptionError:
             raw_calling_format = 'ordinary'
 
         try:
             self.sync_agent_addr = cfg.get(section, 'sync_agent_addr')
-        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+        except (configparser.NoSectionError, configparser.NoOptionError):
             pass
 
         try:
             self.sync_agent_port = cfg.getint(section, 'sync_agent_port')
-        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+        except (configparser.NoSectionError, configparser.NoOptionError):
             pass
 
         try:
             self.sync_meta_wait = cfg.getint(section, 'sync_meta_wait')
-        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+        except (configparser.NoSectionError, configparser.NoOptionError):
             pass
 
 
@@ -198,7 +208,7 @@ class TargetConnection:
 
 class RegionsInfo:
     def __init__(self):
-        self.m = bunch.Bunch()
+        self.m = munch.Munch()
         self.master = None
         self.secondaries = []
 
@@ -216,21 +226,21 @@ class RegionsInfo:
         return self.m[name]
     def get(self):
         return self.m
-    def iteritems(self):
+    def items(self):
-        return self.m.iteritems()
+        return self.m.items()
 
 regions = RegionsInfo()
 
 
 class RegionsConn:
     def __init__(self):
-        self.m = bunch.Bunch()
+        self.m = munch.Munch()
         self.default = None
         self.master = None
         self.secondaries = []
 
-    def iteritems(self):
+    def items(self):
-        return self.m.iteritems()
+        return self.m.items()
 
     def set_default(self, conn):
         self.default = conn
@@ -250,7 +260,7 @@ _multiprocess_can_split_ = True
 
 def setup():
 
-    cfg = ConfigParser.RawConfigParser()
+    cfg = configparser.RawConfigParser()
     try:
         path = os.environ['S3TEST_CONF']
     except KeyError:
@@ -258,8 +268,7 @@ def setup():
             'To run tests, point environment '
             + 'variable S3TEST_CONF to a config file.',
             )
-    with file(path) as f:
+    cfg.read(path)
-        cfg.readfp(f)
 
     global prefix
     global targets
@@ -267,19 +276,19 @@ def setup():
 
     try:
         template = cfg.get('fixtures', 'bucket prefix')
-    except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+    except (configparser.NoSectionError, configparser.NoOptionError):
         template = 'test-{random}-'
     prefix = choose_bucket_prefix(template=template)
 
     try:
         slow_backend = cfg.getboolean('fixtures', 'slow backend')
-    except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+    except (configparser.NoSectionError, configparser.NoOptionError):
         slow_backend = False
 
     # pull the default_region out, if it exists
     try:
         default_region = cfg.get('fixtures', 'default_region')
-    except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
+    except (configparser.NoSectionError, configparser.NoOptionError):
         default_region = None
 
     s3.clear()
@@ -305,20 +314,26 @@ def setup():
         if len(regions.get()) == 0:
             regions.add("default", TargetConfig(cfg, section))
 
-        config[name] = bunch.Bunch()
+        config[name] = munch.Munch()
         for var in [
             'user_id',
             'display_name',
             'email',
+            's3website_domain',
+            'host',
+            'port',
+            'is_secure',
+            'kms_keyid',
+            'storage_classes',
             ]:
             try:
                 config[name][var] = cfg.get(section, var)
-            except ConfigParser.NoOptionError:
+            except configparser.NoOptionError:
                 pass
 
         targets[name] = RegionsConn()
 
-        for (k, conf) in regions.iteritems():
+        for (k, conf) in regions.items():
             conn = boto.s3.connection.S3Connection(
                 aws_access_key_id=cfg.get(section, 'access_key'),
                 aws_secret_access_key=cfg.get(section, 'secret_key'),
@@ -356,6 +371,15 @@ def teardown():
     # remove our buckets here also, to avoid littering
     nuke_prefixed_buckets(prefix=prefix)
 
+@pytest.fixture(scope="package")
+def configfile():
+    setup()
+    yield config
+
+@pytest.fixture(autouse=True)
+def setup_teardown(configfile):
+    yield
+    teardown()
 
 bucket_counter = itertools.count(1)
 
@@ -392,3 +416,83 @@ def get_new_bucket(target=None, name=None, headers=None):
     # ignore that as astronomically unlikely
     bucket = connection.create_bucket(name, location=target.conf.api_name, headers=headers)
     return bucket
+
+def _make_request(method, bucket, key, body=None, authenticated=False, response_headers=None, request_headers=None, expires_in=100000, path_style=True, timeout=None):
+    """
+    issue a request for a specified method, on a specified <bucket,key>,
+    with a specified (optional) body (encrypted per the connection), and
+    return the response (status, reason).
+
+    If key is None, then this will be treated as a bucket-level request.
+
+    If the request or response headers are None, then default values will be
+    provided by later methods.
+    """
+    if not path_style:
+        conn = bucket.connection
+        request_headers['Host'] = conn.calling_format.build_host(conn.server_name(), bucket.name)
+
+    if authenticated:
+        urlobj = None
+        if key is not None:
+            urlobj = key
+        elif bucket is not None:
+            urlobj = bucket
+        else:
+            raise RuntimeError('Unable to find bucket name')
+        url = urlobj.generate_url(expires_in, method=method, response_headers=response_headers, headers=request_headers)
+        o = urlparse(url)
+        path = o.path + '?' + o.query
+    else:
+        bucketobj = None
+        if key is not None:
+            path = '/{obj}'.format(obj=key.name)
+            bucketobj = key.bucket
+        elif bucket is not None:
+            path = '/'
+            bucketobj = bucket
+        else:
+            raise RuntimeError('Unable to find bucket name')
+        if path_style:
+            path = '/{bucket}'.format(bucket=bucketobj.name) + path
+
+    return _make_raw_request(host=s3.main.host, port=s3.main.port, method=method, path=path, body=body, request_headers=request_headers, secure=s3.main.is_secure, timeout=timeout)
+
+def _make_bucket_request(method, bucket, body=None, authenticated=False, response_headers=None, request_headers=None, expires_in=100000, path_style=True, timeout=None):
+    """
+    issue a request for a specified method, on a specified <bucket>,
+    with a specified (optional) body (encrypted per the connection), and
+    return the response (status, reason)
+    """
+    return _make_request(method=method, bucket=bucket, key=None, body=body, authenticated=authenticated, response_headers=response_headers, request_headers=request_headers, expires_in=expires_in, path_style=path_style, timeout=timeout)
+
+def _make_raw_request(host, port, method, path, body=None, request_headers=None, secure=False, timeout=None):
+    """
+    issue a request to a specific host & port, for a specified method, on a
+    specified path with a specified (optional) body (encrypted per the
+    connection), and return the response (status, reason).
+
+    This allows construction of special cases not covered by the bucket/key to
+    URL mapping of _make_request/_make_bucket_request.
+    """
+    if secure:
+        class_ = HTTPSConnection
+    else:
+        class_ = HTTPConnection
+
+    if request_headers is None:
+        request_headers = {}
+
+    c = class_(host, port=port, timeout=timeout)
+
+    # TODO: We might have to modify this in future if we need to interact with
+    # how httplib.request handles Accept-Encoding and Host.
+    c.request(method, path, body=body, headers=request_headers)
+
+    res = c.getresponse()
+    #c.close()
+
+    print(res.status, res.reason)
+    return res
+
+

s3tests/functional/policy.py

@@ -0,0 +1,46 @@
+import json
+
+class Statement(object):
+    def __init__(self, action, resource, principal = {"AWS" : "*"}, effect= "Allow", condition = None):
+        self.principal = principal
+        self.action = action
+        self.resource = resource
+        self.condition = condition
+        self.effect = effect
+
+    def to_dict(self):
+        d = { "Action" : self.action,
+              "Principal" : self.principal,
+              "Effect" : self.effect,
+              "Resource" : self.resource
+        }
+
+        if self.condition is not None:
+            d["Condition"] = self.condition
+
+        return d
+
+class Policy(object):
+    def __init__(self):
+        self.statements = []
+
+    def add_statement(self, s):
+        self.statements.append(s)
+        return self
+
+    def to_json(self):
+        policy_dict = {
+            "Version" : "2012-10-17",
+            "Statement":
+            [s.to_dict() for s in self.statements]
+        }
+
+        return json.dumps(policy_dict)
+
+def make_json_policy(action, resource, principal={"AWS": "*"}, conditions=None):
+    """
+    Helper function to make single statement policies
+    """
+    s = Statement(action, resource, principal, condition=conditions)
+    p = Policy()
+    return p.add_statement(s).to_json()

s3tests/functional/test_utils.py

@@ -1,11 +1,9 @@
-from nose.tools import eq_ as eq
+from . import utils
-
-import utils
 
 def test_generate():
     FIVE_MB = 5 * 1024 * 1024
-    eq(len(''.join(utils.generate_random(0))), 0)
+    assert len(''.join(utils.generate_random(0))) == 0
-    eq(len(''.join(utils.generate_random(1))), 1)
+    assert len(''.join(utils.generate_random(1))) == 1
-    eq(len(''.join(utils.generate_random(FIVE_MB - 1))), FIVE_MB - 1)
+    assert len(''.join(utils.generate_random(FIVE_MB - 1))) == FIVE_MB - 1
-    eq(len(''.join(utils.generate_random(FIVE_MB))), FIVE_MB)
+    assert len(''.join(utils.generate_random(FIVE_MB))) == FIVE_MB
-    eq(len(''.join(utils.generate_random(FIVE_MB + 1))), FIVE_MB + 1)
+    assert len(''.join(utils.generate_random(FIVE_MB + 1))) == FIVE_MB + 1

s3tests/functional/utils.py

@@ -3,8 +3,6 @@ import requests
 import string
 import time
 
-from nose.tools import eq_ as eq
-
 def assert_raises(excClass, callableObj, *args, **kwargs):
     """
     Like unittest.TestCase.assertRaises, but returns the exception.
@@ -28,11 +26,11 @@ def generate_random(size, part_size=5*1024*1024):
     chunk = 1024
     allowed = string.ascii_letters
     for x in range(0, size, part_size):
-        strpart = ''.join([allowed[random.randint(0, len(allowed) - 1)] for _ in xrange(chunk)])
+        strpart = ''.join([allowed[random.randint(0, len(allowed) - 1)] for _ in range(chunk)])
         s = ''
         left = size - x
         this_part_size = min(left, part_size)
-        for y in range(this_part_size / chunk):
+        for y in range(this_part_size // chunk):
             s = s + strpart
         s = s + strpart[:(this_part_size % chunk)]
         yield s
@@ -42,13 +40,22 @@ def generate_random(size, part_size=5*1024*1024):
 # syncs all the regions except for the one passed in
 def region_sync_meta(targets, region):
 
-    for (k, r) in targets.iteritems():
+    for (k, r) in targets.items():
         if r == region:
             continue
         conf = r.conf
         if conf.sync_agent_addr:
             ret = requests.post('http://{addr}:{port}/metadata/incremental'.format(addr = conf.sync_agent_addr, port = conf.sync_agent_port))
-            eq(ret.status_code, 200)
+            assert ret.status_code == 200
         if conf.sync_meta_wait:
             time.sleep(conf.sync_meta_wait)
 
+
+def get_grantee(policy, permission):
+    '''
+    Given an object/bucket policy, extract the grantee with the required permission
+    '''
+
+    for g in policy.acl.grants:
+        if g.permission == permission:
+            return g.id

s3tests/fuzz/headers.py

@@ -1,376 +0,0 @@
-from boto.s3.connection import S3Connection
-from boto.exception import BotoServerError
-from boto.s3.key import Key
-from httplib import BadStatusLine
-from optparse import OptionParser
-from .. import common
-
-import traceback
-import itertools
-import random
-import string
-import struct
-import yaml
-import sys
-import re
-
-
-class DecisionGraphError(Exception):
-    """ Raised when a node in a graph tries to set a header or
-        key that was previously set by another node
-    """
-    def __init__(self, value):
-        self.value = value
-
-    def __str__(self):
-        return repr(self.value)
-
-
-class RecursionError(Exception):
-    """Runaway recursion in string formatting"""
-
-    def __init__(self, msg):
-        self.msg = msg
-
-    def __str__(self):
-        return '{0.__doc__}: {0.msg!r}'.format(self)
-
-
-def assemble_decision(decision_graph, prng):
-    """ Take in a graph describing the possible decision space and a random
-        number generator and traverse the graph to build a decision
-    """
-    return descend_graph(decision_graph, 'start', prng)
-
-
-def descend_graph(decision_graph, node_name, prng):
-    """ Given a graph and a particular node in that graph, set the values in
-        the node's "set" list, pick a choice from the "choice" list, and
-        recurse.  Finally, return dictionary of values
-    """
-    node = decision_graph[node_name]
-
-    try:
-        choice = make_choice(node['choices'], prng)
-        if choice == '':
-            decision = {}
-        else:
-            decision = descend_graph(decision_graph, choice, prng)
-    except IndexError:
-        decision = {}
-
-    for key, choices in node['set'].iteritems():
-        if key in decision:
-            raise DecisionGraphError("Node %s tried to set '%s', but that key was already set by a lower node!" %(node_name, key))
-        decision[key] = make_choice(choices, prng)
-
-    if 'headers' in node:
-        decision.setdefault('headers', [])
-
-        for desc in node['headers']:
-            try:
-                (repetition_range, header, value) = desc
-            except ValueError:
-                (header, value) = desc
-                repetition_range = '1'
-
-            try:
-                size_min, size_max = repetition_range.split('-', 1)
-            except ValueError:
-                size_min = size_max = repetition_range
-
-            size_min = int(size_min)
-            size_max = int(size_max)
-
-            num_reps = prng.randint(size_min, size_max)
-            if header in [h for h, v in decision['headers']]:
-                    raise DecisionGraphError("Node %s tried to add header '%s', but that header already exists!" %(node_name, header))
-            for _ in xrange(num_reps):
-                decision['headers'].append([header, value])
-
-    return decision
-
-
-def make_choice(choices, prng):
-    """ Given a list of (possibly weighted) options or just a single option!,
-        choose one of the options taking weights into account and return the
-        choice
-    """
-    if isinstance(choices, str):
-        return choices
-    weighted_choices = []
-    for option in choices:
-        if option is None:
-            weighted_choices.append('')
-            continue
-        try:
-            (weight, value) = option.split(None, 1)
-            weight = int(weight)
-        except ValueError:
-            weight = 1
-            value = option
-
-        if value == 'null' or value == 'None':
-            value = ''
-
-        for _ in xrange(weight):
-            weighted_choices.append(value)
-
-    return prng.choice(weighted_choices)
-
-
-def expand_headers(decision, prng):
-    expanded_headers = {} 
-    for header in decision['headers']:
-        h = expand(decision, header[0], prng)
-        v = expand(decision, header[1], prng)
-        expanded_headers[h] = v
-    return expanded_headers
-
-
-def expand(decision, value, prng):
-    c = itertools.count()
-    fmt = RepeatExpandingFormatter(prng)
-    new = fmt.vformat(value, [], decision)
-    return new
-
-
-class RepeatExpandingFormatter(string.Formatter):
-    charsets = {
-        'printable_no_whitespace': string.printable.translate(None, string.whitespace),
-        'printable': string.printable,
-        'punctuation': string.punctuation,
-        'whitespace': string.whitespace,
-        'digits': string.digits
-    }
-
-    def __init__(self, prng, _recursion=0):
-        super(RepeatExpandingFormatter, self).__init__()
-        # this class assumes it is always instantiated once per
-        # formatting; use that to detect runaway recursion
-        self.prng = prng
-        self._recursion = _recursion
-
-    def get_value(self, key, args, kwargs):
-        fields = key.split(None, 1)
-        fn = getattr(self, 'special_{name}'.format(name=fields[0]), None)
-        if fn is not None:
-            if len(fields) == 1:
-                fields.append('')
-            return fn(fields[1])
-
-        val = super(RepeatExpandingFormatter, self).get_value(key, args, kwargs)
-        if self._recursion > 5:
-            raise RecursionError(key)
-        fmt = self.__class__(self.prng, _recursion=self._recursion+1)
-
-        n = fmt.vformat(val, args, kwargs)
-        return n
-
-    def special_random(self, args):
-        arg_list = args.split()
-        try:
-            size_min, size_max = arg_list[0].split('-', 1)
-        except ValueError:
-            size_min = size_max = arg_list[0]
-        except IndexError:
-            size_min = '0'
-            size_max = '1000'
-
-        size_min = int(size_min)
-        size_max = int(size_max)
-        length = self.prng.randint(size_min, size_max)
-
-        try:
-            charset_arg = arg_list[1]
-        except IndexError:
-            charset_arg = 'printable'
-
-        if charset_arg == 'binary' or charset_arg == 'binary_no_whitespace':
-            num_bytes = length + 8
-            tmplist = [self.prng.getrandbits(64) for _ in xrange(num_bytes / 8)]
-            tmpstring = struct.pack((num_bytes / 8) * 'Q', *tmplist)
-            if charset_arg == 'binary_no_whitespace':
-                tmpstring = ''.join(c for c in tmpstring if c not in string.whitespace)
-            return tmpstring[0:length]
-        else:
-            charset = self.charsets[charset_arg]
-            return ''.join([self.prng.choice(charset) for _ in xrange(length)]) # Won't scale nicely
-
-
-def parse_options():
-    parser = OptionParser()
-    parser.add_option('-O', '--outfile', help='write output to FILE. Defaults to STDOUT', metavar='FILE')
-    parser.add_option('--seed', dest='seed', type='int',  help='initial seed for the random number generator')
-    parser.add_option('--seed-file', dest='seedfile', help='read seeds for specific requests from FILE', metavar='FILE')
-    parser.add_option('-n', dest='num_requests', type='int',  help='issue NUM requests before stopping', metavar='NUM')
-    parser.add_option('-v', '--verbose', dest='verbose', action="store_true",  help='turn on verbose output')
-    parser.add_option('-d', '--debug', dest='debug', action="store_true",  help='turn on debugging (very verbose) output')
-    parser.add_option('--decision-graph', dest='graph_filename',  help='file in which to find the request decision graph')
-    parser.add_option('--no-cleanup', dest='cleanup', action="store_false", help='turn off teardown so you can peruse the state of buckets after testing')
-
-    parser.set_defaults(num_requests=5)
-    parser.set_defaults(cleanup=True)
-    parser.set_defaults(graph_filename='request_decision_graph.yml')
-    return parser.parse_args()
-
-
-def randomlist(seed=None):
-    """ Returns an infinite generator of random numbers
-    """
-    rng = random.Random(seed)
-    while True:
-        yield rng.randint(0,100000) #100,000 seeds is enough, right?
-
-
-def populate_buckets(conn, alt):
-    """ Creates buckets and keys for fuzz testing and sets appropriate
-        permissions. Returns a dictionary of the bucket and key names.
-    """
-    breadable = common.get_new_bucket(alt)
-    bwritable = common.get_new_bucket(alt)
-    bnonreadable = common.get_new_bucket(alt)
-
-    oreadable = Key(breadable)
-    owritable = Key(bwritable)
-    ononreadable = Key(breadable)
-    oreadable.set_contents_from_string('oreadable body')
-    owritable.set_contents_from_string('owritable body')
-    ononreadable.set_contents_from_string('ononreadable body')
-
-    breadable.set_acl('public-read')
-    bwritable.set_acl('public-read-write')
-    bnonreadable.set_acl('private')
-    oreadable.set_acl('public-read')
-    owritable.set_acl('public-read-write')
-    ononreadable.set_acl('private')
-
-    return dict(
-        bucket_readable=breadable.name,
-        bucket_writable=bwritable.name,
-        bucket_not_readable=bnonreadable.name,
-        bucket_not_writable=breadable.name,
-        object_readable=oreadable.key,
-        object_writable=owritable.key,
-        object_not_readable=ononreadable.key,
-        object_not_writable=oreadable.key,
-    )
-
-
-def _main():
-    """ The main script
-    """
-    (options, args) = parse_options()
-    random.seed(options.seed if options.seed else None)
-    s3_connection = common.s3.main
-    alt_connection = common.s3.alt
-
-    if options.outfile:
-        OUT = open(options.outfile, 'w')
-    else:
-        OUT = sys.stderr
-
-    VERBOSE = DEBUG = open('/dev/null', 'w')
-    if options.verbose:
-        VERBOSE = OUT
-    if options.debug:
-        DEBUG = OUT
-        VERBOSE = OUT
-
-    request_seeds = None
-    if options.seedfile:
-        FH = open(options.seedfile, 'r')
-        request_seeds = [int(line) for line in FH if line != '\n']
-        print>>OUT, 'Seedfile: %s' %options.seedfile
-        print>>OUT, 'Number of requests: %d' %len(request_seeds)
-    else:
-        if options.seed:
-            print>>OUT, 'Initial Seed: %d' %options.seed
-        print>>OUT, 'Number of requests: %d' %options.num_requests
-        random_list = randomlist(options.seed)
-        request_seeds = itertools.islice(random_list, options.num_requests)
-
-    print>>OUT, 'Decision Graph: %s' %options.graph_filename
-
-    graph_file = open(options.graph_filename, 'r')
-    decision_graph = yaml.safe_load(graph_file)
-
-    constants = populate_buckets(s3_connection, alt_connection)
-    print>>VERBOSE, "Test Buckets/Objects:"
-    for key, value in constants.iteritems():
-        print>>VERBOSE, "\t%s: %s" %(key, value)
-
-    print>>OUT, "Begin Fuzzing..."
-    print>>VERBOSE, '='*80
-    for request_seed in request_seeds:
-        print>>VERBOSE, 'Seed is: %r' %request_seed
-        prng = random.Random(request_seed)
-        decision = assemble_decision(decision_graph, prng)
-        decision.update(constants)
-
-        method = expand(decision, decision['method'], prng)
-        path = expand(decision, decision['urlpath'], prng)
-
-        try:
-            body = expand(decision, decision['body'], prng)
-        except KeyError:
-            body = ''
-
-        try:
-            headers = expand_headers(decision, prng)
-        except KeyError:
-            headers = {}
-
-        print>>VERBOSE, "%r %r" %(method[:100], path[:100])
-        for h, v in headers.iteritems():
-            print>>VERBOSE, "%r: %r" %(h[:50], v[:50])
-        print>>VERBOSE, "%r\n" % body[:100]
-
-        print>>DEBUG, 'FULL REQUEST'
-        print>>DEBUG, 'Method: %r' %method
-        print>>DEBUG, 'Path: %r' %path
-        print>>DEBUG, 'Headers:'
-        for h, v in headers.iteritems():
-            print>>DEBUG, "\t%r: %r" %(h, v)
-        print>>DEBUG, 'Body: %r\n' %body
-
-        failed = False # Let's be optimistic, shall we?
-        try:
-            response = s3_connection.make_request(method, path, data=body, headers=headers, override_num_retries=1)
-            body = response.read()
-        except BotoServerError, e:
-            response = e
-            body = e.body
-            failed = True
-        except BadStatusLine, e:
-            print>>OUT, 'FAILED: failed to parse response (BadStatusLine); probably a NUL byte in your request?'
-            print>>VERBOSE, '='*80
-            continue
-
-        if failed:
-            print>>OUT, 'FAILED:'
-            OLD_VERBOSE = VERBOSE
-            OLD_DEBUG = DEBUG
-            VERBOSE = DEBUG = OUT
-        print>>VERBOSE, 'Seed was: %r' %request_seed
-        print>>VERBOSE, 'Response status code: %d %s' %(response.status, response.reason)
-        print>>DEBUG, 'Body:\n%s' %body
-        print>>VERBOSE, '='*80
-        if failed:
-            VERBOSE = OLD_VERBOSE
-            DEBUG = OLD_DEBUG
-
-    print>>OUT, '...done fuzzing'
-
-    if options.cleanup:
-        common.teardown()
-
-
-def main():
-    common.setup()
-    try:
-        _main()
-    except Exception as e:
-        traceback.print_exc()
-        common.teardown()
-

s3tests/fuzz/test/test_fuzzer.py

@@ -1,403 +0,0 @@
-"""
-Unit-test suite for the S3 fuzzer
-
-The fuzzer is a grammar-based random S3 operation generator
-that produces random operation sequences in an effort to
-crash the server.  This unit-test suite does not test
-S3 servers, but rather the fuzzer infrastructure.
-
-It works by running the fuzzer off of a simple grammar,
-and checking the producted requests to ensure that they
-include the expected sorts of operations in the expected
-proportions.
-"""
-import sys
-import itertools
-import nose
-import random
-import string
-import yaml
-
-from ..headers import *
-
-from nose.tools import eq_ as eq
-from nose.tools import assert_true
-from nose.plugins.attrib import attr
-
-from ...functional.utils import assert_raises
-
-_decision_graph = {}
-
-def check_access_denied(fn, *args, **kwargs):
-    e = assert_raises(boto.exception.S3ResponseError, fn, *args, **kwargs)
-    eq(e.status, 403)
-    eq(e.reason, 'Forbidden')
-    eq(e.error_code, 'AccessDenied')
-
-
-def build_graph():
-    graph = {}
-    graph['start'] = {
-        'set': {},
-        'choices': ['node2']
-    }
-    graph['leaf'] = {
-        'set': {
-            'key1': 'value1',
-            'key2': 'value2'
-        },
-        'headers': [
-            ['1-2', 'random-header-{random 5-10 printable}', '{random 20-30 punctuation}']
-        ],
-        'choices': []
-    }
-    graph['node1'] = {
-        'set': {
-            'key3': 'value3',
-            'header_val': [
-                '3 h1',
-                '2 h2',
-                'h3'
-            ]
-        },
-        'headers': [
-            ['1-1', 'my-header', '{header_val}'],
-        ],
-        'choices': ['leaf']
-    }
-    graph['node2'] = {
-        'set': {
-            'randkey': 'value-{random 10-15 printable}',
-            'path': '/{bucket_readable}',
-            'indirect_key1': '{key1}'
-        },
-        'choices': ['leaf']
-    }
-    graph['bad_node'] = {
-        'set': {
-            'key1': 'value1'
-        },
-        'choices': ['leaf']
-    }
-    graph['nonexistant_child_node'] = {
-        'set': {},
-        'choices': ['leafy_greens']
-    }
-    graph['weighted_node'] = {
-        'set': {
-            'k1': [
-                'foo',
-                '2 bar',
-                '1 baz'
-            ]
-        },
-        'choices': [
-            'foo',
-            '2 bar',
-            '1 baz'
-        ]
-    }
-    graph['null_choice_node'] = {
-        'set': {},
-        'choices': [None]
-    }
-    graph['repeated_headers_node'] = {
-        'set': {},
-        'headers': [
-            ['1-2', 'random-header-{random 5-10 printable}', '{random 20-30 punctuation}']
-        ],
-        'choices': ['leaf']
-    }
-    graph['weighted_null_choice_node'] = {
-        'set': {},
-        'choices': ['3 null']
-    }
-    return graph
-
-
-#def test_foo():
-    #graph_file = open('request_decision_graph.yml', 'r')
-    #graph = yaml.safe_load(graph_file)
-    #eq(graph['bucket_put_simple']['set']['grantee'], 0)
-
-
-def test_load_graph():
-    graph_file = open('request_decision_graph.yml', 'r')
-    graph = yaml.safe_load(graph_file)
-    graph['start']
-
-
-def test_descend_leaf_node():
-    graph = build_graph()
-    prng = random.Random(1)
-    decision = descend_graph(graph, 'leaf', prng)
-
-    eq(decision['key1'], 'value1')
-    eq(decision['key2'], 'value2')
-    e = assert_raises(KeyError, lambda x: decision[x], 'key3')
-
-
-def test_descend_node():
-    graph = build_graph()
-    prng = random.Random(1)
-    decision = descend_graph(graph, 'node1', prng)
-
-    eq(decision['key1'], 'value1')
-    eq(decision['key2'], 'value2')
-    eq(decision['key3'], 'value3')
-
-
-def test_descend_bad_node():
-    graph = build_graph()
-    prng = random.Random(1)
-    assert_raises(DecisionGraphError, descend_graph, graph, 'bad_node', prng)
-
-
-def test_descend_nonexistant_child():
-    graph = build_graph()
-    prng = random.Random(1)
-    assert_raises(KeyError, descend_graph, graph, 'nonexistant_child_node', prng)
-
-
-def test_expand_random_printable():
-    prng = random.Random(1)
-    got = expand({}, '{random 10-15 printable}', prng)
-    eq(got, '[/pNI$;92@')
-
-
-def test_expand_random_binary():
-    prng = random.Random(1)
-    got = expand({}, '{random 10-15 binary}', prng)
-    eq(got, '\xdfj\xf1\xd80>a\xcd\xc4\xbb')
-
-
-def test_expand_random_printable_no_whitespace():
-    prng = random.Random(1)
-    for _ in xrange(1000):
-        got = expand({}, '{random 500 printable_no_whitespace}', prng)
-        assert_true(reduce(lambda x, y: x and y, [x not in string.whitespace and x in string.printable for x in got]))
-
-
-def test_expand_random_binary_no_whitespace():
-    prng = random.Random(1)
-    for _ in xrange(1000):
-        got = expand({}, '{random 500 binary_no_whitespace}', prng)
-        assert_true(reduce(lambda x, y: x and y, [x not in string.whitespace for x in got]))
-
-
-def test_expand_random_no_args():
-    prng = random.Random(1)
-    for _ in xrange(1000):
-        got = expand({}, '{random}', prng)
-        assert_true(0 <= len(got) <= 1000)
-        assert_true(reduce(lambda x, y: x and y, [x in string.printable for x in got]))
-
-
-def test_expand_random_no_charset():
-    prng = random.Random(1)
-    for _ in xrange(1000):
-        got = expand({}, '{random 10-30}', prng)
-        assert_true(10 <= len(got) <= 30)
-        assert_true(reduce(lambda x, y: x and y, [x in string.printable for x in got]))
-
-
-def test_expand_random_exact_length():
-    prng = random.Random(1)
-    for _ in xrange(1000):
-        got = expand({}, '{random 10 digits}', prng)
-        assert_true(len(got) == 10)
-        assert_true(reduce(lambda x, y: x and y, [x in string.digits for x in got]))
-
-
-def test_expand_random_bad_charset():
-    prng = random.Random(1)
-    assert_raises(KeyError, expand, {}, '{random 10-30 foo}', prng)
-
-
-def test_expand_random_missing_length():
-    prng = random.Random(1)
-    assert_raises(ValueError, expand, {}, '{random printable}', prng)
-
-
-def test_assemble_decision():
-    graph = build_graph()
-    prng = random.Random(1)
-    decision = assemble_decision(graph, prng)
-
-    eq(decision['key1'], 'value1')
-    eq(decision['key2'], 'value2')
-    eq(decision['randkey'], 'value-{random 10-15 printable}')
-    eq(decision['indirect_key1'], '{key1}')
-    eq(decision['path'], '/{bucket_readable}')
-    assert_raises(KeyError, lambda x: decision[x], 'key3')
-
-
-def test_expand_escape():
-    prng = random.Random(1)
-    decision = dict(
-        foo='{{bar}}',
-        )
-    got = expand(decision, '{foo}', prng)
-    eq(got, '{bar}')
-
-
-def test_expand_indirect():
-    prng = random.Random(1)
-    decision = dict(
-        foo='{bar}',
-        bar='quux',
-        )
-    got = expand(decision, '{foo}', prng)
-    eq(got, 'quux')
-
-
-def test_expand_indirect_double():
-    prng = random.Random(1)
-    decision = dict(
-        foo='{bar}',
-        bar='{quux}',
-        quux='thud',
-        )
-    got = expand(decision, '{foo}', prng)
-    eq(got, 'thud')
-
-
-def test_expand_recursive():
-    prng = random.Random(1)
-    decision = dict(
-        foo='{foo}',
-        )
-    e = assert_raises(RecursionError, expand, decision, '{foo}', prng)
-    eq(str(e), "Runaway recursion in string formatting: 'foo'")
-
-
-def test_expand_recursive_mutual():
-    prng = random.Random(1)
-    decision = dict(
-        foo='{bar}',
-        bar='{foo}',
-        )
-    e = assert_raises(RecursionError, expand, decision, '{foo}', prng)
-    eq(str(e), "Runaway recursion in string formatting: 'foo'")
-
-
-def test_expand_recursive_not_too_eager():
-    prng = random.Random(1)
-    decision = dict(
-        foo='bar',
-        )
-    got = expand(decision, 100*'{foo}', prng)
-    eq(got, 100*'bar')
-
-
-def test_make_choice_unweighted_with_space():
-    prng = random.Random(1)
-    choice = make_choice(['foo bar'], prng)
-    eq(choice, 'foo bar')
-
-def test_weighted_choices():
-    graph = build_graph()
-    prng = random.Random(1)
-
-    choices_made = {}
-    for _ in xrange(1000):
-        choice = make_choice(graph['weighted_node']['choices'], prng)
-        if choices_made.has_key(choice):
-            choices_made[choice] += 1
-        else:
-            choices_made[choice] = 1
-
-    foo_percentage = choices_made['foo'] / 1000.0
-    bar_percentage = choices_made['bar'] / 1000.0
-    baz_percentage = choices_made['baz'] / 1000.0
-    nose.tools.assert_almost_equal(foo_percentage, 0.25, 1)
-    nose.tools.assert_almost_equal(bar_percentage, 0.50, 1)
-    nose.tools.assert_almost_equal(baz_percentage, 0.25, 1)
-
-
-def test_null_choices():
-    graph = build_graph()
-    prng = random.Random(1)
-    choice = make_choice(graph['null_choice_node']['choices'], prng)
-
-    eq(choice, '')
-
-
-def test_weighted_null_choices():
-    graph = build_graph()
-    prng = random.Random(1)
-    choice = make_choice(graph['weighted_null_choice_node']['choices'], prng)
-
-    eq(choice, '')
-
-
-def test_null_child():
-    graph = build_graph()
-    prng = random.Random(1)
-    decision = descend_graph(graph, 'null_choice_node', prng)
-
-    eq(decision, {})
-
-
-def test_weighted_set():
-    graph = build_graph()
-    prng = random.Random(1)
-
-    choices_made = {}
-    for _ in xrange(1000):
-        choice = make_choice(graph['weighted_node']['set']['k1'], prng)
-        if choices_made.has_key(choice):
-            choices_made[choice] += 1
-        else:
-            choices_made[choice] = 1
-
-    foo_percentage = choices_made['foo'] / 1000.0
-    bar_percentage = choices_made['bar'] / 1000.0
-    baz_percentage = choices_made['baz'] / 1000.0
-    nose.tools.assert_almost_equal(foo_percentage, 0.25, 1)
-    nose.tools.assert_almost_equal(bar_percentage, 0.50, 1)
-    nose.tools.assert_almost_equal(baz_percentage, 0.25, 1)
-
-
-def test_header_presence():
-    graph = build_graph()
-    prng = random.Random(1)
-    decision = descend_graph(graph, 'node1', prng)
-
-    c1 = itertools.count()
-    c2 = itertools.count()
-    for header, value in decision['headers']:
-        if header == 'my-header':
-            eq(value, '{header_val}')
-            assert_true(next(c1) < 1)
-        elif header == 'random-header-{random 5-10 printable}':
-            eq(value, '{random 20-30 punctuation}')
-            assert_true(next(c2) < 2)
-        else:
-            raise KeyError('unexpected header found: %s' % header)
-
-    assert_true(next(c1))
-    assert_true(next(c2))
-
-
-def test_duplicate_header():
-    graph = build_graph()
-    prng = random.Random(1)
-    assert_raises(DecisionGraphError, descend_graph, graph, 'repeated_headers_node', prng)
-
-
-def test_expand_headers():
-    graph = build_graph()
-    prng = random.Random(1)
-    decision = descend_graph(graph, 'node1', prng)
-    expanded_headers = expand_headers(decision, prng)
-
-    for header, value in expanded_headers.iteritems():
-        if header == 'my-header':
-            assert_true(value in ['h1', 'h2', 'h3'])
-        elif header.startswith('random-header-'):
-            assert_true(20 <= len(value) <= 30)
-            assert_true(string.strip(value, RepeatExpandingFormatter.charsets['punctuation']) is '')
-        else:
-            raise DecisionGraphError('unexpected header found: "%s"' % header)
-

s3tests/generate_objects.py

@@ -1,117 +0,0 @@
-from boto.s3.key import Key
-from optparse import OptionParser
-from . import realistic
-import traceback
-import random
-from . import common
-import sys
-
-
-def parse_opts():
-    parser = OptionParser()
-    parser.add_option('-O', '--outfile', help='write output to FILE. Defaults to STDOUT', metavar='FILE')
-    parser.add_option('-b', '--bucket', dest='bucket', help='push objects to BUCKET', metavar='BUCKET')
-    parser.add_option('--seed', dest='seed', help='optional seed for the random number generator')
-
-    return parser.parse_args()
-
-
-def get_random_files(quantity, mean, stddev, seed):
-    """Create file-like objects with pseudorandom contents.
-       IN:
-           number of files to create
-           mean file size in bytes
-           standard deviation from mean file size
-           seed for PRNG
-       OUT:
-           list of file handles
-    """
-    file_generator = realistic.files(mean, stddev, seed)
-    return [file_generator.next() for _ in xrange(quantity)]
-
-
-def upload_objects(bucket, files, seed):
-    """Upload a bunch of files to an S3 bucket
-       IN:
-         boto S3 bucket object
-         list of file handles to upload
-         seed for PRNG
-       OUT:
-         list of boto S3 key objects
-    """
-    keys = []
-    name_generator = realistic.names(15, 4, seed=seed)
-
-    for fp in files:
-        print >> sys.stderr, 'sending file with size %dB' % fp.size
-        key = Key(bucket)
-        key.key = name_generator.next()
-        key.set_contents_from_file(fp, rewind=True)
-        key.set_acl('public-read')
-        keys.append(key)
-
-    return keys
-
-
-def _main():
-    '''To run the static content load test, make sure you've bootstrapped your
-       test environment and set up your config.yaml file, then run the following:
-          S3TEST_CONF=config.yaml virtualenv/bin/s3tests-generate-objects.py --seed 1234
-
-        This creates a bucket with your S3 credentials (from config.yaml) and
-        fills it with garbage objects as described in the
-        file_generation.groups section of config.yaml.  It writes a list of
-        URLS to those objects to the file listed in file_generation.url_file
-        in config.yaml.
-
-        Once you have objcts in your bucket, run the siege benchmarking program:
-            siege --rc ./siege.conf -r 5
-
-        This tells siege to read the ./siege.conf config file which tells it to
-        use the urls in ./urls.txt and log to ./siege.log. It hits each url in
-        urls.txt 5 times (-r flag).
-
-        Results are printed to the terminal and written in CSV format to
-        ./siege.log
-    '''
-    (options, args) = parse_opts()
-
-    #SETUP
-    random.seed(options.seed if options.seed else None)
-    conn = common.s3.main
-
-    if options.outfile:
-        OUTFILE = open(options.outfile, 'w')
-    elif common.config.file_generation.url_file:
-        OUTFILE = open(common.config.file_generation.url_file, 'w')
-    else:
-        OUTFILE = sys.stdout
-
-    if options.bucket:
-        bucket = conn.create_bucket(options.bucket)
-    else:
-        bucket = common.get_new_bucket()
-
-    bucket.set_acl('public-read')
-    keys = []
-    print >> OUTFILE, 'bucket: %s' % bucket.name
-    print >> sys.stderr, 'setup complete, generating files'
-    for profile in common.config.file_generation.groups:
-        seed = random.random()
-        files = get_random_files(profile[0], profile[1], profile[2], seed)
-        keys += upload_objects(bucket, files, seed)
-
-    print >> sys.stderr, 'finished sending files. generating urls'
-    for key in keys:
-        print >> OUTFILE, key.generate_url(0, query_auth=False)
-
-    print >> sys.stderr, 'done'
-
-
-def main():
-    common.setup()
-    try:
-        _main()
-    except Exception as e:
-        traceback.print_exc()
-        common.teardown()

s3tests/readwrite.py

@@ -1,265 +0,0 @@
-import gevent
-import gevent.pool
-import gevent.queue
-import gevent.monkey; gevent.monkey.patch_all()
-import itertools
-import optparse
-import os
-import sys
-import time
-import traceback
-import random
-import yaml
-
-import realistic
-import common
-
-NANOSECOND = int(1e9)
-
-def reader(bucket, worker_id, file_names, queue, rand):
-    while True:
-        objname = rand.choice(file_names)
-        key = bucket.new_key(objname)
-
-        fp = realistic.FileValidator()
-        result = dict(
-                type='r',
-                bucket=bucket.name,
-                key=key.name,
-                worker=worker_id,
-                )
-
-        start = time.time()
-        try:
-            key.get_contents_to_file(fp._file)
-        except gevent.GreenletExit:
-            raise
-        except Exception as e:
-            # stop timer ASAP, even on errors
-            end = time.time()
-            result.update(
-                error=dict(
-                    msg=str(e),
-                    traceback=traceback.format_exc(),
-                    ),
-                )
-            # certain kinds of programmer errors make this a busy
-            # loop; let parent greenlet get some time too
-            time.sleep(0)
-        else:
-            end = time.time()
-
-            if not fp.valid():
-                m='md5sum check failed start={s} ({se}) end={e} size={sz} obj={o}'.format(s=time.ctime(start), se=start, e=end, sz=fp._file.tell(), o=objname)
-                result.update(
-                    error=dict(
-                        msg=m,
-                        traceback=traceback.format_exc(),
-                        ),
-                    )
-                print "ERROR:", m
-            else:
-                elapsed = end - start
-                result.update(
-                    start=start,
-                    duration=int(round(elapsed * NANOSECOND)),
-                    )
-        queue.put(result)
-
-def writer(bucket, worker_id, file_names, files, queue, rand):
-    while True:
-        fp = next(files)
-        fp.seek(0)
-        objname = rand.choice(file_names)
-        key = bucket.new_key(objname)
-
-        result = dict(
-            type='w',
-            bucket=bucket.name,
-            key=key.name,
-            worker=worker_id,
-            )
-
-        start = time.time()
-        try:
-            key.set_contents_from_file(fp)
-        except gevent.GreenletExit:
-            raise
-        except Exception as e:
-            # stop timer ASAP, even on errors
-            end = time.time()
-            result.update(
-                error=dict(
-                    msg=str(e),
-                    traceback=traceback.format_exc(),
-                    ),
-                )
-            # certain kinds of programmer errors make this a busy
-            # loop; let parent greenlet get some time too
-            time.sleep(0)
-        else:
-            end = time.time()
-
-            elapsed = end - start
-            result.update(
-                start=start,
-                duration=int(round(elapsed * NANOSECOND)),
-                )
-
-        queue.put(result)
-
-def parse_options():
-    parser = optparse.OptionParser(
-        usage='%prog [OPTS] <CONFIG_YAML',
-        )
-    parser.add_option("--no-cleanup", dest="cleanup", action="store_false",
-        help="skip cleaning up all created buckets", default=True)
-
-    return parser.parse_args()
-
-def write_file(bucket, file_name, fp):
-    """
-    Write a single file to the bucket using the file_name.
-    This is used during the warmup to initialize the files.
-    """
-    key = bucket.new_key(file_name)
-    key.set_contents_from_file(fp)
-
-def main():
-    # parse options
-    (options, args) = parse_options()
-
-    if os.isatty(sys.stdin.fileno()):
-        raise RuntimeError('Need configuration in stdin.')
-    config = common.read_config(sys.stdin)
-    conn = common.connect(config.s3)
-    bucket = None
-
-    try:
-        # setup
-        real_stdout = sys.stdout
-        sys.stdout = sys.stderr
-
-        # verify all required config items are present
-        if 'readwrite' not in config:
-            raise RuntimeError('readwrite section not found in config')
-        for item in ['readers', 'writers', 'duration', 'files', 'bucket']:
-            if item not in config.readwrite:
-                raise RuntimeError("Missing readwrite config item: {item}".format(item=item))
-        for item in ['num', 'size', 'stddev']:
-            if item not in config.readwrite.files:
-                raise RuntimeError("Missing readwrite config item: files.{item}".format(item=item))
-
-        seeds = dict(config.readwrite.get('random_seed', {}))
-        seeds.setdefault('main', random.randrange(2**32))
-
-        rand = random.Random(seeds['main'])
-
-        for name in ['names', 'contents', 'writer', 'reader']:
-            seeds.setdefault(name, rand.randrange(2**32))
-
-        print 'Using random seeds: {seeds}'.format(seeds=seeds)
-
-        # setup bucket and other objects
-        bucket_name = common.choose_bucket_prefix(config.readwrite.bucket, max_len=30)
-        bucket = conn.create_bucket(bucket_name)
-        print "Created bucket: {name}".format(name=bucket.name)
-
-        # check flag for deterministic file name creation
-        if not config.readwrite.get('deterministic_file_names'):
-            print 'Creating random file names'
-            file_names = realistic.names(
-                mean=15,
-                stddev=4,
-                seed=seeds['names'],
-                )
-            file_names = itertools.islice(file_names, config.readwrite.files.num)
-            file_names = list(file_names)
-        else:
-            print 'Creating file names that are deterministic'
-            file_names = []
-            for x in xrange(config.readwrite.files.num):
-                file_names.append('test_file_{num}'.format(num=x))
-
-        files = realistic.files2(
-            mean=1024 * config.readwrite.files.size,
-            stddev=1024 * config.readwrite.files.stddev,
-            seed=seeds['contents'],
-            )
-        q = gevent.queue.Queue()
-
-
-        # warmup - get initial set of files uploaded if there are any writers specified
-        if config.readwrite.writers > 0:
-            print "Uploading initial set of {num} files".format(num=config.readwrite.files.num)
-            warmup_pool = gevent.pool.Pool(size=100)
-            for file_name in file_names:
-                fp = next(files)
-                warmup_pool.spawn(
-                    write_file,
-                    bucket=bucket,
-                    file_name=file_name,
-                    fp=fp,
-                    )
-            warmup_pool.join()
-
-        # main work
-        print "Starting main worker loop."
-        print "Using file size: {size} +- {stddev}".format(size=config.readwrite.files.size, stddev=config.readwrite.files.stddev)
-        print "Spawning {w} writers and {r} readers...".format(w=config.readwrite.writers, r=config.readwrite.readers)
-        group = gevent.pool.Group()
-        rand_writer = random.Random(seeds['writer'])
-
-        # Don't create random files if deterministic_files_names is set and true
-        if not config.readwrite.get('deterministic_file_names'):
-            for x in xrange(config.readwrite.writers):
-                this_rand = random.Random(rand_writer.randrange(2**32))
-                group.spawn(
-                    writer,
-                    bucket=bucket,
-                    worker_id=x,
-                    file_names=file_names,
-                    files=files,
-                    queue=q,
-                    rand=this_rand,
-                    )
-
-        # Since the loop generating readers already uses config.readwrite.readers
-        # and the file names are already generated (randomly or deterministically),
-        # this loop needs no additional qualifiers. If zero readers are specified,
-        # it will behave as expected (no data is read)
-        rand_reader = random.Random(seeds['reader'])
-        for x in xrange(config.readwrite.readers):
-            this_rand = random.Random(rand_reader.randrange(2**32))
-            group.spawn(
-                reader,
-                bucket=bucket,
-                worker_id=x,
-                file_names=file_names,
-                queue=q,
-                rand=this_rand,
-                )
-        def stop():
-            group.kill(block=True)
-            q.put(StopIteration)
-        gevent.spawn_later(config.readwrite.duration, stop)
-
-        # wait for all the tests to finish
-        group.join()
-        print 'post-join, queue size {size}'.format(size=q.qsize())
-
-        if q.qsize() > 0:
-            for temp_dict in q:
-                if 'error' in temp_dict:
-                    raise Exception('exception:\n\t{msg}\n\t{trace}'.format(
-                                    msg=temp_dict['error']['msg'],
-                                    trace=temp_dict['error']['traceback'])
-                                   )
-                else:
-                    yaml.safe_dump(temp_dict, stream=real_stdout)
-
-    finally:
-        # cleanup
-        if options.cleanup:
-            if bucket is not None:
-                common.nuke_bucket(bucket)

s3tests/realistic.py

@@ -1,281 +0,0 @@
-import hashlib
-import random
-import string
-import struct
-import time
-import math
-import tempfile
-import shutil
-import os
-
-
-NANOSECOND = int(1e9)
-
-
-def generate_file_contents(size):
-    """
-    A helper function to generate binary contents for a given size, and
-    calculates the md5 hash of the contents appending itself at the end of the
-    blob.
-    It uses sha1's hexdigest which is 40 chars long. So any binary generated
-    should remove the last 40 chars from the blob to retrieve the original hash
-    and binary so that validity can be proved.
-    """
-    size = int(size)
-    contents = os.urandom(size)
-    content_hash = hashlib.sha1(contents).hexdigest()
-    return contents + content_hash
-
-
-class FileValidator(object):
-
-    def __init__(self, f=None):
-        self._file = tempfile.SpooledTemporaryFile()
-        self.original_hash = None
-        self.new_hash = None
-        if f:
-            f.seek(0)
-            shutil.copyfileobj(f, self._file)
-
-    def valid(self):
-        """
-        Returns True if this file looks valid. The file is valid if the end
-        of the file has the md5 digest for the first part of the file.
-        """
-        self._file.seek(0)
-        contents = self._file.read()
-        self.original_hash, binary = contents[-40:], contents[:-40]
-        self.new_hash = hashlib.sha1(binary).hexdigest()
-        if not self.new_hash == self.original_hash:
-            print 'original  hash: ', self.original_hash
-            print 'new hash: ', self.new_hash
-            print 'size: ', self._file.tell()
-            return False
-        return True
-
-    # XXX not sure if we need all of these
-    def seek(self, offset, whence=os.SEEK_SET):
-        self._file.seek(offset, whence)
-
-    def tell(self):
-        return self._file.tell()
-
-    def read(self, size=-1):
-        return self._file.read(size)
-
-    def write(self, data):
-        self._file.write(data)
-        self._file.seek(0)
-
-
-class RandomContentFile(object):
-    def __init__(self, size, seed):
-        self.size = size
-        self.seed = seed
-        self.random = random.Random(self.seed)
-
-        # Boto likes to seek once more after it's done reading, so we need to save the last chunks/seek value.
-        self.last_chunks = self.chunks = None
-        self.last_seek = None
-
-        # Let seek initialize the rest of it, rather than dup code
-        self.seek(0)
-
-    def _mark_chunk(self):
-        self.chunks.append([self.offset, int(round((time.time() - self.last_seek) * NANOSECOND))])
-
-    def seek(self, offset, whence=os.SEEK_SET):
-        if whence == os.SEEK_SET:
-            self.offset = offset
-        elif whence == os.SEEK_END:
-            self.offset = self.size + offset;
-        elif whence == os.SEEK_CUR:
-            self.offset += offset
-
-        assert self.offset == 0
-
-        self.random.seed(self.seed)
-        self.buffer = ''
-
-        self.hash = hashlib.md5()
-        self.digest_size = self.hash.digest_size
-        self.digest = None
-
-        # Save the last seek time as our start time, and the last chunks
-        self.last_chunks = self.chunks
-        # Before emptying.
-        self.last_seek = time.time()
-        self.chunks = []
-
-    def tell(self):
-        return self.offset
-
-    def _generate(self):
-        # generate and return a chunk of pseudorandom data
-        size = min(self.size, 1*1024*1024) # generate at most 1 MB at a time
-        chunks = int(math.ceil(size/8.0))  # number of 8-byte chunks to create
-
-        l = [self.random.getrandbits(64) for _ in xrange(chunks)]
-        s = struct.pack(chunks*'Q', *l)
-        return s
-
-    def read(self, size=-1):
-        if size < 0:
-            size = self.size - self.offset
-
-        r = []
-
-        random_count = min(size, self.size - self.offset - self.digest_size)
-        if random_count > 0:
-            while len(self.buffer) < random_count:
-                self.buffer += self._generate()
-            self.offset += random_count
-            size -= random_count
-            data, self.buffer = self.buffer[:random_count], self.buffer[random_count:]
-            if self.hash is not None:
-                self.hash.update(data)
-            r.append(data)
-
-        digest_count = min(size, self.size - self.offset)
-        if digest_count > 0:
-            if self.digest is None:
-                self.digest = self.hash.digest()
-                self.hash = None
-            self.offset += digest_count
-            size -= digest_count
-            data = self.digest[:digest_count]
-            r.append(data)
-
-        self._mark_chunk()
-
-        return ''.join(r)
-
-
-class PrecomputedContentFile(object):
-    def __init__(self, f):
-        self._file = tempfile.SpooledTemporaryFile()
-        f.seek(0)
-        shutil.copyfileobj(f, self._file)
-
-        self.last_chunks = self.chunks = None
-        self.seek(0)
-
-    def seek(self, offset, whence=os.SEEK_SET):
-        self._file.seek(offset, whence)
-
-        if self.tell() == 0:
-            # only reset the chunks when seeking to the beginning
-            self.last_chunks = self.chunks
-            self.last_seek = time.time()
-            self.chunks = []
-
-    def tell(self):
-        return self._file.tell()
-
-    def read(self, size=-1):
-        data = self._file.read(size)
-        self._mark_chunk()
-        return data
-
-    def _mark_chunk(self):
-        elapsed = time.time() - self.last_seek
-        elapsed_nsec = int(round(elapsed * NANOSECOND))
-        self.chunks.append([self.tell(), elapsed_nsec])
-
-class FileVerifier(object):
-    def __init__(self):
-        self.size = 0
-        self.hash = hashlib.md5()
-        self.buf = ''
-        self.created_at = time.time()
-        self.chunks = []
-
-    def _mark_chunk(self):
-        self.chunks.append([self.size, int(round((time.time() - self.created_at) * NANOSECOND))])
-
-    def write(self, data):
-        self.size += len(data)
-        self.buf += data
-        digsz = -1*self.hash.digest_size
-        new_data, self.buf = self.buf[0:digsz], self.buf[digsz:]
-        self.hash.update(new_data)
-        self._mark_chunk()
-
-    def valid(self):
-        """
-        Returns True if this file looks valid. The file is valid if the end
-        of the file has the md5 digest for the first part of the file.
-        """
-        if self.size < self.hash.digest_size:
-            return self.hash.digest().startswith(self.buf)
-
-        return self.buf == self.hash.digest()
-
-
-def files(mean, stddev, seed=None):
-    """
-    Yields file-like objects with effectively random contents, where
-    the size of each file follows the normal distribution with `mean`
-    and `stddev`.
-
-    Beware, the file-likeness is very shallow. You can use boto's
-    `key.set_contents_from_file` to send these to S3, but they are not
-    full file objects.
-
-    The last 128 bits are the MD5 digest of the previous bytes, for
-    verifying round-trip data integrity. For example, if you
-    re-download the object and place the contents into a file called
-    ``foo``, the following should print two identical lines:
-
-      python -c 'import sys, hashlib; data=sys.stdin.read(); print hashlib.md5(data[:-16]).hexdigest(); print "".join("%02x" % ord(c) for c in data[-16:])' <foo
-
-    Except for objects shorter than 16 bytes, where the second line
-    will be proportionally shorter.
-    """
-    rand = random.Random(seed)
-    while True:
-        while True:
-            size = int(rand.normalvariate(mean, stddev))
-            if size >= 0:
-                break
-        yield RandomContentFile(size=size, seed=rand.getrandbits(32))
-
-
-def files2(mean, stddev, seed=None, numfiles=10):
-    """
-    Yields file objects with effectively random contents, where the
-    size of each file follows the normal distribution with `mean` and
-    `stddev`.
-
-    Rather than continuously generating new files, this pre-computes and
-    stores `numfiles` files and yields them in a loop.
-    """
-    # pre-compute all the files (and save with TemporaryFiles)
-    fs = []
-    for _ in xrange(numfiles):
-        t = tempfile.SpooledTemporaryFile()
-        t.write(generate_file_contents(random.normalvariate(mean, stddev)))
-        t.seek(0)
-        fs.append(t)
-
-    while True:
-        for f in fs:
-            yield f
-
-
-def names(mean, stddev, charset=None, seed=None):
-    """
-    Yields strings that are somewhat plausible as file names, where
-    the lenght of each filename follows the normal distribution with
-    `mean` and `stddev`.
-    """
-    if charset is None:
-        charset = string.ascii_lowercase
-    rand = random.Random(seed)
-    while True:
-        while True:
-            length = int(rand.normalvariate(mean, stddev))
-            if length > 0:
-                break
-        name = ''.join(rand.choice(charset) for _ in xrange(length))
-        yield name

s3tests/roundtrip.py

@@ -1,219 +0,0 @@
-import gevent
-import gevent.pool
-import gevent.queue
-import gevent.monkey; gevent.monkey.patch_all()
-import itertools
-import optparse
-import os
-import sys
-import time
-import traceback
-import random
-import yaml
-
-import realistic
-import common
-
-NANOSECOND = int(1e9)
-
-def writer(bucket, objname, fp, queue):
-    key = bucket.new_key(objname)
-
-    result = dict(
-        type='w',
-        bucket=bucket.name,
-        key=key.name,
-        )
-
-    start = time.time()
-    try:
-        key.set_contents_from_file(fp)
-    except gevent.GreenletExit:
-        raise
-    except Exception as e:
-        # stop timer ASAP, even on errors
-        end = time.time()
-        result.update(
-            error=dict(
-                msg=str(e),
-                traceback=traceback.format_exc(),
-                ),
-            )
-        # certain kinds of programmer errors make this a busy
-        # loop; let parent greenlet get some time too
-        time.sleep(0)
-    else:
-        end = time.time()
-
-    elapsed = end - start
-    result.update(
-        start=start,
-        duration=int(round(elapsed * NANOSECOND)),
-        chunks=fp.last_chunks,
-        )
-    queue.put(result)
-
-
-def reader(bucket, objname, queue):
-    key = bucket.new_key(objname)
-
-    fp = realistic.FileVerifier()
-    result = dict(
-            type='r',
-            bucket=bucket.name,
-            key=key.name,
-            )
-
-    start = time.time()
-    try:
-        key.get_contents_to_file(fp)
-    except gevent.GreenletExit:
-        raise
-    except Exception as e:
-        # stop timer ASAP, even on errors
-        end = time.time()
-        result.update(
-            error=dict(
-                msg=str(e),
-                traceback=traceback.format_exc(),
-                ),
-            )
-        # certain kinds of programmer errors make this a busy
-        # loop; let parent greenlet get some time too
-        time.sleep(0)
-    else:
-        end = time.time()
-
-        if not fp.valid():
-            result.update(
-                error=dict(
-                    msg='md5sum check failed',
-                    ),
-                )
-
-    elapsed = end - start
-    result.update(
-        start=start,
-        duration=int(round(elapsed * NANOSECOND)),
-        chunks=fp.chunks,
-        )
-    queue.put(result)
-
-def parse_options():
-    parser = optparse.OptionParser(
-        usage='%prog [OPTS] <CONFIG_YAML',
-        )
-    parser.add_option("--no-cleanup", dest="cleanup", action="store_false",
-        help="skip cleaning up all created buckets", default=True)
-
-    return parser.parse_args()
-
-def main():
-    # parse options
-    (options, args) = parse_options()
-
-    if os.isatty(sys.stdin.fileno()):
-        raise RuntimeError('Need configuration in stdin.')
-    config = common.read_config(sys.stdin)
-    conn = common.connect(config.s3)
-    bucket = None
-
-    try:
-        # setup
-        real_stdout = sys.stdout
-        sys.stdout = sys.stderr
-
-        # verify all required config items are present
-        if 'roundtrip' not in config:
-            raise RuntimeError('roundtrip section not found in config')
-        for item in ['readers', 'writers', 'duration', 'files', 'bucket']:
-            if item not in config.roundtrip:
-                raise RuntimeError("Missing roundtrip config item: {item}".format(item=item))
-        for item in ['num', 'size', 'stddev']:
-            if item not in config.roundtrip.files:
-                raise RuntimeError("Missing roundtrip config item: files.{item}".format(item=item))
-
-        seeds = dict(config.roundtrip.get('random_seed', {}))
-        seeds.setdefault('main', random.randrange(2**32))
-
-        rand = random.Random(seeds['main'])
-
-        for name in ['names', 'contents', 'writer', 'reader']:
-            seeds.setdefault(name, rand.randrange(2**32))
-
-        print 'Using random seeds: {seeds}'.format(seeds=seeds)
-
-        # setup bucket and other objects
-        bucket_name = common.choose_bucket_prefix(config.roundtrip.bucket, max_len=30)
-        bucket = conn.create_bucket(bucket_name)
-        print "Created bucket: {name}".format(name=bucket.name)
-        objnames = realistic.names(
-            mean=15,
-            stddev=4,
-            seed=seeds['names'],
-            )
-        objnames = itertools.islice(objnames, config.roundtrip.files.num)
-        objnames = list(objnames)
-        files = realistic.files(
-            mean=1024 * config.roundtrip.files.size,
-            stddev=1024 * config.roundtrip.files.stddev,
-            seed=seeds['contents'],
-            )
-        q = gevent.queue.Queue()
-
-        logger_g = gevent.spawn(yaml.safe_dump_all, q, stream=real_stdout)
-
-        print "Writing {num} objects with {w} workers...".format(
-            num=config.roundtrip.files.num,
-            w=config.roundtrip.writers,
-            )
-        pool = gevent.pool.Pool(size=config.roundtrip.writers)
-        start = time.time()
-        for objname in objnames:
-            fp = next(files)
-            pool.spawn(
-                writer,
-                bucket=bucket,
-                objname=objname,
-                fp=fp,
-                queue=q,
-                )
-        pool.join()
-        stop = time.time()
-        elapsed = stop - start
-        q.put(dict(
-                type='write_done',
-                duration=int(round(elapsed * NANOSECOND)),
-                ))
-
-        print "Reading {num} objects with {w} workers...".format(
-            num=config.roundtrip.files.num,
-            w=config.roundtrip.readers,
-            )
-        # avoid accessing them in the same order as the writing
-        rand.shuffle(objnames)
-        pool = gevent.pool.Pool(size=config.roundtrip.readers)
-        start = time.time()
-        for objname in objnames:
-            pool.spawn(
-                reader,
-                bucket=bucket,
-                objname=objname,
-                queue=q,
-                )
-        pool.join()
-        stop = time.time()
-        elapsed = stop - start
-        q.put(dict(
-                type='read_done',
-                duration=int(round(elapsed * NANOSECOND)),
-                ))
-
-        q.put(StopIteration)
-        logger_g.get()
-
-    finally:
-        # cleanup
-        if options.cleanup:
-            if bucket is not None:
-                common.nuke_bucket(bucket)

s3tests/tests/test_realistic.py

@@ -1,79 +0,0 @@
-from s3tests import realistic
-import shutil
-import tempfile
-
-
-# XXX not used for now
-def create_files(mean=2000):
-    return realistic.files2(
-        mean=1024 * mean,
-        stddev=1024 * 500,
-        seed=1256193726,
-        numfiles=4,
-    )
-
-
-class TestFiles(object):
-    # the size and seed is what we can get when generating a bunch of files
-    # with pseudo random numbers based on sttdev, seed, and mean.
-
-    # this fails, demonstrating the (current) problem
-    #def test_random_file_invalid(self):
-    #    size = 2506764
-    #    seed = 3391518755
-    #    source = realistic.RandomContentFile(size=size, seed=seed)
-    #    t = tempfile.SpooledTemporaryFile()
-    #    shutil.copyfileobj(source, t)
-    #    precomputed = realistic.PrecomputedContentFile(t)
-    #    assert precomputed.valid()
-
-    #    verifier = realistic.FileVerifier()
-    #    shutil.copyfileobj(precomputed, verifier)
-
-    #    assert verifier.valid()
-
-    # this passes
-    def test_random_file_valid(self):
-        size = 2506001
-        seed = 3391518755
-        source = realistic.RandomContentFile(size=size, seed=seed)
-        t = tempfile.SpooledTemporaryFile()
-        shutil.copyfileobj(source, t)
-        precomputed = realistic.PrecomputedContentFile(t)
-
-        verifier = realistic.FileVerifier()
-        shutil.copyfileobj(precomputed, verifier)
-
-        assert verifier.valid()
-
-
-# new implementation
-class TestFileValidator(object):
-
-    def test_new_file_is_valid(self):
-        size = 2506001
-        contents = realistic.generate_file_contents(size)
-        t = tempfile.SpooledTemporaryFile()
-        t.write(contents)
-        t.seek(0)
-        fp = realistic.FileValidator(t)
-        assert fp.valid()
-
-    def test_new_file_is_valid_when_size_is_1(self):
-        size = 1
-        contents = realistic.generate_file_contents(size)
-        t = tempfile.SpooledTemporaryFile()
-        t.write(contents)
-        t.seek(0)
-        fp = realistic.FileValidator(t)
-        assert fp.valid()
-
-    def test_new_file_is_valid_on_several_calls(self):
-        size = 2506001
-        contents = realistic.generate_file_contents(size)
-        t = tempfile.SpooledTemporaryFile()
-        t.write(contents)
-        t.seek(0)
-        fp = realistic.FileValidator(t)
-        assert fp.valid()
-        assert fp.valid()

s3tests_boto3/common.py

@@ -0,0 +1,301 @@
+import boto.s3.connection
+import munch
+import itertools
+import os
+import random
+import string
+import yaml
+import re
+from lxml import etree
+
+from doctest import Example
+from lxml.doctestcompare import LXMLOutputChecker
+
+s3 = munch.Munch()
+config = munch.Munch()
+prefix = ''
+
+bucket_counter = itertools.count(1)
+key_counter = itertools.count(1)
+
+def choose_bucket_prefix(template, max_len=30):
+    """
+    Choose a prefix for our test buckets, so they're easy to identify.
+
+    Use template and feed it more and more random filler, until it's
+    as long as possible but still below max_len.
+    """
+    rand = ''.join(
+        random.choice(string.ascii_lowercase + string.digits)
+        for c in range(255)
+        )
+
+    while rand:
+        s = template.format(random=rand)
+        if len(s) <= max_len:
+            return s
+        rand = rand[:-1]
+
+    raise RuntimeError(
+        'Bucket prefix template is impossible to fulfill: {template!r}'.format(
+            template=template,
+            ),
+        )
+
+def nuke_bucket(bucket):
+    try:
+        bucket.set_canned_acl('private')
+        # TODO: deleted_cnt and the while loop is a work around for rgw
+        # not sending the
+        deleted_cnt = 1
+        while deleted_cnt:
+            deleted_cnt = 0
+            for key in bucket.list():
+                print('Cleaning bucket {bucket} key {key}'.format(
+                    bucket=bucket,
+                    key=key,
+                    ))
+                key.set_canned_acl('private')
+                key.delete()
+                deleted_cnt += 1
+        bucket.delete()
+    except boto.exception.S3ResponseError as e:
+        # TODO workaround for buggy rgw that fails to send
+        # error_code, remove
+        if (e.status == 403
+            and e.error_code is None
+            and e.body == ''):
+            e.error_code = 'AccessDenied'
+        if e.error_code != 'AccessDenied':
+            print('GOT UNWANTED ERROR', e.error_code)
+            raise
+        # seems like we're not the owner of the bucket; ignore
+        pass
+
+def nuke_prefixed_buckets():
+    for name, conn in list(s3.items()):
+        print('Cleaning buckets from connection {name}'.format(name=name))
+        for bucket in conn.get_all_buckets():
+            if bucket.name.startswith(prefix):
+                print('Cleaning bucket {bucket}'.format(bucket=bucket))
+                nuke_bucket(bucket)
+
+    print('Done with cleanup of test buckets.')
+
+def read_config(fp):
+    config = munch.Munch()
+    g = yaml.safe_load_all(fp)
+    for new in g:
+        config.update(munch.Munchify(new))
+    return config
+
+def connect(conf):
+    mapping = dict(
+        port='port',
+        host='host',
+        is_secure='is_secure',
+        access_key='aws_access_key_id',
+        secret_key='aws_secret_access_key',
+        )
+    kwargs = dict((mapping[k],v) for (k,v) in conf.items() if k in mapping)
+    #process calling_format argument
+    calling_formats = dict(
+        ordinary=boto.s3.connection.OrdinaryCallingFormat(),
+        subdomain=boto.s3.connection.SubdomainCallingFormat(),
+        vhost=boto.s3.connection.VHostCallingFormat(),
+        )
+    kwargs['calling_format'] = calling_formats['ordinary']
+    if 'calling_format' in conf:
+        raw_calling_format = conf['calling_format']
+        try:
+            kwargs['calling_format'] = calling_formats[raw_calling_format]
+        except KeyError:
+            raise RuntimeError(
+                'calling_format unknown: %r' % raw_calling_format
+                )
+    # TODO test vhost calling format
+    conn = boto.s3.connection.S3Connection(**kwargs)
+    return conn
+
+def setup():
+    global s3, config, prefix
+    s3.clear()
+    config.clear()
+
+    try:
+        path = os.environ['S3TEST_CONF']
+    except KeyError:
+        raise RuntimeError(
+            'To run tests, point environment '
+            + 'variable S3TEST_CONF to a config file.',
+            )
+    with file(path) as f:
+        config.update(read_config(f))
+
+    # These 3 should always be present.
+    if 's3' not in config:
+        raise RuntimeError('Your config file is missing the s3 section!')
+    if 'defaults' not in config.s3:
+        raise RuntimeError('Your config file is missing the s3.defaults section!')
+    if 'fixtures' not in config:
+        raise RuntimeError('Your config file is missing the fixtures section!')
+
+    template = config.fixtures.get('bucket prefix', 'test-{random}-')
+    prefix = choose_bucket_prefix(template=template)
+    if prefix == '':
+        raise RuntimeError("Empty Prefix! Aborting!")
+
+    defaults = config.s3.defaults
+    for section in list(config.s3.keys()):
+        if section == 'defaults':
+            continue
+
+        conf = {}
+        conf.update(defaults)
+        conf.update(config.s3[section])
+        conn = connect(conf)
+        s3[section] = conn
+
+    # WARNING! we actively delete all buckets we see with the prefix
+    # we've chosen! Choose your prefix with care, and don't reuse
+    # credentials!
+
+    # We also assume nobody else is going to use buckets with that
+    # prefix. This is racy but given enough randomness, should not
+    # really fail.
+    nuke_prefixed_buckets()
+
+def get_new_bucket(connection=None):
+    """
+    Get a bucket that exists and is empty.
+
+    Always recreates a bucket from scratch. This is useful to also
+    reset ACLs and such.
+    """
+    if connection is None:
+        connection = s3.main
+    name = '{prefix}{num}'.format(
+        prefix=prefix,
+        num=next(bucket_counter),
+        )
+    # the only way for this to fail with a pre-existing bucket is if
+    # someone raced us between setup nuke_prefixed_buckets and here;
+    # ignore that as astronomically unlikely
+    bucket = connection.create_bucket(name)
+    return bucket
+
+def teardown():
+    nuke_prefixed_buckets()
+
+def with_setup_kwargs(setup, teardown=None):
+    """Decorator to add setup and/or teardown methods to a test function::
+
+      @with_setup_args(setup, teardown)
+      def test_something():
+          " ... "
+
+    The setup function should return (kwargs) which will be passed to
+    test function, and teardown function.
+
+    Note that `with_setup_kwargs` is useful *only* for test functions, not for test
+    methods or inside of TestCase subclasses.
+    """
+    def decorate(func):
+        kwargs = {}
+
+        def test_wrapped(*args, **kwargs2):
+            k2 = kwargs.copy()
+            k2.update(kwargs2)
+            k2['testname'] = func.__name__
+            func(*args, **k2)
+
+        test_wrapped.__name__ = func.__name__
+
+        def setup_wrapped():
+            k = setup()
+            kwargs.update(k)
+            if hasattr(func, 'setup'):
+                func.setup()
+        test_wrapped.setup = setup_wrapped
+
+        if teardown:
+            def teardown_wrapped():
+                if hasattr(func, 'teardown'):
+                    func.teardown()
+                teardown(**kwargs)
+
+            test_wrapped.teardown = teardown_wrapped
+        else:
+            if hasattr(func, 'teardown'):
+                test_wrapped.teardown = func.teardown()
+        return test_wrapped
+    return decorate
+
+# Demo case for the above, when you run test_gen():
+# _test_gen will run twice,
+# with the following stderr printing
+# setup_func {'b': 2}
+# testcase ('1',) {'b': 2, 'testname': '_test_gen'}
+# teardown_func {'b': 2}
+# setup_func {'b': 2}
+# testcase () {'b': 2, 'testname': '_test_gen'}
+# teardown_func {'b': 2}
+# 
+#def setup_func():
+#    kwargs = {'b': 2}
+#    print("setup_func", kwargs, file=sys.stderr)
+#    return kwargs
+#
+#def teardown_func(**kwargs):
+#    print("teardown_func", kwargs, file=sys.stderr)
+#
+#@with_setup_kwargs(setup=setup_func, teardown=teardown_func)
+#def _test_gen(*args, **kwargs):
+#    print("testcase", args, kwargs, file=sys.stderr)
+#
+#def test_gen():
+#    yield _test_gen, '1'
+#    yield _test_gen
+
+def trim_xml(xml_str):
+    p = etree.XMLParser(remove_blank_text=True)
+    elem = etree.XML(xml_str, parser=p)
+    return etree.tostring(elem)
+
+def normalize_xml(xml, pretty_print=True):
+    if xml is None:
+        return xml
+
+    root = etree.fromstring(xml.encode(encoding='ascii'))
+
+    for element in root.iter('*'):
+        if element.text is not None and not element.text.strip():
+            element.text = None
+        if element.text is not None:
+            element.text = element.text.strip().replace("\n", "").replace("\r", "")
+        if element.tail is not None and not element.tail.strip():
+            element.tail = None
+        if element.tail is not None:
+            element.tail = element.tail.strip().replace("\n", "").replace("\r", "")
+
+    # Sort the elements
+    for parent in root.xpath('//*[./*]'): # Search for parent elements
+          parent[:] = sorted(parent,key=lambda x: x.tag)
+
+    xmlstr = etree.tostring(root, encoding="utf-8", xml_declaration=True, pretty_print=pretty_print)
+    # there are two different DTD URIs
+    xmlstr = re.sub(r'xmlns="[^"]+"', 'xmlns="s3"', xmlstr)
+    xmlstr = re.sub(r'xmlns=\'[^\']+\'', 'xmlns="s3"', xmlstr)
+    for uri in ['http://doc.s3.amazonaws.com/doc/2006-03-01/', 'http://s3.amazonaws.com/doc/2006-03-01/']:
+        xmlstr = xmlstr.replace(uri, 'URI-DTD')
+    #xmlstr = re.sub(r'>\s+', '>', xmlstr, count=0, flags=re.MULTILINE)
+    return xmlstr
+
+def assert_xml_equal(got, want):
+    assert want is not None, 'Wanted XML cannot be None'
+    if got is None:
+        raise AssertionError('Got input to validate was None')
+    checker = LXMLOutputChecker()
+    if not checker.check_output(want, got, 0):
+        message = checker.output_difference(Example("", want), got, 0)
+        raise AssertionError(message)

s3tests_boto3/functional/__init__.py

@@ -0,0 +1,782 @@
+import pytest
+import boto3
+from botocore import UNSIGNED
+from botocore.client import Config
+from botocore.exceptions import ClientError
+from botocore.handlers import disable_signing
+import configparser
+import datetime
+import time
+import os
+import munch
+import random
+import string
+import itertools
+import urllib3
+import re
+
+config = munch.Munch
+
+# this will be assigned by setup()
+prefix = None
+
+def get_prefix():
+    assert prefix is not None
+    return prefix
+
+def choose_bucket_prefix(template, max_len=30):
+    """
+    Choose a prefix for our test buckets, so they're easy to identify.
+
+    Use template and feed it more and more random filler, until it's
+    as long as possible but still below max_len.
+    """
+    rand = ''.join(
+        random.choice(string.ascii_lowercase + string.digits)
+        for c in range(255)
+        )
+
+    while rand:
+        s = template.format(random=rand)
+        if len(s) <= max_len:
+            return s
+        rand = rand[:-1]
+
+    raise RuntimeError(
+        'Bucket prefix template is impossible to fulfill: {template!r}'.format(
+            template=template,
+            ),
+        )
+
+def get_buckets_list(client=None, prefix=None):
+    if client == None:
+        client = get_client()
+    if prefix == None:
+        prefix = get_prefix()
+    response = client.list_buckets()
+    bucket_dicts = response['Buckets']
+    buckets_list = []
+    for bucket in bucket_dicts:
+        if prefix in bucket['Name']:
+            buckets_list.append(bucket['Name'])
+
+    return buckets_list
+
+def get_objects_list(bucket, client=None, prefix=None):
+    if client == None:
+        client = get_client()
+
+    if prefix == None:
+        response = client.list_objects(Bucket=bucket)
+    else:
+        response = client.list_objects(Bucket=bucket, Prefix=prefix)
+    objects_list = []
+
+    if 'Contents' in response:
+        contents = response['Contents']
+        for obj in contents:
+            objects_list.append(obj['Key'])
+
+    return objects_list
+
+# generator function that returns object listings in batches, where each
+# batch is a list of dicts compatible with delete_objects()
+def list_versions(client, bucket, batch_size):
+    kwargs = {'Bucket': bucket, 'MaxKeys': batch_size}
+    truncated = True
+    while truncated:
+        listing = client.list_object_versions(**kwargs)
+
+        kwargs['KeyMarker'] = listing.get('NextKeyMarker')
+        kwargs['VersionIdMarker'] = listing.get('NextVersionIdMarker')
+        truncated = listing['IsTruncated']
+
+        objs = listing.get('Versions', []) + listing.get('DeleteMarkers', [])
+        if len(objs):
+            yield [{'Key': o['Key'], 'VersionId': o['VersionId']} for o in objs]
+
+def nuke_bucket(client, bucket):
+    batch_size = 128
+    max_retain_date = None
+
+    # list and delete objects in batches
+    for objects in list_versions(client, bucket, batch_size):
+        delete = client.delete_objects(Bucket=bucket,
+                Delete={'Objects': objects, 'Quiet': True},
+                BypassGovernanceRetention=True)
+
+        # check for object locks on 403 AccessDenied errors
+        for err in delete.get('Errors', []):
+            if err.get('Code') != 'AccessDenied':
+                continue
+            try:
+                res = client.get_object_retention(Bucket=bucket,
+                        Key=err['Key'], VersionId=err['VersionId'])
+                retain_date = res['Retention']['RetainUntilDate']
+                if not max_retain_date or max_retain_date < retain_date:
+                    max_retain_date = retain_date
+            except ClientError:
+                pass
+
+    if max_retain_date:
+        # wait out the retention period (up to 60 seconds)
+        now = datetime.datetime.now(max_retain_date.tzinfo)
+        if max_retain_date > now:
+            delta = max_retain_date - now
+            if delta.total_seconds() > 60:
+                raise RuntimeError('bucket {} still has objects \
+locked for {} more seconds, not waiting for \
+bucket cleanup'.format(bucket, delta.total_seconds()))
+            print('nuke_bucket', bucket, 'waiting', delta.total_seconds(),
+                    'seconds for object locks to expire')
+            time.sleep(delta.total_seconds())
+
+        for objects in list_versions(client, bucket, batch_size):
+            client.delete_objects(Bucket=bucket,
+                    Delete={'Objects': objects, 'Quiet': True},
+                    BypassGovernanceRetention=True)
+
+    client.delete_bucket(Bucket=bucket)
+
+def nuke_prefixed_buckets(prefix, client=None):
+    if client == None:
+        client = get_client()
+
+    buckets = get_buckets_list(client, prefix)
+
+    err = None
+    for bucket_name in buckets:
+        try:
+            nuke_bucket(client, bucket_name)
+        except Exception as e:
+            # The exception shouldn't be raised when doing cleanup. Pass and continue
+            # the bucket cleanup process. Otherwise left buckets wouldn't be cleared
+            # resulting in some kind of resource leak. err is used to hint user some
+            # exception once occurred.
+            err = e
+            pass
+    if err:
+        raise err
+
+    print('Done with cleanup of buckets in tests.')
+
+def configured_storage_classes():
+    sc = ['STANDARD']
+
+    extra_sc = re.split(r"[\b\W\b]+", config.storage_classes)
+
+    for item in extra_sc:
+        if item != 'STANDARD':
+             sc.append(item)
+
+    sc = [i for i in sc if i]
+    print("storage classes configured: " + str(sc))
+
+    return sc
+
+def configure():
+    cfg = configparser.RawConfigParser()
+    try:
+        path = os.environ['S3TEST_CONF']
+    except KeyError:
+        raise RuntimeError(
+            'To run tests, point environment '
+            + 'variable S3TEST_CONF to a config file.',
+            )
+    cfg.read(path)
+
+    if not cfg.defaults():
+        raise RuntimeError('Your config file is missing the DEFAULT section!')
+    if not cfg.has_section("s3 main"):
+        raise RuntimeError('Your config file is missing the "s3 main" section!')
+    if not cfg.has_section("s3 alt"):
+        raise RuntimeError('Your config file is missing the "s3 alt" section!')
+    if not cfg.has_section("s3 tenant"):
+        raise RuntimeError('Your config file is missing the "s3 tenant" section!')
+
+    global prefix
+
+    defaults = cfg.defaults()
+
+    # vars from the DEFAULT section
+    config.default_host = defaults.get("host")
+    config.default_port = int(defaults.get("port"))
+    config.default_is_secure = cfg.getboolean('DEFAULT', "is_secure")
+
+    proto = 'https' if config.default_is_secure else 'http'
+    config.default_endpoint = "%s://%s:%d" % (proto, config.default_host, config.default_port)
+
+    try:
+        config.default_ssl_verify = cfg.getboolean('DEFAULT', "ssl_verify")
+    except configparser.NoOptionError:
+        config.default_ssl_verify = False
+
+    # Disable InsecureRequestWarning reported by urllib3 when ssl_verify is False
+    if not config.default_ssl_verify:
+        urllib3.disable_warnings()
+
+    # vars from the main section
+    config.main_access_key = cfg.get('s3 main',"access_key")
+    config.main_secret_key = cfg.get('s3 main',"secret_key")
+    config.main_display_name = cfg.get('s3 main',"display_name")
+    config.main_user_id = cfg.get('s3 main',"user_id")
+    config.main_email = cfg.get('s3 main',"email")
+    try:
+        config.main_kms_keyid = cfg.get('s3 main',"kms_keyid")
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.main_kms_keyid = 'testkey-1'
+
+    try:
+        config.main_kms_keyid2 = cfg.get('s3 main',"kms_keyid2")
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.main_kms_keyid2 = 'testkey-2'
+
+    try:
+        config.main_api_name = cfg.get('s3 main',"api_name")
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.main_api_name = ""
+        pass
+
+    try:
+        config.storage_classes = cfg.get('s3 main',"storage_classes")
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.storage_classes = ""
+        pass
+
+    try:
+        config.lc_debug_interval = int(cfg.get('s3 main',"lc_debug_interval"))
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.lc_debug_interval = 10
+
+    config.alt_access_key = cfg.get('s3 alt',"access_key")
+    config.alt_secret_key = cfg.get('s3 alt',"secret_key")
+    config.alt_display_name = cfg.get('s3 alt',"display_name")
+    config.alt_user_id = cfg.get('s3 alt',"user_id")
+    config.alt_email = cfg.get('s3 alt',"email")
+
+    config.tenant_access_key = cfg.get('s3 tenant',"access_key")
+    config.tenant_secret_key = cfg.get('s3 tenant',"secret_key")
+    config.tenant_display_name = cfg.get('s3 tenant',"display_name")
+    config.tenant_user_id = cfg.get('s3 tenant',"user_id")
+    config.tenant_email = cfg.get('s3 tenant',"email")
+    config.tenant_name = cfg.get('s3 tenant',"tenant")
+
+    config.iam_access_key = cfg.get('iam',"access_key")
+    config.iam_secret_key = cfg.get('iam',"secret_key")
+    config.iam_display_name = cfg.get('iam',"display_name")
+    config.iam_user_id = cfg.get('iam',"user_id")
+    config.iam_email = cfg.get('iam',"email")
+
+    config.iam_root_access_key = cfg.get('iam root',"access_key")
+    config.iam_root_secret_key = cfg.get('iam root',"secret_key")
+    config.iam_root_user_id = cfg.get('iam root',"user_id")
+    config.iam_root_email = cfg.get('iam root',"email")
+
+    config.iam_alt_root_access_key = cfg.get('iam alt root',"access_key")
+    config.iam_alt_root_secret_key = cfg.get('iam alt root',"secret_key")
+    config.iam_alt_root_user_id = cfg.get('iam alt root',"user_id")
+    config.iam_alt_root_email = cfg.get('iam alt root',"email")
+
+    # vars from the fixtures section
+    template = cfg.get('fixtures', "bucket prefix", fallback='test-{random}-')
+    prefix = choose_bucket_prefix(template=template)
+    template = cfg.get('fixtures', "iam name prefix", fallback="s3-tests-")
+    config.iam_name_prefix = choose_bucket_prefix(template=template)
+    template = cfg.get('fixtures', "iam path prefix", fallback="/s3-tests/")
+    config.iam_path_prefix = choose_bucket_prefix(template=template)
+
+    if cfg.has_section("s3 cloud"):
+        get_cloud_config(cfg)
+    else:
+        config.cloud_storage_class = None
+
+def setup():
+    alt_client = get_alt_client()
+    tenant_client = get_tenant_client()
+    nuke_prefixed_buckets(prefix=prefix)
+    nuke_prefixed_buckets(prefix=prefix, client=alt_client)
+    nuke_prefixed_buckets(prefix=prefix, client=tenant_client)
+
+def teardown():
+    alt_client = get_alt_client()
+    tenant_client = get_tenant_client()
+    nuke_prefixed_buckets(prefix=prefix)
+    nuke_prefixed_buckets(prefix=prefix, client=alt_client)
+    nuke_prefixed_buckets(prefix=prefix, client=tenant_client)
+    try:
+        iam_client = get_iam_client()
+        list_roles_resp = iam_client.list_roles()
+        for role in list_roles_resp['Roles']:
+            list_policies_resp = iam_client.list_role_policies(RoleName=role['RoleName'])
+            for policy in list_policies_resp['PolicyNames']:
+                del_policy_resp = iam_client.delete_role_policy(
+                                         RoleName=role['RoleName'],
+                                         PolicyName=policy
+                                        )
+            del_role_resp = iam_client.delete_role(RoleName=role['RoleName'])
+        list_oidc_resp = iam_client.list_open_id_connect_providers()
+        for oidcprovider in list_oidc_resp['OpenIDConnectProviderList']:
+            del_oidc_resp = iam_client.delete_open_id_connect_provider(
+                        OpenIDConnectProviderArn=oidcprovider['Arn']
+                    )
+    except:
+        pass
+
+@pytest.fixture(scope="package")
+def configfile():
+    configure()
+    return config
+
+@pytest.fixture(autouse=True)
+def setup_teardown(configfile):
+    setup()
+    yield
+    teardown()
+
+def check_webidentity():
+    cfg = configparser.RawConfigParser()
+    try:
+        path = os.environ['S3TEST_CONF']
+    except KeyError:
+        raise RuntimeError(
+            'To run tests, point environment '
+            + 'variable S3TEST_CONF to a config file.',
+            )
+    cfg.read(path)
+    if not cfg.has_section("webidentity"):
+        raise RuntimeError('Your config file is missing the "webidentity" section!')
+
+    config.webidentity_thumbprint = cfg.get('webidentity', "thumbprint")
+    config.webidentity_aud = cfg.get('webidentity', "aud")
+    config.webidentity_token = cfg.get('webidentity', "token")
+    config.webidentity_realm = cfg.get('webidentity', "KC_REALM")
+    config.webidentity_sub = cfg.get('webidentity', "sub")
+    config.webidentity_azp = cfg.get('webidentity', "azp")
+    config.webidentity_user_token = cfg.get('webidentity', "user_token")
+
+def get_cloud_config(cfg):
+    config.cloud_host = cfg.get('s3 cloud',"host")
+    config.cloud_port = int(cfg.get('s3 cloud',"port"))
+    config.cloud_is_secure = cfg.getboolean('s3 cloud', "is_secure")
+
+    proto = 'https' if config.cloud_is_secure else 'http'
+    config.cloud_endpoint = "%s://%s:%d" % (proto, config.cloud_host, config.cloud_port)
+
+    config.cloud_access_key = cfg.get('s3 cloud',"access_key")
+    config.cloud_secret_key = cfg.get('s3 cloud',"secret_key")
+
+    try:
+        config.cloud_storage_class = cfg.get('s3 cloud', "cloud_storage_class")
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.cloud_storage_class = None
+    
+    try:
+        config.cloud_retain_head_object = cfg.get('s3 cloud',"retain_head_object")
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.cloud_retain_head_object = None
+
+    try:
+        config.cloud_target_path = cfg.get('s3 cloud',"target_path")
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.cloud_target_path = None
+
+    try:
+        config.cloud_target_storage_class = cfg.get('s3 cloud',"target_storage_class")
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.cloud_target_storage_class = 'STANDARD'
+
+    try:
+        config.cloud_regular_storage_class = cfg.get('s3 cloud', "storage_class")
+    except (configparser.NoSectionError, configparser.NoOptionError):
+        config.cloud_regular_storage_class  = None
+
+
+def get_client(client_config=None):
+    if client_config == None:
+        client_config = Config(signature_version='s3v4')
+
+    client = boto3.client(service_name='s3',
+                        aws_access_key_id=config.main_access_key,
+                        aws_secret_access_key=config.main_secret_key,
+                        endpoint_url=config.default_endpoint,
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        config=client_config)
+    return client
+
+def get_v2_client():
+    client = boto3.client(service_name='s3',
+                        aws_access_key_id=config.main_access_key,
+                        aws_secret_access_key=config.main_secret_key,
+                        endpoint_url=config.default_endpoint,
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        config=Config(signature_version='s3'))
+    return client
+
+def get_sts_client(**kwargs):
+    kwargs.setdefault('aws_access_key_id', config.alt_access_key)
+    kwargs.setdefault('aws_secret_access_key', config.alt_secret_key)
+    kwargs.setdefault('config', Config(signature_version='s3v4'))
+
+    client = boto3.client(service_name='sts',
+                          endpoint_url=config.default_endpoint,
+                          region_name='',
+                          use_ssl=config.default_is_secure,
+                          verify=config.default_ssl_verify,
+                          **kwargs)
+    return client
+
+def get_iam_client(**kwargs):
+    kwargs.setdefault('aws_access_key_id', config.iam_access_key)
+    kwargs.setdefault('aws_secret_access_key', config.iam_secret_key)
+
+    client = boto3.client(service_name='iam',
+                        endpoint_url=config.default_endpoint,
+                        region_name='',
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        **kwargs)
+    return client
+
+def get_iam_s3client(**kwargs):
+    kwargs.setdefault('aws_access_key_id', config.iam_access_key)
+    kwargs.setdefault('aws_secret_access_key', config.iam_secret_key)
+    kwargs.setdefault('config', Config(signature_version='s3v4'))
+
+    client = boto3.client(service_name='s3',
+                          endpoint_url=config.default_endpoint,
+                          use_ssl=config.default_is_secure,
+                          verify=config.default_ssl_verify,
+                          **kwargs)
+    return client
+
+def get_iam_root_client(**kwargs):
+    kwargs.setdefault('service_name', 'iam')
+    kwargs.setdefault('aws_access_key_id', config.iam_root_access_key)
+    kwargs.setdefault('aws_secret_access_key', config.iam_root_secret_key)
+
+    return boto3.client(endpoint_url=config.default_endpoint,
+                        region_name='',
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        **kwargs)
+
+def get_iam_alt_root_client(**kwargs):
+    kwargs.setdefault('service_name', 'iam')
+    kwargs.setdefault('aws_access_key_id', config.iam_alt_root_access_key)
+    kwargs.setdefault('aws_secret_access_key', config.iam_alt_root_secret_key)
+
+    return boto3.client(endpoint_url=config.default_endpoint,
+                        region_name='',
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        **kwargs)
+
+def get_alt_client(client_config=None):
+    if client_config == None:
+        client_config = Config(signature_version='s3v4')
+
+    client = boto3.client(service_name='s3',
+                        aws_access_key_id=config.alt_access_key,
+                        aws_secret_access_key=config.alt_secret_key,
+                        endpoint_url=config.default_endpoint,
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        config=client_config)
+    return client
+
+def get_cloud_client(client_config=None):
+    if client_config == None:
+        client_config = Config(signature_version='s3v4')
+
+    client = boto3.client(service_name='s3',
+                        aws_access_key_id=config.cloud_access_key,
+                        aws_secret_access_key=config.cloud_secret_key,
+                        endpoint_url=config.cloud_endpoint,
+                        use_ssl=config.cloud_is_secure,
+                        config=client_config)
+    return client
+
+def get_tenant_client(client_config=None):
+    if client_config == None:
+        client_config = Config(signature_version='s3v4')
+
+    client = boto3.client(service_name='s3',
+                        aws_access_key_id=config.tenant_access_key,
+                        aws_secret_access_key=config.tenant_secret_key,
+                        endpoint_url=config.default_endpoint,
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        config=client_config)
+    return client
+
+def get_v2_tenant_client():
+    client_config = Config(signature_version='s3')
+    client = boto3.client(service_name='s3',
+                          aws_access_key_id=config.tenant_access_key,
+                          aws_secret_access_key=config.tenant_secret_key,
+                          endpoint_url=config.default_endpoint,
+                          use_ssl=config.default_is_secure,
+                          verify=config.default_ssl_verify,
+                          config=client_config)
+    return client
+
+def get_tenant_iam_client():
+
+    client = boto3.client(service_name='iam',
+                          region_name='us-east-1',
+                          aws_access_key_id=config.tenant_access_key,
+                          aws_secret_access_key=config.tenant_secret_key,
+                          endpoint_url=config.default_endpoint,
+                          verify=config.default_ssl_verify,
+                          use_ssl=config.default_is_secure)
+    return client
+
+def get_alt_iam_client():
+
+    client = boto3.client(service_name='iam',
+                          region_name='',
+                          aws_access_key_id=config.alt_access_key,
+                          aws_secret_access_key=config.alt_secret_key,
+                          endpoint_url=config.default_endpoint,
+                          verify=config.default_ssl_verify,
+                          use_ssl=config.default_is_secure)
+    return client
+
+def get_unauthenticated_client():
+    client = boto3.client(service_name='s3',
+                        aws_access_key_id='',
+                        aws_secret_access_key='',
+                        endpoint_url=config.default_endpoint,
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        config=Config(signature_version=UNSIGNED))
+    return client
+
+def get_bad_auth_client(aws_access_key_id='badauth'):
+    client = boto3.client(service_name='s3',
+                        aws_access_key_id=aws_access_key_id,
+                        aws_secret_access_key='roflmao',
+                        endpoint_url=config.default_endpoint,
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        config=Config(signature_version='s3v4'))
+    return client
+
+def get_svc_client(client_config=None, svc='s3'):
+    if client_config == None:
+        client_config = Config(signature_version='s3v4')
+
+    client = boto3.client(service_name=svc,
+                        aws_access_key_id=config.main_access_key,
+                        aws_secret_access_key=config.main_secret_key,
+                        endpoint_url=config.default_endpoint,
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify,
+                        config=client_config)
+    return client
+
+bucket_counter = itertools.count(1)
+
+def get_new_bucket_name():
+    """
+    Get a bucket name that probably does not exist.
+
+    We make every attempt to use a unique random prefix, so if a
+    bucket by this name happens to exist, it's ok if tests give
+    false negatives.
+    """
+    name = '{prefix}{num}'.format(
+        prefix=prefix,
+        num=next(bucket_counter),
+        )
+    return name
+
+def get_new_bucket_resource(name=None):
+    """
+    Get a bucket that exists and is empty.
+
+    Always recreates a bucket from scratch. This is useful to also
+    reset ACLs and such.
+    """
+    s3 = boto3.resource('s3',
+                        aws_access_key_id=config.main_access_key,
+                        aws_secret_access_key=config.main_secret_key,
+                        endpoint_url=config.default_endpoint,
+                        use_ssl=config.default_is_secure,
+                        verify=config.default_ssl_verify)
+    if name is None:
+        name = get_new_bucket_name()
+    bucket = s3.Bucket(name)
+    bucket_location = bucket.create()
+    return bucket
+
+def get_new_bucket(client=None, name=None):
+    """
+    Get a bucket that exists and is empty.
+
+    Always recreates a bucket from scratch. This is useful to also
+    reset ACLs and such.
+    """
+    if client is None:
+        client = get_client()
+    if name is None:
+        name = get_new_bucket_name()
+
+    client.create_bucket(Bucket=name)
+    return name
+
+def get_parameter_name():
+    parameter_name=""
+    rand = ''.join(
+        random.choice(string.ascii_lowercase + string.digits)
+        for c in range(255)
+        )
+    while rand:
+        parameter_name = '{random}'.format(random=rand)
+        if len(parameter_name) <= 10:
+            return parameter_name
+        rand = rand[:-1]
+    return parameter_name
+
+def get_sts_user_id():
+    return config.alt_user_id
+
+def get_config_is_secure():
+    return config.default_is_secure
+
+def get_config_host():
+    return config.default_host
+
+def get_config_port():
+    return config.default_port
+
+def get_config_endpoint():
+    return config.default_endpoint
+
+def get_config_ssl_verify():
+    return config.default_ssl_verify
+
+def get_main_aws_access_key():
+    return config.main_access_key
+
+def get_main_aws_secret_key():
+    return config.main_secret_key
+
+def get_main_display_name():
+    return config.main_display_name
+
+def get_main_user_id():
+    return config.main_user_id
+
+def get_main_email():
+    return config.main_email
+
+def get_main_api_name():
+    return config.main_api_name
+
+def get_main_kms_keyid():
+    return config.main_kms_keyid
+
+def get_secondary_kms_keyid():
+    return config.main_kms_keyid2
+
+def get_alt_aws_access_key():
+    return config.alt_access_key
+
+def get_alt_aws_secret_key():
+    return config.alt_secret_key
+
+def get_alt_display_name():
+    return config.alt_display_name
+
+def get_alt_user_id():
+    return config.alt_user_id
+
+def get_alt_email():
+    return config.alt_email
+
+def get_tenant_aws_access_key():
+    return config.tenant_access_key
+
+def get_tenant_aws_secret_key():
+    return config.tenant_secret_key
+
+def get_tenant_display_name():
+    return config.tenant_display_name
+
+def get_tenant_name():
+    return config.tenant_name
+
+def get_tenant_user_id():
+    return config.tenant_user_id
+
+def get_tenant_email():
+    return config.tenant_email
+
+def get_thumbprint():
+    return config.webidentity_thumbprint
+
+def get_aud():
+    return config.webidentity_aud
+
+def get_sub():
+    return config.webidentity_sub
+
+def get_azp():
+    return config.webidentity_azp
+
+def get_token():
+    return config.webidentity_token
+
+def get_realm_name():
+    return config.webidentity_realm
+
+def get_iam_name_prefix():
+    return config.iam_name_prefix
+
+def make_iam_name(name):
+    return config.iam_name_prefix + name
+
+def get_iam_path_prefix():
+    return config.iam_path_prefix
+
+def get_iam_access_key():
+    return config.iam_access_key
+
+def get_iam_secret_key():
+    return config.iam_secret_key
+
+def get_iam_root_user_id():
+    return config.iam_root_user_id
+
+def get_iam_root_email():
+    return config.iam_root_email
+
+def get_iam_alt_root_user_id():
+    return config.iam_alt_root_user_id
+
+def get_iam_alt_root_email():
+    return config.iam_alt_root_email
+
+def get_user_token():
+    return config.webidentity_user_token
+
+def get_cloud_storage_class():
+    return config.cloud_storage_class
+
+def get_cloud_retain_head_object():
+    return config.cloud_retain_head_object
+
+def get_cloud_regular_storage_class():
+    return config.cloud_regular_storage_class
+
+def get_cloud_target_path():
+    return config.cloud_target_path
+
+def get_cloud_target_storage_class():
+    return config.cloud_target_storage_class
+
+def get_lc_debug_interval():
+    return config.lc_debug_interval

s3tests_boto3/functional/iam.py

@@ -0,0 +1,199 @@
+from botocore.exceptions import ClientError
+import pytest
+
+from . import (
+    configfile,
+    get_iam_root_client,
+    get_iam_root_user_id,
+    get_iam_root_email,
+    get_iam_alt_root_client,
+    get_iam_alt_root_user_id,
+    get_iam_alt_root_email,
+    get_iam_path_prefix,
+)
+
+def nuke_user_keys(client, name):
+    p = client.get_paginator('list_access_keys')
+    for response in p.paginate(UserName=name):
+        for key in response['AccessKeyMetadata']:
+            try:
+                client.delete_access_key(UserName=name, AccessKeyId=key['AccessKeyId'])
+            except:
+                pass
+
+def nuke_user_policies(client, name):
+    p = client.get_paginator('list_user_policies')
+    for response in p.paginate(UserName=name):
+        for policy in response['PolicyNames']:
+            try:
+                client.delete_user_policy(UserName=name, PolicyName=policy)
+            except:
+                pass
+
+def nuke_attached_user_policies(client, name):
+    p = client.get_paginator('list_attached_user_policies')
+    for response in p.paginate(UserName=name):
+        for policy in response['AttachedPolicies']:
+            try:
+                client.detach_user_policy(UserName=name, PolicyArn=policy['PolicyArn'])
+            except:
+                pass
+
+def nuke_user(client, name):
+    # delete access keys, user policies, etc
+    try:
+        nuke_user_keys(client, name)
+    except:
+        pass
+    try:
+        nuke_user_policies(client, name)
+    except:
+        pass
+    try:
+        nuke_attached_user_policies(client, name)
+    except:
+        pass
+    client.delete_user(UserName=name)
+
+def nuke_users(client, **kwargs):
+    p = client.get_paginator('list_users')
+    for response in p.paginate(**kwargs):
+        for user in response['Users']:
+            try:
+                nuke_user(client, user['UserName'])
+            except:
+                pass
+
+def nuke_group_policies(client, name):
+    p = client.get_paginator('list_group_policies')
+    for response in p.paginate(GroupName=name):
+        for policy in response['PolicyNames']:
+            try:
+                client.delete_group_policy(GroupName=name, PolicyName=policy)
+            except:
+                pass
+
+def nuke_attached_group_policies(client, name):
+    p = client.get_paginator('list_attached_group_policies')
+    for response in p.paginate(GroupName=name):
+        for policy in response['AttachedPolicies']:
+            try:
+                client.detach_group_policy(GroupName=name, PolicyArn=policy['PolicyArn'])
+            except:
+                pass
+
+def nuke_group_users(client, name):
+    p = client.get_paginator('get_group')
+    for response in p.paginate(GroupName=name):
+        for user in response['Users']:
+            try:
+                client.remove_user_from_group(GroupName=name, UserName=user['UserName'])
+            except:
+                pass
+
+def nuke_group(client, name):
+    # delete group policies and remove all users
+    try:
+        nuke_group_policies(client, name)
+    except:
+        pass
+    try:
+        nuke_attached_group_policies(client, name)
+    except:
+        pass
+    try:
+        nuke_group_users(client, name)
+    except:
+        pass
+    client.delete_group(GroupName=name)
+
+def nuke_groups(client, **kwargs):
+    p = client.get_paginator('list_groups')
+    for response in p.paginate(**kwargs):
+        for user in response['Groups']:
+            try:
+                nuke_group(client, user['GroupName'])
+            except:
+                pass
+
+def nuke_role_policies(client, name):
+    p = client.get_paginator('list_role_policies')
+    for response in p.paginate(RoleName=name):
+        for policy in response['PolicyNames']:
+            try:
+                client.delete_role_policy(RoleName=name, PolicyName=policy)
+            except:
+                pass
+
+def nuke_attached_role_policies(client, name):
+    p = client.get_paginator('list_attached_role_policies')
+    for response in p.paginate(RoleName=name):
+        for policy in response['AttachedPolicies']:
+            try:
+                client.detach_role_policy(RoleName=name, PolicyArn=policy['PolicyArn'])
+            except:
+                pass
+
+def nuke_role(client, name):
+    # delete role policies, etc
+    try:
+        nuke_role_policies(client, name)
+    except:
+        pass
+    try:
+        nuke_attached_role_policies(client, name)
+    except:
+        pass
+    client.delete_role(RoleName=name)
+
+def nuke_roles(client, **kwargs):
+    p = client.get_paginator('list_roles')
+    for response in p.paginate(**kwargs):
+        for role in response['Roles']:
+            try:
+                nuke_role(client, role['RoleName'])
+            except:
+                pass
+
+def nuke_oidc_providers(client, prefix):
+    result = client.list_open_id_connect_providers()
+    for provider in result['OpenIDConnectProviderList']:
+        arn = provider['Arn']
+        if f':oidc-provider{prefix}' in arn:
+            try:
+                client.delete_open_id_connect_provider(OpenIDConnectProviderArn=arn)
+            except:
+                pass
+
+
+# fixture for iam account root user
+@pytest.fixture
+def iam_root(configfile):
+    client = get_iam_root_client()
+    try:
+        arn = client.get_user()['User']['Arn']
+        if not arn.endswith(':root'):
+            pytest.skip('[iam root] user does not have :root arn')
+    except ClientError as e:
+        pytest.skip('[iam root] user does not belong to an account')
+
+    yield client
+    nuke_users(client, PathPrefix=get_iam_path_prefix())
+    nuke_groups(client, PathPrefix=get_iam_path_prefix())
+    nuke_roles(client, PathPrefix=get_iam_path_prefix())
+    nuke_oidc_providers(client, get_iam_path_prefix())
+
+# fixture for iam alt account root user
+@pytest.fixture
+def iam_alt_root(configfile):
+    client = get_iam_alt_root_client()
+    try:
+        arn = client.get_user()['User']['Arn']
+        if not arn.endswith(':root'):
+            pytest.skip('[iam alt root] user does not have :root arn')
+    except ClientError as e:
+        pytest.skip('[iam alt root] user does not belong to an account')
+
+    yield client
+    nuke_users(client, PathPrefix=get_iam_path_prefix())
+    nuke_roles(client, PathPrefix=get_iam_path_prefix())

s3tests_boto3/functional/policy.py

@@ -0,0 +1,46 @@
+import json
+
+class Statement(object):
+    def __init__(self, action, resource, principal = {"AWS" : "*"}, effect= "Allow", condition = None):
+        self.principal = principal
+        self.action = action
+        self.resource = resource
+        self.condition = condition
+        self.effect = effect
+
+    def to_dict(self):
+        d = { "Action" : self.action,
+              "Principal" : self.principal,
+              "Effect" : self.effect,
+              "Resource" : self.resource
+        }
+
+        if self.condition is not None:
+            d["Condition"] = self.condition
+
+        return d
+
+class Policy(object):
+    def __init__(self):
+        self.statements = []
+
+    def add_statement(self, s):
+        self.statements.append(s)
+        return self
+
+    def to_json(self):
+        policy_dict = {
+            "Version" : "2012-10-17",
+            "Statement":
+            [s.to_dict() for s in self.statements]
+        }
+
+        return json.dumps(policy_dict)
+
+def make_json_policy(action, resource, principal={"AWS": "*"}, effect="Allow", conditions=None):
+    """
+    Helper function to make single statement policies
+    """
+    s = Statement(action, resource, principal, effect=effect, condition=conditions)
+    p = Policy()
+    return p.add_statement(s).to_json()

s3tests_boto3/functional/rgw_interactive.py

@@ -0,0 +1,92 @@
+#!/usr/bin/python
+import boto3
+import os
+import random
+import string
+import itertools
+
+host = "localhost"
+port = 8000
+
+## AWS access key
+access_key = "0555b35654ad1656d804"
+
+## AWS secret key
+secret_key = "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
+
+prefix = "YOURNAMEHERE-1234-"
+
+endpoint_url = "http://%s:%d" % (host, port)
+
+client = boto3.client(service_name='s3',
+                    aws_access_key_id=access_key,
+                    aws_secret_access_key=secret_key,
+                    endpoint_url=endpoint_url,
+                    use_ssl=False,
+                    verify=False)
+
+s3 = boto3.resource('s3', 
+                    use_ssl=False,
+                    verify=False,
+                    endpoint_url=endpoint_url, 
+                    aws_access_key_id=access_key,
+                    aws_secret_access_key=secret_key)
+
+def choose_bucket_prefix(template, max_len=30):
+    """
+    Choose a prefix for our test buckets, so they're easy to identify.
+
+    Use template and feed it more and more random filler, until it's
+    as long as possible but still below max_len.
+    """
+    rand = ''.join(
+        random.choice(string.ascii_lowercase + string.digits)
+        for c in range(255)
+        )
+
+    while rand:
+        s = template.format(random=rand)
+        if len(s) <= max_len:
+            return s
+        rand = rand[:-1]
+
+    raise RuntimeError(
+        'Bucket prefix template is impossible to fulfill: {template!r}'.format(
+            template=template,
+            ),
+        )
+
+bucket_counter = itertools.count(1)
+
+def get_new_bucket_name():
+    """
+    Get a bucket name that probably does not exist.
+
+    We make every attempt to use a unique random prefix, so if a
+    bucket by this name happens to exist, it's ok if tests give
+    false negatives.
+    """
+    name = '{prefix}{num}'.format(
+        prefix=prefix,
+        num=next(bucket_counter),
+        )
+    return name
+
+def get_new_bucket(session=boto3, name=None, headers=None):
+    """
+    Get a bucket that exists and is empty.
+
+    Always recreates a bucket from scratch. This is useful to also
+    reset ACLs and such.
+    """
+    s3 = session.resource('s3', 
+                        use_ssl=False,
+                        verify=False,
+                        endpoint_url=endpoint_url, 
+                        aws_access_key_id=access_key,
+                        aws_secret_access_key=secret_key)
+    if name is None:
+        name = get_new_bucket_name()
+    bucket = s3.Bucket(name)
+    bucket_location = bucket.create()
+    return bucket

s3tests_boto3/functional/test_headers.py

@@ -0,0 +1,572 @@
+import boto3
+import pytest
+from botocore.exceptions import ClientError
+from email.utils import formatdate
+
+from .utils import assert_raises
+from .utils import _get_status_and_error_code
+from .utils import _get_status
+
+from . import (
+    configfile,
+    setup_teardown,
+    get_client,
+    get_v2_client,
+    get_new_bucket,
+    get_new_bucket_name,
+    )
+
+def _add_header_create_object(headers, client=None):
+    """ Create a new bucket, add an object w/header customizations
+    """
+    bucket_name = get_new_bucket()
+    if client == None:
+        client = get_client()
+    key_name = 'foo'
+
+    # pass in custom headers before PutObject call
+    add_headers = (lambda **kwargs: kwargs['params']['headers'].update(headers))
+    client.meta.events.register('before-call.s3.PutObject', add_headers)
+    client.put_object(Bucket=bucket_name, Key=key_name)
+
+    return bucket_name, key_name
+
+
+def _add_header_create_bad_object(headers, client=None):
+    """ Create a new bucket, add an object with a header. This should cause a failure 
+    """
+    bucket_name = get_new_bucket()
+    if client == None:
+        client = get_client()
+    key_name = 'foo'
+
+    # pass in custom headers before PutObject call
+    add_headers = (lambda **kwargs: kwargs['params']['headers'].update(headers))
+    client.meta.events.register('before-call.s3.PutObject', add_headers)
+    e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key=key_name, Body='bar')
+
+    return e
+
+
+def _remove_header_create_object(remove, client=None):
+    """ Create a new bucket, add an object without a header
+    """
+    bucket_name = get_new_bucket()
+    if client == None:
+        client = get_client()
+    key_name = 'foo'
+
+    # remove custom headers before PutObject call
+    def remove_header(**kwargs):
+        if (remove in kwargs['params']['headers']):
+            del kwargs['params']['headers'][remove]
+
+    client.meta.events.register('before-call.s3.PutObject', remove_header)
+    client.put_object(Bucket=bucket_name, Key=key_name)
+
+    return bucket_name, key_name
+
+def _remove_header_create_bad_object(remove, client=None):
+    """ Create a new bucket, add an object without a header. This should cause a failure
+    """
+    bucket_name = get_new_bucket()
+    if client == None:
+        client = get_client()
+    key_name = 'foo'
+
+    # remove custom headers before PutObject call
+    def remove_header(**kwargs):
+        if (remove in kwargs['params']['headers']):
+            del kwargs['params']['headers'][remove]
+
+    client.meta.events.register('before-call.s3.PutObject', remove_header)
+    e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key=key_name, Body='bar')
+
+    return e
+
+
+def _add_header_create_bucket(headers, client=None):
+    """ Create a new bucket, w/header customizations
+    """
+    bucket_name = get_new_bucket_name()
+    if client == None:
+        client = get_client()
+
+    # pass in custom headers before PutObject call
+    add_headers = (lambda **kwargs: kwargs['params']['headers'].update(headers))
+    client.meta.events.register('before-call.s3.CreateBucket', add_headers)
+    client.create_bucket(Bucket=bucket_name)
+
+    return bucket_name
+
+
+def _add_header_create_bad_bucket(headers=None, client=None):
+    """ Create a new bucket, w/header customizations that should cause a failure 
+    """
+    bucket_name = get_new_bucket_name()
+    if client == None:
+        client = get_client()
+
+    # pass in custom headers before PutObject call
+    add_headers = (lambda **kwargs: kwargs['params']['headers'].update(headers))
+    client.meta.events.register('before-call.s3.CreateBucket', add_headers)
+    e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name)
+
+    return e
+
+
+def _remove_header_create_bucket(remove, client=None):
+    """ Create a new bucket, without a header
+    """
+    bucket_name = get_new_bucket_name()
+    if client == None:
+        client = get_client()
+
+    # remove custom headers before PutObject call
+    def remove_header(**kwargs):
+        if (remove in kwargs['params']['headers']):
+            del kwargs['params']['headers'][remove]
+
+    client.meta.events.register('before-call.s3.CreateBucket', remove_header)
+    client.create_bucket(Bucket=bucket_name)
+
+    return bucket_name
+
+def _remove_header_create_bad_bucket(remove, client=None):
+    """ Create a new bucket, without a header. This should cause a failure
+    """
+    bucket_name = get_new_bucket_name()
+    if client == None:
+        client = get_client()
+
+    # remove custom headers before PutObject call
+    def remove_header(**kwargs):
+        if (remove in kwargs['params']['headers']):
+            del kwargs['params']['headers'][remove]
+
+    client.meta.events.register('before-call.s3.CreateBucket', remove_header)
+    e = assert_raises(ClientError, client.create_bucket, Bucket=bucket_name)
+
+    return e
+
+#
+# common tests
+#
+
+@pytest.mark.auth_common
+def test_object_create_bad_md5_invalid_short():
+    e = _add_header_create_bad_object({'Content-MD5':'YWJyYWNhZGFicmE='})
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 400
+    assert error_code == 'InvalidDigest'
+
+@pytest.mark.auth_common
+def test_object_create_bad_md5_bad():
+    e = _add_header_create_bad_object({'Content-MD5':'rL0Y20xC+Fzt72VPzMSk2A=='})
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 400
+    assert error_code == 'BadDigest'
+
+@pytest.mark.auth_common
+def test_object_create_bad_md5_empty():
+    e = _add_header_create_bad_object({'Content-MD5':''})
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 400
+    assert error_code == 'InvalidDigest'
+
+@pytest.mark.auth_common
+def test_object_create_bad_md5_none():
+    bucket_name, key_name = _remove_header_create_object('Content-MD5')
+    client = get_client()
+    client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+@pytest.mark.auth_common
+def test_object_create_bad_expect_mismatch():
+    bucket_name, key_name = _add_header_create_object({'Expect': 200})
+    client = get_client()
+    client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+@pytest.mark.auth_common
+def test_object_create_bad_expect_empty():
+    bucket_name, key_name = _add_header_create_object({'Expect': ''})
+    client = get_client()
+    client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+@pytest.mark.auth_common
+def test_object_create_bad_expect_none():
+    bucket_name, key_name = _remove_header_create_object('Expect')
+    client = get_client()
+    client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the content-length header
+@pytest.mark.fails_on_rgw
+def test_object_create_bad_contentlength_empty():
+    e = _add_header_create_bad_object({'Content-Length':''})
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 400
+
+@pytest.mark.auth_common
+@pytest.mark.fails_on_mod_proxy_fcgi
+def test_object_create_bad_contentlength_negative():
+    client = get_client()
+    bucket_name = get_new_bucket()
+    key_name = 'foo'
+    e = assert_raises(ClientError, client.put_object, Bucket=bucket_name, Key=key_name, ContentLength=-1)
+    status = _get_status(e.response)
+    assert status == 400
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the content-length header
+@pytest.mark.fails_on_rgw
+def test_object_create_bad_contentlength_none():
+    remove = 'Content-Length'
+    e = _remove_header_create_bad_object('Content-Length')
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 411
+    assert error_code == 'MissingContentLength'
+
+@pytest.mark.auth_common
+def test_object_create_bad_contenttype_invalid():
+    bucket_name, key_name = _add_header_create_object({'Content-Type': 'text/plain'})
+    client = get_client()
+    client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+@pytest.mark.auth_common
+def test_object_create_bad_contenttype_empty():
+    client = get_client()
+    key_name = 'foo'
+    bucket_name = get_new_bucket()
+    client.put_object(Bucket=bucket_name, Key=key_name, Body='bar', ContentType='')
+
+@pytest.mark.auth_common
+def test_object_create_bad_contenttype_none():
+    bucket_name = get_new_bucket()
+    key_name = 'foo'
+    client = get_client()
+    # as long as ContentType isn't specified in put_object it isn't going into the request
+    client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the authorization header
+@pytest.mark.fails_on_rgw
+def test_object_create_bad_authorization_empty():
+    e = _add_header_create_bad_object({'Authorization': ''})
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to pass both the 'Date' and 'X-Amz-Date' header during signing and not 'X-Amz-Date' before
+@pytest.mark.fails_on_rgw
+def test_object_create_date_and_amz_date():
+    date = formatdate(usegmt=True)
+    bucket_name, key_name = _add_header_create_object({'Date': date, 'X-Amz-Date': date})
+    client = get_client()
+    client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to pass both the 'Date' and 'X-Amz-Date' header during signing and not 'X-Amz-Date' before
+@pytest.mark.fails_on_rgw
+def test_object_create_amz_date_and_no_date():
+    date = formatdate(usegmt=True)
+    bucket_name, key_name = _add_header_create_object({'Date': '', 'X-Amz-Date': date})
+    client = get_client()
+    client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+# the teardown is really messed up here. check it out
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the authorization header
+@pytest.mark.fails_on_rgw
+def test_object_create_bad_authorization_none():
+    e = _remove_header_create_bad_object('Authorization')
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the content-length header
+@pytest.mark.fails_on_rgw
+def test_bucket_create_contentlength_none():
+    remove = 'Content-Length'
+    _remove_header_create_bucket(remove)
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the content-length header
+@pytest.mark.fails_on_rgw
+def test_object_acl_create_contentlength_none():
+    bucket_name = get_new_bucket()
+    client = get_client()
+    client.put_object(Bucket=bucket_name, Key='foo', Body='bar')
+
+    remove = 'Content-Length'
+    def remove_header(**kwargs):
+        if (remove in kwargs['params']['headers']):
+            del kwargs['params']['headers'][remove]
+
+    client.meta.events.register('before-call.s3.PutObjectAcl', remove_header)
+    client.put_object_acl(Bucket=bucket_name, Key='foo', ACL='public-read')
+
+@pytest.mark.auth_common
+def test_bucket_put_bad_canned_acl():
+    bucket_name = get_new_bucket()
+    client = get_client()
+
+    headers = {'x-amz-acl': 'public-ready'}
+    add_headers = (lambda **kwargs: kwargs['params']['headers'].update(headers))
+    client.meta.events.register('before-call.s3.PutBucketAcl', add_headers)
+
+    e = assert_raises(ClientError, client.put_bucket_acl, Bucket=bucket_name, ACL='public-read')
+    status = _get_status(e.response)
+    assert status == 400
+
+@pytest.mark.auth_common
+def test_bucket_create_bad_expect_mismatch():
+    bucket_name = get_new_bucket_name()
+    client = get_client()
+
+    headers = {'Expect': 200}
+    add_headers = (lambda **kwargs: kwargs['params']['headers'].update(headers))
+    client.meta.events.register('before-call.s3.CreateBucket', add_headers)
+    client.create_bucket(Bucket=bucket_name)
+
+@pytest.mark.auth_common
+def test_bucket_create_bad_expect_empty():
+    headers = {'Expect': ''}
+    _add_header_create_bucket(headers)
+
+@pytest.mark.auth_common
+# TODO: The request isn't even making it to the RGW past the frontend
+# This test had 'fails_on_rgw' before the move to boto3
+@pytest.mark.fails_on_rgw
+def test_bucket_create_bad_contentlength_empty():
+    headers = {'Content-Length': ''}
+    e = _add_header_create_bad_bucket(headers)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 400
+
+@pytest.mark.auth_common
+@pytest.mark.fails_on_mod_proxy_fcgi
+def test_bucket_create_bad_contentlength_negative():
+    headers = {'Content-Length': '-1'}
+    e = _add_header_create_bad_bucket(headers)
+    status = _get_status(e.response)
+    assert status == 400
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the content-length header
+@pytest.mark.fails_on_rgw
+def test_bucket_create_bad_contentlength_none():
+    remove = 'Content-Length'
+    _remove_header_create_bucket(remove)
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to manipulate the authorization header
+@pytest.mark.fails_on_rgw
+def test_bucket_create_bad_authorization_empty():
+    headers = {'Authorization': ''}
+    e = _add_header_create_bad_bucket(headers)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+@pytest.mark.auth_common
+# TODO: remove 'fails_on_rgw' and once we have learned how to manipulate the authorization header
+@pytest.mark.fails_on_rgw
+def test_bucket_create_bad_authorization_none():
+    e = _remove_header_create_bad_bucket('Authorization')
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+@pytest.mark.auth_aws2
+def test_object_create_bad_md5_invalid_garbage_aws2():
+    v2_client = get_v2_client()
+    headers = {'Content-MD5': 'AWS HAHAHA'}
+    e = _add_header_create_bad_object(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 400
+    assert error_code == 'InvalidDigest'
+
+@pytest.mark.auth_aws2
+# TODO: remove 'fails_on_rgw' and once we have learned how to manipulate the Content-Length header
+@pytest.mark.fails_on_rgw
+def test_object_create_bad_contentlength_mismatch_below_aws2():
+    v2_client = get_v2_client()
+    content = 'bar'
+    length = len(content) - 1
+    headers = {'Content-Length': str(length)}
+    e = _add_header_create_bad_object(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 400
+    assert error_code == 'BadDigest'
+
+@pytest.mark.auth_aws2
+# TODO: remove 'fails_on_rgw' and once we have learned how to manipulate the authorization header
+@pytest.mark.fails_on_rgw
+def test_object_create_bad_authorization_incorrect_aws2():
+    v2_client = get_v2_client()
+    headers = {'Authorization': 'AWS AKIAIGR7ZNNBHC5BKSUB:FWeDfwojDSdS2Ztmpfeubhd9isU='}
+    e = _add_header_create_bad_object(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'InvalidDigest'
+
+@pytest.mark.auth_aws2
+# TODO: remove 'fails_on_rgw' and once we have learned how to manipulate the authorization header
+@pytest.mark.fails_on_rgw
+def test_object_create_bad_authorization_invalid_aws2():
+    v2_client = get_v2_client()
+    headers = {'Authorization': 'AWS HAHAHA'}
+    e = _add_header_create_bad_object(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 400
+    assert error_code == 'InvalidArgument'
+
+@pytest.mark.auth_aws2
+def test_object_create_bad_ua_empty_aws2():
+    v2_client = get_v2_client()
+    headers = {'User-Agent': ''}
+    bucket_name, key_name = _add_header_create_object(headers, v2_client)
+    v2_client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+@pytest.mark.auth_aws2
+def test_object_create_bad_ua_none_aws2():
+    v2_client = get_v2_client()
+    remove = 'User-Agent'
+    bucket_name, key_name = _remove_header_create_object(remove, v2_client)
+    v2_client.put_object(Bucket=bucket_name, Key=key_name, Body='bar')
+
+@pytest.mark.auth_aws2
+def test_object_create_bad_date_invalid_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': 'Bad Date'}
+    e = _add_header_create_bad_object(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+@pytest.mark.auth_aws2
+def test_object_create_bad_date_empty_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': ''}
+    e = _add_header_create_bad_object(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+@pytest.mark.auth_aws2
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the date header
+@pytest.mark.fails_on_rgw
+def test_object_create_bad_date_none_aws2():
+    v2_client = get_v2_client()
+    remove = 'x-amz-date'
+    e = _remove_header_create_bad_object(remove, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+@pytest.mark.auth_aws2
+def test_object_create_bad_date_before_today_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': 'Tue, 07 Jul 2010 21:53:04 GMT'}
+    e = _add_header_create_bad_object(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'RequestTimeTooSkewed'
+
+@pytest.mark.auth_aws2
+def test_object_create_bad_date_before_epoch_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': 'Tue, 07 Jul 1950 21:53:04 GMT'}
+    e = _add_header_create_bad_object(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+@pytest.mark.auth_aws2
+def test_object_create_bad_date_after_end_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': 'Tue, 07 Jul 9999 21:53:04 GMT'}
+    e = _add_header_create_bad_object(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'RequestTimeTooSkewed'
+
+@pytest.mark.auth_aws2
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the date header
+@pytest.mark.fails_on_rgw
+def test_bucket_create_bad_authorization_invalid_aws2():
+    v2_client = get_v2_client()
+    headers = {'Authorization': 'AWS HAHAHA'}
+    e = _add_header_create_bad_bucket(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 400
+    assert error_code == 'InvalidArgument'
+
+@pytest.mark.auth_aws2
+def test_bucket_create_bad_ua_empty_aws2():
+    v2_client = get_v2_client()
+    headers = {'User-Agent': ''}
+    _add_header_create_bucket(headers, v2_client)
+
+@pytest.mark.auth_aws2
+def test_bucket_create_bad_ua_none_aws2():
+    v2_client = get_v2_client()
+    remove = 'User-Agent'
+    _remove_header_create_bucket(remove, v2_client)
+
+@pytest.mark.auth_aws2
+def test_bucket_create_bad_date_invalid_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': 'Bad Date'}
+    e = _add_header_create_bad_bucket(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+@pytest.mark.auth_aws2
+def test_bucket_create_bad_date_empty_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': ''}
+    e = _add_header_create_bad_bucket(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+@pytest.mark.auth_aws2
+# TODO: remove 'fails_on_rgw' and once we have learned how to remove the date header
+@pytest.mark.fails_on_rgw
+def test_bucket_create_bad_date_none_aws2():
+    v2_client = get_v2_client()
+    remove = 'x-amz-date'
+    e = _remove_header_create_bad_bucket(remove, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+@pytest.mark.auth_aws2
+def test_bucket_create_bad_date_before_today_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': 'Tue, 07 Jul 2010 21:53:04 GMT'}
+    e = _add_header_create_bad_bucket(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'RequestTimeTooSkewed'
+
+@pytest.mark.auth_aws2
+def test_bucket_create_bad_date_after_today_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': 'Tue, 07 Jul 2030 21:53:04 GMT'}
+    e = _add_header_create_bad_bucket(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'RequestTimeTooSkewed'
+
+@pytest.mark.auth_aws2
+def test_bucket_create_bad_date_before_epoch_aws2():
+    v2_client = get_v2_client()
+    headers = {'x-amz-date': 'Tue, 07 Jul 1950 21:53:04 GMT'}
+    e = _add_header_create_bad_bucket(headers, v2_client)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'

s3tests_boto3/functional/test_sns.py

@@ -0,0 +1,159 @@
+import json
+import pytest
+from botocore.exceptions import ClientError
+from . import (
+    configfile,
+    get_iam_root_client,
+    get_iam_alt_root_client,
+    get_new_bucket_name,
+    get_prefix,
+    nuke_prefixed_buckets,
+)
+from .iam import iam_root, iam_alt_root
+from .utils import assert_raises, _get_status_and_error_code
+
+def get_new_topic_name():
+    return get_new_bucket_name()
+
+def nuke_topics(client, prefix):
+    p = client.get_paginator('list_topics')
+    for response in p.paginate():
+        for topic in response['Topics']:
+            arn = topic['TopicArn']
+            if prefix not in arn:
+                pass
+            try:
+                client.delete_topic(TopicArn=arn)
+            except:
+                pass
+
+@pytest.fixture
+def sns(iam_root):
+    client = get_iam_root_client(service_name='sns')
+    yield client
+    nuke_topics(client, get_prefix())
+
+@pytest.fixture
+def sns_alt(iam_alt_root):
+    client = get_iam_alt_root_client(service_name='sns')
+    yield client
+    nuke_topics(client, get_prefix())
+
+@pytest.fixture
+def s3(iam_root):
+    client = get_iam_root_client(service_name='s3')
+    yield client
+    nuke_prefixed_buckets(get_prefix(), client)
+
+@pytest.fixture
+def s3_alt(iam_alt_root):
+    client = get_iam_alt_root_client(service_name='s3')
+    yield client
+    nuke_prefixed_buckets(get_prefix(), client)
+
+
+@pytest.mark.iam_account
+@pytest.mark.sns
+def test_account_topic(sns):
+    name = get_new_topic_name()
+
+    response = sns.create_topic(Name=name)
+    arn = response['TopicArn']
+    assert arn.startswith('arn:aws:sns:')
+    assert arn.endswith(f':{name}')
+
+    response = sns.list_topics()
+    assert arn in [p['TopicArn'] for p in response['Topics']]
+
+    sns.set_topic_attributes(TopicArn=arn, AttributeName='Policy', AttributeValue='')
+
+    response = sns.get_topic_attributes(TopicArn=arn)
+    assert 'Attributes' in response
+
+    sns.delete_topic(TopicArn=arn)
+
+    response = sns.list_topics()
+    assert arn not in [p['TopicArn'] for p in response['Topics']]
+
+    with pytest.raises(sns.exceptions.NotFoundException):
+        sns.get_topic_attributes(TopicArn=arn)
+    sns.delete_topic(TopicArn=arn)
+
+@pytest.mark.iam_account
+@pytest.mark.sns
+def test_cross_account_topic(sns, sns_alt):
+    name = get_new_topic_name()
+    arn = sns.create_topic(Name=name)['TopicArn']
+
+    # not visible to any alt user apis
+    with pytest.raises(sns.exceptions.NotFoundException):
+        sns_alt.get_topic_attributes(TopicArn=arn)
+    with pytest.raises(sns.exceptions.NotFoundException):
+        sns_alt.set_topic_attributes(TopicArn=arn, AttributeName='Policy', AttributeValue='')
+
+    # delete returns success
+    sns_alt.delete_topic(TopicArn=arn)
+
+    response = sns_alt.list_topics()
+    assert arn not in [p['TopicArn'] for p in response['Topics']]
+
+@pytest.mark.iam_account
+@pytest.mark.sns
+def test_account_topic_publish(sns, s3):
+    name = get_new_topic_name()
+
+    response = sns.create_topic(Name=name)
+    topic_arn = response['TopicArn']
+
+    bucket = get_new_bucket_name()
+    s3.create_bucket(Bucket=bucket)
+
+    config = {'TopicConfigurations': [{
+        'Id': 'id',
+        'TopicArn': topic_arn,
+        'Events': [ 's3:ObjectCreated:*' ],
+        }]}
+    s3.put_bucket_notification_configuration(
+            Bucket=bucket, NotificationConfiguration=config)
+
+@pytest.mark.iam_account
+@pytest.mark.iam_cross_account
+@pytest.mark.sns
+def test_cross_account_topic_publish(sns, s3_alt, iam_alt_root):
+    name = get_new_topic_name()
+
+    response = sns.create_topic(Name=name)
+    topic_arn = response['TopicArn']
+
+    bucket = get_new_bucket_name()
+    s3_alt.create_bucket(Bucket=bucket)
+
+    config = {'TopicConfigurations': [{
+        'Id': 'id',
+        'TopicArn': topic_arn,
+        'Events': [ 's3:ObjectCreated:*' ],
+        }]}
+
+    # expect AccessDenies because no resource policy allows cross-account access
+    e = assert_raises(ClientError, s3_alt.put_bucket_notification_configuration,
+                      Bucket=bucket, NotificationConfiguration=config)
+    status, error_code = _get_status_and_error_code(e.response)
+    assert status == 403
+    assert error_code == 'AccessDenied'
+
+    # add topic policy to allow the alt user
+    alt_principal = iam_alt_root.get_user()['User']['Arn']
+    policy = json.dumps({
+        'Version': '2012-10-17',
+        'Statement': [{
+            'Effect': 'Allow',
+            'Principal': {'AWS': alt_principal},
+            'Action': 'sns:Publish',
+            'Resource': topic_arn
+            }]
+        })
+    sns.set_topic_attributes(TopicArn=topic_arn, AttributeName='Policy',
+                             AttributeValue=policy)
+
+    s3_alt.put_bucket_notification_configuration(
+            Bucket=bucket, NotificationConfiguration=config)

s3tests_boto3/functional/test_utils.py

@@ -0,0 +1,9 @@
+from . import utils
+
+def test_generate():
+    FIVE_MB = 5 * 1024 * 1024
+    assert len(''.join(utils.generate_random(0))) == 0
+    assert len(''.join(utils.generate_random(1))) == 1
+    assert len(''.join(utils.generate_random(FIVE_MB - 1))) == FIVE_MB - 1
+    assert len(''.join(utils.generate_random(FIVE_MB))) == FIVE_MB
+    assert len(''.join(utils.generate_random(FIVE_MB + 1))) == FIVE_MB + 1

s3tests_boto3/functional/utils.py

@@ -0,0 +1,47 @@
+import random
+import requests
+import string
+import time
+
+def assert_raises(excClass, callableObj, *args, **kwargs):
+    """
+    Like unittest.TestCase.assertRaises, but returns the exception.
+    """
+    try:
+        callableObj(*args, **kwargs)
+    except excClass as e:
+        return e
+    else:
+        if hasattr(excClass, '__name__'):
+            excName = excClass.__name__
+        else:
+            excName = str(excClass)
+        raise AssertionError("%s not raised" % excName)
+
+def generate_random(size, part_size=5*1024*1024):
+    """
+    Generate the specified number random data.
+    (actually each MB is a repetition of the first KB)
+    """
+    chunk = 1024
+    allowed = string.ascii_letters
+    for x in range(0, size, part_size):
+        strpart = ''.join([allowed[random.randint(0, len(allowed) - 1)] for _ in range(chunk)])
+        s = ''
+        left = size - x
+        this_part_size = min(left, part_size)
+        for y in range(this_part_size // chunk):
+            s = s + strpart
+        s = s + strpart[:(this_part_size % chunk)]
+        yield s
+        if (x == size):
+            return
+
+def _get_status(response):
+    status = response['ResponseMetadata']['HTTPStatusCode']
+    return status
+
+def _get_status_and_error_code(response):
+    status = response['ResponseMetadata']['HTTPStatusCode']
+    error_code = response['Error']['Code']
+    return status, error_code

setup.py

@@ -14,20 +14,10 @@ setup(
 
     install_requires=[
         'boto >=2.0b4',
+        'boto3 >=1.0.0',
         'PyYAML',
-        'bunch >=1.0.0',
+        'munch >=2.0.0',
         'gevent >=1.0',
         'isodate >=0.4.4',
         ],
-
-    entry_points={
-        'console_scripts': [
-            's3tests-generate-objects = s3tests.generate_objects:main',
-            's3tests-test-readwrite = s3tests.readwrite:main',
-            's3tests-test-roundtrip = s3tests.roundtrip:main',
-            's3tests-fuzz-headers = s3tests.fuzz.headers:main',
-            's3tests-analysis-rwstats = s3tests.analysis.rwstats:main',
-            ],
-        },
-
     )

siege.conf

@@ -1,382 +0,0 @@
-# Updated by Siege 2.69, May-24-2010
-# Copyright 2000-2007 by Jeffrey Fulmer, et al.
-# 
-# Siege configuration file -- edit as necessary
-# For more information about configuring and running
-# this program, visit: http://www.joedog.org/
-
-#
-# Variable declarations. You can set variables here 
-# for use in the directives below. Example:
-# PROXY = proxy.joedog.org
-# Reference variables inside ${} or $(), example: 
-# proxy-host = ${PROXY} 
-# You can also reference ENVIRONMENT variables without
-# actually declaring them, example:
-# logfile = $(HOME)/var/siege.log
-
-#
-# Signify verbose mode, true turns on verbose output
-# ex: verbose = true|false
-#
-verbose = true
-
-#
-# CSV Verbose format: with this option, you can choose 
-# to format verbose output in traditional siege format 
-# or comma separated format. The latter will allow you 
-# to redirect output to a file for import into a spread
-# sheet, i.e., siege > file.csv 
-# ex: csv = true|false (default false)
-#
-csv = true
-
-#
-# Full URL verbose format: By default siege displays
-# the URL path and not the full URL. With this option, 
-# you # can instruct siege to show the complete URL.
-# ex: fullurl = true|false (default false)
-# 
-# fullurl = true
-
-#
-# Display id: in verbose mode, display the siege user
-# id associated with the HTTP transaction information
-# ex: display-id = true|false
-#
-# display-id = 
-
-#
-# Show logfile location.  By default, siege displays the
-# logfile location at the end of every run when logging
-# You can turn this message off with this directive.
-# ex: show-logfile = false
-#
-show-logfile = true
-
-#
-# Default logging status, true turns logging on.
-# ex: logging = true|false
-#
-logging = true
-
-#
-# Logfile, the default siege logfile is $PREFIX/var/siege.log
-# This directive allows you to choose an alternative log file.
-# Environment variables may be used as shown in the examples:
-# ex: logfile = /home/jeff/var/log/siege.log
-#     logfile = ${HOME}/var/log/siege.log
-#     logfile = ${LOGFILE}
-#
-logfile = ./siege.log
-
-#
-# HTTP protocol.  Options HTTP/1.1 and HTTP/1.0.
-# Some webservers have broken implementation of the
-# 1.1 protocol which skews throughput evaluations.
-# If you notice some siege clients hanging for
-# extended periods of time, change this to HTTP/1.0
-# ex: protocol = HTTP/1.1
-#     protocol = HTTP/1.0
-#
-protocol = HTTP/1.1
-
-#
-# Chunked encoding is required by HTTP/1.1 protocol
-# but siege allows you to turn it off as desired.
-# 
-# ex: chunked = true 
-# 
-chunked = true
-
-# 
-# Cache revalidation.
-# Siege supports cache revalidation for both ETag and 
-# Last-modified headers. If a copy is still fresh, the 
-# server responds with 304. 
-# HTTP/1.1 200   0.00 secs:    2326 bytes ==> /apache_pb.gif
-# HTTP/1.1 304   0.00 secs:       0 bytes ==> /apache_pb.gif
-# HTTP/1.1 304   0.00 secs:       0 bytes ==> /apache_pb.gif
-# 
-# ex: cache = true
-#
-cache = false
-
-#
-# Connection directive. Options "close" and "keep-alive"
-# Starting with release 2.57b3, siege implements persistent 
-# connections in accordance to RFC 2068 using both chunked
-# encoding and content-length directives to determine the 
-# page size. To run siege with persistent connections set
-# the connection directive to keep-alive. (Default close)
-# CAUTION: use the keep-alive directive with care.
-# DOUBLE CAUTION: this directive does not work well on HPUX
-# TRIPLE CAUTION: don't use keep-alives until further notice
-# ex: connection = close
-#     connection = keep-alive
-#
-connection = close
-
-#
-# Default number of simulated  concurrent users
-# ex: concurrent = 25
-#
-concurrent = 15
-
-#
-# Default duration of the siege.  The right hand argument has
-# a modifier which specifies the time units, H=hours, M=minutes,
-# and S=seconds. If a modifier is not specified, then minutes 
-# are assumed.
-# ex: time = 50M
-#
-# time =
-
-#
-# Repetitions.  The length of siege may be specified in client
-# reps rather then a time duration.  Instead of specifying a time
-# span, you can tell each siege instance to hit the server X number
-# of times.  So if you chose 'reps = 20' and you've selected 10 
-# concurrent users, then siege will hit the server 200 times.
-# ex: reps = 20
-#
-# reps = 
-
-#
-# Default URLs file, set at configuration time, the default
-# file is PREFIX/etc/urls.txt.  So if you configured siege
-# with --prefix=/usr/local then the urls.txt file is installed
-# int /usr/local/etc/urls.txt.  Use the "file = " directive to
-# configure an alternative URLs file. You may use environment
-# variables as shown in the examples below:
-# ex: file = /export/home/jdfulmer/MYURLS.txt
-#     file = $HOME/etc/urls.txt
-#     file = $URLSFILE
-#
-file = ./urls.txt
-
-#
-# Default URL, this is a single URL that you want to test. This
-# is usually set at the command line with the -u option.  When
-# used, this option overrides the urls.txt (-f FILE/--file=FILE)
-# option. You will HAVE to comment this out for in order to use
-# the urls.txt file option.
-# ex: url = https://shemp.whoohoo.com/docs/index.jsp
-#
-# url =
-
-#
-# Default delay value, see the siege(1) man page.  
-# This value is used for load testing, it is not used 
-# for benchmarking.
-# ex: delay = 3
-#
-delay = 1
-
-#
-# Connection timeout value. Set the value in seconds for 
-# socket connection timeouts. The default value is 30 seconds.
-# ex: timeout = 30
-#
-# timeout = 
-
-#
-# Session expiration: This directive allows you to delete all
-# cookies after you pass through the URLs. This means siege will
-# grab a new session with each run through its URLs. The default
-# value is false.
-# ex: expire-session = true
-#
-# expire-session = 
-
-#
-# Failures: This is the number of total connection failures allowed
-# before siege aborts. Connection failures (timeouts, socket failures, 
-# etc.) are combined with 400 and 500 level errors in the final stats, 
-# but those errors do not count against the abort total.  If you set 
-# this total to 10, then siege will abort after ten socket timeouts, 
-# but it will NOT abort after ten 404s. This is designed to prevent 
-# a run-away mess on an unattended siege. The default value is 1024
-# ex: failures = 50
-#
-# failures = 
-
-#
-# Internet simulation. If true, siege clients will hit
-# the URLs in the urls.txt file randomly, thereby simulating
-# internet usage.  If false, siege will run through the 
-# urls.txt file in order from first to last and back again.
-# ex: internet = true
-#
-internet = false
-
-#
-# Default benchmarking value, If true, there is NO delay
-# between server requests, siege runs as fast as the web
-# server and the network will let it.  Set this to false 
-# for load testing.
-# ex: benchmark = true
-# 
-benchmark = false
-
-#
-# Set the siege User-Agent to identify yourself at the
-# host, the default is: JoeDog/1.00 [en] (X11; I; Siege #.##) 
-# But that wreaks of corporate techno speak.  Feel free
-# to make it more interesting :-)  Since Limey is recovering
-# from minor surgery as I write this, I'll dedicate the 
-# example to him...
-# ex: user-agent = Limey The Bulldog
-# 
-# user-agent =
-
-#
-# Accept-encoding. This option allows you to specify 
-# acceptable encodings returned by the server. Use this
-# directive to turn on compression. By default we accept
-# gzip compression.
-#
-# ex: accept-encoding = *
-#     accept-encoding = gzip
-#     accept-encoding = compress;q=0.5;gzip;q=1
-accept-encoding = gzip
-
-#
-# TURN OFF THAT ANNOYING SPINNER!
-# Siege spawns a thread and runs a spinner to entertain you
-# as it collects and computes its stats. If you don't like 
-# this feature, you may turn it off here.
-# ex: spinner = false 
-#
-spinner = true
-
-#
-# WWW-Authenticate login. When siege hits a webpage
-# that requires basic authentication, it will search its 
-# logins for authentication which matches the specific realm 
-# requested by the server. If it finds a match, it will send 
-# that login information. If it fails to match the realm, it 
-# will send the default login information. (Default is "all"). 
-# You may configure siege with several logins as long as no
-# two realms match. The format for logins is:
-# username:password[:realm] where "realm" is optional.
-# If you do not supply a realm, then it will default to "all"
-# ex: login = jdfulmer:topsecret:Admin
-#     login = jeff:supersecret
-#
-# login = 
-
-#
-# WWW-Authenticate username and password.  When siege
-# hits a webpage that requires authentication, it will
-# send this user name and password to the server.  Note
-# this is NOT form based authentication.  You will have
-# to construct URLs for that.
-# ex: username = jdfulmer
-#     password = whoohoo
-#
-# username = 
-# password =
-
-#
-# ssl-cert
-# This optional feature allows you to specify a path to a client
-# certificate. It is not neccessary to specify a certificate in
-# order to use https. If you don't know why you would want one, 
-# then you probably don't need this feature.  Use openssl to 
-# generate a certificate and key with the following command:
-#   $ openssl req -nodes -new -days 365 -newkey rsa:1024 \
-#                 -keyout key.pem -out cert.pem
-# Specify a path to cert.pem as follows:
-# ex: ssl-cert = /home/jeff/.certs/cert.pem
-#
-# ssl-cert = 
- 
-#
-# ssl-key
-# Use this option to specify the key you generated with the command
-# above. ex: ssl-key = /home/jeff/.certs/key.pem 
-# You may actually skip this option and combine both your cert and 
-# your key in a single file:
-#   $ cat key.pem > client.pem
-#   $ cat cert.pem >> client.pem
-# Now set the path for ssl-cert:
-# ex: ssl-cert = /home/jeff/.certs/client.pem
-# (in this scenario, you comment out ssl-key)
-#
-# ssl-key = 
-
-#
-# ssl-timeout 
-# This option sets a connection timeout for the ssl library
-# ex: ssl-timeout = 30
-# 
-# ssl-timeout = 
-
-#
-# ssl-ciphers
-# You can use this feature to select a specific ssl cipher
-# for HTTPs. To view the ones available with your library run
-# the following command: openssl ciphers
-# ex: ssl-ciphers = EXP-RC4-MD5
-#
-# ssl-ciphers = 
-
-#
-# Login URL. This is the first URL to be hit by every siege
-# client. This feature was designed to allow you to login to 
-# a server and establish a session. It will only be hit once
-# so if you need to hit this URL more then once, make sure it
-# also appears in your urls.txt file.
-#
-# ex: login-url = http://eos.haha.com/login.jsp POST name=jeff&pass=foo
-#
-# login-url = 
-
-#
-# Proxy protocol. This option allows you to select a proxy
-# server stress testing. The proxy will request the URL(s) 
-# specified by -u"my.url.org" OR from the urls.txt file.
-# 
-# ex: proxy-host = proxy.whoohoo.org
-#     proxy-port = 8080
-#
-# proxy-host = 
-# proxy-port = 
-
-#
-# Proxy-Authenticate. When scout hits a proxy server which
-# requires username and password authentication, it will this
-# username and password to the server. The format is username,
-# password and optional realm each separated by a colon. You
-# may enter more than one proxy-login as long as each one has
-# a different realm. If you do not enter a realm, then scout
-# will send that login information to all proxy challenges. If
-# you have more than one proxy-login, then scout will attempt
-# to match the login to the realm.
-# ex: proxy-login: jeff:secret:corporate
-#     proxy-login: jeff:whoohoo
-#
-# proxy-login = 
-
-#
-# Redirection support.  This option allows to to control
-# whether a Location: hint will be followed.  Most users
-# will want to follow redirection information, but sometimes
-# it's desired to just get the Location information.
-#
-# ex: follow-location = false
-#
-# follow-location = 
-
-# Zero-length data.  siege can be configured to disregard
-# results in which zero bytes are read after the headers.
-# Alternatively, such results can be counted in the final
-# tally of outcomes.
-#
-# ex: zero-data-ok = false 
-#
-# zero-data-ok =
-
-#
-# end of siegerc

tox.ini

@@ -0,0 +1,9 @@
+[tox]
+envlist = py
+
+[testenv]
+deps = -rrequirements.txt
+passenv =
+  S3TEST_CONF
+  S3_USE_SIGV4
+commands = pytest {posargs}