dkirillov/s3-tests
1
0
Fork 0
forked from TrueCloudLab/s3-tests
Code Pull requests Activity
711 commits 127 branches 0 tags 8.9 MiB
Commit graph

711 commits

This branch
This branch
All branches
Author SHA1 Message Date
Tommi Virtanen
f3b5ed6ccc Say "worker_id" instead of name, as I keep confusing it with filenames. 2011-07-20 11:15:06 -07:00
Tommi Virtanen
1eb38e09e4 Link worker greenlets to parent, so we abort if they have an unhandled exception. 2011-07-20 11:06:30 -07:00
Tommi Virtanen
9ecaf63fbe Code layout cleanup. 2011-07-20 11:06:30 -07:00
Tommi Virtanen
deb8be9957 Make readwrite test put objects one by one, so it can report keys. 2011-07-20 11:06:30 -07:00
Tommi Virtanen
9e707b897d Remove the concept of batches from readwrite test. 
That's an internal implementation detail caused by
the generate_objects api.
 2011-07-20 11:06:30 -07:00
Tommi Virtanen
4cd4131dd7 Don't eat exceptions, that's a bad habit. 2011-07-20 11:06:30 -07:00
Tommi Virtanen
648ddbb761 Log readwrite test results in yaml. 
This temporarily removes the statistics calculation; that
belongs somewhere else anyway, so it can be run on aggregated
results etc.
 2011-07-20 11:06:30 -07:00
Tommi Virtanen
ed966af0a7 Use more idiomatic Python. 2011-07-20 11:06:30 -07:00
Tommi Virtanen
ad444b33ee Make queue and name arguments not optional; they're always passed in anyway. 2011-07-20 11:06:30 -07:00
Tommi Virtanen
71cfd4956c Move end test logic out of individual workers. 
Parent waits for the duration anyway, just make it unconditionally
kill the workers. They already had timeouts aborting them; they may
be in the middle of an operation, but we really don't care, time's up.

This also avoids oddities with the queue where there may be items
after the StopIteration.

This also avoids workers potentially not exiting within the 1-second
timeout the old code had.
 2011-07-20 11:06:30 -07:00
Tommi Virtanen
dff6b568e4 Use gevent Groups to manage groups of workers. 
This mostly adds convenience functions, but it's nice to have.
 2011-07-20 11:06:30 -07:00
Stephon Striplin
510b6e91d4 refactor and fix some acl_grant tests 2011-07-19 11:16:11 -07:00
Tommi Virtanen
fe749adebf pep8 cleanup. 2011-07-19 09:09:51 -07:00
Tommi Virtanen
e0575e4efe Pyflakes cleanup. 2011-07-19 09:05:05 -07:00
Tommi Virtanen
c7bb78b666 Remove dead code. 2011-07-19 09:04:20 -07:00
Tommi Virtanen
5c25c66649 Merge branch 'roundtrip' 2011-07-19 09:00:44 -07:00
Tommi Virtanen
56ff0f6374 Revert "Add assert to RandomContentFile, ensure size >= digest_size" 
This reverts commit fb8f110e69.

realistic.files was meant to work just fine with short files. Commit
91c4f88860 has a fix for the verifying
side, to support short files.

Conflicts:

	s3tests/realistic.py
 2011-07-19 08:56:17 -07:00
Wesley Spikes
0297d61e40 Swap stdout/stderr -- debug messages then go to stderr, and yaml data to stdout 2011-07-18 17:04:36 -07:00
Wesley Spikes
ce6c57841a Now that we no longer have the need to retry, get rid of that logic. 
Also fix a typo -- s/neads_first_read/needs_first_read/
 2011-07-18 16:15:40 -07:00
Wesley Spikes
c154d98b96 Resolve the random AssertionError 
By doing a copy-new on the key, it seems to work around the random
AssertionError's I've been seeing. Here's hoping this fully
resolves it.
 2011-07-18 16:15:40 -07:00
Wesley Spikes
44cb2f256e Adding unlimited=bool to files_varied -- if set to true, it will never issue StopIteration 2011-07-18 16:15:40 -07:00
Wesley Spikes
43b2f8695d Cleanup output just a little bit more. 2011-07-18 16:15:40 -07:00
Wesley Spikes
841b098619 Refactor/cleanup TransferResults class. Also cleans the output to be in the format TV seems to want. 2011-07-18 16:15:40 -07:00
Wesley Spikes
3c07a4a363 Update roundtrip._main -- while loop should auto-replenish all pools, 
and not ones so named.
 2011-07-18 16:15:40 -07:00
Wesley Spikes
fb8f110e69 Add assert to RandomContentFile, ensure size >= digest_size 
It clearly will not work to have the size smaller than the digest
size, so we shouldn't allow it.
 2011-07-18 16:15:40 -07:00
Wesley Spikes
d0bb148dea Rename _call_doit to _real_run -- it's a bit more clear that way, IMHO 2011-07-18 16:15:40 -07:00
Stephon Striplin
5136608e76 fix test_bucket_acl_grant_nonexist_user 
* Change user_id to avoid name conflict
* Check for error_code
 2011-07-15 14:57:46 -07:00
Stephon Striplin
03a80e6121 add acl grant userid tests 
* Refactor some acl grant user_id code and run through the full suite of
  acls
 2011-07-15 14:34:08 -07:00
Wesley Spikes
653c2b2200 Adding new testing tool, roundtrip 2011-07-15 12:06:02 -07:00
Wesley Spikes
6afc7fdcdb Adding last_read to RandomContentFile and first_write to FileVerifier, so we can measure latency. 2011-07-15 12:05:02 -07:00
Wesley Spikes
12716d854b Adding generator files_varied to s3tests.realistic 
Given a tuple of tuples, construct several files() generators, and
yield from those randomly. Randomness is weighted based on the number
of files remaining in each group.
 2011-07-15 12:03:23 -07:00
Wesley Spikes
4959c5e1a4 Work-around for RGW failing to send IsTruncated on listing all keys. 
Without this, it fails to delete the bucket when there's more than 1000 keys.
 2011-07-15 11:47:57 -07:00
Wesley Spikes
b6e0287810 Add context, get_next_key, and fill_pools to s3tests.common 
context is a generic Bunch that we can store items in to be shared
across modules, without invoking the wrath of the globals -- yuck!

get_next_key will generate a sequentially numbered key for us to use

fill_pools will take a variable list of gevent.pool.Pool objects and
top them all off
 2011-07-15 11:45:38 -07:00
Wesley Spikes
298bc99c5d Packagify s3tests.common so we can add more magic into it later. 2011-07-15 11:44:05 -07:00
Wesley Spikes
e4effe5479 realistic.names shouldn't return a 0-length filename 2011-07-15 11:08:38 -07:00
Yehuda Sadeh
3303b0f33b fix 100-continue test to support https 2011-07-14 16:26:46 -07:00
Steven Berler
c312832c70 use time.time() instead of time.clock() 
We want to measure actual time, not cpu time.
 2011-07-14 11:40:26 -07:00
Kyle Marsh
3d13883f62 dhq-qa: remove query auth from generated urls 
Object generation for static load testing now sets the ACL on the test
bucket and keys to 'public-read' so we don't need an expiring, temp url.
 2011-07-14 11:37:54 -07:00
Steven Berler
7b51f55a1b fix zero devision error in readwrite test 
Was dividing by 0 if somehow it managed to upload or download a file
in under .01 seconds.
 2011-07-14 11:00:48 -07:00
Steven Berler
91c4f88860 fix random file verifier with small files 
The file verifier will now correctly verify small files (smaller than
the md5 digest size), as long as the data that is there matches what
the digest would be.
 2011-07-13 15:59:38 -07:00
Tommi Virtanen
91a39caf46 Fix bug where prefix was None. 
Accessing a non-constant global from another module is tricky,
as the import binds to the original object. Rebinding the name
in the original module does not change it in importers. Use a
getter as a quick workaround.
 2011-07-13 14:49:07 -07:00
Tommi Virtanen
02a4c82dfa Make generate_objects.py and rand_readwrite.py inside the package. 
Handle dependencies properly. Now e.g. "./bootstrap &&
./virtualenv/bin/s3tests-test-readwrite" should just work.
 2011-07-13 13:52:54 -07:00
Tommi Virtanen
33b25c577e Move setup/teardown into s3tests.functional __init__, to share across modules. 2011-07-11 15:24:39 -07:00
Tommi Virtanen
e9ad6f7ee6 Move test_s3 into s3tests.functional, so it can be split to multiple files nicely. 2011-07-11 15:24:39 -07:00
Stephon Striplin
e253e328a7 remove unused variable 2011-07-11 14:57:46 -07:00
Stephon Striplin
83830e2951 add xml-based acl tests 2011-07-11 14:22:31 -07:00
Stephon Striplin
cd84d65e9c add test_object_acl_* and test_bucket_acl_* tests 2011-07-11 14:22:31 -07:00
Stephon Striplin
f929069189 add several test_object_set_get_metadata_* tests 2011-07-11 14:22:31 -07:00
Stephon Striplin
bb3f214369 add test_object_set_get_metadata_empty 2011-07-11 14:20:14 -07:00
Stephon Striplin
bbbe6c45c3 add test_object_copy_not_owned_bucket 2011-07-11 14:18:46 -07:00