Steven Berler
a6c16990df
remove extraneous #TODO
2011-07-26 14:20:28 -07:00
Steven Berler
0c0c590d47
use stderr for non-yml output in readwrite tool
...
stdout should now only contain the yml output.
2011-07-26 14:20:28 -07:00
Steven Berler
33876b0dae
fix yml format in readwrite tool
...
Now will print the chunks out one line per chunk instead of splitting
them accross multiple lines.
2011-07-26 14:20:28 -07:00
Steven Berler
1e991bc4bc
adding chunks to readwrite tool output
2011-07-26 14:20:28 -07:00
Steven Berler
8cf9c57012
fix readwrite tool file sizes
...
Makes the file sizes specified correctly in KB instead of B
2011-07-26 14:20:28 -07:00
Steven Berler
15cf598b12
readwrite tool improvements
...
Now generates a list of filenames to use duing the setup, and passes this
list to the workers rather than having the workers list the bucket contents
or generate file names themselves.
Added a "warmup" stage that first writes out all the files so that content
exists for each filename.
Added setting the random seed as a command line parameter and uses this seed
for both generating the file names and the file content generator.
2011-07-26 14:20:28 -07:00
Steven Berler
b465596063
adding max_amount to realistic.names generator
...
Set max_amount to limit the number of names yeilded by the generator.
2011-07-26 14:20:28 -07:00
Stephon Striplin
a775f5da04
add Content-Type object tests
2011-07-25 14:12:50 -07:00
Stephon Striplin
2e351e9eb8
add header-manipulation tests
...
These tests are based on explicitly modifying headers in some way.
2011-07-25 11:25:39 -07:00
Stephon Striplin
ee28900620
add test_object_create_unreadable
2011-07-25 11:25:39 -07:00
Yehuda Sadeh
9b3cdf753f
test consistency of bucket acls, exercises cache
2011-07-21 14:10:05 -07:00
Stephon Striplin
338728ecd2
add test_bucket_acl_no_grants
...
When the owner has no grants on their bucket, they should not be able to
write to it, but still read and write the ACL. Amazon allows users to
read the bucket, and our behavior is consistent with this.
2011-07-20 12:30:23 -07:00
Tommi Virtanen
b54e1cbe99
Handle and report errors in readwrite test workers.
2011-07-20 11:32:52 -07:00
Tommi Virtanen
f3b5ed6ccc
Say "worker_id" instead of name, as I keep confusing it with filenames.
2011-07-20 11:15:06 -07:00
Tommi Virtanen
1eb38e09e4
Link worker greenlets to parent, so we abort if they have an unhandled exception.
2011-07-20 11:06:30 -07:00
Tommi Virtanen
9ecaf63fbe
Code layout cleanup.
2011-07-20 11:06:30 -07:00
Tommi Virtanen
deb8be9957
Make readwrite test put objects one by one, so it can report keys.
2011-07-20 11:06:30 -07:00
Tommi Virtanen
9e707b897d
Remove the concept of batches from readwrite test.
...
That's an internal implementation detail caused by
the generate_objects api.
2011-07-20 11:06:30 -07:00
Tommi Virtanen
4cd4131dd7
Don't eat exceptions, that's a bad habit.
2011-07-20 11:06:30 -07:00
Tommi Virtanen
648ddbb761
Log readwrite test results in yaml.
...
This temporarily removes the statistics calculation; that
belongs somewhere else anyway, so it can be run on aggregated
results etc.
2011-07-20 11:06:30 -07:00
Tommi Virtanen
ed966af0a7
Use more idiomatic Python.
2011-07-20 11:06:30 -07:00
Tommi Virtanen
ad444b33ee
Make queue and name arguments not optional; they're always passed in anyway.
2011-07-20 11:06:30 -07:00
Tommi Virtanen
71cfd4956c
Move end test logic out of individual workers.
...
Parent waits for the duration anyway, just make it unconditionally
kill the workers. They already had timeouts aborting them; they may
be in the middle of an operation, but we really don't care, time's up.
This also avoids oddities with the queue where there may be items
after the StopIteration.
This also avoids workers potentially not exiting within the 1-second
timeout the old code had.
2011-07-20 11:06:30 -07:00
Tommi Virtanen
dff6b568e4
Use gevent Groups to manage groups of workers.
...
This mostly adds convenience functions, but it's nice to have.
2011-07-20 11:06:30 -07:00
Stephon Striplin
510b6e91d4
refactor and fix some acl_grant tests
2011-07-19 11:16:11 -07:00
Tommi Virtanen
fe749adebf
pep8 cleanup.
2011-07-19 09:09:51 -07:00
Tommi Virtanen
e0575e4efe
Pyflakes cleanup.
2011-07-19 09:05:05 -07:00
Tommi Virtanen
c7bb78b666
Remove dead code.
2011-07-19 09:04:20 -07:00
Tommi Virtanen
5c25c66649
Merge branch 'roundtrip'
2011-07-19 09:00:44 -07:00
Tommi Virtanen
56ff0f6374
Revert "Add assert to RandomContentFile, ensure size >= digest_size"
...
This reverts commit fb8f110e6991c4f88860
2011-07-19 08:56:17 -07:00
Wesley Spikes
0297d61e40
Swap stdout/stderr -- debug messages then go to stderr, and yaml data to stdout
2011-07-18 17:04:36 -07:00
Wesley Spikes
ce6c57841a
Now that we no longer have the need to retry, get rid of that logic.
...
Also fix a typo -- s/neads_first_read/needs_first_read/
2011-07-18 16:15:40 -07:00
Wesley Spikes
c154d98b96
Resolve the random AssertionError
...
By doing a copy-new on the key, it seems to work around the random
AssertionError's I've been seeing. Here's hoping this fully
resolves it.
2011-07-18 16:15:40 -07:00
Wesley Spikes
44cb2f256e
Adding unlimited=bool to files_varied -- if set to true, it will never issue StopIteration
2011-07-18 16:15:40 -07:00
Wesley Spikes
43b2f8695d
Cleanup output just a little bit more.
2011-07-18 16:15:40 -07:00
Wesley Spikes
841b098619
Refactor/cleanup TransferResults class. Also cleans the output to be in the format TV seems to want.
2011-07-18 16:15:40 -07:00
Wesley Spikes
3c07a4a363
Update roundtrip._main -- while loop should auto-replenish all pools,
...
and not ones so named.
2011-07-18 16:15:40 -07:00
Wesley Spikes
fb8f110e69
Add assert to RandomContentFile, ensure size >= digest_size
...
It clearly will not work to have the size smaller than the digest
size, so we shouldn't allow it.
2011-07-18 16:15:40 -07:00
Wesley Spikes
d0bb148dea
Rename _call_doit to _real_run -- it's a bit more clear that way, IMHO
2011-07-18 16:15:40 -07:00
Stephon Striplin
5136608e76
fix test_bucket_acl_grant_nonexist_user
...
* Change user_id to avoid name conflict
* Check for error_code
2011-07-15 14:57:46 -07:00
Stephon Striplin
03a80e6121
add acl grant userid tests
...
* Refactor some acl grant user_id code and run through the full suite of
acls
2011-07-15 14:34:08 -07:00
Wesley Spikes
653c2b2200
Adding new testing tool, roundtrip
2011-07-15 12:06:02 -07:00
Wesley Spikes
6afc7fdcdb
Adding last_read to RandomContentFile and first_write to FileVerifier, so we can measure latency.
2011-07-15 12:05:02 -07:00
Wesley Spikes
12716d854b
Adding generator files_varied to s3tests.realistic
...
Given a tuple of tuples, construct several files() generators, and
yield from those randomly. Randomness is weighted based on the number
of files remaining in each group.
2011-07-15 12:03:23 -07:00
Wesley Spikes
4959c5e1a4
Work-around for RGW failing to send IsTruncated on listing all keys.
...
Without this, it fails to delete the bucket when there's more than 1000 keys.
2011-07-15 11:47:57 -07:00
Wesley Spikes
b6e0287810
Add context, get_next_key, and fill_pools to s3tests.common
...
context is a generic Bunch that we can store items in to be shared
across modules, without invoking the wrath of the globals -- yuck!
get_next_key will generate a sequentially numbered key for us to use
fill_pools will take a variable list of gevent.pool.Pool objects and
top them all off
2011-07-15 11:45:38 -07:00
Wesley Spikes
298bc99c5d
Packagify s3tests.common so we can add more magic into it later.
2011-07-15 11:44:05 -07:00
Wesley Spikes
e4effe5479
realistic.names shouldn't return a 0-length filename
2011-07-15 11:08:38 -07:00
Yehuda Sadeh
3303b0f33b
fix 100-continue test to support https
2011-07-14 16:26:46 -07:00
Steven Berler
c312832c70
use time.time() instead of time.clock()
...
We want to measure actual time, not cpu time.
2011-07-14 11:40:26 -07:00
