776 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Steven Berler	680be12750	added sample yaml config for readwrite tool	2011-07-26 14:20:28 -07:00
Steven Berler	55a72f002b	moved most readwrite tool command-line opts to config file ... The only remaining command-line options are setting the random seed and --no-cleanup.	2011-07-26 14:20:28 -07:00
Steven Berler	a6c16990df	remove extraneous #TODO	2011-07-26 14:20:28 -07:00
Steven Berler	0c0c590d47	use stderr for non-yml output in readwrite tool ... stdout should now only contain the yml output.	2011-07-26 14:20:28 -07:00
Steven Berler	33876b0dae	fix yml format in readwrite tool ... Now will print the chunks out one line per chunk instead of splitting them accross multiple lines.	2011-07-26 14:20:28 -07:00
Steven Berler	1e991bc4bc	adding chunks to readwrite tool output	2011-07-26 14:20:28 -07:00
Steven Berler	8cf9c57012	fix readwrite tool file sizes ... Makes the file sizes specified correctly in KB instead of B	2011-07-26 14:20:28 -07:00
Steven Berler	15cf598b12	readwrite tool improvements ... Now generates a list of filenames to use duing the setup, and passes this list to the workers rather than having the workers list the bucket contents or generate file names themselves. Added a "warmup" stage that first writes out all the files so that content exists for each filename. Added setting the random seed as a command line parameter and uses this seed for both generating the file names and the file content generator.	2011-07-26 14:20:28 -07:00
Steven Berler	b465596063	adding max_amount to realistic.names generator ... Set max_amount to limit the number of names yeilded by the generator.	2011-07-26 14:20:28 -07:00
Stephon Striplin	a775f5da04	add Content-Type object tests	2011-07-25 14:12:50 -07:00
Stephon Striplin	2e351e9eb8	add header-manipulation tests ... These tests are based on explicitly modifying headers in some way.	2011-07-25 11:25:39 -07:00
Stephon Striplin	ee28900620	add test_object_create_unreadable	2011-07-25 11:25:39 -07:00
Yehuda Sadeh	9b3cdf753f	test consistency of bucket acls, exercises cache	2011-07-21 14:10:05 -07:00
Stephon Striplin	338728ecd2	add test_bucket_acl_no_grants ... When the owner has no grants on their bucket, they should not be able to write to it, but still read and write the ACL. Amazon allows users to read the bucket, and our behavior is consistent with this.	2011-07-20 12:30:23 -07:00
Tommi Virtanen	b54e1cbe99	Handle and report errors in readwrite test workers.	2011-07-20 11:32:52 -07:00
Tommi Virtanen	f3b5ed6ccc	Say "worker_id" instead of name, as I keep confusing it with filenames.	2011-07-20 11:15:06 -07:00
Tommi Virtanen	1eb38e09e4	Link worker greenlets to parent, so we abort if they have an unhandled exception.	2011-07-20 11:06:30 -07:00
Tommi Virtanen	9ecaf63fbe	Code layout cleanup.	2011-07-20 11:06:30 -07:00
Tommi Virtanen	deb8be9957	Make readwrite test put objects one by one, so it can report keys.	2011-07-20 11:06:30 -07:00
Tommi Virtanen	9e707b897d	Remove the concept of batches from readwrite test. ... That's an internal implementation detail caused by the generate_objects api.	2011-07-20 11:06:30 -07:00
Tommi Virtanen	4cd4131dd7	Don't eat exceptions, that's a bad habit.	2011-07-20 11:06:30 -07:00
Tommi Virtanen	648ddbb761	Log readwrite test results in yaml. ... This temporarily removes the statistics calculation; that belongs somewhere else anyway, so it can be run on aggregated results etc.	2011-07-20 11:06:30 -07:00
Tommi Virtanen	ed966af0a7	Use more idiomatic Python.	2011-07-20 11:06:30 -07:00
Tommi Virtanen	ad444b33ee	Make queue and name arguments not optional; they're always passed in anyway.	2011-07-20 11:06:30 -07:00
Tommi Virtanen	71cfd4956c	Move end test logic out of individual workers. ... Parent waits for the duration anyway, just make it unconditionally kill the workers. They already had timeouts aborting them; they may be in the middle of an operation, but we really don't care, time's up. This also avoids oddities with the queue where there may be items after the StopIteration. This also avoids workers potentially not exiting within the 1-second timeout the old code had.	2011-07-20 11:06:30 -07:00
Tommi Virtanen	dff6b568e4	Use gevent Groups to manage groups of workers. ... This mostly adds convenience functions, but it's nice to have.	2011-07-20 11:06:30 -07:00
Stephon Striplin	510b6e91d4	refactor and fix some acl_grant tests	2011-07-19 11:16:11 -07:00
Tommi Virtanen	fe749adebf	pep8 cleanup.	2011-07-19 09:09:51 -07:00
Tommi Virtanen	e0575e4efe	Pyflakes cleanup.	2011-07-19 09:05:05 -07:00
Tommi Virtanen	c7bb78b666	Remove dead code.	2011-07-19 09:04:20 -07:00
Tommi Virtanen	5c25c66649	Merge branch 'roundtrip'	2011-07-19 09:00:44 -07:00
Tommi Virtanen	56ff0f6374	Revert "Add assert to RandomContentFile, ensure size >= digest_size" ... This reverts commit fb8f110e69. realistic.files was meant to work just fine with short files. Commit 91c4f88860 has a fix for the verifying side, to support short files. Conflicts: s3tests/realistic.py	2011-07-19 08:56:17 -07:00
Wesley Spikes	0297d61e40	Swap stdout/stderr -- debug messages then go to stderr, and yaml data to stdout	2011-07-18 17:04:36 -07:00
Wesley Spikes	ce6c57841a	Now that we no longer have the need to retry, get rid of that logic. ... Also fix a typo -- s/neads_first_read/needs_first_read/	2011-07-18 16:15:40 -07:00
Wesley Spikes	c154d98b96	Resolve the random AssertionError ... By doing a copy-new on the key, it seems to work around the random AssertionError's I've been seeing. Here's hoping this fully resolves it.	2011-07-18 16:15:40 -07:00
Wesley Spikes	44cb2f256e	Adding unlimited=bool to files_varied -- if set to true, it will never issue StopIteration	2011-07-18 16:15:40 -07:00
Wesley Spikes	43b2f8695d	Cleanup output just a little bit more.	2011-07-18 16:15:40 -07:00
Wesley Spikes	841b098619	Refactor/cleanup TransferResults class. Also cleans the output to be in the format TV seems to want.	2011-07-18 16:15:40 -07:00
Wesley Spikes	3c07a4a363	Update roundtrip._main -- while loop should auto-replenish all pools, ... and not ones so named.	2011-07-18 16:15:40 -07:00
Wesley Spikes	fb8f110e69	Add assert to RandomContentFile, ensure size >= digest_size ... It clearly will not work to have the size smaller than the digest size, so we shouldn't allow it.	2011-07-18 16:15:40 -07:00
Wesley Spikes	d0bb148dea	Rename _call_doit to _real_run -- it's a bit more clear that way, IMHO	2011-07-18 16:15:40 -07:00
Stephon Striplin	5136608e76	fix test_bucket_acl_grant_nonexist_user ... * Change user_id to avoid name conflict * Check for error_code	2011-07-15 14:57:46 -07:00
Stephon Striplin	03a80e6121	add acl grant userid tests ... * Refactor some acl grant user_id code and run through the full suite of acls	2011-07-15 14:34:08 -07:00
Wesley Spikes	653c2b2200	Adding new testing tool, roundtrip	2011-07-15 12:06:02 -07:00
Wesley Spikes	6afc7fdcdb	Adding last_read to RandomContentFile and first_write to FileVerifier, so we can measure latency.	2011-07-15 12:05:02 -07:00
Wesley Spikes	12716d854b	Adding generator files_varied to s3tests.realistic ... Given a tuple of tuples, construct several files() generators, and yield from those randomly. Randomness is weighted based on the number of files remaining in each group.	2011-07-15 12:03:23 -07:00
Wesley Spikes	4959c5e1a4	Work-around for RGW failing to send IsTruncated on listing all keys. ... Without this, it fails to delete the bucket when there's more than 1000 keys.	2011-07-15 11:47:57 -07:00
Wesley Spikes	b6e0287810	Add context, get_next_key, and fill_pools to s3tests.common ... context is a generic Bunch that we can store items in to be shared across modules, without invoking the wrath of the globals -- yuck! get_next_key will generate a sequentially numbered key for us to use fill_pools will take a variable list of gevent.pool.Pool objects and top them all off	2011-07-15 11:45:38 -07:00
Wesley Spikes	298bc99c5d	Packagify s3tests.common so we can add more magic into it later.	2011-07-15 11:44:05 -07:00
Wesley Spikes	e4effe5479	realistic.names shouldn't return a 0-length filename	2011-07-15 11:08:38 -07:00