TrueCloudLab/frostfs-testcases
25
0
Fork 21
Code Issues 2 Pull requests 3 Releases Activity Actions 1

Fixed arguments order in get_filtered_logs #167

Closed
mkadilov wants to merge 30 commits from (deleted):bug/log_analyzer_fix_for_ft_1_3 into support/v0.37
pull from: (deleted):bug/log_analyzer_fix_for_ft_1_3
merge into: TrueCloudLab:support/v0.37
Conversation 0 Commits 30 Files changed 46 +2467 -2516
46 changed files with 2467 additions and 2516 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
CONTRIBUTING.md
View file

@ -3,8 +3,8 @@
First, thank you for contributing! We love and encourage pull requests from
everyone. Please follow the guidelines:
- Check the open [issues](https://github.com/TrueCloudLab/frostfs-testcases/issues) and
[pull requests](https://github.com/TrueCloudLab/frostfs-testcases/pulls) for existing
- Check the open [issues](https://git.frostfs.info/TrueCloudLab/frostfs-testcases/issues) and
[pull requests](https://git.frostfs.info/TrueCloudLab/frostfs-testcases/pulls) for existing
discussions.
- Open an issue first, to discuss a new feature or enhancement.
@ -26,9 +26,9 @@ Start by forking the `frostfs-testcases` repository, make changes in a branch an
send a pull request. We encourage pull requests to discuss code changes. Here
are the steps in details:
### Set up your GitHub Repository
### Set up your Git Repository
Fork [FrosfFS testcases upstream](https://github.com/TrueCloudLab/frostfs-testcases/fork) source
Fork [FrosfFS testcases upstream](https://git.frostfs.info/TrueCloudLab/frostfs-testcases/forks) source
repository to your own personal repository. Copy the URL of your fork and clone it:
```shell
@ -38,7 +38,7 @@ $ git clone <url of your fork>
### Set up git remote as ``upstream``
```sh
$ cd frostfs-testcases
$ git remote add upstream https://github.com/TrueCloudLab/frostfs-testcases
$ git remote add upstream https://git.frostfs.info/TrueCloudLab/frostfs-testcases
$ git fetch upstream
```
@ -94,8 +94,8 @@ $ git push origin feature/123-something_awesome
### Create a Pull Request
Pull requests can be created via GitHub. Refer to [this
document](https://help.github.com/articles/creating-a-pull-request/) for
Pull requests can be created via Git. Refer to [this
document](https://docs.codeberg.org/collaborating/pull-requests-and-git-flow/) for
detailed steps on how to create a pull request. After a Pull Request gets peer
reviewed and approved, it will be merged.

12
README.md
View file

@ -9,27 +9,27 @@ These tests rely on resources and utility modules that have been originally deve
### Initial preparation
1. Install frostfs-cli
- `git clone git@github.com:TrueCloudLab/frostfs-node.git`
- `git clone git@git.frostfs.info:TrueCloudLab/frostfs-node.git`
- `cd frostfs-node`
- `make`
- `sudo cp bin/frostfs-cli /usr/local/bin/frostfs-cli`
2. Install frostfs-s3-authmate
- `git clone git@github.com:TrueCloudLab/frostfs-s3-gw.git`
- `git clone git@git.frostfs.info:TrueCloudLab/frostfs-s3-gw.git`
- `cd frostfs-s3-gw`
- `make`
- `sudo cp bin/frostfs-s3-authmate /usr/local/bin/frostfs-s3-authmate`
3. Install neo-go
- `git clone git@github.com:nspcc-dev/neo-go.git`
- `git clone git@git.frostfs.info:TrueCloudLab/neo-go.git`
- `cd neo-go`
- `git checkout v0.101.0` (or the current version in the frostfs-dev-env)
- `make`
- `sudo cp bin/neo-go /usr/local/bin/neo-go`
or download binary from releases: https://github.com/nspcc-dev/neo-go/releases
or download binary from releases: https://git.frostfs.info/TrueCloudLab/neo-go/releases
4. Clone frostfs-dev-env
`git clone git@github.com:TrueCloudLab/frostfs-dev-env.git`
`git clone git@git.frostfs.info:TrueCloudLab/frostfs-dev-env.git`
Note that we expect frostfs-dev-env to be located under
the `<testcases_root_dir>/../frostfs-dev-env` directory. If you put this repo in any other place,
@ -44,7 +44,7 @@ python3.10-dev
libssl-dev
```
As we use frostfs-dev-env, you'll also need to install
[prerequisites](https://github.com/TrueCloudLab/frostfs-dev-env#prerequisites) of this repository.
[prerequisites](https://git.frostfs.info/TrueCloudLab/frostfs-dev-env#prerequisites) of this repository.
6. Prepare virtualenv

20
pytest_tests/pytest.ini → pytest.ini
View file

@ -13,6 +13,7 @@ markers =
# controlling markers
no_healthcheck: skip healthcheck for this test
# functional markers
maintenance: tests for change mode node
container: tests for container creation
grpc_api: standard gRPC API tests
grpc_control: tests related to using frostfs-cli control commands
@ -47,3 +48,22 @@ markers =
check_binaries: check frostfs installed binaries versions
payments: tests for payment associated operations
load: performance tests
simple: tests with simple characteristics
complex: tests with complex characteristics
aws: AWS related tests
boto3: tests using the boto3
policy: policy tests
failover_baremetal: failover tests on hardware (baremetal)
failover_server: server failover tests
failover_storage: storage failover tests
failover_empty_map: failover tests for an empty map
failover_empty_map_offlne: offline failover tests for an empty map
failover_empty_map_stop_service: failover tests for stopped empty map service
failover_data_loss: failover tests in case of data loss
metabase_loss: tests for metadata loss
write_cache_loss: tests for write cache loss
time: time tests
replication: replication tests
static_session_container: tests for a static session in a container
shard: shard management tests
logs_after_session: tests after a session with logs

16
pytest_tests/helpers/object_access.py
View file

@ -1,6 +1,6 @@
from typing import Optional
import allure
from frostfs_testlib import reporter
from frostfs_testlib.resources.cli import CLI_DEFAULT_TIMEOUT
from frostfs_testlib.resources.error_patterns import OBJECT_ACCESS_DENIED
from frostfs_testlib.shell import Shell
@ -31,7 +31,7 @@ def can_get_object(
wallet_config: Optional[str] = None,
xhdr: Optional[dict] = None,
) -> bool:
with allure.step("Try get object from container"):
with reporter.step("Try get object from container"):
try:
got_file_path = get_object_from_random_node(
wallet,
@ -63,7 +63,7 @@ def can_put_object(
xhdr: Optional[dict] = None,
attributes: Optional[dict] = None,
) -> bool:
with allure.step("Try put object to container"):
with reporter.step("Try put object to container"):
try:
put_object_to_random_node(
wallet,
@ -94,7 +94,7 @@ def can_delete_object(
wallet_config: Optional[str] = None,
xhdr: Optional[dict] = None,
) -> bool:
with allure.step("Try delete object from container"):
with reporter.step("Try delete object from container"):
try:
delete_object(
wallet,
@ -125,7 +125,7 @@ def can_get_head_object(
xhdr: Optional[dict] = None,
timeout: Optional[str] = CLI_DEFAULT_TIMEOUT,
) -> bool:
with allure.step("Try get head of object"):
with reporter.step("Try get head of object"):
try:
head_object(
wallet,
@ -157,7 +157,7 @@ def can_get_range_of_object(
xhdr: Optional[dict] = None,
timeout: Optional[str] = CLI_DEFAULT_TIMEOUT,
) -> bool:
with allure.step("Try get range of object"):
with reporter.step("Try get range of object"):
try:
get_range(
wallet,
@ -190,7 +190,7 @@ def can_get_range_hash_of_object(
xhdr: Optional[dict] = None,
timeout: Optional[str] = CLI_DEFAULT_TIMEOUT,
) -> bool:
with allure.step("Try get range hash of object"):
with reporter.step("Try get range hash of object"):
try:
get_range_hash(
wallet,
@ -223,7 +223,7 @@ def can_search_object(
xhdr: Optional[dict] = None,
timeout: Optional[str] = CLI_DEFAULT_TIMEOUT,
) -> bool:
with allure.step("Try search object in container"):
with reporter.step("Try search object in container"):
try:
oids = search_object(
wallet,

4
pytest_tests/helpers/utility.py
View file

@ -1,6 +1,6 @@
import time
import allure
from frostfs_testlib import reporter
from frostfs_testlib.resources.common import STORAGE_GC_TIME
from frostfs_testlib.utils import datetime_utils
@ -33,5 +33,5 @@ def placement_policy_from_container(container_info: str) -> str:
def wait_for_gc_pass_on_storage_nodes() -> None:
wait_time = datetime_utils.parse_time(STORAGE_GC_TIME)
with allure.step(f"Wait {wait_time}s until GC completes on storage nodes"):
with reporter.step(f"Wait {wait_time}s until GC completes on storage nodes"):
time.sleep(wait_time)

18
pytest_tests/testsuites/acl/conftest.py
View file

@ -3,8 +3,8 @@ import uuid
from dataclasses import dataclass
from typing import Optional
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.common import DEFAULT_WALLET_CONFIG, DEFAULT_WALLET_PASS
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.shell import Shell
@ -39,9 +39,7 @@ class Wallets:
@pytest.fixture(scope="module")
def wallets(default_wallet: str, temp_directory: str, cluster: Cluster) -> Wallets:
other_wallets_paths = [
os.path.join(temp_directory, f"{str(uuid.uuid4())}.json") for _ in range(2)
]
other_wallets_paths = [os.path.join(temp_directory, f"{str(uuid.uuid4())}.json") for _ in range(2)]
for other_wallet_path in other_wallets_paths:
wallet_utils.init_wallet(other_wallet_path, DEFAULT_WALLET_PASS)
@ -72,11 +70,7 @@ def wallets(default_wallet: str, temp_directory: str, cluster: Cluster) -> Walle
if role == EACLRole.SYSTEM:
continue
for wallet in wallets:
allure.attach.file(
wallet.wallet_path,
os.path.basename(wallet.wallet_path),
allure.attachment_type.JSON,
)
reporter.attach(wallet.wallet_path, os.path.basename(wallet.wallet_path))
return wallets_collection
@ -91,7 +85,7 @@ def eacl_container_with_objects(
wallets: Wallets, client_shell: Shell, cluster: Cluster, file_path: str
) -> tuple[str, list[str], str]:
user_wallet = wallets.get_wallet()
with allure.step("Create eACL public container"):
with reporter.step("Create eACL public container"):
cid = create_container(
user_wallet.wallet_path,
basic_acl=PUBLIC_ACL,
@ -99,7 +93,7 @@ def eacl_container_with_objects(
endpoint=cluster.default_rpc_endpoint,
)
with allure.step("Add test objects to container"):
with reporter.step("Add test objects to container"):
objects_oids = [
put_object_to_random_node(
user_wallet.wallet_path,
@ -114,5 +108,5 @@ def eacl_container_with_objects(
yield cid, objects_oids, file_path
# with allure.step('Delete eACL public container'):
# with reporter.step('Delete eACL public container'):
# delete_container(user_wallet, cid)

35
pytest_tests/testsuites/acl/test_acl.py
View file

@ -1,5 +1,6 @@
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PRIVATE_ACL_F, PUBLIC_ACL_F, READONLY_ACL_F
from frostfs_testlib.shell import Shell
from frostfs_testlib.steps.cli.container import create_container
@ -23,7 +24,7 @@ class TestACLBasic(ClusterTestBase):
@pytest.fixture(scope="function")
def public_container(self, wallets):
user_wallet = wallets.get_wallet()
with allure.step("Create public container"):
with reporter.step("Create public container"):
cid_public = create_container(
user_wallet.wallet_path,
basic_acl=PUBLIC_ACL_F,
@ -33,13 +34,13 @@ class TestACLBasic(ClusterTestBase):
yield cid_public
# with allure.step('Delete public container'):
# with reporter.step('Delete public container'):
# delete_container(user_wallet.wallet_path, cid_public)
@pytest.fixture(scope="function")
def private_container(self, wallets: Wallets):
user_wallet = wallets.get_wallet()
with allure.step("Create private container"):
with reporter.step("Create private container"):
cid_private = create_container(
user_wallet.wallet_path,
basic_acl=PRIVATE_ACL_F,
@ -49,13 +50,13 @@ class TestACLBasic(ClusterTestBase):
yield cid_private
# with allure.step('Delete private container'):
# with reporter.step('Delete private container'):
# delete_container(user_wallet.wallet_path, cid_private)
@pytest.fixture(scope="function")
def read_only_container(self, wallets: Wallets):
user_wallet = wallets.get_wallet()
with allure.step("Create public readonly container"):
with reporter.step("Create public readonly container"):
cid_read_only = create_container(
user_wallet.wallet_path,
basic_acl=READONLY_ACL_F,
@ -65,7 +66,7 @@ class TestACLBasic(ClusterTestBase):
yield cid_read_only
# with allure.step('Delete public readonly container'):
# with reporter.step('Delete public readonly container'):
# delete_container(user_wallet.wallet_path, cid_read_only)
@allure.title("Operations with basic ACL on public container (obj_size={object_size})")
@ -77,7 +78,7 @@ class TestACLBasic(ClusterTestBase):
other_wallet = wallets.get_wallet(role=EACLRole.OTHERS)
cid = public_container
for wallet, desc in ((user_wallet, "owner"), (other_wallet, "other users")):
with allure.step("Add test objects to container"):
with reporter.step("Add test objects to container"):
# We create new objects for each wallet because check_full_access_to_container
# deletes the object
owner_object_oid = put_object_to_random_node(
@ -96,7 +97,7 @@ class TestACLBasic(ClusterTestBase):
cluster=self.cluster,
attributes={"created": "other"},
)
with allure.step(f"Check {desc} has full access to public container"):
with reporter.step(f"Check {desc} has full access to public container"):
check_full_access_to_container(
wallet.wallet_path,
cid,
@ -122,13 +123,13 @@ class TestACLBasic(ClusterTestBase):
user_wallet = wallets.get_wallet()
other_wallet = wallets.get_wallet(role=EACLRole.OTHERS)
cid = private_container
with allure.step("Add test objects to container"):
with reporter.step("Add test objects to container"):
owner_object_oid = put_object_to_random_node(
user_wallet.wallet_path, file_path, cid, shell=self.shell, cluster=self.cluster
)
with allure.step("Check only owner has full access to private container"):
with allure.step("Check no one except owner has access to operations with container"):
with reporter.step("Check only owner has full access to private container"):
with reporter.step("Check no one except owner has access to operations with container"):
check_no_access_to_container(
other_wallet.wallet_path,
cid,
@ -138,7 +139,7 @@ class TestACLBasic(ClusterTestBase):
cluster=self.cluster,
)
with allure.step("Check owner has full access to private container"):
with reporter.step("Check owner has full access to private container"):
check_full_access_to_container(
user_wallet.wallet_path,
cid,
@ -149,9 +150,7 @@ class TestACLBasic(ClusterTestBase):
)
@allure.title("Operations with basic ACL on READONLY container (obj_size={object_size})")
def test_basic_acl_readonly(
self, wallets: Wallets, client_shell: Shell, read_only_container: str, file_path: str
):
def test_basic_acl_readonly(self, wallets: Wallets, client_shell: Shell, read_only_container: str, file_path: str):
"""
Test basic ACL Operations for Read-Only Container.
"""
@ -159,12 +158,12 @@ class TestACLBasic(ClusterTestBase):
other_wallet = wallets.get_wallet(role=EACLRole.OTHERS)
cid = read_only_container
with allure.step("Add test objects to container"):
with reporter.step("Add test objects to container"):
object_oid = put_object_to_random_node(
user_wallet.wallet_path, file_path, cid, shell=client_shell, cluster=self.cluster
)
with allure.step("Check other has read-only access to operations with container"):
with reporter.step("Check other has read-only access to operations with container"):
check_read_only_container(
other_wallet.wallet_path,
cid,
@ -174,7 +173,7 @@ class TestACLBasic(ClusterTestBase):
cluster=self.cluster,
)
with allure.step("Check owner has full access to public container"):
with reporter.step("Check owner has full access to public container"):
check_full_access_to_container(
user_wallet.wallet_path,
cid,

57
pytest_tests/testsuites/acl/test_bearer.py
View file

@ -1,11 +1,7 @@
import allure
import pytest
from frostfs_testlib.steps.acl import (
create_eacl,
form_bearertoken_file,
set_eacl,
wait_for_cache_expired,
)
from frostfs_testlib import reporter
from frostfs_testlib.steps.acl import create_eacl, form_bearertoken_file, set_eacl, wait_for_cache_expired
from frostfs_testlib.storage.dataclasses.acl import EACLAccess, EACLOperation, EACLRole, EACLRule
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
@ -34,7 +30,7 @@ class TestACLBearer(ClusterTestBase):
deny_wallet = wallets.get_wallet(role)
endpoint = self.cluster.default_rpc_endpoint
with allure.step(f"Check {role.value} has full access to container without bearer token"):
with reporter.step(f"Check {role.value} has full access to container without bearer token"):
check_full_access_to_container(
deny_wallet.wallet_path,
cid,
@ -45,29 +41,22 @@ class TestACLBearer(ClusterTestBase):
cluster=self.cluster,
)
with allure.step(f"Set deny all operations for {role.value} via eACL"):
eacl = [
EACLRule(access=EACLAccess.DENY, role=role, operation=op) for op in EACLOperation
]
with reporter.step(f"Set deny all operations for {role.value} via eACL"):
eacl = [EACLRule(access=EACLAccess.DENY, role=role, operation=op) for op in EACLOperation]
eacl_file = create_eacl(cid, eacl, shell=self.shell)
set_eacl(user_wallet.wallet_path, cid, eacl_file, shell=self.shell, endpoint=endpoint)
wait_for_cache_expired()
with allure.step(f"Create bearer token for {role.value} with all operations allowed"):
with reporter.step(f"Create bearer token for {role.value} with all operations allowed"):
bearer = form_bearertoken_file(
user_wallet.wallet_path,
cid,
[
EACLRule(operation=op, access=EACLAccess.ALLOW, role=role)
for op in EACLOperation
],
[EACLRule(operation=op, access=EACLAccess.ALLOW, role=role) for op in EACLOperation],
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
)
with allure.step(
f"Check {role.value} without token has no access to all operations with container"
):
with reporter.step(f"Check {role.value} without token has no access to all operations with container"):
check_no_access_to_container(
deny_wallet.wallet_path,
cid,
@ -78,9 +67,7 @@ class TestACLBearer(ClusterTestBase):
cluster=self.cluster,
)
with allure.step(
f"Check {role.value} with token has access to all operations with container"
):
with reporter.step(f"Check {role.value} with token has access to all operations with container"):
check_full_access_to_container(
deny_wallet.wallet_path,
cid,
@ -92,17 +79,13 @@ class TestACLBearer(ClusterTestBase):
cluster=self.cluster,
)
with allure.step(f"Set allow all operations for {role.value} via eACL"):
eacl = [
EACLRule(access=EACLAccess.ALLOW, role=role, operation=op) for op in EACLOperation
]
with reporter.step(f"Set allow all operations for {role.value} via eACL"):
eacl = [EACLRule(access=EACLAccess.ALLOW, role=role, operation=op) for op in EACLOperation]
eacl_file = create_eacl(cid, eacl, shell=self.shell)
set_eacl(user_wallet.wallet_path, cid, eacl_file, shell=self.shell, endpoint=endpoint)
wait_for_cache_expired()
with allure.step(
f"Check {role.value} without token has access to all operations with container"
):
with reporter.step(f"Check {role.value} without token has access to all operations with container"):
check_full_access_to_container(
deny_wallet.wallet_path,
cid,
@ -145,19 +128,13 @@ class TestACLBearer(ClusterTestBase):
}
deny_map_with_bearer = {
EACLRole.USER: [
op for op in deny_map[EACLRole.USER] if op not in bearer_map[EACLRole.USER]
],
EACLRole.OTHERS: [
op for op in deny_map[EACLRole.OTHERS] if op not in bearer_map[EACLRole.OTHERS]
],
EACLRole.USER: [op for op in deny_map[EACLRole.USER] if op not in bearer_map[EACLRole.USER]],
EACLRole.OTHERS: [op for op in deny_map[EACLRole.OTHERS] if op not in bearer_map[EACLRole.OTHERS]],
}
eacl_deny = []
for role, operations in deny_map.items():
eacl_deny += [
EACLRule(access=EACLAccess.DENY, role=role, operation=op) for op in operations
]
eacl_deny += [EACLRule(access=EACLAccess.DENY, role=role, operation=op) for op in operations]
set_eacl(
user_wallet.wallet_path,
cid,
@ -167,7 +144,7 @@ class TestACLBearer(ClusterTestBase):
)
wait_for_cache_expired()
with allure.step("Check rule consistency without bearer"):
with reporter.step("Check rule consistency without bearer"):
check_custom_access_to_container(
user_wallet.wallet_path,
cid,
@ -189,7 +166,7 @@ class TestACLBearer(ClusterTestBase):
cluster=self.cluster,
)
with allure.step("Check rule consistency using bearer token"):
with reporter.step("Check rule consistency using bearer token"):
bearer_user = form_bearertoken_file(
user_wallet.wallet_path,
cid,

89
pytest_tests/testsuites/acl/test_eacl.py
View file

@ -1,5 +1,6 @@
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.acl import create_eacl, set_eacl, wait_for_cache_expired
from frostfs_testlib.steps.cli.container import create_container
@ -9,10 +10,7 @@ from frostfs_testlib.storage.dataclasses.acl import EACLAccess, EACLOperation, E
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
from frostfs_testlib.utils.failover_utils import wait_object_replication
from pytest_tests.helpers.container_access import (
check_full_access_to_container,
check_no_access_to_container,
)
from pytest_tests.helpers.container_access import check_full_access_to_container, check_no_access_to_container
from pytest_tests.helpers.object_access import (
can_delete_object,
can_get_head_object,
@ -25,7 +23,6 @@ from pytest_tests.helpers.object_access import (
from pytest_tests.testsuites.acl.conftest import Wallets
@pytest.mark.sanity
@pytest.mark.acl
@pytest.mark.acl_extended
class TestEACLContainer(ClusterTestBase):
@ -34,7 +31,7 @@ class TestEACLContainer(ClusterTestBase):
user_wallet = wallets.get_wallet()
storage_nodes = self.cluster.storage_nodes
node_count = len(storage_nodes)
with allure.step("Create eACL public container with full placement rule"):
with reporter.step("Create eACL public container with full placement rule"):
full_placement_rule = f"REP {node_count} IN X CBF 1 SELECT {node_count} FROM * AS X"
cid = create_container(
wallet=user_wallet.wallet_path,
@ -44,7 +41,7 @@ class TestEACLContainer(ClusterTestBase):
endpoint=self.cluster.default_rpc_endpoint,
)
with allure.step("Add test object to container"):
with reporter.step("Add test object to container"):
oid = put_object_to_random_node(
user_wallet.wallet_path, file_path, cid, shell=self.shell, cluster=self.cluster
)
@ -58,6 +55,7 @@ class TestEACLContainer(ClusterTestBase):
yield cid, oid, file_path
@pytest.mark.sanity
@allure.title("Deny operations (role={deny_role.value}, obj_size={object_size})")
@pytest.mark.parametrize("deny_role", [EACLRole.USER, EACLRole.OTHERS])
def test_extended_acl_deny_all_operations(
@ -74,11 +72,8 @@ class TestEACLContainer(ClusterTestBase):
not_deny_role_str = "user" if deny_role == EACLRole.OTHERS else "all others"
cid, object_oids, file_path = eacl_container_with_objects
with allure.step(f"Deny all operations for {deny_role_str} via eACL"):
eacl_deny = [
EACLRule(access=EACLAccess.DENY, role=deny_role, operation=op)
for op in EACLOperation
]
with reporter.step(f"Deny all operations for {deny_role_str} via eACL"):
eacl_deny = [EACLRule(access=EACLAccess.DENY, role=deny_role, operation=op) for op in EACLOperation]
set_eacl(
user_wallet.wallet_path,
cid,
@ -88,10 +83,8 @@ class TestEACLContainer(ClusterTestBase):
)
wait_for_cache_expired()
with allure.step(f"Check only {not_deny_role_str} has full access to container"):
with allure.step(
f"Check {deny_role_str} has not access to any operations with container"
):
with reporter.step(f"Check only {not_deny_role_str} has full access to container"):
with reporter.step(f"Check {deny_role_str} has not access to any operations with container"):
check_no_access_to_container(
deny_role_wallet.wallet_path,
cid,
@ -101,9 +94,7 @@ class TestEACLContainer(ClusterTestBase):
cluster=self.cluster,
)
with allure.step(
f"Check {not_deny_role_wallet} has full access to eACL public container"
):
with reporter.step(f"Check {not_deny_role_wallet} has full access to eACL public container"):
check_full_access_to_container(
not_deny_role_wallet.wallet_path,
cid,
@ -113,11 +104,8 @@ class TestEACLContainer(ClusterTestBase):
cluster=self.cluster,
)
with allure.step(f"Allow all operations for {deny_role_str} via eACL"):
eacl_deny = [
EACLRule(access=EACLAccess.ALLOW, role=deny_role, operation=op)
for op in EACLOperation
]
with reporter.step(f"Allow all operations for {deny_role_str} via eACL"):
eacl_deny = [EACLRule(access=EACLAccess.ALLOW, role=deny_role, operation=op) for op in EACLOperation]
set_eacl(
user_wallet.wallet_path,
cid,
@ -127,7 +115,7 @@ class TestEACLContainer(ClusterTestBase):
)
wait_for_cache_expired()
with allure.step("Check all have full access to eACL public container"):
with reporter.step("Check all have full access to eACL public container"):
check_full_access_to_container(
user_wallet.wallet_path,
cid,
@ -153,7 +141,7 @@ class TestEACLContainer(ClusterTestBase):
other_wallet, other_wallet_allow = wallets.get_wallets_list(EACLRole.OTHERS)[0:2]
cid, object_oids, file_path = eacl_container_with_objects
with allure.step("Deny all operations for others except single wallet via eACL"):
with reporter.step("Deny all operations for others except single wallet via eACL"):
eacl = [
EACLRule(
access=EACLAccess.ALLOW,
@ -162,10 +150,7 @@ class TestEACLContainer(ClusterTestBase):
)
for op in EACLOperation
]
eacl += [
EACLRule(access=EACLAccess.DENY, role=EACLRole.OTHERS, operation=op)
for op in EACLOperation
]
eacl += [EACLRule(access=EACLAccess.DENY, role=EACLRole.OTHERS, operation=op) for op in EACLOperation]
set_eacl(
user_wallet.wallet_path,
cid,
@ -175,8 +160,8 @@ class TestEACLContainer(ClusterTestBase):
)
wait_for_cache_expired()
with allure.step("Check only owner and allowed other have full access to public container"):
with allure.step("Check other has not access to operations with container"):
with reporter.step("Check only owner and allowed other have full access to public container"):
with reporter.step("Check other has not access to operations with container"):
check_no_access_to_container(
other_wallet.wallet_path,
cid,
@ -186,7 +171,7 @@ class TestEACLContainer(ClusterTestBase):
cluster=self.cluster,
)
with allure.step("Check owner has full access to public container"):
with reporter.step("Check owner has full access to public container"):
check_full_access_to_container(
user_wallet.wallet_path,
cid,
@ -196,7 +181,7 @@ class TestEACLContainer(ClusterTestBase):
cluster=self.cluster,
)
with allure.step("Check allowed other has full access to public container"):
with reporter.step("Check allowed other has full access to public container"):
check_full_access_to_container(
other_wallet_allow.wallet_path,
cid,
@ -217,15 +202,9 @@ class TestEACLContainer(ClusterTestBase):
storage_nodes = self.cluster.storage_nodes
storage_node = self.cluster.storage_nodes[0]
with allure.step("Deny all operations for user via eACL"):
eacl_deny = [
EACLRule(access=EACLAccess.DENY, role=EACLRole.USER, operation=op)
for op in EACLOperation
]
eacl_deny += [
EACLRule(access=EACLAccess.DENY, role=EACLRole.OTHERS, operation=op)
for op in EACLOperation
]
with reporter.step("Deny all operations for user via eACL"):
eacl_deny = [EACLRule(access=EACLAccess.DENY, role=EACLRole.USER, operation=op) for op in EACLOperation]
eacl_deny += [EACLRule(access=EACLAccess.DENY, role=EACLRole.OTHERS, operation=op) for op in EACLOperation]
set_eacl(
user_wallet.wallet_path,
cid,
@ -235,11 +214,11 @@ class TestEACLContainer(ClusterTestBase):
)
wait_for_cache_expired()
with allure.step("Drop object to check replication"):
with reporter.step("Drop object to check replication"):
drop_object(storage_node, cid=cid, oid=oid)
storage_wallet_path = storage_node.get_wallet_path()
with allure.step("Wait for dropped object replicated"):
with reporter.step("Wait for dropped object replicated"):
wait_object_replication(
cid,
oid,
@ -249,16 +228,14 @@ class TestEACLContainer(ClusterTestBase):
)
@allure.title("Operations with extended ACL for SYSTEM (obj_size={object_size})")
def test_extended_actions_system(
self, wallets: Wallets, eacl_container_with_objects: tuple[str, list[str], str]
):
def test_extended_actions_system(self, wallets: Wallets, eacl_container_with_objects: tuple[str, list[str], str]):
user_wallet = wallets.get_wallet()
ir_wallet, storage_wallet = wallets.get_wallets_list(role=EACLRole.SYSTEM)[:2]
cid, object_oids, file_path = eacl_container_with_objects
endpoint = self.cluster.default_rpc_endpoint
with allure.step("Check IR and STORAGE rules compliance"):
with reporter.step("Check IR and STORAGE rules compliance"):
assert not can_put_object(
ir_wallet.wallet_path,
cid,
@ -385,15 +362,14 @@ class TestEACLContainer(ClusterTestBase):
wallet_config=storage_wallet.config_path,
)
with allure.step("Deny all operations for SYSTEM via eACL"):
with reporter.step("Deny all operations for SYSTEM via eACL"):
set_eacl(
user_wallet.wallet_path,
cid,
create_eacl(
cid=cid,
rules_list=[
EACLRule(access=EACLAccess.DENY, role=EACLRole.SYSTEM, operation=op)
for op in EACLOperation
EACLRule(access=EACLAccess.DENY, role=EACLRole.SYSTEM, operation=op) for op in EACLOperation
],
shell=self.shell,
),
@ -402,7 +378,7 @@ class TestEACLContainer(ClusterTestBase):
)
wait_for_cache_expired()
with allure.step("Check IR and STORAGE rules compliance with deny eACL"):
with reporter.step("Check IR and STORAGE rules compliance with deny eACL"):
assert not can_put_object(
wallet=ir_wallet.wallet_path,
cid=cid,
@ -536,15 +512,14 @@ class TestEACLContainer(ClusterTestBase):
wallet_config=storage_wallet.config_path,
)
with allure.step("Allow all operations for SYSTEM via eACL"):
with reporter.step("Allow all operations for SYSTEM via eACL"):
set_eacl(
user_wallet.wallet_path,
cid,
create_eacl(
cid=cid,
rules_list=[
EACLRule(access=EACLAccess.ALLOW, role=EACLRole.SYSTEM, operation=op)
for op in EACLOperation
EACLRule(access=EACLAccess.ALLOW, role=EACLRole.SYSTEM, operation=op) for op in EACLOperation
],
shell=self.shell,
),
@ -553,7 +528,7 @@ class TestEACLContainer(ClusterTestBase):
)
wait_for_cache_expired()
with allure.step("Check IR and STORAGE rules compliance with allow eACL"):
with reporter.step("Check IR and STORAGE rules compliance with allow eACL"):
assert not can_put_object(
wallet=ir_wallet.wallet_path,
cid=cid,

148
pytest_tests/testsuites/acl/test_eacl_filters.py
View file

@ -1,12 +1,8 @@
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.acl import (
create_eacl,
form_bearertoken_file,
set_eacl,
wait_for_cache_expired,
)
from frostfs_testlib.steps.acl import create_eacl, form_bearertoken_file, set_eacl, wait_for_cache_expired
from frostfs_testlib.steps.cli.container import create_container, delete_container
from frostfs_testlib.steps.cli.object import put_object_to_random_node
from frostfs_testlib.storage.dataclasses.acl import (
@ -21,15 +17,11 @@ from frostfs_testlib.storage.dataclasses.acl import (
)
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
from pytest_tests.helpers.container_access import (
check_full_access_to_container,
check_no_access_to_container,
)
from pytest_tests.helpers.container_access import check_full_access_to_container, check_no_access_to_container
from pytest_tests.helpers.object_access import can_get_head_object, can_get_object, can_put_object
from pytest_tests.testsuites.acl.conftest import Wallets
@pytest.mark.sanity
@pytest.mark.acl
@pytest.mark.acl_filters
class TestEACLFilters(ClusterTestBase):
@ -46,18 +38,14 @@ class TestEACLFilters(ClusterTestBase):
"x_key": "other_value",
"check_key": "other_value",
}
REQ_EQUAL_FILTER = EACLFilter(
key="check_key", value="check_value", header_type=EACLHeaderType.REQUEST
)
REQ_EQUAL_FILTER = EACLFilter(key="check_key", value="check_value", header_type=EACLHeaderType.REQUEST)
NOT_REQ_EQUAL_FILTER = EACLFilter(
key="check_key",
value="other_value",
match_type=EACLMatchType.STRING_NOT_EQUAL,
header_type=EACLHeaderType.REQUEST,
)
OBJ_EQUAL_FILTER = EACLFilter(
key="check_key", value="check_value", header_type=EACLHeaderType.OBJECT
)
OBJ_EQUAL_FILTER = EACLFilter(key="check_key", value="check_value", header_type=EACLHeaderType.OBJECT)
NOT_OBJ_EQUAL_FILTER = EACLFilter(
key="check_key",
value="other_value",
@ -74,7 +62,7 @@ class TestEACLFilters(ClusterTestBase):
@pytest.fixture(scope="function")
def eacl_container_with_objects(self, wallets: Wallets, file_path: str):
user_wallet = wallets.get_wallet()
with allure.step("Create eACL public container"):
with reporter.step("Create eACL public container"):
cid = create_container(
user_wallet.wallet_path,
basic_acl=PUBLIC_ACL,
@ -82,7 +70,7 @@ class TestEACLFilters(ClusterTestBase):
endpoint=self.cluster.default_rpc_endpoint,
)
with allure.step("Add test objects to container"):
with reporter.step("Add test objects to container"):
objects_with_header = [
put_object_to_random_node(
user_wallet.wallet_path,
@ -120,7 +108,7 @@ class TestEACLFilters(ClusterTestBase):
yield cid, objects_with_header, objects_with_other_header, objects_without_header, file_path
with allure.step("Delete eACL public container"):
with reporter.step("Delete eACL public container"):
delete_container(
user_wallet.wallet_path,
cid,
@ -128,12 +116,9 @@ class TestEACLFilters(ClusterTestBase):
endpoint=self.cluster.default_rpc_endpoint,
)
@allure.title(
"Operations with request filter (match_type={match_type}, obj_size={object_size})"
)
@pytest.mark.parametrize(
"match_type", [EACLMatchType.STRING_EQUAL, EACLMatchType.STRING_NOT_EQUAL]
)
@pytest.mark.sanity
@allure.title("Operations with request filter (match_type={match_type}, obj_size={object_size})")
@pytest.mark.parametrize("match_type", [EACLMatchType.STRING_EQUAL, EACLMatchType.STRING_NOT_EQUAL])
def test_extended_acl_filters_request(
self,
wallets: Wallets,
@ -150,7 +135,7 @@ class TestEACLFilters(ClusterTestBase):
file_path,
) = eacl_container_with_objects
with allure.step("Deny all operations for other with eACL request filter"):
with reporter.step("Deny all operations for other with eACL request filter"):
equal_filter = EACLFilter(**self.REQ_EQUAL_FILTER.__dict__)
equal_filter.match_type = match_type
eacl_deny = [
@ -174,12 +159,8 @@ class TestEACLFilters(ClusterTestBase):
# Filter denies requests where "check_key {match_type} ATTRIBUTE", so when match_type
# is STRING_EQUAL, then requests with "check_key=OTHER_ATTRIBUTE" will be allowed while
# requests with "check_key=ATTRIBUTE" will be denied, and vice versa
allow_headers = (
self.OTHER_ATTRIBUTE if match_type == EACLMatchType.STRING_EQUAL else self.ATTRIBUTE
)
deny_headers = (
self.ATTRIBUTE if match_type == EACLMatchType.STRING_EQUAL else self.OTHER_ATTRIBUTE
)
allow_headers = self.OTHER_ATTRIBUTE if match_type == EACLMatchType.STRING_EQUAL else self.ATTRIBUTE
deny_headers = self.ATTRIBUTE if match_type == EACLMatchType.STRING_EQUAL else self.OTHER_ATTRIBUTE
# We test on 3 groups of objects with various headers,
# but eACL rule should ignore object headers and
# work only based on request headers
@ -188,7 +169,7 @@ class TestEACLFilters(ClusterTestBase):
objects_with_other_header,
objects_without_header,
):
with allure.step("Check other has full access when sending request without headers"):
with reporter.step("Check other has full access when sending request without headers"):
check_full_access_to_container(
other_wallet.wallet_path,
cid,
@ -198,9 +179,7 @@ class TestEACLFilters(ClusterTestBase):
cluster=self.cluster,
)
with allure.step(
"Check other has full access when sending request with allowed headers"
):
with reporter.step("Check other has full access when sending request with allowed headers"):
check_full_access_to_container(
other_wallet.wallet_path,
cid,
@ -211,7 +190,7 @@ class TestEACLFilters(ClusterTestBase):
xhdr=allow_headers,
)
with allure.step("Check other has no access when sending request with denied headers"):
with reporter.step("Check other has no access when sending request with denied headers"):
check_no_access_to_container(
other_wallet.wallet_path,
cid,
@ -222,17 +201,13 @@ class TestEACLFilters(ClusterTestBase):
xhdr=deny_headers,
)
with allure.step(
"Check other has full access when sending request "
"with denied headers and using bearer token"
with reporter.step(
"Check other has full access when sending request " "with denied headers and using bearer token"
):
bearer_other = form_bearertoken_file(
user_wallet.wallet_path,
cid,
[
EACLRule(operation=op, access=EACLAccess.ALLOW, role=EACLRole.OTHERS)
for op in EACLOperation
],
[EACLRule(operation=op, access=EACLAccess.ALLOW, role=EACLRole.OTHERS) for op in EACLOperation],
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
)
@ -247,12 +222,8 @@ class TestEACLFilters(ClusterTestBase):
bearer=bearer_other,
)
@allure.title(
"Operations with deny user headers filter (match_type={match_type}, obj_size={object_size})"
)
@pytest.mark.parametrize(
"match_type", [EACLMatchType.STRING_EQUAL, EACLMatchType.STRING_NOT_EQUAL]
)
@allure.title("Operations with deny user headers filter (match_type={match_type}, obj_size={object_size})")
@pytest.mark.parametrize("match_type", [EACLMatchType.STRING_EQUAL, EACLMatchType.STRING_NOT_EQUAL])
def test_extended_acl_deny_filters_object(
self,
wallets: Wallets,
@ -269,7 +240,7 @@ class TestEACLFilters(ClusterTestBase):
file_path,
) = eacl_container_with_objects
with allure.step("Deny all operations for other with object filter"):
with reporter.step("Deny all operations for other with object filter"):
equal_filter = EACLFilter(**self.OBJ_EQUAL_FILTER.__dict__)
equal_filter.match_type = match_type
eacl_deny = [
@ -290,22 +261,14 @@ class TestEACLFilters(ClusterTestBase):
)
wait_for_cache_expired()
allow_objects = (
objects_with_other_header
if match_type == EACLMatchType.STRING_EQUAL
else objects_with_header
)
deny_objects = (
objects_with_header
if match_type == EACLMatchType.STRING_EQUAL
else objects_with_other_header
)
allow_objects = objects_with_other_header if match_type == EACLMatchType.STRING_EQUAL else objects_with_header
deny_objects = objects_with_header if match_type == EACLMatchType.STRING_EQUAL else objects_with_other_header
# We will attempt requests with various headers,
# but eACL rule should ignore request headers and validate
# only object headers
for xhdr in (self.ATTRIBUTE, self.OTHER_ATTRIBUTE, None):
with allure.step("Check other have full access to objects without attributes"):
with reporter.step("Check other have full access to objects without attributes"):
check_full_access_to_container(
other_wallet.wallet_path,
cid,
@ -316,7 +279,7 @@ class TestEACLFilters(ClusterTestBase):
xhdr=xhdr,
)
with allure.step("Check other have full access to objects without deny attribute"):
with reporter.step("Check other have full access to objects without deny attribute"):
check_full_access_to_container(
other_wallet.wallet_path,
cid,
@ -327,7 +290,7 @@ class TestEACLFilters(ClusterTestBase):
xhdr=xhdr,
)
with allure.step("Check other have no access to objects with deny attribute"):
with reporter.step("Check other have no access to objects with deny attribute"):
with pytest.raises(AssertionError):
assert can_get_head_object(
other_wallet.wallet_path,
@ -348,9 +311,7 @@ class TestEACLFilters(ClusterTestBase):
xhdr=xhdr,
)
with allure.step(
"Check other have access to objects with deny attribute and using bearer token"
):
with reporter.step("Check other have access to objects with deny attribute and using bearer token"):
bearer_other = form_bearertoken_file(
user_wallet.wallet_path,
cid,
@ -376,10 +337,8 @@ class TestEACLFilters(ClusterTestBase):
bearer=bearer_other,
)
allow_attribute = (
self.OTHER_ATTRIBUTE if match_type == EACLMatchType.STRING_EQUAL else self.ATTRIBUTE
)
with allure.step("Check other can PUT objects without denied attribute"):
allow_attribute = self.OTHER_ATTRIBUTE if match_type == EACLMatchType.STRING_EQUAL else self.ATTRIBUTE
with reporter.step("Check other can PUT objects without denied attribute"):
assert can_put_object(
other_wallet.wallet_path,
cid,
@ -388,14 +347,10 @@ class TestEACLFilters(ClusterTestBase):
cluster=self.cluster,
attributes=allow_attribute,
)
assert can_put_object(
other_wallet.wallet_path, cid, file_path, shell=self.shell, cluster=self.cluster
)
assert can_put_object(other_wallet.wallet_path, cid, file_path, shell=self.shell, cluster=self.cluster)
deny_attribute = (
self.ATTRIBUTE if match_type == EACLMatchType.STRING_EQUAL else self.OTHER_ATTRIBUTE
)
with allure.step("Check other can not PUT objects with denied attribute"):
deny_attribute = self.ATTRIBUTE if match_type == EACLMatchType.STRING_EQUAL else self.OTHER_ATTRIBUTE
with reporter.step("Check other can not PUT objects with denied attribute"):
with pytest.raises(AssertionError):
assert can_put_object(
other_wallet.wallet_path,
@ -406,9 +361,7 @@ class TestEACLFilters(ClusterTestBase):
attributes=deny_attribute,
)
with allure.step(
"Check other can PUT objects with denied attribute and using bearer token"
):
with reporter.step("Check other can PUT objects with denied attribute and using bearer token"):
bearer_other_for_put = form_bearertoken_file(
user_wallet.wallet_path,
cid,
@ -432,12 +385,8 @@ class TestEACLFilters(ClusterTestBase):
bearer=bearer_other_for_put,
)
@allure.title(
"Operations with allow eACL user headers filters (match_type={match_type}, obj_size={object_size})"
)
@pytest.mark.parametrize(
"match_type", [EACLMatchType.STRING_EQUAL, EACLMatchType.STRING_NOT_EQUAL]
)
@allure.title("Operations with allow eACL user headers filters (match_type={match_type}, obj_size={object_size})")
@pytest.mark.parametrize("match_type", [EACLMatchType.STRING_EQUAL, EACLMatchType.STRING_NOT_EQUAL])
def test_extended_acl_allow_filters_object(
self,
wallets: Wallets,
@ -454,9 +403,7 @@ class TestEACLFilters(ClusterTestBase):
file_path,
) = eacl_container_with_objects
with allure.step(
"Deny all operations for others except few operations allowed by object filter"
):
with reporter.step("Deny all operations for others except few operations allowed by object filter"):
equal_filter = EACLFilter(**self.OBJ_EQUAL_FILTER.__dict__)
equal_filter.match_type = match_type
eacl = [
@ -491,7 +438,7 @@ class TestEACLFilters(ClusterTestBase):
allow_attribute = self.OTHER_ATTRIBUTE
deny_attribute = self.ATTRIBUTE
with allure.step("Check other cannot get and put objects without attributes"):
with reporter.step("Check other cannot get and put objects without attributes"):
oid = objects_without_header.pop()
with pytest.raises(AssertionError):
assert can_get_head_object(
@ -511,13 +458,9 @@ class TestEACLFilters(ClusterTestBase):
cluster=self.cluster,
)
with pytest.raises(AssertionError):
assert can_put_object(
other_wallet.wallet_path, cid, file_path, shell=self.shell, cluster=self.cluster
)
assert can_put_object(other_wallet.wallet_path, cid, file_path, shell=self.shell, cluster=self.cluster)
with allure.step(
"Check other can get and put objects without attributes and using bearer token"
):
with reporter.step("Check other can get and put objects without attributes and using bearer token"):
bearer_other = form_bearertoken_file(
user_wallet.wallet_path,
cid,
@ -558,7 +501,7 @@ class TestEACLFilters(ClusterTestBase):
bearer=bearer_other,
)
with allure.step("Check other can get objects with attributes matching the filter"):
with reporter.step("Check other can get objects with attributes matching the filter"):
oid = allow_objects.pop()
assert can_get_head_object(
other_wallet.wallet_path,
@ -584,7 +527,7 @@ class TestEACLFilters(ClusterTestBase):
attributes=allow_attribute,
)
with allure.step("Check other cannot get objects without attributes matching the filter"):
with reporter.step("Check other cannot get objects without attributes matching the filter"):
with pytest.raises(AssertionError):
assert can_get_head_object(
other_wallet.wallet_path,
@ -612,9 +555,8 @@ class TestEACLFilters(ClusterTestBase):
cluster=self.cluster,
)
with allure.step(
"Check other can get objects without attributes matching the filter "
"and using bearer token"
with reporter.step(
"Check other can get objects without attributes matching the filter " "and using bearer token"
):
oid = deny_objects.pop()
assert can_get_head_object(

159
pytest_tests/testsuites/conftest.py
View file

@ -1,15 +1,19 @@
import logging
import os
import random
import shutil
from datetime import datetime, timedelta, timezone
from importlib.metadata import entry_points
from typing import Optional
import allure
import pytest
import yaml
from dateutil import parser
from frostfs_testlib import plugins, reporter
from frostfs_testlib.healthcheck.interfaces import Healthcheck
from frostfs_testlib.hosting import Hosting
from frostfs_testlib.reporter import AllureHandler, get_reporter
from frostfs_testlib.reporter import AllureHandler, StepsLogger
from frostfs_testlib.resources.common import (
ASSETS_DIR,
COMPLEX_OBJECT_CHUNKS_COUNT,
@ -17,12 +21,12 @@ from frostfs_testlib.resources.common import (
DEFAULT_WALLET_PASS,
SIMPLE_OBJECT_SIZE,
)
from frostfs_testlib.s3.interfaces import S3ClientWrapper, VersioningStatus
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper, VersioningStatus
from frostfs_testlib.shell import LocalShell, Shell
from frostfs_testlib.steps.cli.container import list_containers
from frostfs_testlib.steps.cli.object import get_netmap_netinfo
from frostfs_testlib.steps.node_management import storage_node_healthcheck
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage import get_service_registry
from frostfs_testlib.storage.cluster import Cluster, ClusterNode
from frostfs_testlib.storage.controllers.cluster_state_controller import ClusterStateController
from frostfs_testlib.storage.dataclasses.frostfs_services import StorageNode
@ -44,17 +48,55 @@ def pytest_configure(config: pytest.Config):
config.option.markexpr = f"logs_after_session or ({markers})"
number_key = pytest.StashKey[str]()
start_time = pytest.StashKey[int]()
test_outcome = pytest.StashKey[str]()
# pytest hook. Do not rename
def pytest_collection_modifyitems(items: list[pytest.Item]):
# Make network tests last based on @pytest.mark.node_mgmt and logs_test to be latest
def priority(item: pytest.Item) -> int:
is_node_mgmt_test = 1 if item.get_closest_marker("node_mgmt") else 0
is_logs_check_test = 100 if item.get_closest_marker("logs_after_session") else 0
return is_node_mgmt_test + is_logs_check_test
is_system_time_test = 10 if item.get_closest_marker("system_time") else 0
return is_node_mgmt_test + is_logs_check_test + is_system_time_test
items.sort(key=lambda item: priority(item))
# pytest hook. Do not rename
def pytest_collection_finish(session: pytest.Session):
items_total = len(session.items)
for number, item in enumerate(session.items, 1):
item.stash[number_key] = f"[{number}/{items_total}]"
item.stash[test_outcome] = ""
item.stash[start_time] = 0
# pytest hook. Do not rename
def pytest_runtest_setup(item: pytest.Item):
item.stash[start_time] = int(datetime.now().timestamp())
logger.info(f"STARTED {item.stash[number_key]}: {item.name}")
# pytest hook. Do not rename
def pytest_runtest_makereport(item: pytest.Item, call: pytest.CallInfo):
if call.excinfo is not None:
if call.excinfo.typename == "Skipped":
item.stash[start_time] = int(datetime.now().timestamp())
item.stash[test_outcome] += f"SKIPPED on {call.when}; "
else:
item.stash[test_outcome] += f"FAILED on {call.when}; "
if call.when == "teardown":
duration = int(datetime.now().timestamp()) - item.stash[start_time]
if not item.stash[test_outcome]:
outcome = "PASSED "
else:
outcome = item.stash[test_outcome]
logger.info(f"ENDED {item.stash[number_key]}: {item.name}: {outcome}(duration={duration}s)")
# pytest hook. Do not rename
def pytest_generate_tests(metafunc: pytest.Metafunc):
if (
TEST_CYCLES_COUNT <= 1
@ -73,8 +115,17 @@ def pytest_generate_tests(metafunc: pytest.Metafunc):
@pytest.fixture(scope="session")
def configure_testlib():
get_reporter().register_handler(AllureHandler())
reporter.get_reporter().register_handler(AllureHandler())
reporter.get_reporter().register_handler(StepsLogger())
logging.getLogger("paramiko").setLevel(logging.INFO)
# Register Services for cluster
registry = get_service_registry()
services = entry_points(group="frostfs.testlib.services")
for svc in services:
registry.register_service(svc.name, svc.load())
yield
@ -144,7 +195,10 @@ def complex_object_size(max_object_size: int) -> ObjectSize:
# By default we want all tests to be executed with both object sizes
# This can be overriden in choosen tests if needed
@pytest.fixture(scope="session", params=["simple", "complex"])
@pytest.fixture(
scope="session",
params=[pytest.param("simple", marks=pytest.mark.simple), pytest.param("complex", marks=pytest.mark.complex)],
)
def object_size(
simple_object_size: ObjectSize, complex_object_size: ObjectSize, request: pytest.FixtureRequest
) -> ObjectSize:
@ -171,7 +225,7 @@ def cluster(temp_directory: str, hosting: Hosting, client_shell: Shell) -> Clust
yield cluster
@allure.step("[Class]: Provide S3 policy")
@reporter.step("[Class]: Provide S3 policy")
@pytest.fixture(scope="class")
def s3_policy(request: pytest.FixtureRequest):
policy = None
@ -182,13 +236,29 @@ def s3_policy(request: pytest.FixtureRequest):
@pytest.fixture(scope="session")
def cluster_state_controller(client_shell: Shell, cluster: Cluster) -> ClusterStateController:
controller = ClusterStateController(client_shell, cluster)
@allure.title("[Session] Create healthcheck object")
def healthcheck(cluster: Cluster) -> Healthcheck:
healthcheck_cls = plugins.load_plugin(
"frostfs.testlib.healthcheck", cluster.cluster_nodes[0].host.config.healthcheck_plugin_name
)
return healthcheck_cls()
@pytest.fixture(scope="session")
def cluster_state_controller(client_shell: Shell, cluster: Cluster, healthcheck: Healthcheck) -> ClusterStateController:
controller = ClusterStateController(client_shell, cluster, healthcheck)
yield controller
@allure.step("[Class]: Create S3 client")
@pytest.fixture(scope="class")
@reporter.step("[Class]: Create S3 client")
@pytest.fixture(
scope="class",
params=[
pytest.param(AwsCliClient, marks=pytest.mark.aws),
pytest.param(Boto3ClientWrapper, marks=pytest.mark.boto3),
],
)
def s3_client(
default_wallet: str,
client_shell: Shell,
@ -223,35 +293,43 @@ def versioning_status(request: pytest.FixtureRequest) -> VersioningStatus:
return VersioningStatus.UNDEFINED
@allure.step("Create/delete bucket")
@reporter.step("Create/delete bucket")
@pytest.fixture
def bucket(s3_client: S3ClientWrapper, versioning_status: VersioningStatus):
def bucket(s3_client: S3ClientWrapper, versioning_status: VersioningStatus, request: pytest.FixtureRequest):
bucket_name = s3_client.create_bucket()
if versioning_status:
s3_helper.set_bucket_versioning(s3_client, bucket_name, versioning_status)
yield bucket_name
if "sanity" not in request.config.option.markexpr:
s3_helper.delete_bucket_with_objects(s3_client, bucket_name)
@allure.step("Create two buckets")
@reporter.step("Create two buckets")
@pytest.fixture
def two_buckets(s3_client: S3ClientWrapper):
def two_buckets(s3_client: S3ClientWrapper, request: pytest.FixtureRequest):
bucket_1 = s3_client.create_bucket()
bucket_2 = s3_client.create_bucket()
yield bucket_1, bucket_2
if "sanity" not in request.config.option.markexpr:
for bucket_name in [bucket_1, bucket_2]:
s3_helper.delete_bucket_with_objects(s3_client, bucket_name)
@allure.step("[Autouse/Session] Check binary versions")
@reporter.step("[Autouse/Session] Check binary versions")
@pytest.fixture(scope="session", autouse=True)
def check_binary_versions(hosting: Hosting, client_shell: Shell, request: pytest.FixtureRequest):
local_versions = version_utils.get_local_binaries_versions(client_shell)
remote_versions = version_utils.get_remote_binaries_versions(hosting)
all_versions = {**local_versions, **remote_versions}
all_versions = {
**local_versions,
**{binary_name: binary["version"] for binary_name, binary in remote_versions.items()},
}
environment_dir = request.config.getoption("--alluredir")
if not environment_dir:
@ -261,23 +339,23 @@ def check_binary_versions(hosting: Hosting, client_shell: Shell, request: pytest
env_utils.save_env_properties(file_path, all_versions)
@allure.step("Prepare tmp directory")
@reporter.step("Prepare tmp directory")
@pytest.fixture(scope="session")
def temp_directory():
with allure.step("Prepare tmp directory"):
def temp_directory(configure_testlib):
with reporter.step("Prepare tmp directory"):
full_path = os.path.join(os.getcwd(), ASSETS_DIR)
shutil.rmtree(full_path, ignore_errors=True)
os.mkdir(full_path)
yield full_path
with allure.step("Remove tmp directory"):
with reporter.step("Remove tmp directory"):
shutil.rmtree(full_path)
@allure.step("[Autouse/Session] Test session start time")
@reporter.step("[Autouse/Session] Test session start time")
@pytest.fixture(scope="session", autouse=True)
def session_start_time():
def session_start_time(configure_testlib):
start_time = datetime.utcnow()
return start_time
@ -285,23 +363,22 @@ def session_start_time():
@allure.title("[Autouse/Session] After deploy healthcheck")
@pytest.fixture(scope="session", autouse=True)
def after_deploy_healthcheck(cluster: Cluster):
with allure.step("Wait for cluster readiness after deploy"):
with reporter.step("Wait for cluster readiness after deploy"):
parallel(readiness_on_node, cluster.cluster_nodes)
SERVICE_ACTIVE_TIME = 20
@wait_for_success(60 * SERVICE_ACTIVE_TIME * 3, 60)
@allure.step("Check readiness on node {cluster_node}")
@wait_for_success(60 * SERVICE_ACTIVE_TIME * 3, 60, title="Wait for {cluster_node} readiness")
def readiness_on_node(cluster_node: ClusterNode):
# TODO: Move to healtcheck classes
svc_name = cluster_node.service(StorageNode).get_service_systemctl_name()
with allure.step(f"Check service {svc_name} is active"):
with reporter.step(f"Check service {svc_name} is active"):
result = cluster_node.host.get_shell().exec(f"systemctl is-active {svc_name}")
assert "active" == result.stdout.strip(), f"Service {svc_name} should be in active state"
with allure.step(f"Check service {svc_name} is active more than {SERVICE_ACTIVE_TIME} minutes"):
with reporter.step(f"Check service {svc_name} is active more than {SERVICE_ACTIVE_TIME} minutes"):
result = cluster_node.host.get_shell().exec(
f"systemctl show {svc_name} --property ActiveEnterTimestamp | cut -d '=' -f 2"
)
@ -319,30 +396,22 @@ def readiness_on_node(cluster_node: ClusterNode):
@allure.title("[Autouse/Test] Run health check for all nodes")
@pytest.fixture(autouse=True)
def run_health_check(cluster: Cluster, request: pytest.FixtureRequest):
def run_health_check(healthcheck: Healthcheck, cluster: Cluster, request: pytest.FixtureRequest):
if request.node.get_closest_marker("no_healthcheck"):
# Skip healthcheck for tests marked with no_healthcheck
return
parallel(healthcheck_on_node, cluster.cluster_nodes)
parallel(healthcheck.storage_healthcheck, cluster.cluster_nodes)
@allure.title("Run health check for {cluster_node}")
def healthcheck_on_node(cluster_node: ClusterNode):
health_check = storage_node_healthcheck(cluster_node.storage_node)
assert (
health_check.health_status == "READY" and health_check.network_status == "ONLINE"
), f"Node {cluster_node} is not healthy"
@allure.step("Prepare wallet and deposit")
@reporter.step("Prepare wallet and deposit")
@pytest.fixture(scope="session")
def default_wallet(wallet_factory: WalletFactory) -> str:
wallet = wallet_factory.create_wallet(password=DEFAULT_WALLET_PASS)
allure.attach.file(wallet.path, os.path.basename(wallet.path), allure.attachment_type.JSON)
reporter.attach(wallet.path, os.path.basename(wallet.path))
return wallet.path
@allure.step("[Class]: Container placement policy for keys")
@reporter.step("[Class]: Container placement policy for keys")
@pytest.fixture(scope="class")
def auth_container_placement_policy(cluster: Cluster, request: pytest.FixtureRequest):
placeholders = {
@ -355,3 +424,11 @@ def auth_container_placement_policy(cluster: Cluster, request: pytest.FixtureReq
for key, value in placeholders.items():
placement_policy = placement_policy.replace(key, str(value))
return placement_policy
@pytest.fixture()
@allure.title("Select random node for testing")
def node_under_test(cluster: Cluster) -> ClusterNode:
selected_node = random.choice(cluster.cluster_nodes)
reporter.attach(f"{selected_node}", "Selected node")
return selected_node

36
pytest_tests/testsuites/container/test_container.py
View file

@ -2,6 +2,7 @@ import json
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PRIVATE_ACL_F
from frostfs_testlib.steps.cli.container import (
create_container,
@ -38,9 +39,7 @@ class TestContainer(ClusterTestBase):
endpoint=self.cluster.default_rpc_endpoint,
)
containers = list_containers(
wallet, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint
)
containers = list_containers(wallet, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint)
assert cid in containers, f"Expected container {cid} in containers: {containers}"
container_info: str = get_container(
@ -50,9 +49,7 @@ class TestContainer(ClusterTestBase):
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
)
container_info = (
container_info.casefold()
) # To ignore case when comparing with expected values
container_info = container_info.casefold() # To ignore case when comparing with expected values
info_to_check = {
f"basic ACL: {PRIVATE_ACL_F} (private)",
@ -62,7 +59,7 @@ class TestContainer(ClusterTestBase):
if name:
info_to_check.add(f"Name={name}")
with allure.step("Check container has correct information"):
with reporter.step("Check container has correct information"):
expected_policy = placement_rule.casefold()
actual_policy = placement_policy_from_container(container_info)
assert (
@ -71,11 +68,9 @@ class TestContainer(ClusterTestBase):
for info in info_to_check:
expected_info = info.casefold()
assert (
expected_info in container_info
), f"Expected {expected_info} in container info:\n{container_info}"
assert expected_info in container_info, f"Expected {expected_info} in container info:\n{container_info}"
with allure.step("Delete container and check it was deleted"):
with reporter.step("Delete container and check it was deleted"):
delete_container(
wallet,
cid,
@ -84,9 +79,7 @@ class TestContainer(ClusterTestBase):
await_mode=True,
)
self.tick_epoch()
wait_for_container_deletion(
wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint
)
wait_for_container_deletion(wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint)
@allure.title("Parallel container creation and deletion")
def test_container_creation_deletion_parallel(self, default_wallet: str):
@ -95,7 +88,7 @@ class TestContainer(ClusterTestBase):
placement_rule = "REP 2 IN X CBF 1 SELECT 2 FROM * AS X"
cids: list[str] = []
with allure.step(f"Create {containers_count} containers"):
with reporter.step(f"Create {containers_count} containers"):
for _ in range(containers_count):
cids.append(
create_container(
@ -108,7 +101,7 @@ class TestContainer(ClusterTestBase):
)
)
with allure.step("Wait for containers occur in container list"):
with reporter.step("Wait for containers occur in container list"):
for cid in cids:
wait_for_container_creation(
wallet,
@ -118,12 +111,9 @@ class TestContainer(ClusterTestBase):
endpoint=self.cluster.default_rpc_endpoint,
)
with allure.step("Delete containers and check they were deleted"):
with reporter.step("Delete containers and check they were deleted"):
for cid in cids:
delete_container(
wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint
)
delete_container(wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint)
self.tick_epoch()
wait_for_container_deletion(
wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint
)
for cid in cids:
wait_for_container_deletion(wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint)

656
pytest_tests/testsuites/container/test_policy.py
View file

File diff suppressed because it is too large Load diff

22
pytest_tests/testsuites/failovers/conftest.py
View file

@ -1,18 +1,12 @@
import random
import os
from datetime import datetime
import allure
import pytest
from frostfs_testlib.storage.cluster import Cluster, ClusterNode
from frostfs_testlib.storage.cluster import ClusterNode
from frostfs_testlib.storage.controllers import ShardsWatcher
@pytest.fixture()
@allure.title("Select random node for testing")
def node_under_test(cluster: Cluster) -> ClusterNode:
selected_node = random.choice(cluster.cluster_nodes)
allure.attach(f"{selected_node}", "Selected node", allure.attachment_type.TEXT)
return selected_node
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.utils.file_utils import generate_file
@pytest.fixture()
@ -27,3 +21,11 @@ def shards_watcher(node_under_test: ClusterNode) -> ShardsWatcher:
def test_start_time() -> datetime:
start_time = datetime.utcnow()
return start_time
@pytest.fixture()
@allure.title("Generate simple size file")
def simple_file(simple_object_size: ObjectSize) -> str:
path_file = generate_file(size=simple_object_size.value)
yield path_file
os.remove(path_file)

79
pytest_tests/testsuites/failovers/test_discrete_time.py Normal file
View file

@ -0,0 +1,79 @@
import datetime
from time import sleep
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.common import MORPH_BLOCK_TIME
from frostfs_testlib.steps.cli.object import neo_go_query_height
from frostfs_testlib.storage.controllers import ClusterStateController
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
from frostfs_testlib.utils import datetime_utils
@pytest.mark.time
@pytest.mark.failover
class TestTime(ClusterTestBase):
@reporter.step("Neo-go should continue to release blocks")
def check_nodes_block(self, cluster_state_controller: ClusterStateController):
count_blocks = {}
with reporter.step("Get current block id"):
for cluster_node in self.cluster.cluster_nodes:
cluster_state_controller.get_node_date(cluster_node)
count_blocks[cluster_node] = neo_go_query_height(
shell=cluster_node.host.get_shell(), endpoint=cluster_node.morph_chain.get_http_endpoint()
)["Latest block"]
with reporter.step("Wait for 3 blocks"):
sleep(datetime_utils.parse_time(MORPH_BLOCK_TIME) * 3)
with reporter.step("Current block id should be higher than before"):
for cluster_node in self.cluster.cluster_nodes:
shell = cluster_node.host.get_shell()
now_block = neo_go_query_height(shell=shell, endpoint=cluster_node.morph_chain.get_http_endpoint())[
"Latest block"
]
assert count_blocks[cluster_node] < now_block
@pytest.fixture()
def node_time_synchronizer(self, cluster_state_controller: ClusterStateController) -> None:
cluster_state_controller.set_sync_date_all_nodes(status="inactive")
yield
cluster_state_controller.set_sync_date_all_nodes(status="active")
@allure.title("Changing hardware and system time")
def test_system_time(self, cluster_state_controller: ClusterStateController, node_time_synchronizer: None):
cluster_nodes = self.cluster.cluster_nodes
timezone_utc = datetime.timezone.utc
node_1, node_2, node_3 = cluster_nodes[0:3]
with reporter.step("On node 1, move the system time forward by 5 days"):
cluster_state_controller.change_node_date(
node_1, (datetime.datetime.now(timezone_utc) + datetime.timedelta(days=5))
)
self.check_nodes_block(cluster_state_controller)
with reporter.step("On node 2, move the system time back 5 days."):
cluster_state_controller.change_node_date(
node_2, (datetime.datetime.now(timezone_utc) - datetime.timedelta(days=5))
)
self.check_nodes_block(cluster_state_controller)
with reporter.step("On node 3, move the system time forward by 10 days"):
cluster_state_controller.change_node_date(
node_3, (datetime.datetime.now(timezone_utc) + datetime.timedelta(days=10))
)
self.check_nodes_block(cluster_state_controller)
with reporter.step("Return the time on all nodes to the current one"):
for cluster_node in self.cluster.cluster_nodes:
cluster_state_controller.restore_node_date(cluster_node)
self.check_nodes_block(cluster_state_controller)
with reporter.step("Reboot all nodes"):
cluster_state_controller.shutdown_cluster(mode="soft")
cluster_state_controller.start_stopped_hosts()
self.check_nodes_block(cluster_state_controller)

101
pytest_tests/testsuites/failovers/test_failover_network.py
View file

@ -5,8 +5,9 @@ from time import sleep
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.healthcheck.interfaces import Healthcheck
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.s3 import AwsCliClient
from frostfs_testlib.steps.cli.container import create_container
from frostfs_testlib.steps.cli.object import (
get_object,
@ -21,7 +22,8 @@ from frostfs_testlib.storage.controllers import ClusterStateController
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.storage.dataclasses.storage_object_info import Interfaces, StorageObjectInfo
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
from frostfs_testlib.utils.failover_utils import wait_all_storage_nodes_returned, wait_object_replication
from frostfs_testlib.testing.parallel import parallel
from frostfs_testlib.utils.failover_utils import wait_object_replication
from frostfs_testlib.utils.file_utils import generate_file, get_file_hash
logger = logging.getLogger("NeoLogger")
@ -38,26 +40,21 @@ OBJECT_ATTRIBUTES = [
]
def pytest_generate_tests(metafunc: pytest.Metafunc):
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize("s3_client", [AwsCliClient], indirect=True)
@pytest.mark.failover
@pytest.mark.failover_network
class TestFailoverNetwork(ClusterTestBase):
@pytest.fixture(autouse=True)
@allure.title("Restore network")
def restore_network(self, cluster_state_controller: ClusterStateController):
def restore_network(self, healthcheck: Healthcheck, cluster_state_controller: ClusterStateController):
yield
with allure.step(f"Count blocked nodes {len(blocked_nodes)}"):
with reporter.step(f"Count blocked nodes {len(blocked_nodes)}"):
not_empty = len(blocked_nodes) != 0
for node in list(blocked_nodes):
with allure.step(f"Restore network for {node}"):
with reporter.step(f"Restore network for {node}"):
cluster_state_controller.restore_traffic(mode="ports", node=node)
blocked_nodes.remove(node)
if not_empty:
wait_all_storage_nodes_returned(self.shell, self.cluster)
parallel(healthcheck.storage_healthcheck, self.cluster.cluster_nodes)
@pytest.fixture()
@allure.title("Restore drop traffic to system")
@ -83,7 +80,7 @@ class TestFailoverNetwork(ClusterTestBase):
file_path = generate_file(simple_object_size.value)
file_hash = get_file_hash(file_path)
with allure.step("Create container"):
with reporter.step("Create container"):
placement_rule = "REP 1 CBF 1"
cid = create_container(
wallet=default_wallet,
@ -95,7 +92,7 @@ class TestFailoverNetwork(ClusterTestBase):
storage_objects = []
with allure.step("Put object"):
with reporter.step("Put object"):
for attribute in OBJECT_ATTRIBUTES:
oid = put_object_to_random_node(
wallet=default_wallet,
@ -154,7 +151,7 @@ class TestFailoverNetwork(ClusterTestBase):
excluded_nodes = []
for node in nodes_to_block:
with allure.step(f"Block incoming traffic at node {node} on port {PORTS_TO_BLOCK}"):
with reporter.step(f"Block incoming traffic at node {node} on port {PORTS_TO_BLOCK}"):
block_node = [
cluster_node for cluster_node in self.cluster.cluster_nodes if cluster_node.storage_node == node
]
@ -167,7 +164,7 @@ class TestFailoverNetwork(ClusterTestBase):
ports=PORTS_TO_BLOCK,
)
with allure.step(f"Check object is not stored on node {node}"):
with reporter.step(f"Check object is not stored on node {node}"):
new_nodes = wait_object_replication(
cid,
oid,
@ -177,12 +174,12 @@ class TestFailoverNetwork(ClusterTestBase):
)
assert node not in new_nodes
with allure.step("Check object data is not corrupted"):
with reporter.step("Check object data is not corrupted"):
got_file_path = get_object(wallet, cid, oid, endpoint=new_nodes[0].get_rpc_endpoint(), shell=self.shell)
assert get_file_hash(source_file_path) == get_file_hash(got_file_path)
for node in nodes_to_block:
with allure.step(f"Unblock incoming traffic at host {node} on port {PORTS_TO_BLOCK}"):
with reporter.step(f"Unblock incoming traffic at host {node} on port {PORTS_TO_BLOCK}"):
cluster_state_controller.restore_traffic(mode="ports", node=node)
block_node = [
cluster_node for cluster_node in self.cluster.cluster_nodes if cluster_node.storage_node == node
@ -190,7 +187,7 @@ class TestFailoverNetwork(ClusterTestBase):
blocked_nodes.remove(*block_node)
sleep(wakeup_node_timeout)
with allure.step("Check object data is not corrupted"):
with reporter.step("Check object data is not corrupted"):
new_nodes = wait_object_replication(cid, oid, 2, shell=self.shell, nodes=self.cluster.storage_nodes)
got_file_path = get_object(wallet, cid, oid, shell=self.shell, endpoint=new_nodes[0].get_rpc_endpoint())
@ -208,7 +205,7 @@ class TestFailoverNetwork(ClusterTestBase):
):
storage_object = storage_objects[0]
with allure.step("Search nodes with object"):
with reporter.step("Search nodes with object"):
nodes_with_object = get_object_nodes(
cluster=self.cluster,
wallet=default_wallet,
@ -218,20 +215,20 @@ class TestFailoverNetwork(ClusterTestBase):
endpoint=self.cluster.default_rpc_endpoint,
)
with allure.step("Get data interface to node"):
with reporter.step("Get data interface to node"):
config_interfaces = list(nodes_with_object[0].host.config.interfaces.keys())
with allure.step(f"Get data in {config_interfaces}"):
with reporter.step(f"Get data in {config_interfaces}"):
data_interfaces = [interface for interface in config_interfaces if "data" in interface]
with allure.step("Block data interfaces for node"):
with reporter.step("Block data interfaces for node"):
for interface in data_interfaces:
cluster_state_controller.down_interface(nodes=nodes_with_object, interface=interface)
with allure.step("Tick epoch and wait 2 block"):
with reporter.step("Tick epoch and wait 2 block"):
nodes_without_an_object = list(set(self.cluster.cluster_nodes) - set(nodes_with_object))
self.tick_epochs(1, alive_node=nodes_without_an_object[0].storage_node, wait_block=2)
with allure.step("Get object for target nodes to data interfaces, expect false"):
with reporter.step("Get object for target nodes to data interfaces, expect false"):
with pytest.raises(RuntimeError, match="return code: 1"):
get_object(
wallet=default_wallet,
@ -241,7 +238,7 @@ class TestFailoverNetwork(ClusterTestBase):
endpoint=nodes_with_object[0].storage_node.get_rpc_endpoint(),
)
with allure.step(f"Get object others nodes, expect true"):
with reporter.step(f"Get object others nodes, expect true"):
input_file = get_object(
wallet=default_wallet,
cid=storage_object.cid,
@ -251,7 +248,7 @@ class TestFailoverNetwork(ClusterTestBase):
)
file_wait_delete.append(input_file)
with allure.step("Restore interface and tick 1 epoch, wait 2 block"):
with reporter.step("Restore interface and tick 1 epoch, wait 2 block"):
cluster_state_controller.restore_interfaces()
self.tick_epochs(1, alive_node=nodes_without_an_object[0].storage_node, wait_block=2)
@ -268,7 +265,7 @@ class TestFailoverNetwork(ClusterTestBase):
):
storage_object = storage_objects[0]
with allure.step("Search nodes with object"):
with reporter.step("Search nodes with object"):
nodes_with_object = get_object_nodes(
cluster=self.cluster,
wallet=default_wallet,
@ -278,20 +275,20 @@ class TestFailoverNetwork(ClusterTestBase):
endpoint=self.cluster.default_rpc_endpoint,
)
with allure.step("Get internal interface to node"):
with reporter.step("Get internal interface to node"):
config_interfaces = list(nodes_with_object[0].host.config.interfaces.keys())
with allure.step(f"Get internal in {config_interfaces}"):
with reporter.step(f"Get internal in {config_interfaces}"):
internal_interfaces = [interface for interface in config_interfaces if "internal" in interface]
with allure.step("Block internal interfaces for node"):
with reporter.step("Block internal interfaces for node"):
for interface in internal_interfaces:
cluster_state_controller.down_interface(nodes=nodes_with_object, interface=interface)
with allure.step("Tick epoch and wait 2 block"):
with reporter.step("Tick epoch and wait 2 block"):
nodes_without_an_object = list(set(self.cluster.cluster_nodes) - set(nodes_with_object))
self.tick_epochs(1, alive_node=nodes_without_an_object[0].storage_node, wait_block=2)
with allure.step("Get object others node, expect false"):
with reporter.step("Get object others node, expect false"):
with pytest.raises(RuntimeError, match="return code: 1"):
get_object(
wallet=default_wallet,
@ -301,7 +298,7 @@ class TestFailoverNetwork(ClusterTestBase):
endpoint=nodes_without_an_object[0].storage_node.get_rpc_endpoint(),
)
with allure.step("Put object, others node, expect false"):
with reporter.step("Put object, others node, expect false"):
with pytest.raises(RuntimeError, match="return code: 1"):
put_object(
wallet=default_wallet,
@ -311,7 +308,7 @@ class TestFailoverNetwork(ClusterTestBase):
endpoint=nodes_without_an_object[0].storage_node.get_rpc_endpoint(),
)
with allure.step(f"Get object nodes with object, expect true"):
with reporter.step(f"Get object nodes with object, expect true"):
input_file = get_object(
wallet=default_wallet,
cid=storage_object.cid,
@ -321,7 +318,7 @@ class TestFailoverNetwork(ClusterTestBase):
)
file_wait_delete.append(input_file)
with allure.step(f"Put object nodes with object, expect true"):
with reporter.step(f"Put object nodes with object, expect true"):
temp_file_path = generate_file(simple_object_size.value)
_ = put_object(
wallet=default_wallet,
@ -331,7 +328,7 @@ class TestFailoverNetwork(ClusterTestBase):
endpoint=nodes_with_object[0].storage_node.get_rpc_endpoint(),
)
file_wait_delete.append(temp_file_path)
with allure.step("Restore interface and tick 1 epoch, wait 2 block"):
with reporter.step("Restore interface and tick 1 epoch, wait 2 block"):
cluster_state_controller.restore_interfaces()
self.tick_epochs(1, alive_node=nodes_without_an_object[0].storage_node, wait_block=2)
@ -354,13 +351,13 @@ class TestFailoverNetwork(ClusterTestBase):
other_interface: Interfaces,
):
cluster_nodes = self.cluster.cluster_nodes
with allure.step(f"Block {block_interface.value} interfaces"):
with reporter.step(f"Block {block_interface.value} interfaces"):
cluster_state_controller.down_interface(cluster_nodes, block_interface.value)
with allure.step("Tick 1 epoch and wait 2 block for sync all nodes"):
with reporter.step("Tick 1 epoch and wait 2 block for sync all nodes"):
self.tick_epochs(1, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with allure.step("Create container"):
with reporter.step("Create container"):
cid = create_container(
wallet=default_wallet,
shell=self.shell,
@ -368,7 +365,7 @@ class TestFailoverNetwork(ClusterTestBase):
rule="REP 4 CBF 1",
)
with allure.step("Put object"):
with reporter.step("Put object"):
file_path = generate_file(simple_object_size.value)
file_wait_delete.append(file_path)
@ -380,7 +377,7 @@ class TestFailoverNetwork(ClusterTestBase):
endpoint=f"{cluster_nodes[0].get_data_interface(other_interface.value)[0]}:8080",
)
with allure.step("Get object"):
with reporter.step("Get object"):
file_get_path = get_object(
wallet=default_wallet,
cid=cid,
@ -390,7 +387,7 @@ class TestFailoverNetwork(ClusterTestBase):
)
file_wait_delete.append(file_get_path)
with allure.step("Restore interfaces all nodes"):
with reporter.step("Restore interfaces all nodes"):
cluster_state_controller.restore_interfaces()
self.tick_epochs(1, alive_node=cluster_nodes[0].storage_node, wait_block=2)
@ -411,19 +408,19 @@ class TestFailoverNetwork(ClusterTestBase):
cluster_nodes = self.cluster.cluster_nodes
latest_block = {}
with allure.step("Get block all nodes"):
with reporter.step("Get block all nodes"):
for cluster_node in cluster_nodes:
latest_block[cluster_node] = neo_go_query_height(
shell=cluster_node.host.get_shell(), endpoint=cluster_node.morph_chain.get_http_endpoint()
)
with allure.step(f"Block {interface} interfaces"):
with reporter.step(f"Block {interface} interfaces"):
cluster_state_controller.down_interface(cluster_nodes, interface.value)
with allure.step("Tick 1 epoch and wait 2 block for sync all nodes"):
with reporter.step("Tick 1 epoch and wait 2 block for sync all nodes"):
self.tick_epochs(1, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with allure.step("Create container"):
with reporter.step("Create container"):
cid = create_container(
wallet=default_wallet,
shell=self.shell,
@ -431,7 +428,7 @@ class TestFailoverNetwork(ClusterTestBase):
rule="REP 4 CBF 1",
)
with allure.step(f"Put object, after down {interface}"):
with reporter.step(f"Put object, after down {interface}"):
file_path = generate_file(simple_object_size.value)
file_wait_delete.append(file_path)
@ -443,7 +440,7 @@ class TestFailoverNetwork(ClusterTestBase):
endpoint=self.cluster.default_rpc_endpoint,
)
with allure.step("Get object"):
with reporter.step("Get object"):
file_get_path = get_object(
wallet=default_wallet,
cid=cid,
@ -455,19 +452,19 @@ class TestFailoverNetwork(ClusterTestBase):
now_block = {}
with allure.step("Get actual block"):
with reporter.step("Get actual block"):
for cluster_node in cluster_nodes:
now_block[cluster_node] = neo_go_query_height(
shell=cluster_node.host.get_shell(), endpoint=cluster_node.morph_chain.get_http_endpoint()
)
with allure.step(f"Compare block"):
with reporter.step(f"Compare block"):
for cluster_node, items in now_block.items():
with allure.step(
with reporter.step(
f"Node - {cluster_node.host_ip}, old block - {latest_block[cluster_node]['Latest block']}, "
f"now block - {now_block[cluster_node]['Latest block']}"
):
assert latest_block[cluster_node]["Latest block"] < now_block[cluster_node]["Latest block"]
with allure.step("Restore interfaces all nodes"):
with reporter.step("Restore interfaces all nodes"):
cluster_state_controller.restore_interfaces()
self.tick_epochs(1, alive_node=self.cluster.cluster_nodes[0].storage_node, wait_block=2)

220
pytest_tests/testsuites/failovers/test_failover_server.py
View file

@ -1,14 +1,14 @@
import logging
import os.path
import random
import time
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.common import MORPH_BLOCK_TIME
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.cli.container import StorageContainer, StorageContainerInfo, create_container
from frostfs_testlib.steps.cli.object import get_object
from frostfs_testlib.steps.cli.object import get_object, get_object_nodes, put_object
from frostfs_testlib.steps.node_management import check_node_in_map, check_node_not_in_map
from frostfs_testlib.storage.cluster import ClusterNode, StorageNode
from frostfs_testlib.storage.controllers import ClusterStateController
@ -18,7 +18,7 @@ from frostfs_testlib.storage.dataclasses.wallet import WalletInfo
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
from frostfs_testlib.testing.test_control import wait_for_success
from frostfs_testlib.utils import datetime_utils
from frostfs_testlib.utils.failover_utils import wait_for_host_offline, wait_object_replication
from frostfs_testlib.utils.failover_utils import wait_object_replication
from frostfs_testlib.utils.file_utils import get_file_hash
from pytest import FixtureRequest
@ -36,7 +36,7 @@ class TestFailoverServer(ClusterTestBase):
def wait_node_in_map(self, *args, **kwargs):
check_node_in_map(*args, **kwargs)
@allure.step("Create {count_containers} containers and {count_files} objects")
@reporter.step("Create {count_containers} containers and {count_files} objects")
@pytest.fixture
def containers(
self,
@ -64,7 +64,23 @@ class TestFailoverServer(ClusterTestBase):
return containers
@allure.step("Create object and delete after test")
@reporter.step("Creation container")
@pytest.fixture()
def container(self, default_wallet: str) -> StorageContainer:
select = len(self.cluster.cluster_nodes)
placement_rule = f"REP {select - 1} CBF 1 SELECT {select} FROM *"
cont_id = create_container(
default_wallet,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
rule=placement_rule,
basic_acl=PUBLIC_ACL,
)
wallet = WalletInfo(path=default_wallet)
storage_cont_info = StorageContainerInfo(id=cont_id, wallet_file=wallet)
return StorageContainer(storage_container_info=storage_cont_info, shell=self.shell, cluster=self.cluster)
@reporter.step("Create object and delete after test")
@pytest.fixture(scope="class")
def storage_objects(
self,
@ -75,7 +91,7 @@ class TestFailoverServer(ClusterTestBase):
) -> StorageObjectInfo:
count_object = request.param
object_sizes = [simple_object_size, complex_object_size]
object_list = []
object_list: list[StorageObjectInfo] = []
for cont in containers:
for _ in range(count_object):
object_list.append(cont.generate_object(size=random.choice(object_sizes).value))
@ -85,22 +101,24 @@ class TestFailoverServer(ClusterTestBase):
yield object_list
@allure.step("Select random node to stop and start it after test")
@reporter.step("Select random node to stop and start it after test")
@pytest.fixture
def node_to_stop(
self, node_under_test: ClusterNode, cluster_state_controller: ClusterStateController
) -> ClusterNode:
yield node_under_test
with allure.step(f"start {node_under_test.storage_node}"):
with reporter.step(f"start {node_under_test.storage_node}"):
cluster_state_controller.start_stopped_hosts()
@allure.step("Upload object with nodes and compare")
@reporter.step("Upload object with nodes and compare")
def get_corrupted_objects_list(
self, nodes: list[StorageNode], storage_objects: list[StorageObjectInfo]
) -> list[StorageObjectInfo]:
corrupted_objects = []
errors_get = []
for node in nodes:
for storage_object in storage_objects:
try:
got_file_path = get_object(
storage_object.wallet_file_path,
storage_object.cid,
@ -112,7 +130,10 @@ class TestFailoverServer(ClusterTestBase):
if storage_object.file_hash != get_file_hash(got_file_path):
corrupted_objects.append(storage_object)
os.remove(got_file_path)
except RuntimeError:
errors_get.append(storage_object.oid)
assert len(errors_get) == 0, f"Get failed - {errors_get}"
return corrupted_objects
def check_objects_replication(
@ -129,55 +150,63 @@ class TestFailoverServer(ClusterTestBase):
attempts=60,
)
@pytest.fixture()
def object_and_nodes(
self, simple_object_size: ObjectSize, container: StorageContainer, default_wallet: str
) -> tuple[StorageObjectInfo, list[ClusterNode]]:
object_info = container.generate_object(simple_object_size.value)
object_nodes = get_object_nodes(
cluster=self.cluster,
wallet=default_wallet,
cid=object_info.cid,
oid=object_info.oid,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
)
return object_info, object_nodes
@pytest.fixture()
def up_stop_nodes(self, cluster_state_controller: ClusterStateController):
yield
cluster_state_controller.start_stopped_hosts()
@allure.title("Full shutdown node")
@pytest.mark.parametrize("containers, storage_objects", [(5, 10)], indirect=True)
def test_complete_node_shutdown(
self,
containers: list[StorageContainer],
storage_objects: list[StorageObjectInfo],
default_wallet: str,
node_to_stop: ClusterNode,
cluster_state_controller: ClusterStateController,
):
with allure.step(f"Remove {node_to_stop} from the list of nodes"):
with reporter.step(f"Remove {node_to_stop} from the list of nodes"):
alive_nodes = list(set(self.cluster.cluster_nodes) - {node_to_stop})
storage_nodes = [cluster.storage_node for cluster in alive_nodes]
with allure.step("Tick epoch"):
self.tick_epochs(1, storage_nodes[0])
with allure.step("Wait 2 block time"):
time.sleep(datetime_utils.parse_time(MORPH_BLOCK_TIME) * 2)
with reporter.step("Tick epoch and wait for 2 blocks"):
self.tick_epochs(1, storage_nodes[0], wait_block=2)
with reporter.step(f"Stop node"):
cluster_state_controller.stop_node_host(node=node_to_stop, mode="hard")
with allure.step(f"Check if the node {node_to_stop.storage_node} has stopped"):
wait_for_host_offline(self.shell, node_to_stop.storage_node)
with allure.step("Verify that there are no corrupted objects"):
with reporter.step("Verify that there are no corrupted objects"):
corrupted_objects_list = self.get_corrupted_objects_list(storage_nodes, storage_objects)
assert not corrupted_objects_list
with allure.step(f"check {node_to_stop.storage_node} in map"):
with reporter.step(f"check {node_to_stop.storage_node} in map"):
self.wait_node_in_map(node_to_stop.storage_node, self.shell, alive_node=storage_nodes[0])
count_tick_epoch = int(alive_nodes[0].ir_node.get_netmap_cleaner_threshold()) + 4
with allure.step(f"Tick {count_tick_epoch} epoch, in {storage_nodes[0]} node"):
for tick in range(count_tick_epoch):
self.tick_epoch(storage_nodes[0])
time.sleep(datetime_utils.parse_time(MORPH_BLOCK_TIME) * 2)
with reporter.step(f"Tick {count_tick_epoch} epochs and wait for 2 blocks"):
self.tick_epochs(count_tick_epoch, storage_nodes[0], wait_block=2)
with allure.step(f"Check if the node {node_to_stop.storage_node} has stopped"):
wait_for_host_offline(self.shell, node_to_stop.storage_node)
with allure.step(f"Check {node_to_stop} in not map"):
with reporter.step(f"Check {node_to_stop} in not map"):
self.wait_node_not_in_map(node_to_stop.storage_node, self.shell, alive_node=storage_nodes[0])
with allure.step(f"Verify that there are no corrupted objects after {count_tick_epoch} epoch"):
with reporter.step(f"Verify that there are no corrupted objects after {count_tick_epoch} epoch"):
corrupted_objects_list = self.get_corrupted_objects_list(storage_nodes, storage_objects)
assert not corrupted_objects_list
@ -185,35 +214,136 @@ class TestFailoverServer(ClusterTestBase):
@pytest.mark.parametrize("containers, storage_objects", [(5, 10)], indirect=True)
def test_temporarily_disable_a_node(
self,
containers: list[StorageContainer],
storage_objects: list[StorageObjectInfo],
default_wallet: str,
node_to_stop: ClusterNode,
cluster_state_controller: ClusterStateController,
):
with allure.step(f"Remove {node_to_stop} from the list of nodes"):
with reporter.step(f"Remove {node_to_stop} from the list of nodes"):
storage_nodes = list(set(self.cluster.storage_nodes) - {node_to_stop.storage_node})
with allure.step("Tick epoch"):
self.tick_epochs(1, storage_nodes[0])
with allure.step("Wait 2 block time"):
time.sleep(datetime_utils.parse_time(MORPH_BLOCK_TIME) * 2)
with reporter.step("Tick epoch and wait for 2 blocks"):
self.tick_epochs(1, storage_nodes[0], wait_block=2)
with reporter.step(f"Stop node"):
cluster_state_controller.stop_node_host(node=node_to_stop, mode="hard")
with allure.step(f"Check if the node {node_to_stop} has stopped"):
wait_for_host_offline(self.shell, node_to_stop.storage_node)
with allure.step("Verify that there are no corrupted objects"):
with reporter.step("Verify that there are no corrupted objects"):
corrupted_objects_list = self.get_corrupted_objects_list(storage_nodes, storage_objects)
assert not corrupted_objects_list
with allure.step(f"Check {node_to_stop} in map"):
with reporter.step(f"Check {node_to_stop} in map"):
self.wait_node_in_map(node_to_stop.storage_node, self.shell, alive_node=storage_nodes[0])
cluster_state_controller.start_node_host(node_to_stop)
with allure.step("Verify that there are no corrupted objects"):
with reporter.step("Verify that there are no corrupted objects"):
corrupted_objects_list = self.get_corrupted_objects_list(storage_nodes, storage_objects)
assert not corrupted_objects_list
@allure.title("Not enough nodes in the container with policy - 'REP 3 CBF 1 SELECT 4 FROM *'")
def test_not_enough_nodes_in_container_rep_3(
self,
container: list[StorageContainer],
object_and_nodes: tuple[StorageObjectInfo, list[ClusterNode]],
default_wallet: str,
cluster_state_controller: ClusterStateController,
simple_file: str,
up_stop_nodes: None,
):
object_info, object_nodes = object_and_nodes
node_not_object = list(set(self.cluster.cluster_nodes) - set(object_nodes))[0]
with reporter.step("Stop all nodes with object, except 1"):
for cluster_node in object_nodes[1:]:
cluster_state_controller.stop_node_host(node=cluster_node, mode="hard")
with reporter.step("Get object"):
with reporter.step(f"Get operation to {node_not_object} where it does not exist, expect success"):
get_object(
wallet=default_wallet,
cid=object_info.cid,
oid=object_info.oid,
shell=self.shell,
endpoint=node_not_object.storage_node.get_rpc_endpoint(),
)
with reporter.step(f"Get operation to {object_nodes[0]} with object, expect success"):
get_object(
wallet=default_wallet,
cid=object_info.cid,
oid=object_info.oid,
shell=self.shell,
endpoint=object_nodes[0].storage_node.get_rpc_endpoint(),
)
with reporter.step(f"Put operation to node with object, expect error"):
with pytest.raises(RuntimeError):
put_object(
wallet=default_wallet,
path=simple_file,
cid=object_info.cid,
shell=self.shell,
endpoint=node_not_object.storage_node.get_rpc_endpoint(),
)
@allure.title("Not enough nodes in the container with policy - 'REP 2 CBF 2 SELECT 4 FROM *'")
def test_not_enough_nodes_in_container_rep_2(
self,
default_wallet: str,
cluster_state_controller: ClusterStateController,
simple_file: str,
up_stop_nodes: None,
):
with reporter.step("Creating a container with a full network map"):
select = len(self.cluster.cluster_nodes)
placement_rule = f"REP {select - 2} IN X CBF 2 SELECT {select} FROM * AS X"
cid_1 = create_container(
default_wallet,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
rule=placement_rule,
basic_acl=PUBLIC_ACL,
)
with reporter.step("Put object"):
oid = put_object(
wallet=default_wallet,
path=simple_file,
cid=cid_1,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
)
with reporter.step("Search nodes with object"):
object_nodes = get_object_nodes(
cluster=self.cluster,
wallet=default_wallet,
cid=cid_1,
oid=oid,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
)
with reporter.step("Turn off random node with object"):
cluster_state_controller.stop_node_host(node=random.choice(object_nodes[1:]), mode="hard")
with reporter.step("Checking PUT operation"):
oid_2 = put_object(
wallet=default_wallet,
path=simple_file,
cid=cid_1,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
)
with reporter.step("Checking GET operation"):
get_file = get_object(
wallet=default_wallet,
cid=cid_1,
oid=oid_2,
shell=self.shell,
endpoint=object_nodes[0].storage_node.get_rpc_endpoint(),
)
os.remove(get_file)
with reporter.step("Checking creating container"):
_ = create_container(
default_wallet,
shell=self.shell,
endpoint=object_nodes[0].storage_node.get_rpc_endpoint(),
rule=placement_rule,
basic_acl=PUBLIC_ACL,
)

340
pytest_tests/testsuites/failovers/test_failover_storage.py
View file

@ -5,11 +5,10 @@ from time import sleep
import allure
import pytest
from frostfs_testlib.hosting import Host
from frostfs_testlib import reporter
from frostfs_testlib.resources.common import MORPH_BLOCK_TIME
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper, VersioningStatus
from frostfs_testlib.shell import CommandOptions
from frostfs_testlib.s3 import S3ClientWrapper, VersioningStatus
from frostfs_testlib.steps.cli.container import StorageContainer, StorageContainerInfo, create_container
from frostfs_testlib.steps.cli.object import get_object, put_object_to_random_node
from frostfs_testlib.steps.node_management import (
@ -21,7 +20,7 @@ from frostfs_testlib.steps.node_management import (
wait_for_node_to_be_ready,
)
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage.cluster import Cluster, ClusterNode, StorageNode
from frostfs_testlib.storage.cluster import Cluster, ClusterNode, S3Gate, StorageNode
from frostfs_testlib.storage.controllers import ClusterStateController, ShardsWatcher
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.storage.dataclasses.storage_object_info import StorageObjectInfo
@ -37,11 +36,6 @@ logger = logging.getLogger("NeoLogger")
stopped_nodes: list[StorageNode] = []
def pytest_generate_tests(metafunc: pytest.Metafunc):
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize("s3_client", [AwsCliClient, Boto3ClientWrapper], indirect=True)
@pytest.fixture(scope="function")
@allure.title("Provide File Keeper")
def file_keeper():
@ -50,33 +44,18 @@ def file_keeper():
keeper.restore_files()
@allure.step("Return all stopped hosts")
@reporter.step("Return all stopped hosts")
@pytest.fixture(scope="function", autouse=True)
def after_run_return_all_stopped_hosts(cluster_state_controller: ClusterStateController) -> str:
yield
cluster_state_controller.start_stopped_hosts()
@allure.step("Return all stopped storage services after test")
@reporter.step("Return all stopped services after test")
@pytest.fixture(scope="function")
def after_run_return_all_stopped_services(cluster_state_controller: ClusterStateController):
yield
cluster_state_controller.start_stopped_storage_services()
@allure.step("Return all stopped S3 GateWay services after test")
@pytest.fixture(scope="function")
def after_run_return_all_stopped_s3(cluster_state_controller: ClusterStateController):
yield
cluster_state_controller.start_stopped_s3_gates()
def panic_reboot_host(host: Host) -> None:
shell = host.get_shell()
shell.exec('sudo sh -c "echo 1 > /proc/sys/kernel/sysrq"')
options = CommandOptions(close_stdin=True, timeout=1, check=False)
shell.exec('sudo sh -c "echo b > /proc/sysrq-trigger"', options)
cluster_state_controller.start_all_stopped_services()
@pytest.mark.failover
@ -99,7 +78,7 @@ class TestFailoverStorage(ClusterTestBase):
source_file_path = generate_file(simple_object_size.value)
stopped_hosts_nodes = []
with allure.step(f"Create container and put object"):
with reporter.step(f"Create container and put object"):
cid = create_container(
wallet,
shell=self.shell,
@ -109,10 +88,10 @@ class TestFailoverStorage(ClusterTestBase):
)
oid = put_object_to_random_node(wallet, source_file_path, cid, shell=self.shell, cluster=self.cluster)
with allure.step(f"Wait for replication and get nodes with object"):
with reporter.step(f"Wait for replication and get nodes with object"):
nodes_with_object = wait_object_replication(cid, oid, 2, shell=self.shell, nodes=self.cluster.storage_nodes)
with allure.step(f"Stop 2 nodes with object and wait replication one by one"):
with reporter.step(f"Stop 2 nodes with object and wait replication one by one"):
for storage_node in random.sample(nodes_with_object, 2):
stopped_hosts_nodes.append(storage_node)
@ -127,126 +106,65 @@ class TestFailoverStorage(ClusterTestBase):
nodes=list(set(self.cluster.storage_nodes) - {*stopped_hosts_nodes}),
)
with allure.step("Check object data is not corrupted"):
with reporter.step("Check object data is not corrupted"):
got_file_path = get_object(
wallet, cid, oid, endpoint=replicated_nodes[0].get_rpc_endpoint(), shell=self.shell
)
assert get_file_hash(source_file_path) == get_file_hash(got_file_path)
with allure.step("Return all hosts"):
with reporter.step("Return all hosts"):
cluster_state_controller.start_stopped_hosts()
with allure.step("Check object data is not corrupted"):
with reporter.step("Check object data is not corrupted"):
replicated_nodes = wait_object_replication(cid, oid, 2, shell=self.shell, nodes=self.cluster.storage_nodes)
got_file_path = get_object(
wallet, cid, oid, shell=self.shell, endpoint=replicated_nodes[0].get_rpc_endpoint()
)
assert get_file_hash(source_file_path) == get_file_hash(got_file_path)
@allure.title("Panic reboot nodes (sequenced_reboots={sequence})")
@pytest.mark.parametrize("sequence", [True, False])
@pytest.mark.failover_panic
def test_panic_storage_node_host(
self,
default_wallet,
require_multiple_hosts,
sequence: bool,
simple_object_size: ObjectSize,
cluster_state_controller: ClusterStateController,
):
wallet = default_wallet
placement_rule = "REP 2 IN X CBF 2 SELECT 2 FROM * AS X"
source_file_path = generate_file(simple_object_size.value)
cid = create_container(
wallet,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
rule=placement_rule,
basic_acl=PUBLIC_ACL,
)
oid = put_object_to_random_node(wallet, source_file_path, cid, shell=self.shell, cluster=self.cluster)
nodes_with_object = wait_object_replication(cid, oid, 2, shell=self.shell, nodes=self.cluster.storage_nodes)
allure.attach(
"\n".join([str(node) for node in nodes_with_object]),
"Current nodes with object",
allure.attachment_type.TEXT,
)
new_nodes: list[StorageNode] = []
with allure.step(f"Hard reboot hosts via magic SysRq option"):
for storage_node in nodes_with_object:
cluster_state_controller.panic_reboot_host(self.cluster.node(storage_node))
if sequence:
new_nodes = wait_object_replication(
cid,
oid,
2,
shell=self.shell,
nodes=self.cluster.storage_nodes,
)
allure.attach(
"\n".join([str(new_node) for new_node in new_nodes]),
f"Nodes with object after {storage_node} fail",
allure.attachment_type.TEXT,
)
if not sequence:
new_nodes = wait_object_replication(cid, oid, 2, shell=self.shell, nodes=self.cluster.storage_nodes)
allure.attach(
"\n".join([str(new_node) for new_node in new_nodes]),
"Nodes with object after nodes fail",
allure.attachment_type.TEXT,
)
got_file_path = get_object(wallet, cid, oid, shell=self.shell, endpoint=new_nodes[0].get_rpc_endpoint())
assert get_file_hash(source_file_path) == get_file_hash(got_file_path)
@allure.title("Do not ignore unhealthy tree endpoints (s3_client={s3_client})")
def test_unhealthy_tree(
self,
s3_client: S3ClientWrapper,
simple_object_size: ObjectSize,
cluster_state_controller: ClusterStateController,
after_run_return_all_stopped_s3,
after_run_return_all_stopped_services,
):
default_node = self.cluster.cluster_nodes[0]
with allure.step("Turn S3 GW off on default node"):
cluster_state_controller.stop_s3_gate(default_node)
with reporter.step("Turn S3 GW off on default node"):
cluster_state_controller.stop_service_of_type(default_node, S3Gate)
with allure.step("Turn off storage on default node"):
cluster_state_controller.stop_storage_service(default_node)
with reporter.step("Turn off storage on default node"):
cluster_state_controller.stop_service_of_type(default_node, StorageNode)
with allure.step("Turn on S3 GW on default node"):
cluster_state_controller.start_s3_gate(default_node)
with reporter.step("Turn on S3 GW on default node"):
cluster_state_controller.start_service_of_type(default_node, S3Gate)
with allure.step("Turn on storage on default node"):
cluster_state_controller.start_storage_service(default_node)
with reporter.step("Turn on storage on default node"):
cluster_state_controller.start_service_of_type(default_node, StorageNode)
with allure.step("Create bucket with REP 1 SELECT 1 policy"):
with reporter.step("Create bucket with REP 1 SELECT 1 policy"):
bucket = s3_client.create_bucket(
location_constraint="load-1-1",
)
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
put_object = s3_client.put_object(bucket, file_path)
s3_helper.check_objects_in_bucket(s3_client, bucket, expected_objects=[file_name])
with allure.step("Turn off all storage nodes except default"):
with reporter.step("Turn off all storage nodes except default"):
for node in self.cluster.cluster_nodes[1:]:
with allure.step(f"Stop storage service on node: {node}"):
cluster_state_controller.stop_storage_service(node)
with reporter.step(f"Stop storage service on node: {node}"):
cluster_state_controller.stop_service_of_type(node, StorageNode)
with allure.step("Check that object is available"):
with reporter.step("Check that object is available"):
s3_helper.check_objects_in_bucket(s3_client, bucket, expected_objects=[file_name])
with allure.step("Start storage nodes"):
cluster_state_controller.start_stopped_storage_services()
with reporter.step("Start storage nodes"):
cluster_state_controller.start_all_stopped_services()
@pytest.mark.failover
@ -256,11 +174,11 @@ class TestEmptyMap(ClusterTestBase):
A set of tests for makes map empty and verify that we can read objects after that
"""
@allure.step("Teardown after EmptyMap offline test")
@reporter.step("Teardown after EmptyMap offline test")
@pytest.fixture()
def empty_map_offline_teardown(self):
yield
with allure.step("Return all storage nodes to network map"):
with reporter.step("Return all storage nodes to network map"):
for node in stopped_nodes:
include_node_to_network_map(node, node, shell=self.shell, cluster=self.cluster)
stopped_nodes.remove(node)
@ -295,32 +213,32 @@ class TestEmptyMap(ClusterTestBase):
objects_list = s3_client.list_objects(bucket)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
s3_client.put_object(bucket, file_path)
with allure.step("Check that object exists in bucket"):
with reporter.step("Check that object exists in bucket"):
s3_helper.check_objects_in_bucket(s3_client, bucket, bucket_objects)
storage_nodes = self.cluster.storage_nodes
with allure.step("Exclude all storage nodes from network map"):
with reporter.step("Exclude all storage nodes from network map"):
for node in storage_nodes:
stopped_nodes.append(node)
exclude_node_from_network_map(node, node, shell=self.shell, cluster=self.cluster)
with allure.step("Return all storage nodes to network map"):
with reporter.step("Return all storage nodes to network map"):
for node in storage_nodes:
include_node_to_network_map(node, node, shell=self.shell, cluster=self.cluster)
stopped_nodes.remove(node)
with allure.step("Check that we can read object"):
with reporter.step("Check that we can read object"):
s3_helper.check_objects_in_bucket(s3_client, bucket, bucket_objects)
@allure.step("Teardown after EmptyMap stop service test")
@reporter.step("Teardown after EmptyMap stop service test")
@pytest.fixture()
def empty_map_stop_service_teardown(self, cluster_state_controller: ClusterStateController):
yield
with allure.step("Return all storage nodes to network map"):
cluster_state_controller.start_stopped_storage_services()
with reporter.step("Return all storage nodes to network map"):
cluster_state_controller.start_all_stopped_services()
for node in stopped_nodes:
check_node_in_map(node, shell=self.shell, alive_node=node)
@ -358,32 +276,32 @@ class TestEmptyMap(ClusterTestBase):
objects_list = s3_client.list_objects(bucket)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
s3_client.put_object(bucket, file_path)
with allure.step("Check that object exists in bucket"):
with reporter.step("Check that object exists in bucket"):
s3_helper.check_objects_in_bucket(s3_client, bucket, bucket_objects)
with allure.step("Stop all storage nodes"):
cluster_state_controller.stop_all_storage_services()
with reporter.step("Stop all storage nodes"):
cluster_state_controller.stop_services_of_type(StorageNode)
with allure.step("Remove all nodes from network map"):
with reporter.step("Remove all nodes from network map"):
remove_nodes_from_map_morph(
shell=self.shell, cluster=self.cluster, remove_nodes=self.cluster.services(StorageNode)
)
with allure.step("Return all storage nodes to network map"):
with reporter.step("Return all storage nodes to network map"):
self.return_nodes_after_stop_with_check_empty_map(cluster_state_controller)
with allure.step("Check that object exists in bucket"):
with reporter.step("Check that object exists in bucket"):
s3_helper.check_objects_in_bucket(s3_client, bucket, bucket_objects)
@allure.step("Return all nodes to cluster with check empty map first")
@reporter.step("Return all nodes to cluster with check empty map first")
def return_nodes_after_stop_with_check_empty_map(self, cluster_state_controller: ClusterStateController) -> None:
first_node = self.cluster.cluster_nodes[0].service(StorageNode)
with allure.step("Start first node and check network map"):
cluster_state_controller.start_storage_service(self.cluster.cluster_nodes[0])
with reporter.step("Start first node and check network map"):
cluster_state_controller.start_service_of_type(self.cluster.cluster_nodes[0], StorageNode)
wait_for_node_to_be_ready(first_node)
for check_node in self.cluster.storage_nodes:
@ -392,7 +310,7 @@ class TestEmptyMap(ClusterTestBase):
for node in self.cluster.cluster_nodes[1:]:
storage_node = node.service(StorageNode)
cluster_state_controller.start_storage_service(node)
cluster_state_controller.start_service_of_type(node, StorageNode)
wait_for_node_to_be_ready(storage_node)
sleep(datetime_utils.parse_time(MORPH_BLOCK_TIME))
@ -414,35 +332,33 @@ class TestEmptyMap(ClusterTestBase):
file_name = s3_helper.object_key_from_file_path(file_path)
object_versions = []
with allure.step("Put object into one bucket"):
with reporter.step("Put object into one bucket"):
put_object = s3_client.put_object(bucket, file_path)
s3_helper.check_objects_in_bucket(s3_client, bucket, expected_objects=[file_name])
object_versions.append(put_object)
with allure.step("Stop all storage nodes"):
for node in self.cluster.cluster_nodes:
with allure.step(f"Stop storage service on node: {node}"):
cluster_state_controller.stop_storage_service(node)
with reporter.step("Stop all storage nodes"):
cluster_state_controller.stop_services_of_type(StorageNode)
with allure.step("Delete blobovnicza and fstree from all nodes"):
with reporter.step("Delete blobovnicza and fstree from all nodes"):
for node in self.cluster.storage_nodes:
node.delete_blobovnicza()
node.delete_fstree()
with allure.step("Start all storage nodes"):
cluster_state_controller.start_stopped_storage_services()
with reporter.step("Start all storage nodes"):
cluster_state_controller.start_all_stopped_services()
# need to get Delete Marker first
with allure.step("Delete the object from the bucket"):
with reporter.step("Delete the object from the bucket"):
delete_object = s3_client.delete_object(bucket, file_name)
object_versions.append(delete_object["VersionId"])
# and now delete all versions of object (including Delete Markers)
with allure.step("Delete all versions of the object from the bucket"):
with reporter.step("Delete all versions of the object from the bucket"):
for version in object_versions:
delete_object = s3_client.delete_object(bucket, file_name, version_id=version)
with allure.step("Delete bucket"):
with reporter.step("Delete bucket"):
s3_client.delete_bucket(bucket)
@allure.title("Object loss from fstree/blobovnicza (versioning=disabled, s3_client={s3_client})")
@ -457,27 +373,25 @@ class TestEmptyMap(ClusterTestBase):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
with allure.step("Put object into one bucket"):
with reporter.step("Put object into one bucket"):
s3_client.put_object(bucket, file_path)
s3_helper.check_objects_in_bucket(s3_client, bucket, expected_objects=[file_name])
with allure.step("Stop all storage nodes"):
for node in self.cluster.cluster_nodes:
with allure.step(f"Stop storage service on node: {node}"):
cluster_state_controller.stop_storage_service(node)
with reporter.step("Stop all storage nodes"):
cluster_state_controller.stop_services_of_type(StorageNode)
with allure.step("Delete blobovnicza and fstree from all nodes"):
with reporter.step("Delete blobovnicza and fstree from all nodes"):
for node in self.cluster.storage_nodes:
node.delete_blobovnicza()
node.delete_fstree()
with allure.step("Start all storage nodes"):
cluster_state_controller.start_stopped_storage_services()
with reporter.step("Start all storage nodes"):
cluster_state_controller.start_all_stopped_services()
with allure.step("Delete the object from the bucket"):
with reporter.step("Delete the object from the bucket"):
s3_client.delete_object(bucket, file_name)
with allure.step("Delete bucket"):
with reporter.step("Delete bucket"):
s3_client.delete_bucket(bucket)
@pytest.mark.skip(reason="Need to increase cache lifetime")
@ -502,38 +416,36 @@ class TestEmptyMap(ClusterTestBase):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
with allure.step("Put object into one bucket"):
with reporter.step("Put object into one bucket"):
s3_client.put_object(bucket, file_path)
s3_helper.check_objects_in_bucket(s3_client, bucket, expected_objects=[file_name])
with allure.step("Stop all storage nodes"):
for node in self.cluster.cluster_nodes:
with allure.step(f"Stop storage service on node: {node}"):
cluster_state_controller.stop_storage_service(node)
with reporter.step("Stop all storage nodes"):
cluster_state_controller.stop_services_of_type(StorageNode)
with allure.step("Delete pilorama.db from all nodes"):
with reporter.step("Delete pilorama.db from all nodes"):
for node in self.cluster.storage_nodes:
node.delete_pilorama()
with allure.step("Start all storage nodes"):
cluster_state_controller.start_stopped_storage_services()
with reporter.step("Start all storage nodes"):
cluster_state_controller.start_all_stopped_services()
with allure.step("Check list objects first time"):
with reporter.step("Check list objects first time"):
objects_list = s3_client.list_objects(bucket)
assert objects_list, f"Expected not empty bucket"
with allure.step("Check list objects second time"):
with reporter.step("Check list objects second time"):
objects_list = s3_client.list_objects(bucket)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
with allure.step("Delete bucket"):
with reporter.step("Delete bucket"):
s3_client.delete_bucket(bucket)
@pytest.mark.failover
@pytest.mark.failover_data_loss
class TestStorageDataLoss(ClusterTestBase):
@allure.step("Get list of all piloramas on node")
@reporter.step("Get list of all piloramas on node")
def get_piloramas_list(self, node: StorageNode) -> list:
data_directory_path = node.get_data_directory()
@ -559,10 +471,10 @@ class TestStorageDataLoss(ClusterTestBase):
):
allure.dynamic.description(after_run_return_all_stopped_services)
with allure.step("Create bucket"):
with reporter.step("Create bucket"):
bucket = s3_client.create_bucket()
with allure.step("Put objects into bucket"):
with reporter.step("Put objects into bucket"):
simple_object_path = generate_file(simple_object_size.value)
simple_object_key = s3_helper.object_key_from_file_path(simple_object_path)
@ -572,40 +484,44 @@ class TestStorageDataLoss(ClusterTestBase):
s3_client.put_object(bucket, simple_object_path)
s3_client.put_object(bucket, complex_object_path)
with allure.step("Check objects are in bucket"):
with reporter.step("Check objects are in bucket"):
s3_helper.check_objects_in_bucket(
s3_client, bucket, expected_objects=[simple_object_key, complex_object_key]
)
with allure.step("Stop storage services on all nodes"):
cluster_state_controller.stop_all_storage_services()
with reporter.step("Stop storage services on all nodes"):
cluster_state_controller.stop_services_of_type(StorageNode)
with allure.step("Delete metabase from all nodes"):
with reporter.step("Delete metabase from all nodes"):
for node in cluster_state_controller.cluster.storage_nodes:
node.delete_metabase()
with allure.step("Enable resync_metabase option for storage services"):
with reporter.step("Enable resync_metabase option for storage services"):
for storage_node in cluster_state_controller.cluster.storage_nodes:
with allure.step(f"Enable resync_metabase option for {storage_node}"):
with reporter.step(f"Enable resync_metabase option for {storage_node}"):
config_file_path, config = storage_node.get_shards_config()
if not config["storage"]["shard"]["default"]["resync_metabase"]:
file_keeper.add(storage_node, config_file_path)
config["storage"]["shard"]["default"]["resync_metabase"] = True
storage_node.save_config(config, config_file_path)
with allure.step("Start storage services on all nodes"):
cluster_state_controller.start_stopped_storage_services()
with reporter.step("Start storage services on all nodes"):
cluster_state_controller.start_all_stopped_services()
with allure.step("Delete objects from bucket"):
with allure.step("Delete simple object from bucket"):
with reporter.step("Wait for tree rebalance"):
# TODO: Use product metric when we have proper ones for this check
sleep(30)
with reporter.step("Delete objects from bucket"):
with reporter.step("Delete simple object from bucket"):
with expect_not_raises():
s3_client.delete_object(bucket, simple_object_key)
with allure.step("Delete complex object from bucket"):
with reporter.step("Delete complex object from bucket"):
with expect_not_raises():
s3_client.delete_object(bucket, complex_object_key)
with allure.step("Delete bucket"):
with reporter.step("Delete bucket"):
with expect_not_raises():
s3_client.delete_bucket(bucket)
@ -624,7 +540,7 @@ class TestStorageDataLoss(ClusterTestBase):
exception_messages = []
allure.dynamic.description(after_run_return_all_stopped_services)
with allure.step(f"Create container on node {node_under_test}"):
with reporter.step(f"Create container on node {node_under_test}"):
locode = node_under_test.storage_node.get_un_locode()
placement_rule = f"""REP 1 IN X
CBF 1
@ -642,7 +558,7 @@ class TestStorageDataLoss(ClusterTestBase):
cluster_state_controller.cluster,
)
with allure.step(f"Put couple objects to container on node {node_under_test}"):
with reporter.step(f"Put couple objects to container on node {node_under_test}"):
storage_objects: list[StorageObjectInfo] = []
for _ in range(5):
storage_object = container.generate_object(
@ -651,19 +567,19 @@ class TestStorageDataLoss(ClusterTestBase):
)
storage_objects.append(storage_object)
with allure.step("Take shards snapshot"):
with reporter.step("Take shards snapshot"):
shards_watcher.take_shards_snapshot()
with allure.step(f"Stop storage service on node {node_under_test}"):
cluster_state_controller.stop_storage_service(node_under_test)
with reporter.step(f"Stop storage service on node {node_under_test}"):
cluster_state_controller.stop_service_of_type(node_under_test, StorageNode)
with allure.step(f"Delete write cache from node {node_under_test}"):
with reporter.step(f"Delete write cache from node {node_under_test}"):
node_under_test.storage_node.delete_write_cache()
with allure.step(f"Start storage service on node {node_under_test}"):
cluster_state_controller.start_storage_service(node_under_test)
with reporter.step(f"Start storage service on node {node_under_test}"):
cluster_state_controller.start_all_stopped_services()
with allure.step("Objects should be available"):
with reporter.step("Objects should be available"):
for storage_object in storage_objects:
get_object(
storage_object.wallet_file_path,
@ -673,26 +589,26 @@ class TestStorageDataLoss(ClusterTestBase):
node_under_test.storage_node.get_rpc_endpoint(),
)
with allure.step("No shards should have new errors"):
with reporter.step("No shards should have new errors"):
shards_watcher.take_shards_snapshot()
shards_with_errors = shards_watcher.get_shards_with_new_errors()
if shards_with_errors:
exception_messages.append(f"Shards have new errors: {shards_with_errors}")
with allure.step("No shards should have degraded status"):
with reporter.step("No shards should have degraded status"):
snapshot = shards_watcher.get_shards_snapshot()
for shard in snapshot:
status = snapshot[shard]["mode"]
if status != "read-write":
exception_messages.append(f"Shard {shard} changed status to {status}")
with allure.step("No related errors should be in log"):
with reporter.step("No related errors should be in log"):
if node_under_test.host.is_message_in_logs(
message_regex=r"\Wno such file or directory\W", since=test_start_time
):
exception_messages.append(f"Node {node_under_test} have shard errors in logs")
with allure.step("Pass test if no errors found"):
with reporter.step("Pass test if no errors found"):
assert not exception_messages, "\n".join(exception_messages)
@allure.title(
@ -708,15 +624,15 @@ class TestStorageDataLoss(ClusterTestBase):
):
# TODO: need to check that s3 gate is connected to localhost (such metric will be supported in 1.3)
with allure.step("Stop one node and wait for rebalance connection of s3 gate to storage service"):
with reporter.step("Stop one node and wait for rebalance connection of s3 gate to storage service"):
current_node = self.cluster.cluster_nodes[0]
cluster_state_controller.stop_storage_service(current_node)
cluster_state_controller.stop_service_of_type(current_node, StorageNode)
# waiting for rebalance connection of s3 gate to storage service
sleep(60)
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
with allure.step("Put object into one bucket"):
with reporter.step("Put object into one bucket"):
put_object = s3_client.put_object(bucket, file_path)
s3_helper.check_objects_in_bucket(s3_client, bucket, expected_objects=[file_name])
@ -733,7 +649,7 @@ class TestStorageDataLoss(ClusterTestBase):
)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Check bucket versioning"):
with reporter.step("Check bucket versioning"):
bucket_versioning = s3_client.get_bucket_versioning_status(bucket)
assert bucket_versioning == "Enabled", "Bucket should have enabled versioning"
@ -741,61 +657,59 @@ class TestStorageDataLoss(ClusterTestBase):
file_name = s3_helper.object_key_from_file_path(file_path)
object_versions = []
with allure.step("Put object into one bucket"):
with reporter.step("Put object into one bucket"):
put_object = s3_client.put_object(bucket, file_path)
s3_helper.check_objects_in_bucket(s3_client, bucket, expected_objects=[file_name])
object_versions.append(put_object)
node_to_check = self.cluster.storage_nodes[0]
piloramas_list_before_removing = {}
with allure.step("Get list of all pilorama.db"):
with reporter.step("Get list of all pilorama.db"):
piloramas_list_before_removing = self.get_piloramas_list(node_to_check)
with allure.step("Stop all storage nodes"):
for node in self.cluster.cluster_nodes:
with allure.step(f"Stop storage service on node: {node}"):
cluster_state_controller.stop_storage_service(node)
with reporter.step("Stop all storage nodes"):
cluster_state_controller.stop_services_of_type(StorageNode)
with allure.step("Delete pilorama.db from one node"):
with reporter.step("Delete pilorama.db from one node"):
node_to_check.delete_pilorama()
with allure.step("Start all storage nodes"):
cluster_state_controller.start_stopped_storage_services()
with reporter.step("Start all storage nodes"):
cluster_state_controller.start_all_stopped_services()
with allure.step("Tick epoch to trigger sync and then wait for 1 minute"):
with reporter.step("Tick epoch to trigger sync and then wait for 1 minute"):
self.tick_epochs(1)
sleep(120)
piloramas_list_afrer_removing = {}
with allure.step("Get list of all pilorama.db after sync"):
with reporter.step("Get list of all pilorama.db after sync"):
piloramas_list_afrer_removing = self.get_piloramas_list(node_to_check)
assert piloramas_list_afrer_removing == piloramas_list_before_removing, "List of pilorama.db is different"
with allure.step("Check bucket versioning"):
with reporter.step("Check bucket versioning"):
bucket_versioning = s3_client.get_bucket_versioning_status(bucket)
assert bucket_versioning == "Enabled", "Bucket should have enabled versioning"
with allure.step("Check list objects"):
with reporter.step("Check list objects"):
objects_list = s3_client.list_objects(bucket)
assert objects_list, f"Expected not empty bucket"
with allure.step("Delete the object from the bucket"):
with reporter.step("Delete the object from the bucket"):
delete_object = s3_client.delete_object(bucket, file_name)
assert "DeleteMarker" in delete_object.keys(), "Delete markers not found"
with allure.step("Check list objects"):
with reporter.step("Check list objects"):
objects_list = s3_client.list_objects_versions(bucket)
assert objects_list, f"Expected not empty bucket"
object_versions.append(delete_object["VersionId"])
# and now delete all versions of object (including Delete Markers)
with allure.step("Delete all versions of the object from the bucket"):
with reporter.step("Delete all versions of the object from the bucket"):
for version in object_versions:
delete_object = s3_client.delete_object(bucket, file_name, version_id=version)
with allure.step("Check list objects"):
with reporter.step("Check list objects"):
objects_list = s3_client.list_objects_versions(bucket)
assert not objects_list, f"Expected empty bucket"
with allure.step("Delete bucket"):
with reporter.step("Delete bucket"):
s3_client.delete_bucket(bucket)

652
pytest_tests/testsuites/management/test_node_management.py Normal file
View file

@ -0,0 +1,652 @@
import logging
import os
import random
from time import sleep
from typing import Optional, Tuple
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.cli import FrostfsCli
from frostfs_testlib.cli.netmap_parser import NetmapParser
from frostfs_testlib.resources.cli import FROSTFS_CLI_EXEC
from frostfs_testlib.resources.common import DEFAULT_WALLET_CONFIG, MORPH_BLOCK_TIME
from frostfs_testlib.resources.error_patterns import OBJECT_NOT_FOUND
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.cli.container import create_container, search_nodes_with_container
from frostfs_testlib.steps.cli.object import (
delete_object,
get_object,
get_object_from_random_node,
head_object,
put_object,
put_object_to_random_node,
search_object,
)
from frostfs_testlib.steps.node_management import (
check_node_in_map,
delete_node_data,
drop_object,
exclude_node_from_network_map,
get_locode_from_random_node,
include_node_to_network_map,
node_shard_list,
node_shard_set_mode,
storage_node_set_status,
wait_for_node_to_be_ready,
)
from frostfs_testlib.steps.storage_policy import get_nodes_with_object
from frostfs_testlib.storage.cluster import ClusterNode, StorageNode
from frostfs_testlib.storage.controllers import ClusterStateController
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.storage.dataclasses.storage_object_info import ModeNode
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
from frostfs_testlib.utils import datetime_utils, string_utils
from frostfs_testlib.utils.failover_utils import wait_object_replication
from frostfs_testlib.utils.file_utils import generate_file
from pytest_tests.helpers.utility import wait_for_gc_pass_on_storage_nodes
logger = logging.getLogger("NeoLogger")
check_nodes: list[StorageNode] = []
@allure.title("Add one node to cluster")
@pytest.mark.add_nodes
@pytest.mark.node_mgmt
class TestNodeManagement(ClusterTestBase):
@pytest.fixture
@allure.title("Create container and pick the node with data")
def create_container_and_pick_node(
self, default_wallet: str, simple_object_size: ObjectSize
) -> Tuple[str, StorageNode]:
file_path = generate_file(simple_object_size.value)
placement_rule = "REP 1 IN X CBF 1 SELECT 1 FROM * AS X"
endpoint = self.cluster.default_rpc_endpoint
cid = create_container(
default_wallet,
shell=self.shell,
endpoint=endpoint,
rule=placement_rule,
basic_acl=PUBLIC_ACL,
)
oid = put_object_to_random_node(default_wallet, file_path, cid, self.shell, self.cluster)
nodes = get_nodes_with_object(cid, oid, shell=self.shell, nodes=self.cluster.storage_nodes)
assert len(nodes) == 1
node = nodes[0]
yield cid, node
shards = node_shard_list(node)
assert shards
for shard in shards:
node_shard_set_mode(node, shard, "read-write")
node_shard_list(node)
@reporter.step("Tick epoch with retries")
def tick_epoch_with_retries(self, attempts: int = 3, timeout: int = 3, wait_block: int = None):
for attempt in range(attempts):
try:
self.tick_epoch(wait_block=wait_block)
except RuntimeError:
sleep(timeout)
if attempt >= attempts - 1:
raise
continue
return
@pytest.fixture
def after_run_start_all_nodes(self):
yield
self.return_nodes()
@pytest.fixture
def return_nodes_after_test_run(self):
yield
self.return_nodes()
@reporter.step("Return node to cluster")
def return_nodes(self, alive_node: Optional[StorageNode] = None) -> None:
for node in list(check_nodes):
with reporter.step(f"Start node {node}"):
node.start_service()
with reporter.step(f"Waiting status ready for node {node}"):
wait_for_node_to_be_ready(node)
# We need to wait for node to establish notifications from morph-chain
# Otherwise it will hang up when we will try to set status
sleep(datetime_utils.parse_time(MORPH_BLOCK_TIME))
with reporter.step(f"Move node {node} to online state"):
storage_node_set_status(node, status="online", retries=2)
check_nodes.remove(node)
sleep(datetime_utils.parse_time(MORPH_BLOCK_TIME))
self.tick_epoch_with_retries(3, wait_block=2)
check_node_in_map(node, shell=self.shell, alive_node=alive_node)
@allure.title("Add one node to cluster")
@pytest.mark.add_nodes
def test_add_nodes(
self,
default_wallet: str,
simple_object_size: ObjectSize,
return_nodes_after_test_run,
):
"""
This test remove one node from frostfs_testlib.storage.cluster then add it back. Test uses base control operations with storage nodes (healthcheck, netmap-snapshot, set-status).
"""
wallet = default_wallet
placement_rule_3 = "REP 3 IN X CBF 1 SELECT 3 FROM * AS X"
placement_rule_4 = "REP 4 IN X CBF 1 SELECT 4 FROM * AS X"
source_file_path = generate_file(simple_object_size.value)
storage_nodes = self.cluster.storage_nodes
random_node = random.choice(storage_nodes[1:])
alive_node = random.choice(
[storage_node for storage_node in storage_nodes if storage_node.id != random_node.id]
)
check_node_in_map(random_node, shell=self.shell, alive_node=alive_node)
# Add node to recovery list before messing with it
check_nodes.append(random_node)
exclude_node_from_network_map(random_node, alive_node, shell=self.shell, cluster=self.cluster)
delete_node_data(random_node)
cid = create_container(
wallet,
rule=placement_rule_3,
basic_acl=PUBLIC_ACL,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
oid = put_object(
wallet,
source_file_path,
cid,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
wait_object_replication(cid, oid, 3, shell=self.shell, nodes=storage_nodes)
self.return_nodes(alive_node)
with reporter.step("Check data could be replicated to new node"):
random_node = random.choice(list(set(storage_nodes) - {random_node, alive_node}))
# Add node to recovery list before messing with it
check_nodes.append(random_node)
exclude_node_from_network_map(random_node, alive_node, shell=self.shell, cluster=self.cluster)
wait_object_replication(
cid,
oid,
3,
shell=self.shell,
nodes=list(set(storage_nodes) - {random_node}),
)
include_node_to_network_map(random_node, alive_node, shell=self.shell, cluster=self.cluster)
wait_object_replication(cid, oid, 3, shell=self.shell, nodes=storage_nodes)
with reporter.step("Check container could be created with new node"):
cid = create_container(
wallet,
rule=placement_rule_4,
basic_acl=PUBLIC_ACL,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
oid = put_object(
wallet,
source_file_path,
cid,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
wait_object_replication(cid, oid, 4, shell=self.shell, nodes=storage_nodes)
@pytest.mark.node_mgmt
@allure.title("Drop object using control command")
def test_drop_object(self, default_wallet, complex_object_size: ObjectSize, simple_object_size: ObjectSize):
"""
Test checks object could be dropped using `frostfs-cli control drop-objects` command.
"""
wallet = default_wallet
endpoint = self.cluster.default_rpc_endpoint
file_path_simple = generate_file(simple_object_size.value)
file_path_complex = generate_file(complex_object_size.value)
locode = get_locode_from_random_node(self.cluster)
rule = f"REP 1 IN SE CBF 1 SELECT 1 FROM LOC AS SE FILTER 'UN-LOCODE' EQ '{locode}' AS LOC"
cid = create_container(wallet, rule=rule, shell=self.shell, endpoint=endpoint)
oid_simple = put_object_to_random_node(wallet, file_path_simple, cid, shell=self.shell, cluster=self.cluster)
oid_complex = put_object_to_random_node(wallet, file_path_complex, cid, shell=self.shell, cluster=self.cluster)
for oid in (oid_simple, oid_complex):
get_object_from_random_node(wallet, cid, oid, shell=self.shell, cluster=self.cluster)
head_object(wallet, cid, oid, shell=self.shell, endpoint=endpoint)
nodes_with_object = get_nodes_with_object(cid, oid_simple, shell=self.shell, nodes=self.cluster.storage_nodes)
random_node = random.choice(nodes_with_object)
for oid in (oid_simple, oid_complex):
with reporter.step(f"Drop object {oid}"):
get_object_from_random_node(wallet, cid, oid, shell=self.shell, cluster=self.cluster)
head_object(wallet, cid, oid, shell=self.shell, endpoint=endpoint)
drop_object(random_node, cid, oid)
self.wait_for_obj_dropped(wallet, cid, oid, endpoint, get_object)
self.wait_for_obj_dropped(wallet, cid, oid, endpoint, head_object)
@pytest.mark.node_mgmt
@pytest.mark.skip(reason="Need to clarify scenario")
@allure.title("Control Operations with storage nodes")
def test_shards(
self,
default_wallet,
create_container_and_pick_node,
simple_object_size: ObjectSize,
):
wallet = default_wallet
file_path = generate_file(simple_object_size.value)
cid, node = create_container_and_pick_node
original_oid = put_object_to_random_node(wallet, file_path, cid, self.shell, self.cluster)
# for mode in ('read-only', 'degraded'):
for mode in ("degraded",):
shards = node_shard_list(node)
assert shards
for shard in shards:
node_shard_set_mode(node, shard, mode)
shards = node_shard_list(node)
assert shards
with pytest.raises(RuntimeError):
put_object_to_random_node(wallet, file_path, cid, self.shell, self.cluster)
with pytest.raises(RuntimeError):
delete_object(wallet, cid, original_oid, self.shell, self.cluster.default_rpc_endpoint)
get_object_from_random_node(wallet, cid, original_oid, self.shell, self.cluster)
for shard in shards:
node_shard_set_mode(node, shard, "read-write")
shards = node_shard_list(node)
assert shards
oid = put_object_to_random_node(wallet, file_path, cid, self.shell, self.cluster)
delete_object(wallet, cid, oid, self.shell, self.cluster.default_rpc_endpoint)
@pytest.mark.node_mgmt
@allure.title("Put object with stopped node")
def test_stop_node(self, default_wallet, return_nodes_after_test_run, simple_object_size: ObjectSize):
wallet = default_wallet
placement_rule = "REP 3 IN X SELECT 4 FROM * AS X"
source_file_path = generate_file(simple_object_size.value)
storage_nodes = self.cluster.storage_nodes
random_node = random.choice(storage_nodes[1:])
alive_node = random.choice(
[storage_node for storage_node in storage_nodes if storage_node.id != random_node.id]
)
cid = create_container(
wallet,
rule=placement_rule,
basic_acl=PUBLIC_ACL,
shell=self.shell,
endpoint=random_node.get_rpc_endpoint(),
)
with reporter.step("Stop the random node"):
check_nodes.append(random_node)
random_node.stop_service()
with reporter.step("Try to put an object and expect success"):
put_object(
wallet,
source_file_path,
cid,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
self.return_nodes(alive_node)
@reporter.step("Wait for object to be dropped")
def wait_for_obj_dropped(self, wallet: str, cid: str, oid: str, endpoint: str, checker) -> None:
for _ in range(3):
try:
checker(wallet, cid, oid, shell=self.shell, endpoint=endpoint)
wait_for_gc_pass_on_storage_nodes()
except Exception as err:
if string_utils.is_str_match_pattern(err, OBJECT_NOT_FOUND):
return
raise AssertionError(f'Expected "{OBJECT_NOT_FOUND}" error, got\n{err}')
raise AssertionError(f"Object {oid} was not dropped from node")
@pytest.mark.maintenance
class TestMaintenanceMode(ClusterTestBase):
change_node: ClusterNode = None
@pytest.fixture()
@allure.title("Init Frostfs CLI remote")
def frostfs_cli_remote(self, node_under_test: ClusterNode) -> FrostfsCli:
host = node_under_test.host
service_config = host.get_service_config(node_under_test.storage_node.name)
wallet_path = service_config.attributes["wallet_path"]
wallet_password = service_config.attributes["wallet_password"]
shell = host.get_shell()
wallet_config_path = f"/tmp/{node_under_test.storage_node.name}-config.yaml"
wallet_config = f'wallet: {wallet_path}\npassword: "{wallet_password}"'
shell.exec(f"echo '{wallet_config}' > {wallet_config_path}")
cli = FrostfsCli(shell=shell, frostfs_cli_exec_path=FROSTFS_CLI_EXEC, config_file=wallet_config_path)
return cli
@pytest.fixture()
@allure.title("Init Frostfs CLI remote")
def frostfs_cli(self) -> FrostfsCli:
cli = FrostfsCli(shell=self.shell, frostfs_cli_exec_path=FROSTFS_CLI_EXEC, config_file=DEFAULT_WALLET_CONFIG)
return cli
@pytest.fixture()
def restore_online_mode_node(self, cluster_state_controller: ClusterStateController, default_wallet: str):
yield
cluster_state_controller.set_mode_node(cluster_node=self.change_node, wallet=default_wallet, status="online")
self.tick_epoch(wait_block=2)
def basic_operations(self, wallet, cid, oid, shell, endpoint, matchs, object_size):
file_path = generate_file(object_size)
default_kw = {"wallet": wallet, "cid": cid, "shell": shell, "endpoint": endpoint}
operations = {
get_object: {"oid": oid},
search_object: {},
delete_object: {"oid": oid},
put_object: {"path": file_path},
}
for index, operation, kw in enumerate(operations.items()):
with reporter.step(f"Run {operation.__name__} object, waiting response - {matchs[index]}"):
default_kw.update(kw)
if operation == search_object and "Found" in matchs[index]:
operation(**default_kw)
continue
with pytest.raises(RuntimeError, match=matchs[index]):
operation(**default_kw)
os.remove(file_path)
@allure.title("Test of basic node operations in maintenance mode")
def test_maintenance_mode(
self,
default_wallet: str,
simple_object_size: ObjectSize,
cluster_state_controller: ClusterStateController,
restore_online_mode_node: None,
):
with reporter.step("Create container and create\put object"):
cid = create_container(
wallet=default_wallet,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
rule="REP 1 CBF 1",
)
node = search_nodes_with_container(
wallet=default_wallet,
cid=cid,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
cluster=self.cluster,
)
self.change_node = node[0]
file_path = generate_file(simple_object_size.value)
oid = put_object(
wallet=default_wallet,
path=file_path,
cid=cid,
shell=self.shell,
endpoint=self.change_node.storage_node.get_rpc_endpoint(),
)
with reporter.step("Enable MaintenanceModeAllowed:"):
cluster_state_controller.set_mode_node(
cluster_node=self.change_node, wallet=default_wallet, status="maintenance"
)
other_nodes = list(set(self.cluster.cluster_nodes) - set(node))
node_and_match = {
self.change_node: ["node is under maintenance"] * 4,
other_nodes[0]: [
"object not found",
"Found 0 objects",
"object not found",
"node is under maintenance",
],
}
with reporter.step("Run basic operations"):
for cluster_node, matchs in node_and_match.items():
self.basic_operations(
wallet=default_wallet,
cid=cid,
oid=oid,
shell=self.shell,
endpoint=cluster_node.storage_node.get_rpc_endpoint(),
matchs=matchs,
object_size=simple_object_size.value,
)
@pytest.mark.sanity
@allure.title("MAINTENANCE and OFFLINE mode transitions")
def test_mode_transitions(
self,
cluster_state_controller: ClusterStateController,
node_under_test: ClusterNode,
default_wallet: str,
frostfs_cli: FrostfsCli,
restore_online_mode_node: None,
):
self.change_node = node_under_test
cluster_nodes = list(set(self.cluster.cluster_nodes) - {node_under_test})
with reporter.step("Tick epoch"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Set node mode to offline"):
cluster_state_controller.set_mode_node(
cluster_node=node_under_test,
wallet=default_wallet,
status=ModeNode.OFFLINE.value,
)
with reporter.step("Tick epoch to update the network map"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Check node mode = offline, after update the network map"):
netmap = frostfs_cli.netmap.snapshot(
rpc_endpoint=cluster_nodes[0].storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout
netmap = NetmapParser.snapshot_all_nodes(netmap)
assert node_under_test.host_ip not in [
netmap_node.node for netmap_node in netmap
], f"Node {node_under_test.host_ip} not in state offline"
with reporter.step("Restart storage service"):
cluster_state_controller.stop_storage_service(node_under_test)
cluster_state_controller.start_storage_service(node_under_test)
with reporter.step("Tick epoch after restart storage service and set mode to offline"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with (reporter.step("Check node mode = online, after restart storage service")):
netmap = frostfs_cli.netmap.snapshot(
rpc_endpoint=cluster_nodes[0].storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout
node_state = NetmapParser.snapshot_one_node(netmap, node_under_test).node_status
assert node_state == ModeNode.ONLINE.value.upper(), (
f"Node actual state - {node_state}, " f"expect - {ModeNode.ONLINE.value}"
)
with reporter.step("Tick epoch"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Set node mode to maintenance"):
cluster_state_controller.set_mode_node(
cluster_node=node_under_test, wallet=default_wallet, status=ModeNode.MAINTENANCE.value
)
with reporter.step("Restart storage service"):
cluster_state_controller.stop_storage_service(node_under_test)
cluster_state_controller.start_storage_service(node_under_test)
with reporter.step("Tick epoch after restart storage service"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Check node mode = maintenance, after restart storage service and tick epoch"):
netmap = frostfs_cli.netmap.snapshot(
rpc_endpoint=cluster_nodes[0].storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout
node_state = NetmapParser.snapshot_one_node(netmap, node_under_test).node_status
assert node_state == ModeNode.MAINTENANCE.value.upper(), (
f"Node actual state - {node_state}, " f"expect - {ModeNode.MAINTENANCE.value}"
)
with reporter.step("Tick epoch"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Set node mode to offline"):
netmap = frostfs_cli.netmap.snapshot(
rpc_endpoint=cluster_nodes[0].storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout
node_state = NetmapParser.snapshot_one_node(netmap, node_under_test).node_status
assert node_state == ModeNode.OFFLINE.value.upper(), (
f"Node actual state - {node_state}, " f"expect - {ModeNode.OFFLINE.value}"
)
with reporter.step("Stop storage service"):
cluster_state_controller.stop_storage_service(node_under_test)
with reporter.step("Tick epoch"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Start storage service"):
cluster_state_controller.start_storage_service(node_under_test)
with reporter.step("Check node mode = offline, after tick epoch and start storage service"):
netmap = frostfs_cli.netmap.snapshot(
rpc_endpoint=cluster_nodes[0].storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout
node_state = NetmapParser.snapshot_one_node(netmap, node_under_test).node_status
assert node_state == ModeNode.OFFLINE.value.upper(), (
f"Node actual state - {node_state}, " f"expect - {ModeNode.OFFLINE.value}"
)
with reporter.step("Tick epoch"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Check node mode = online, after start storage service and tick epoch"):
netmap = frostfs_cli.netmap.snapshot(
rpc_endpoint=cluster_nodes[0].storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout
node_state = NetmapParser.snapshot_one_node(netmap, node_under_test).node_status
assert node_state == ModeNode.ONLINE.value.upper(), (
f"Node actual state - {node_state}, " f"expect - {ModeNode.ONLINE.value}"
)
with reporter.step("Tick epoch"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Set node mode to maintenance"):
cluster_state_controller.set_mode_node(
cluster_node=node_under_test, wallet=default_wallet, status=ModeNode.MAINTENANCE.value
)
with reporter.step("Stop storage service"):
cluster_state_controller.stop_storage_service(node_under_test)
with reporter.step("Tick epoch"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Start storage service"):
cluster_state_controller.start_storage_service(node_under_test)
with reporter.step("Check node mode = maintenance"):
netmap = frostfs_cli.netmap.snapshot(
rpc_endpoint=cluster_nodes[0].storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout
node_state = NetmapParser.snapshot_one_node(netmap, node_under_test).node_status
assert (
node_state == ModeNode.MAINTENANCE.value.upper()
), f"Node actual state - {node_state}, expect - {ModeNode.MAINTENANCE.value}"
with reporter.step("Tick epoch"):
self.tick_epochs(epochs_to_tick=2, alive_node=cluster_nodes[0].storage_node, wait_block=2)
with reporter.step("Check node mode = maintenance"):
netmap = frostfs_cli.netmap.snapshot(
rpc_endpoint=cluster_nodes[0].storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout
node_state = NetmapParser.snapshot_one_node(netmap, node_under_test).node_status
assert node_state == ModeNode.MAINTENANCE.value.upper(), (
f"Node actual state - {node_state}, " f"expect - {ModeNode.MAINTENANCE.value}"
)
@allure.title("A node cannot go into maintenance if maintenance is prohibited globally in the network")
def test_maintenance_globally_forbidden(
self,
cluster_state_controller: ClusterStateController,
node_under_test: ClusterNode,
frostfs_cli_remote: FrostfsCli,
frostfs_cli: FrostfsCli,
default_wallet: str,
restore_online_mode_node: None,
):
self.change_node = node_under_test
control_endpoint = node_under_test.service(StorageNode).get_control_endpoint()
with reporter.step("Set MaintenanceModeAllowed = false"):
cluster_state_controller.set_maintenance_mode_allowed("false", node_under_test)
with reporter.step("Set status node - maintenance"):
with pytest.raises(RuntimeError, match="maintenance mode is not allowed by the network"):
frostfs_cli_remote.control.set_status(endpoint=control_endpoint, status="maintenance")
with reporter.step("Set MaintenanceModeAllowed = true"):
cluster_state_controller.set_maintenance_mode_allowed("true", node_under_test)
with reporter.step("Set status node - maintenance"):
output = frostfs_cli_remote.control.set_status(endpoint=control_endpoint, status="maintenance")
assert "update request successfully sent" in output.stdout, f"Response = {output}"
with reporter.step("Tick epoch"):
self.tick_epoch(wait_block=2)
with reporter.step("Check state node = maintenance "):
netmap_node = NetmapParser.snapshot_one_node(
frostfs_cli.netmap.snapshot(
rpc_endpoint=node_under_test.storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout,
node_under_test,
)
assert (
netmap_node.node_status == ModeNode.MAINTENANCE.value.upper()
), f"Node actual state - {netmap_node.node_status}, expect - {ModeNode.MAINTENANCE.value}"
with reporter.step("Set status node - online "):
frostfs_cli_remote.control.set_status(endpoint=control_endpoint, status="online")
with reporter.step("Tick epoch"):
self.tick_epoch()
with reporter.step("Check state node: online"):
netmap_node = NetmapParser.snapshot_one_node(
frostfs_cli.netmap.snapshot(
rpc_endpoint=node_under_test.storage_node.get_rpc_endpoint(), wallet=default_wallet
).stdout,
node_under_test,
)
assert (
netmap_node.node_status == ModeNode.ONLINE.value.upper()
), f"Node actual state - {netmap_node.node_status}, expect - {ModeNode.ONLINE.value}"

324
pytest_tests/testsuites/network/test_node_management.py
View file

@ -1,324 +0,0 @@
import logging
import random
from time import sleep
from typing import Optional, Tuple
import allure
import pytest
from frostfs_testlib.resources.common import FROSTFS_CONTRACT_CACHE_TIMEOUT, MORPH_BLOCK_TIME
from frostfs_testlib.resources.error_patterns import OBJECT_NOT_FOUND
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.cli.container import create_container, get_container
from frostfs_testlib.steps.cli.object import (
delete_object,
get_object,
get_object_from_random_node,
head_object,
put_object,
put_object_to_random_node,
)
from frostfs_testlib.steps.epoch import tick_epoch
from frostfs_testlib.steps.node_management import (
check_node_in_map,
delete_node_data,
drop_object,
exclude_node_from_network_map,
get_locode_from_random_node,
include_node_to_network_map,
node_shard_list,
node_shard_set_mode,
storage_node_healthcheck,
storage_node_set_status,
wait_for_node_to_be_ready,
)
from frostfs_testlib.steps.storage_policy import get_nodes_with_object, get_simple_object_copies
from frostfs_testlib.storage.cluster import StorageNode
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
from frostfs_testlib.utils import datetime_utils, string_utils
from frostfs_testlib.utils.failover_utils import wait_object_replication
from frostfs_testlib.utils.file_utils import generate_file
from pytest_tests.helpers.utility import placement_policy_from_container, wait_for_gc_pass_on_storage_nodes
logger = logging.getLogger("NeoLogger")
check_nodes: list[StorageNode] = []
@allure.title("Add one node to cluster")
@pytest.mark.add_nodes
@pytest.mark.node_mgmt
class TestNodeManagement(ClusterTestBase):
@pytest.fixture
@allure.title("Create container and pick the node with data")
def create_container_and_pick_node(
self, default_wallet: str, simple_object_size: ObjectSize
) -> Tuple[str, StorageNode]:
file_path = generate_file(simple_object_size.value)
placement_rule = "REP 1 IN X CBF 1 SELECT 1 FROM * AS X"
endpoint = self.cluster.default_rpc_endpoint
cid = create_container(
default_wallet,
shell=self.shell,
endpoint=endpoint,
rule=placement_rule,
basic_acl=PUBLIC_ACL,
)
oid = put_object_to_random_node(default_wallet, file_path, cid, self.shell, self.cluster)
nodes = get_nodes_with_object(cid, oid, shell=self.shell, nodes=self.cluster.storage_nodes)
assert len(nodes) == 1
node = nodes[0]
yield cid, node
shards = node_shard_list(node)
assert shards
for shard in shards:
node_shard_set_mode(node, shard, "read-write")
node_shard_list(node)
@allure.step("Tick epoch with retries")
def tick_epoch_with_retries(self, attempts: int = 3, timeout: int = 3, wait_block: int = None):
for attempt in range(attempts):
try:
self.tick_epoch(wait_block=wait_block)
except RuntimeError:
sleep(timeout)
if attempt >= attempts - 1:
raise
continue
return
@pytest.fixture
def after_run_start_all_nodes(self):
yield
self.return_nodes()
@pytest.fixture
def return_nodes_after_test_run(self):
yield
self.return_nodes()
@allure.step("Return node to cluster")
def return_nodes(self, alive_node: Optional[StorageNode] = None) -> None:
for node in list(check_nodes):
with allure.step(f"Start node {node}"):
node.start_service()
with allure.step(f"Waiting status ready for node {node}"):
wait_for_node_to_be_ready(node)
# We need to wait for node to establish notifications from morph-chain
# Otherwise it will hang up when we will try to set status
sleep(datetime_utils.parse_time(MORPH_BLOCK_TIME))
with allure.step(f"Move node {node} to online state"):
storage_node_set_status(node, status="online", retries=2)
check_nodes.remove(node)
sleep(datetime_utils.parse_time(MORPH_BLOCK_TIME))
self.tick_epoch_with_retries(3, wait_block=2)
check_node_in_map(node, shell=self.shell, alive_node=alive_node)
@allure.title("Add one node to cluster")
@pytest.mark.add_nodes
def test_add_nodes(
self,
default_wallet: str,
simple_object_size: ObjectSize,
return_nodes_after_test_run,
):
"""
This test remove one node from frostfs_testlib.storage.cluster then add it back. Test uses base control operations with storage nodes (healthcheck, netmap-snapshot, set-status).
"""
wallet = default_wallet
placement_rule_3 = "REP 3 IN X CBF 1 SELECT 3 FROM * AS X"
placement_rule_4 = "REP 4 IN X CBF 1 SELECT 4 FROM * AS X"
source_file_path = generate_file(simple_object_size.value)
storage_nodes = self.cluster.storage_nodes
random_node = random.choice(storage_nodes[1:])
alive_node = random.choice(
[storage_node for storage_node in storage_nodes if storage_node.id != random_node.id]
)
check_node_in_map(random_node, shell=self.shell, alive_node=alive_node)
# Add node to recovery list before messing with it
check_nodes.append(random_node)
exclude_node_from_network_map(random_node, alive_node, shell=self.shell, cluster=self.cluster)
delete_node_data(random_node)
cid = create_container(
wallet,
rule=placement_rule_3,
basic_acl=PUBLIC_ACL,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
oid = put_object(
wallet,
source_file_path,
cid,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
wait_object_replication(cid, oid, 3, shell=self.shell, nodes=storage_nodes)
self.return_nodes(alive_node)
with allure.step("Check data could be replicated to new node"):
random_node = random.choice(list(set(storage_nodes) - {random_node, alive_node}))
# Add node to recovery list before messing with it
check_nodes.append(random_node)
exclude_node_from_network_map(random_node, alive_node, shell=self.shell, cluster=self.cluster)
wait_object_replication(
cid,
oid,
3,
shell=self.shell,
nodes=list(set(storage_nodes) - {random_node}),
)
include_node_to_network_map(random_node, alive_node, shell=self.shell, cluster=self.cluster)
wait_object_replication(cid, oid, 3, shell=self.shell, nodes=storage_nodes)
with allure.step("Check container could be created with new node"):
cid = create_container(
wallet,
rule=placement_rule_4,
basic_acl=PUBLIC_ACL,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
oid = put_object(
wallet,
source_file_path,
cid,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
wait_object_replication(cid, oid, 4, shell=self.shell, nodes=storage_nodes)
@pytest.mark.node_mgmt
@allure.title("Drop object using control command")
def test_drop_object(self, default_wallet, complex_object_size: ObjectSize, simple_object_size: ObjectSize):
"""
Test checks object could be dropped using `frostfs-cli control drop-objects` command.
"""
wallet = default_wallet
endpoint = self.cluster.default_rpc_endpoint
file_path_simple = generate_file(simple_object_size.value)
file_path_complex = generate_file(complex_object_size.value)
locode = get_locode_from_random_node(self.cluster)
rule = f"REP 1 IN SE CBF 1 SELECT 1 FROM LOC AS SE FILTER 'UN-LOCODE' EQ '{locode}' AS LOC"
cid = create_container(wallet, rule=rule, shell=self.shell, endpoint=endpoint)
oid_simple = put_object_to_random_node(wallet, file_path_simple, cid, shell=self.shell, cluster=self.cluster)
oid_complex = put_object_to_random_node(wallet, file_path_complex, cid, shell=self.shell, cluster=self.cluster)
for oid in (oid_simple, oid_complex):
get_object_from_random_node(wallet, cid, oid, shell=self.shell, cluster=self.cluster)
head_object(wallet, cid, oid, shell=self.shell, endpoint=endpoint)
nodes_with_object = get_nodes_with_object(cid, oid_simple, shell=self.shell, nodes=self.cluster.storage_nodes)
random_node = random.choice(nodes_with_object)
for oid in (oid_simple, oid_complex):
with allure.step(f"Drop object {oid}"):
get_object_from_random_node(wallet, cid, oid, shell=self.shell, cluster=self.cluster)
head_object(wallet, cid, oid, shell=self.shell, endpoint=endpoint)
drop_object(random_node, cid, oid)
self.wait_for_obj_dropped(wallet, cid, oid, endpoint, get_object)
self.wait_for_obj_dropped(wallet, cid, oid, endpoint, head_object)
@pytest.mark.node_mgmt
@pytest.mark.skip(reason="Need to clarify scenario")
@allure.title("Control Operations with storage nodes")
def test_shards(
self,
default_wallet,
create_container_and_pick_node,
simple_object_size: ObjectSize,
):
wallet = default_wallet
file_path = generate_file(simple_object_size.value)
cid, node = create_container_and_pick_node
original_oid = put_object_to_random_node(wallet, file_path, cid, self.shell, self.cluster)
# for mode in ('read-only', 'degraded'):
for mode in ("degraded",):
shards = node_shard_list(node)
assert shards
for shard in shards:
node_shard_set_mode(node, shard, mode)
shards = node_shard_list(node)
assert shards
with pytest.raises(RuntimeError):
put_object_to_random_node(wallet, file_path, cid, self.shell, self.cluster)
with pytest.raises(RuntimeError):
delete_object(wallet, cid, original_oid, self.shell, self.cluster.default_rpc_endpoint)
get_object_from_random_node(wallet, cid, original_oid, self.shell, self.cluster)
for shard in shards:
node_shard_set_mode(node, shard, "read-write")
shards = node_shard_list(node)
assert shards
oid = put_object_to_random_node(wallet, file_path, cid, self.shell, self.cluster)
delete_object(wallet, cid, oid, self.shell, self.cluster.default_rpc_endpoint)
@pytest.mark.node_mgmt
@allure.title("Put object with stopped node")
def test_stop_node(self, default_wallet, return_nodes_after_test_run, simple_object_size: ObjectSize):
wallet = default_wallet
placement_rule = "REP 3 IN X SELECT 4 FROM * AS X"
source_file_path = generate_file(simple_object_size.value)
storage_nodes = self.cluster.storage_nodes
random_node = random.choice(storage_nodes[1:])
alive_node = random.choice(
[storage_node for storage_node in storage_nodes if storage_node.id != random_node.id]
)
cid = create_container(
wallet,
rule=placement_rule,
basic_acl=PUBLIC_ACL,
shell=self.shell,
endpoint=random_node.get_rpc_endpoint(),
)
with allure.step("Stop the random node"):
check_nodes.append(random_node)
random_node.stop_service()
with allure.step("Try to put an object and expect success"):
put_object(
wallet,
source_file_path,
cid,
shell=self.shell,
endpoint=alive_node.get_rpc_endpoint(),
)
self.return_nodes(alive_node)
@allure.step("Wait for object to be dropped")
def wait_for_obj_dropped(self, wallet: str, cid: str, oid: str, endpoint: str, checker) -> None:
for _ in range(3):
try:
checker(wallet, cid, oid, shell=self.shell, endpoint=endpoint)
wait_for_gc_pass_on_storage_nodes()
except Exception as err:
if string_utils.is_str_match_pattern(err, OBJECT_NOT_FOUND):
return
raise AssertionError(f'Expected "{OBJECT_NOT_FOUND}" error, got\n{err}')
raise AssertionError(f"Object {oid} was not dropped from node")

88
pytest_tests/testsuites/object/test_object_api.py
View file

@ -4,6 +4,7 @@ import sys
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.error_patterns import (
INVALID_LENGTH_SPECIFIER,
INVALID_OFFSET_SPECIFIER,
@ -79,9 +80,7 @@ def generate_ranges(
range_length = random.randint(RANGE_MIN_LEN, RANGE_MAX_LEN)
range_start = random.randint(offset, offset + length)
file_ranges_to_test.append(
(range_start, min(range_length, storage_object.size - range_start))
)
file_ranges_to_test.append((range_start, min(range_length, storage_object.size - range_start)))
file_ranges_to_test.extend(STATIC_RANGES.get(storage_object.size, []))
@ -104,7 +103,7 @@ def storage_objects(
storage_objects = []
with allure.step("Put objects"):
with reporter.step("Put objects"):
# We need to upload objects multiple times with different attributes
for attributes in OBJECT_ATTRIBUTES:
storage_object_id = put_object_to_random_node(
@ -144,7 +143,7 @@ class TestObjectApi(ClusterTestBase):
Validate object storage policy
"""
with allure.step("Validate storage policy for objects"):
with reporter.step("Validate storage policy for objects"):
for storage_object in storage_objects:
if storage_object.size == simple_object_size.value:
copies = get_simple_object_copies(
@ -170,7 +169,7 @@ class TestObjectApi(ClusterTestBase):
Validate get object native API
"""
with allure.step("Get objects and compare hashes"):
with reporter.step("Get objects and compare hashes"):
for storage_object in storage_objects:
file_path = get_object_from_random_node(
storage_object.wallet_file_path,
@ -191,7 +190,7 @@ class TestObjectApi(ClusterTestBase):
storage_object_1 = storage_objects[0]
storage_object_2 = storage_objects[1]
with allure.step("Head object and validate"):
with reporter.step("Head object and validate"):
head_object(
storage_object_1.wallet_file_path,
storage_object_1.cid,
@ -224,7 +223,7 @@ class TestObjectApi(ClusterTestBase):
(COMMON_ATTRIBUTE, oids[1:3]),
]
with allure.step("Search objects"):
with reporter.step("Search objects"):
# Search with no attributes
result = search_object(
wallet,
@ -250,9 +249,7 @@ class TestObjectApi(ClusterTestBase):
assert sorted(expected_oids) == sorted(result)
@allure.title("Search objects with removed items (obj_size={object_size})")
def test_object_search_should_return_tombstone_items(
self, default_wallet: str, object_size: ObjectSize
):
def test_object_search_should_return_tombstone_items(self, default_wallet: str, object_size: ObjectSize):
"""
Validate object search with removed items
"""
@ -260,7 +257,7 @@ class TestObjectApi(ClusterTestBase):
wallet = default_wallet
cid = create_container(wallet, self.shell, self.cluster.default_rpc_endpoint)
with allure.step("Upload file"):
with reporter.step("Upload file"):
file_path = generate_file(object_size.value)
file_hash = get_file_hash(file_path)
@ -273,34 +270,24 @@ class TestObjectApi(ClusterTestBase):
file_hash=file_hash,
)
with allure.step("Search object"):
with reporter.step("Search object"):
# Root Search object should return root object oid
result = search_object(
wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint, root=True
)
result = search_object(wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint, root=True)
assert result == [storage_object.oid]
with allure.step("Delete file"):
with reporter.step("Delete file"):
delete_objects([storage_object], self.shell, self.cluster)
with allure.step("Search deleted object with --root"):
with reporter.step("Search deleted object with --root"):
# Root Search object should return nothing
result = search_object(
wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint, root=True
)
result = search_object(wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint, root=True)
assert len(result) == 0
with allure.step("Search deleted object with --phy should return only tombstones"):
with reporter.step("Search deleted object with --phy should return only tombstones"):
# Physical Search object should return only tombstones
result = search_object(
wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint, phy=True
)
assert (
storage_object.tombstone in result
), "Search result should contain tombstone of removed object"
assert (
storage_object.oid not in result
), "Search result should not contain ObjectId of removed object"
result = search_object(wallet, cid, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint, phy=True)
assert storage_object.tombstone in result, "Search result should contain tombstone of removed object"
assert storage_object.oid not in result, "Search result should not contain ObjectId of removed object"
for tombstone_oid in result:
header = head_object(
wallet,
@ -315,7 +302,6 @@ class TestObjectApi(ClusterTestBase):
), f"Object wasn't deleted properly. Found object {tombstone_oid} with type {object_type}"
@allure.title("Get range hash by native API (obj_size={object_size})")
@pytest.mark.sanity
@pytest.mark.grpc_api
def test_object_get_range_hash(self, storage_objects: list[StorageObjectInfo], max_object_size):
"""
@ -327,14 +313,12 @@ class TestObjectApi(ClusterTestBase):
oids = [storage_object.oid for storage_object in storage_objects[:2]]
file_path = storage_objects[0].file_path
file_ranges_to_test = generate_ranges(
storage_objects[0], max_object_size, self.shell, self.cluster
)
file_ranges_to_test = generate_ranges(storage_objects[0], max_object_size, self.shell, self.cluster)
logging.info(f"Ranges used in test {file_ranges_to_test}")
for range_start, range_len in file_ranges_to_test:
range_cut = f"{range_start}:{range_len}"
with allure.step(f"Get range hash ({range_cut})"):
with reporter.step(f"Get range hash ({range_cut})"):
for oid in oids:
range_hash = get_range_hash(
wallet,
@ -349,7 +333,6 @@ class TestObjectApi(ClusterTestBase):
), f"Expected range hash to match {range_cut} slice of file payload"
@allure.title("Get range by native API (obj_size={object_size})")
@pytest.mark.sanity
@pytest.mark.grpc_api
def test_object_get_range(self, storage_objects: list[StorageObjectInfo], max_object_size):
"""
@ -361,14 +344,12 @@ class TestObjectApi(ClusterTestBase):
oids = [storage_object.oid for storage_object in storage_objects[:2]]
file_path = storage_objects[0].file_path
file_ranges_to_test = generate_ranges(
storage_objects[0], max_object_size, self.shell, self.cluster
)
file_ranges_to_test = generate_ranges(storage_objects[0], max_object_size, self.shell, self.cluster)
logging.info(f"Ranges used in test {file_ranges_to_test}")
for range_start, range_len in file_ranges_to_test:
range_cut = f"{range_start}:{range_len}"
with allure.step(f"Get range ({range_cut})"):
with reporter.step(f"Get range ({range_cut})"):
for oid in oids:
_, range_content = get_range(
wallet,
@ -379,14 +360,11 @@ class TestObjectApi(ClusterTestBase):
range_cut=range_cut,
)
assert (
get_file_content(
file_path, content_len=range_len, mode="rb", offset=range_start
)
get_file_content(file_path, content_len=range_len, mode="rb", offset=range_start)
== range_content
), f"Expected range content to match {range_cut} slice of file payload"
@allure.title("[NEGATIVE] Get invalid range by native API (obj_size={object_size})")
@pytest.mark.sanity
@pytest.mark.grpc_api
def test_object_get_range_negatives(
self,
@ -421,12 +399,8 @@ class TestObjectApi(ClusterTestBase):
for range_start, range_len, expected_error in file_ranges_to_test:
range_cut = f"{range_start}:{range_len}"
expected_error = (
expected_error.format(range=range_cut)
if "{range}" in expected_error
else expected_error
)
with allure.step(f"Get range ({range_cut})"):
expected_error = expected_error.format(range=range_cut) if "{range}" in expected_error else expected_error
with reporter.step(f"Get range ({range_cut})"):
for oid in oids:
with pytest.raises(Exception, match=expected_error):
get_range(
@ -472,12 +446,8 @@ class TestObjectApi(ClusterTestBase):
for range_start, range_len, expected_error in file_ranges_to_test:
range_cut = f"{range_start}:{range_len}"
expected_error = (
expected_error.format(range=range_cut)
if "{range}" in expected_error
else expected_error
)
with allure.step(f"Get range hash ({range_cut})"):
expected_error = expected_error.format(range=range_cut) if "{range}" in expected_error else expected_error
with reporter.step(f"Get range hash ({range_cut})"):
for oid in oids:
with pytest.raises(Exception, match=expected_error):
get_range_hash(
@ -491,9 +461,7 @@ class TestObjectApi(ClusterTestBase):
def check_header_is_presented(self, head_info: dict, object_header: dict) -> None:
for key_to_check, val_to_check in object_header.items():
assert (
key_to_check in head_info["header"]["attributes"]
), f"Key {key_to_check} is found in {head_object}"
assert key_to_check in head_info["header"]["attributes"], f"Key {key_to_check} is found in {head_object}"
assert head_info["header"]["attributes"].get(key_to_check) == str(
val_to_check
), f"Value {val_to_check} is equal"

20
pytest_tests/testsuites/object/test_object_api_bearer.py
View file

@ -1,5 +1,6 @@
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import EACL_PUBLIC_READ_WRITE
from frostfs_testlib.shell import Shell
from frostfs_testlib.steps.acl import form_bearertoken_file
@ -28,10 +29,7 @@ def bearer_token_file_all_allow(default_wallet: str, client_shell: Shell, cluste
bearer = form_bearertoken_file(
default_wallet,
"",
[
EACLRule(operation=op, access=EACLAccess.ALLOW, role=EACLRole.OTHERS)
for op in EACLOperation
],
[EACLRule(operation=op, access=EACLAccess.ALLOW, role=EACLRole.OTHERS) for op in EACLOperation],
shell=client_shell,
endpoint=cluster.default_rpc_endpoint,
)
@ -85,9 +83,7 @@ def storage_objects(
@pytest.mark.smoke
@pytest.mark.bearer
class TestObjectApiWithBearerToken(ClusterTestBase):
@allure.title(
"Object can be deleted from any node using s3gate wallet with bearer token (obj_size={object_size})"
)
@allure.title("Object can be deleted from any node using s3gate wallet with bearer token (obj_size={object_size})")
@pytest.mark.parametrize(
"user_container",
[SINGLE_PLACEMENT_RULE],
@ -99,7 +95,7 @@ class TestObjectApiWithBearerToken(ClusterTestBase):
bearer_token_file_all_allow: str,
):
s3_gate_wallet = self.cluster.s3_gates[0]
with allure.step("Try to delete each object from first storage node"):
with reporter.step("Try to delete each object from first storage node"):
for storage_object in storage_objects:
with expect_not_raises():
delete_object(
@ -112,9 +108,7 @@ class TestObjectApiWithBearerToken(ClusterTestBase):
wallet_config=s3_gate_wallet.get_wallet_config_path(),
)
@allure.title(
"Object can be fetched from any node using s3gate wallet with bearer token (obj_size={object_size})"
)
@allure.title("Object can be fetched from any node using s3gate wallet with bearer token (obj_size={object_size})")
@pytest.mark.parametrize(
"user_container",
[REP_2_FOR_3_NODES_PLACEMENT_RULE],
@ -127,13 +121,13 @@ class TestObjectApiWithBearerToken(ClusterTestBase):
bearer_token_file_all_allow: str,
):
s3_gate_wallet = self.cluster.s3_gates[0]
with allure.step("Put one object to container"):
with reporter.step("Put one object to container"):
epoch = self.get_epoch()
storage_object = user_container.generate_object(
object_size.value, epoch + 3, bearer_token=bearer_token_file_all_allow
)
with allure.step("Try to fetch object from each storage node"):
with reporter.step("Try to fetch object from each storage node"):
for node in self.cluster.storage_nodes:
with expect_not_raises():
get_object(

19
pytest_tests/testsuites/object/test_object_lifetime.py
View file

@ -2,13 +2,10 @@ import logging
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.error_patterns import OBJECT_NOT_FOUND
from frostfs_testlib.steps.cli.container import create_container
from frostfs_testlib.steps.cli.object import (
get_object_from_random_node,
head_object,
put_object_to_random_node,
)
from frostfs_testlib.steps.cli.object import get_object_from_random_node, head_object, put_object_to_random_node
from frostfs_testlib.steps.epoch import get_epoch
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
@ -36,31 +33,29 @@ class TestObjectApiLifetime(ClusterTestBase):
file_hash = get_file_hash(file_path)
epoch = get_epoch(self.shell, self.cluster)
oid = put_object_to_random_node(
wallet, file_path, cid, self.shell, self.cluster, expire_at=epoch + 1
)
oid = put_object_to_random_node(wallet, file_path, cid, self.shell, self.cluster, expire_at=epoch + 1)
got_file = get_object_from_random_node(wallet, cid, oid, self.shell, self.cluster)
assert get_file_hash(got_file) == file_hash
with allure.step("Tick two epochs"):
with reporter.step("Tick two epochs"):
for _ in range(2):
self.tick_epoch()
# Wait for GC, because object with expiration is counted as alive until GC removes it
wait_for_gc_pass_on_storage_nodes()
with allure.step("Check object deleted because it expires on epoch"):
with reporter.step("Check object deleted because it expires on epoch"):
with pytest.raises(Exception, match=OBJECT_NOT_FOUND):
head_object(wallet, cid, oid, self.shell, self.cluster.default_rpc_endpoint)
with pytest.raises(Exception, match=OBJECT_NOT_FOUND):
get_object_from_random_node(wallet, cid, oid, self.shell, self.cluster)
with allure.step("Tick additional epoch"):
with reporter.step("Tick additional epoch"):
self.tick_epoch()
wait_for_gc_pass_on_storage_nodes()
with allure.step("Check object deleted because it expires on previous epoch"):
with reporter.step("Check object deleted because it expires on previous epoch"):
with pytest.raises(Exception, match=OBJECT_NOT_FOUND):
head_object(wallet, cid, oid, self.shell, self.cluster.default_rpc_endpoint)
with pytest.raises(Exception, match=OBJECT_NOT_FOUND):

146
pytest_tests/testsuites/object/test_object_lock.py
View file

@ -3,6 +3,7 @@ import re
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.common import STORAGE_GC_TIME
from frostfs_testlib.resources.error_patterns import (
LIFETIME_REQUIRED,
@ -14,11 +15,7 @@ from frostfs_testlib.resources.error_patterns import (
OBJECT_NOT_FOUND,
)
from frostfs_testlib.shell import Shell
from frostfs_testlib.steps.cli.container import (
StorageContainer,
StorageContainerInfo,
create_container,
)
from frostfs_testlib.steps.cli.container import StorageContainer, StorageContainerInfo, create_container
from frostfs_testlib.steps.cli.object import delete_object, head_object, lock_object
from frostfs_testlib.steps.complex_object_actions import get_link_object, get_storage_object_chunks
from frostfs_testlib.steps.epoch import ensure_fresh_epoch, get_epoch, tick_epoch
@ -27,10 +24,7 @@ from frostfs_testlib.steps.storage_object import delete_objects
from frostfs_testlib.steps.storage_policy import get_nodes_with_object
from frostfs_testlib.storage.cluster import Cluster
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.storage.dataclasses.storage_object_info import (
LockObjectInfo,
StorageObjectInfo,
)
from frostfs_testlib.storage.dataclasses.storage_object_info import LockObjectInfo, StorageObjectInfo
from frostfs_testlib.storage.dataclasses.wallet import WalletFactory, WalletInfo
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
from frostfs_testlib.testing.test_control import expect_not_raises, wait_for_success
@ -48,7 +42,7 @@ FIXTURE_OBJECT_LIFETIME = 10
scope="module",
)
def user_wallet(wallet_factory: WalletFactory):
with allure.step("Create user wallet with container"):
with reporter.step("Create user wallet with container"):
wallet_file = wallet_factory.create_wallet()
return wallet_file
@ -57,9 +51,7 @@ def user_wallet(wallet_factory: WalletFactory):
scope="module",
)
def user_container(user_wallet: WalletInfo, client_shell: Shell, cluster: Cluster):
container_id = create_container(
user_wallet.path, shell=client_shell, endpoint=cluster.default_rpc_endpoint
)
container_id = create_container(user_wallet.path, shell=client_shell, endpoint=cluster.default_rpc_endpoint)
return StorageContainer(StorageContainerInfo(container_id, user_wallet), client_shell, cluster)
@ -75,7 +67,7 @@ def locked_storage_object(
"""
Intention of this fixture is to provide storage object which is NOT expected to be deleted during test act phase
"""
with allure.step("Creating locked object"):
with reporter.step("Creating locked object"):
current_epoch = ensure_fresh_epoch(client_shell, cluster)
expiration_epoch = current_epoch + FIXTURE_LOCK_LIFETIME
@ -91,19 +83,17 @@ def locked_storage_object(
lifetime=FIXTURE_LOCK_LIFETIME,
)
storage_object.locks = [
LockObjectInfo(
storage_object.cid, lock_object_id, FIXTURE_LOCK_LIFETIME, expiration_epoch
)
LockObjectInfo(storage_object.cid, lock_object_id, FIXTURE_LOCK_LIFETIME, expiration_epoch)
]
yield storage_object
with allure.step("Delete created locked object"):
with reporter.step("Delete created locked object"):
current_epoch = get_epoch(client_shell, cluster)
epoch_diff = expiration_epoch - current_epoch + 1
if epoch_diff > 0:
with allure.step(f"Tick {epoch_diff} epochs"):
with reporter.step(f"Tick {epoch_diff} epochs"):
for _ in range(epoch_diff):
tick_epoch(client_shell, cluster)
try:
@ -117,17 +107,13 @@ def locked_storage_object(
except Exception as ex:
ex_message = str(ex)
# It's okay if object already removed
if not re.search(OBJECT_NOT_FOUND, ex_message) and not re.search(
OBJECT_ALREADY_REMOVED, ex_message
):
if not re.search(OBJECT_NOT_FOUND, ex_message) and not re.search(OBJECT_ALREADY_REMOVED, ex_message):
raise ex
logger.debug(ex_message)
@wait_for_success(datetime_utils.parse_time(STORAGE_GC_TIME))
def check_object_not_found(
wallet_file_path: str, cid: str, oid: str, shell: Shell, rpc_endpoint: str
):
def check_object_not_found(wallet_file_path: str, cid: str, oid: str, shell: Shell, rpc_endpoint: str):
with pytest.raises(Exception, match=OBJECT_NOT_FOUND):
head_object(
wallet_file_path,
@ -138,9 +124,7 @@ def check_object_not_found(
)
def verify_object_available(
wallet_file_path: str, cid: str, oid: str, shell: Shell, rpc_endpoint: str
):
def verify_object_available(wallet_file_path: str, cid: str, oid: str, shell: Shell, rpc_endpoint: str):
with expect_not_raises():
head_object(
wallet_file_path,
@ -151,18 +135,15 @@ def verify_object_available(
)
@pytest.mark.sanity
@pytest.mark.grpc_object_lock
class TestObjectLockWithGrpc(ClusterTestBase):
@pytest.fixture()
def new_locked_storage_object(
self, user_container: StorageContainer, object_size: ObjectSize
) -> StorageObjectInfo:
def new_locked_storage_object(self, user_container: StorageContainer, object_size: ObjectSize) -> StorageObjectInfo:
"""
Intention of this fixture is to provide new storage object for tests which may delete or corrupt the object or it's complementary objects
So we need a new one each time we ask for it
"""
with allure.step("Creating locked object"):
with reporter.step("Creating locked object"):
current_epoch = self.get_epoch()
storage_object = user_container.generate_object(
@ -284,6 +265,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
expire_at=wrong_expire_at,
)
@pytest.mark.sanity
@allure.title("Expired object is deleted when locks are expired (obj_size={object_size})")
def test_expired_object_should_be_deleted_after_locks_are_expired(
self,
@ -295,11 +277,9 @@ class TestObjectLockWithGrpc(ClusterTestBase):
"""
current_epoch = self.ensure_fresh_epoch()
storage_object = user_container.generate_object(
object_size.value, expire_at=current_epoch + 1
)
storage_object = user_container.generate_object(object_size.value, expire_at=current_epoch + 1)
with allure.step("Lock object for couple epochs"):
with reporter.step("Lock object for couple epochs"):
lock_object(
storage_object.wallet_file_path,
storage_object.cid,
@ -317,7 +297,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
expire_at=current_epoch + 2,
)
with allure.step("Check object is not deleted at expiration time"):
with reporter.step("Check object is not deleted at expiration time"):
self.tick_epochs(2)
# Must wait to ensure object is not deleted
wait_for_gc_pass_on_storage_nodes()
@ -330,7 +310,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.cluster.default_rpc_endpoint,
)
with allure.step("Wait for object to be deleted after third epoch"):
with reporter.step("Wait for object to be deleted after third epoch"):
self.tick_epoch()
check_object_not_found(
storage_object.wallet_file_path,
@ -353,11 +333,9 @@ class TestObjectLockWithGrpc(ClusterTestBase):
current_epoch = ensure_fresh_epoch(self.shell, self.cluster)
storage_objects: list[StorageObjectInfo] = []
with allure.step("Generate three objects"):
with reporter.step("Generate three objects"):
for _ in range(3):
storage_objects.append(
user_container.generate_object(object_size.value, expire_at=current_epoch + 5)
)
storage_objects.append(user_container.generate_object(object_size.value, expire_at=current_epoch + 5))
lock_object(
storage_objects[0].wallet_file_path,
@ -369,7 +347,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
)
for storage_object in storage_objects:
with allure.step(f"Try to delete object {storage_object.oid}"):
with reporter.step(f"Try to delete object {storage_object.oid}"):
with pytest.raises(Exception, match=OBJECT_IS_LOCKED):
delete_object(
storage_object.wallet_file_path,
@ -379,7 +357,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.cluster.default_rpc_endpoint,
)
with allure.step("Tick two epochs"):
with reporter.step("Tick two epochs"):
self.tick_epoch()
self.tick_epoch()
@ -398,16 +376,12 @@ class TestObjectLockWithGrpc(ClusterTestBase):
current_epoch = self.ensure_fresh_epoch()
storage_object = user_container.generate_object(
object_size.value, expire_at=current_epoch + 1
)
storage_object = user_container.generate_object(object_size.value, expire_at=current_epoch + 1)
expiration_epoch = current_epoch - 1
with pytest.raises(
Exception,
match=LOCK_OBJECT_EXPIRATION.format(
expiration_epoch=expiration_epoch, current_epoch=current_epoch
),
match=LOCK_OBJECT_EXPIRATION.format(expiration_epoch=expiration_epoch, current_epoch=current_epoch),
):
lock_object(
storage_object.wallet_file_path,
@ -418,6 +392,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
expire_at=expiration_epoch,
)
@pytest.mark.sanity
@allure.title("Delete object when lock is expired by lifetime (obj_size={object_size})")
@expect_not_raises()
def test_after_lock_expiration_with_lifetime_user_should_be_able_to_delete_object(
@ -430,9 +405,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
"""
current_epoch = self.ensure_fresh_epoch()
storage_object = user_container.generate_object(
object_size.value, expire_at=current_epoch + 5
)
storage_object = user_container.generate_object(object_size.value, expire_at=current_epoch + 5)
lock_object(
storage_object.wallet_file_path,
@ -466,9 +439,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
current_epoch = self.ensure_fresh_epoch()
storage_object = user_container.generate_object(
object_size.value, expire_at=current_epoch + 5
)
storage_object = user_container.generate_object(object_size.value, expire_at=current_epoch + 5)
lock_object(
storage_object.wallet_file_path,
@ -505,11 +476,9 @@ class TestObjectLockWithGrpc(ClusterTestBase):
Complex object chunks should also be protected from deletion
"""
chunk_object_ids = get_storage_object_chunks(
locked_storage_object, self.shell, self.cluster
)
chunk_object_ids = get_storage_object_chunks(locked_storage_object, self.shell, self.cluster)
for chunk_object_id in chunk_object_ids:
with allure.step(f"Try to delete chunk object {chunk_object_id}"):
with reporter.step(f"Try to delete chunk object {chunk_object_id}"):
with pytest.raises(Exception, match=OBJECT_IS_LOCKED):
delete_object(
locked_storage_object.wallet_file_path,
@ -527,9 +496,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
["complex"],
indirect=True,
)
def test_link_object_of_locked_complex_object_can_be_dropped(
self, new_locked_storage_object: StorageObjectInfo
):
def test_link_object_of_locked_complex_object_can_be_dropped(self, new_locked_storage_object: StorageObjectInfo):
link_object_id = get_link_object(
new_locked_storage_object.wallet_file_path,
new_locked_storage_object.cid,
@ -538,7 +505,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.cluster.storage_nodes,
)
with allure.step(f"Drop link object with id {link_object_id} from nodes"):
with reporter.step(f"Drop link object with id {link_object_id} from nodes"):
nodes_with_object = get_nodes_with_object(
new_locked_storage_object.cid,
link_object_id,
@ -557,15 +524,11 @@ class TestObjectLockWithGrpc(ClusterTestBase):
["complex"],
indirect=True,
)
def test_chunks_of_locked_complex_object_can_be_dropped(
self, new_locked_storage_object: StorageObjectInfo
):
chunk_objects = get_storage_object_chunks(
new_locked_storage_object, self.shell, self.cluster
)
def test_chunks_of_locked_complex_object_can_be_dropped(self, new_locked_storage_object: StorageObjectInfo):
chunk_objects = get_storage_object_chunks(new_locked_storage_object, self.shell, self.cluster)
for chunk_object_id in chunk_objects:
with allure.step(f"Drop chunk object with id {chunk_object_id} from nodes"):
with reporter.step(f"Drop chunk object with id {chunk_object_id} from nodes"):
nodes_with_object = get_nodes_with_object(
new_locked_storage_object.cid,
chunk_object_id,
@ -613,7 +576,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.cluster.storage_nodes,
is_direct=False,
)
with allure.step(f"Try to delete link object {link_object_id}"):
with reporter.step(f"Try to delete link object {link_object_id}"):
with pytest.raises(Exception, match=OBJECT_IS_LOCKED):
delete_object(
locked_storage_object.wallet_file_path,
@ -630,11 +593,9 @@ class TestObjectLockWithGrpc(ClusterTestBase):
object_size: ObjectSize,
):
current_epoch = self.ensure_fresh_epoch()
storage_object = user_container.generate_object(
object_size.value, expire_at=current_epoch + 1
)
storage_object = user_container.generate_object(object_size.value, expire_at=current_epoch + 1)
with allure.step("Apply first lock to object for 3 epochs"):
with reporter.step("Apply first lock to object for 3 epochs"):
lock_object_id_0 = lock_object(
storage_object.wallet_file_path,
storage_object.cid,
@ -646,7 +607,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.tick_epochs(2)
with allure.step("Check first lock is still available"):
with reporter.step("Check first lock is still available"):
verify_object_available(
storage_object.wallet_file_path,
storage_object.cid,
@ -655,7 +616,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.cluster.default_rpc_endpoint,
)
with allure.step("Apply second lock to object for 3 more epochs"):
with reporter.step("Apply second lock to object for 3 more epochs"):
lock_object_id_1 = lock_object(
storage_object.wallet_file_path,
storage_object.cid,
@ -667,7 +628,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.tick_epochs(2)
with allure.step("Verify first lock is expired and removed"):
with reporter.step("Verify first lock is expired and removed"):
check_object_not_found(
storage_object.wallet_file_path,
storage_object.cid,
@ -676,7 +637,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.cluster.default_rpc_endpoint,
)
with allure.step("Verify second lock is still available"):
with reporter.step("Verify second lock is still available"):
verify_object_available(
storage_object.wallet_file_path,
storage_object.cid,
@ -685,7 +646,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.cluster.default_rpc_endpoint,
)
with allure.step("Apply third lock to object for 3 more epochs"):
with reporter.step("Apply third lock to object for 3 more epochs"):
lock_object(
storage_object.wallet_file_path,
storage_object.cid,
@ -695,7 +656,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
expire_at=current_epoch + 7,
)
with allure.step("Verify object is deleted after all locks are expired"):
with reporter.step("Verify object is deleted after all locks are expired"):
self.tick_epochs(4)
check_object_not_found(
storage_object.wallet_file_path,
@ -705,9 +666,8 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.cluster.default_rpc_endpoint,
)
@allure.title(
"Two expired objects with one lock are deleted after lock expiration (obj_size={object_size})"
)
@pytest.mark.sanity
@allure.title("Two expired objects with one lock are deleted after lock expiration (obj_size={object_size})")
def test_two_objects_expiration_with_one_lock(
self,
user_container: StorageContainer,
@ -717,17 +677,15 @@ class TestObjectLockWithGrpc(ClusterTestBase):
current_epoch = self.ensure_fresh_epoch()
storage_objects: list[StorageObjectInfo] = []
with allure.step("Generate two objects"):
with reporter.step("Generate two objects"):
for epoch_i in range(2):
storage_objects.append(
user_container.generate_object(
object_size.value, expire_at=current_epoch + epoch_i + 3
)
user_container.generate_object(object_size.value, expire_at=current_epoch + epoch_i + 3)
)
self.tick_epoch()
with allure.step("Lock objects for 4 epochs"):
with reporter.step("Lock objects for 4 epochs"):
lock_object(
storage_objects[0].wallet_file_path,
storage_objects[0].cid,
@ -737,10 +695,10 @@ class TestObjectLockWithGrpc(ClusterTestBase):
expire_at=current_epoch + 4,
)
with allure.step("Verify objects are available during next three epochs"):
with reporter.step("Verify objects are available during next three epochs"):
for epoch_i in range(3):
self.tick_epoch()
with allure.step(f"Check objects at epoch {current_epoch + epoch_i + 2}"):
with reporter.step(f"Check objects at epoch {current_epoch + epoch_i + 2}"):
for storage_object in storage_objects:
verify_object_available(
storage_object.wallet_file_path,
@ -750,7 +708,7 @@ class TestObjectLockWithGrpc(ClusterTestBase):
self.cluster.default_rpc_endpoint,
)
with allure.step("Verify objects are deleted after lock was expired"):
with reporter.step("Verify objects are deleted after lock was expired"):
self.tick_epoch()
for storage_object in storage_objects:
check_object_not_found(

10
pytest_tests/testsuites/replication/test_replication.py
View file

@ -4,6 +4,7 @@ from time import sleep
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.shell import Shell
from frostfs_testlib.steps.cli.container import create_container, delete_container
from frostfs_testlib.steps.cli.object import delete_object, head_object, put_object
@ -20,6 +21,7 @@ OBJECT_ATTRIBUTES = {"common_key": "common_value"}
WAIT_FOR_REPLICATION = 60
# Adding failover mark because it may make cluster unhealthy
@pytest.mark.sanity
@pytest.mark.failover
@pytest.mark.replication
class TestReplication(ClusterTestBase):
@ -55,7 +57,7 @@ class TestReplication(ClusterTestBase):
file_path = generate_file(object_size.value)
with allure.step("Put object"):
with reporter.step("Put object"):
oid = put_object(
wallet=default_wallet,
path=file_path,
@ -68,7 +70,7 @@ class TestReplication(ClusterTestBase):
)
cluster_state_controller.start_node_host(node_for_rep)
with allure.step(f"Wait for replication."):
with reporter.step(f"Wait for replication."):
object_nodes = wait_object_replication(
cid=cid,
oid=oid,
@ -77,7 +79,7 @@ class TestReplication(ClusterTestBase):
nodes=self.cluster.storage_nodes,
)
with allure.step("Check attributes"):
with reporter.step("Check attributes"):
for node in object_nodes:
header_info = head_object(
wallet=default_wallet,
@ -96,7 +98,7 @@ class TestReplication(ClusterTestBase):
f"expected attribute value: {attribute_value}"
)
with allure.step("Cleanup"):
with reporter.step("Cleanup"):
delete_object(
wallet=default_wallet,
cid=cid,

22
pytest_tests/testsuites/services/http_gate/test_http_bearer.py
View file

@ -2,6 +2,7 @@ import logging
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.acl import (
bearer_token_base64_from_file,
@ -21,9 +22,8 @@ from frostfs_testlib.utils.file_utils import generate_file
logger = logging.getLogger("NeoLogger")
@pytest.mark.sanity
@pytest.mark.skip("Skipped temporarly")
@pytest.mark.http_gate
@pytest.mark.skip("Skipped due to deprecated PUT via http")
@pytest.mark.http_put
class Test_http_bearer(ClusterTestBase):
PLACEMENT_RULE = "REP 2 IN X CBF 1 SELECT 2 FROM * AS X"
@ -45,10 +45,8 @@ class Test_http_bearer(ClusterTestBase):
@pytest.fixture(scope="class")
def eacl_deny_for_others(self, user_container: str) -> None:
with allure.step(f"Set deny all operations for {EACLRole.OTHERS} via eACL"):
eacl = EACLRule(
access=EACLAccess.DENY, role=EACLRole.OTHERS, operation=EACLOperation.PUT
)
with reporter.step(f"Set deny all operations for {EACLRole.OTHERS} via eACL"):
eacl = EACLRule(access=EACLAccess.DENY, role=EACLRole.OTHERS, operation=EACLOperation.PUT)
set_eacl(
self.wallet,
user_container,
@ -60,14 +58,11 @@ class Test_http_bearer(ClusterTestBase):
@pytest.fixture(scope="class")
def bearer_token_no_limit_for_others(self, user_container: str) -> str:
with allure.step(f"Create bearer token for {EACLRole.OTHERS} with all operations allowed"):
with reporter.step(f"Create bearer token for {EACLRole.OTHERS} with all operations allowed"):
bearer = form_bearertoken_file(
self.wallet,
user_container,
[
EACLRule(operation=op, access=EACLAccess.ALLOW, role=EACLRole.OTHERS)
for op in EACLOperation
],
[EACLRule(operation=op, access=EACLAccess.ALLOW, role=EACLRole.OTHERS) for op in EACLOperation],
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
sign=False,
@ -94,7 +89,6 @@ class Test_http_bearer(ClusterTestBase):
error_pattern="access to object operation denied",
)
@pytest.mark.skip("Temp disable for v0.37")
def test_put_with_bearer_when_eacl_restrict(
self,
object_size: ObjectSize,
@ -105,9 +99,7 @@ class Test_http_bearer(ClusterTestBase):
eacl_deny_for_others
bearer = bearer_token_no_limit_for_others
file_path = generate_file(object_size.value)
with allure.step(
f"Put object with bearer token for {EACLRole.OTHERS}, then get and verify hashes"
):
with reporter.step(f"Put object with bearer token for {EACLRole.OTHERS}, then get and verify hashes"):
headers = [f" -H 'Authorization: Bearer {bearer}'"]
oid = upload_via_http_gate_curl(
cid=user_container,

85
pytest_tests/testsuites/services/http_gate/test_http_gate.py
View file

@ -1,7 +1,6 @@
import os
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.cli.container import create_container
from frostfs_testlib.steps.cli.object import put_object_to_random_node
@ -27,11 +26,11 @@ OBJECT_NOT_FOUND_ERROR = "not found"
@allure.link(
"https://github.com/TrueCloudLab/frostfs-http-gw#frostfs-http-gateway",
"https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#frostfs-http-gateway",
name="frostfs-http-gateway",
)
@allure.link("https://github.com/TrueCloudLab/frostfs-http-gw#uploading", name="uploading")
@allure.link("https://github.com/TrueCloudLab/frostfs-http-gw#downloading", name="downloading")
@allure.link("https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#uploading", name="uploading")
@allure.link("https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#downloading", name="downloading")
@pytest.mark.sanity
@pytest.mark.http_gate
class TestHttpGate(ClusterTestBase):
@ -44,16 +43,14 @@ class TestHttpGate(ClusterTestBase):
TestHttpGate.wallet = default_wallet
@allure.title("Put over gRPC, Get over HTTP")
def test_put_grpc_get_http(
self, complex_object_size: ObjectSize, simple_object_size: ObjectSize
):
def test_put_grpc_get_http(self, complex_object_size: ObjectSize, simple_object_size: ObjectSize):
"""
Test that object can be put using gRPC interface and get using HTTP.
Steps:
1. Create simple and large objects.
2. Put objects using gRPC (frostfs-cli).
3. Download objects using HTTP gate (https://github.com/TrueCloudLab/frostfs-http-gw#downloading).
3. Download objects using HTTP gate (https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#downloading).
4. Get objects using gRPC (frostfs-cli).
5. Compare hashes for got objects.
6. Compare hashes for got and original objects.
@ -71,7 +68,7 @@ class TestHttpGate(ClusterTestBase):
file_path_simple = generate_file(simple_object_size.value)
file_path_large = generate_file(complex_object_size.value)
with allure.step("Put objects using gRPC"):
with reporter.step("Put objects using gRPC"):
oid_simple = put_object_to_random_node(
wallet=self.wallet,
path=file_path_simple,
@ -101,30 +98,27 @@ class TestHttpGate(ClusterTestBase):
@allure.link(
"https://github.com/TrueCloudLab/frostfs-http-gw#frostfs-http-gateway",
"https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#frostfs-http-gateway",
name="frostfs-http-gateway",
)
@allure.link("https://github.com/TrueCloudLab/frostfs-http-gw#uploading", name="uploading")
@allure.link("https://github.com/TrueCloudLab/frostfs-http-gw#downloading", name="downloading")
@pytest.mark.sanity
@allure.link("https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#uploading", name="uploading")
@allure.link("https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#downloading", name="downloading")
@pytest.mark.skip("Skipped temporarly")
@pytest.mark.http_gate
@pytest.mark.http_put
@pytest.mark.skip("Skipped due to deprecated PUT via http")
class TestHttpPut(ClusterTestBase):
@allure.link("https://github.com/TrueCloudLab/frostfs-http-gw#uploading", name="uploading")
@allure.link("https://github.com/TrueCloudLab/frostfs-http-gw#downloading", name="downloading")
@allure.link("https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#uploading", name="uploading")
@allure.link("https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#downloading", name="downloading")
@allure.title("Put over HTTP, Get over HTTP")
@pytest.mark.smoke
def test_put_http_get_http(
self, complex_object_size: ObjectSize, simple_object_size: ObjectSize
):
def test_put_http_get_http(self, complex_object_size: ObjectSize, simple_object_size: ObjectSize):
"""
Test that object can be put and get using HTTP interface.
Steps:
1. Create simple and large objects.
2. Upload objects using HTTP (https://github.com/TrueCloudLab/frostfs-http-gw#uploading).
3. Download objects using HTTP gate (https://github.com/TrueCloudLab/frostfs-http-gw#downloading).
2. Upload objects using HTTP (https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#uploading).
3. Download objects using HTTP gate (https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#downloading).
4. Compare hashes for got and original objects.
Expected result:
@ -140,7 +134,7 @@ class TestHttpPut(ClusterTestBase):
file_path_simple = generate_file(simple_object_size.value)
file_path_large = generate_file(complex_object_size.value)
with allure.step("Put objects using HTTP"):
with reporter.step("Put objects using HTTP"):
oid_simple = upload_via_http_gate(
cid=cid, path=file_path_simple, endpoint=self.cluster.default_http_gate_endpoint
)
@ -161,36 +155,32 @@ class TestHttpPut(ClusterTestBase):
)
@allure.link(
"https://github.com/TrueCloudLab/frostfs-http-gw#by-attributes",
"https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#by-attributes",
name="download by attributes",
)
@pytest.mark.skip("Skipped due to deprecated PUT via http")
@allure.title("Put over HTTP, Get over HTTP with header")
@allure.title("Put over HTTP, Get over HTTP with {id} header")
@pytest.mark.parametrize(
"attributes",
"attributes,id",
[
{"fileName": "simple_obj_filename"},
{"file-Name": "simple obj filename"},
{"cat%jpeg": "cat%jpeg"},
({"fileName": "simple_obj_filename"}, "simple"),
({"file-Name": "simple obj filename"}, "hyphen"),
({"cat%jpeg": "cat%jpeg"}, "percent"),
],
ids=["simple", "hyphen", "percent"],
)
def test_put_http_get_http_with_headers(
self, attributes: dict, simple_object_size: ObjectSize, request: pytest.FixtureRequest
):
def test_put_http_get_http_with_headers(self, attributes: dict, simple_object_size: ObjectSize, id: str):
"""
Test that object can be downloaded using different attributes in HTTP header.
Steps:
1. Create simple and large objects.
2. Upload objects using HTTP with particular attributes in the header.
3. Download objects by attributes using HTTP gate (https://github.com/TrueCloudLab/frostfs-http-gw#by-attributes).
3. Download objects by attributes using HTTP gate (https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#by-attributes).
4. Compare hashes for got and original objects.
Expected result:
Hashes must be the same.
"""
allure.dynamic.title(f"Put over HTTP, Get over HTTP with {request.node.callspec.id} header")
cid = create_container(
self.wallet,
shell=self.shell,
@ -200,7 +190,7 @@ class TestHttpPut(ClusterTestBase):
)
file_path = generate_file(simple_object_size.value)
with allure.step("Put objects using HTTP with attribute"):
with reporter.step("Put objects using HTTP with attribute"):
headers = attr_into_header(attributes)
oid = upload_via_http_gate(
cid=cid,
@ -240,7 +230,7 @@ class TestHttpPut(ClusterTestBase):
valid_until = min_valid_epoch + gap_until
headers = {"X-Attribute-System-Expiration-Epoch": str(valid_until)}
with allure.step("Put objects using HTTP with attribute Expiration-Epoch"):
with reporter.step("Put objects using HTTP with attribute Expiration-Epoch"):
oid = upload_via_http_gate(
cid=cid,
path=file_path,
@ -252,7 +242,7 @@ class TestHttpPut(ClusterTestBase):
oids_to_be_valid.append(oid)
else:
oids_to_be_expired.append(oid)
with allure.step("This object can be got"):
with reporter.step("This object can be got"):
get_via_http_gate(
cid=cid,
oid=oid,
@ -266,7 +256,7 @@ class TestHttpPut(ClusterTestBase):
wait_for_gc_pass_on_storage_nodes()
for oid in oids_to_be_expired:
with allure.step(f"{oid} shall be expired and cannot be got"):
with reporter.step(f"{oid} shall be expired and cannot be got"):
try_to_get_object_and_expect_error(
cid=cid,
oid=oid,
@ -275,7 +265,7 @@ class TestHttpPut(ClusterTestBase):
http_hostname=self.cluster.default_http_hostname[0],
)
for oid in oids_to_be_valid:
with allure.step(f"{oid} shall be valid and can be got"):
with reporter.step(f"{oid} shall be valid and can be got"):
get_via_http_gate(
cid=cid,
oid=oid,
@ -319,7 +309,7 @@ class TestHttpPut(ClusterTestBase):
http_hostname=self.cluster.default_http_hostname[0],
)
with allure.step("Verify hashes"):
with reporter.step("Verify hashes"):
assert get_file_hash(f"{dir_path}/file1") == get_file_hash(file_path_simple)
assert get_file_hash(f"{dir_path}/file2") == get_file_hash(file_path_large)
@ -340,10 +330,8 @@ class TestHttpPut(ClusterTestBase):
file_path = generate_file(complex_object_size.value)
with allure.step("Put objects using HTTP"):
oid_gate = upload_via_http_gate(
cid=cid, path=file_path, endpoint=self.cluster.default_http_gate_endpoint
)
with reporter.step("Put objects using HTTP"):
oid_gate = upload_via_http_gate(cid=cid, path=file_path, endpoint=self.cluster.default_http_gate_endpoint)
oid_curl = upload_via_http_gate_curl(
cid=cid,
filepath=file_path,
@ -372,11 +360,8 @@ class TestHttpPut(ClusterTestBase):
object_getter=get_via_http_curl,
)
@pytest.mark.skip("Skipped due to deprecated PUT via http")
@allure.title("Put/Get over HTTP using Curl utility")
def test_put_http_get_http_curl(
self, complex_object_size: ObjectSize, simple_object_size: ObjectSize
):
def test_put_http_get_http_curl(self, complex_object_size: ObjectSize, simple_object_size: ObjectSize):
"""
Test checks upload and download over HTTP using curl utility.
"""
@ -390,7 +375,7 @@ class TestHttpPut(ClusterTestBase):
file_path_simple = generate_file(simple_object_size.value)
file_path_large = generate_file(complex_object_size.value)
with allure.step("Put objects using curl utility"):
with reporter.step("Put objects using curl utility"):
oid_simple = upload_via_http_gate_curl(
cid=cid, filepath=file_path_simple, endpoint=self.cluster.default_http_gate_endpoint
)

33
pytest_tests/testsuites/services/http_gate/test_http_headers.py
View file

@ -3,6 +3,7 @@ import os
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.cli.container import (
create_container,
@ -27,10 +28,9 @@ OBJECT_ALREADY_REMOVED_ERROR = "object already removed"
logger = logging.getLogger("NeoLogger")
@pytest.mark.sanity
@pytest.mark.skip("Skipped temporarly")
@pytest.mark.http_gate
@pytest.mark.http_put
@pytest.mark.skip("Skipped due to deprecated PUT via http")
class Test_http_headers(ClusterTestBase):
PLACEMENT_RULE = "REP 2 IN X CBF 1 SELECT 4 FROM * AS X"
obj1_keys = ["Writer", "Chapter1", "Chapter2"]
@ -79,9 +79,7 @@ class Test_http_headers(ClusterTestBase):
yield storage_objects
@allure.title("Get object1 by attribute")
def test_object1_can_be_get_by_attr(
self, storage_objects_with_attributes: list[StorageObjectInfo]
):
def test_object1_can_be_get_by_attr(self, storage_objects_with_attributes: list[StorageObjectInfo]):
"""
Test to get object#1 by attribute and comapre hashes
@ -91,7 +89,7 @@ class Test_http_headers(ClusterTestBase):
storage_object_1 = storage_objects_with_attributes[0]
with allure.step(
with reporter.step(
f'Download object#1 via wget with attributes Chapter2: {storage_object_1.attributes["Chapter2"]} and compare hashes'
):
get_object_by_attr_and_verify_hashes(
@ -104,9 +102,7 @@ class Test_http_headers(ClusterTestBase):
)
@allure.title("Get object2 with different attributes, then delete object2 and get object1")
def test_object2_can_be_get_by_attr(
self, storage_objects_with_attributes: list[StorageObjectInfo]
):
def test_object2_can_be_get_by_attr(self, storage_objects_with_attributes: list[StorageObjectInfo]):
"""
Test to get object2 with different attributes, then delete object2 and get object1 using 1st attribute. Note: obj1 and obj2 have the same attribute#1,
and when obj2 is deleted you can get obj1 by 1st attribute
@ -120,7 +116,7 @@ class Test_http_headers(ClusterTestBase):
storage_object_1 = storage_objects_with_attributes[0]
storage_object_2 = storage_objects_with_attributes[1]
with allure.step(
with reporter.step(
f'Download object#2 via wget with attributes [chapter2={storage_object_2.attributes["chapter2"]}] / [Ch@pter1={storage_object_2.attributes["Ch@pter1"]}] and compare hashes'
):
selected_attributes_object2 = [
@ -136,7 +132,7 @@ class Test_http_headers(ClusterTestBase):
endpoint=self.cluster.default_http_gate_endpoint,
http_hostname=self.cluster.default_http_hostname[0],
)
with allure.step("Delete object#2 and verify is the container deleted"):
with reporter.step("Delete object#2 and verify is the container deleted"):
delete_object(
wallet=self.wallet,
cid=storage_object_2.cid,
@ -153,7 +149,7 @@ class Test_http_headers(ClusterTestBase):
)
storage_objects_with_attributes.remove(storage_object_2)
with allure.step(
with reporter.step(
f'Download object#1 with attributes [Writer={storage_object_1.attributes["Writer"]}] and compare hashes'
):
key_value_pair = {"Writer": storage_object_1.attributes["Writer"]}
@ -167,9 +163,7 @@ class Test_http_headers(ClusterTestBase):
)
@allure.title("[NEGATIVE] Put object and get right after container is deleted")
def test_negative_put_and_get_object3(
self, storage_objects_with_attributes: list[StorageObjectInfo]
):
def test_negative_put_and_get_object3(self, storage_objects_with_attributes: list[StorageObjectInfo]):
"""
Test to attempt to put object and try to download it right after the container has been deleted
@ -182,7 +176,7 @@ class Test_http_headers(ClusterTestBase):
"""
storage_object_1 = storage_objects_with_attributes[0]
with allure.step(
with reporter.step(
"[Negative] Allocate and attemt to put object#3 via http with attributes: [Writer=Leo Tolstoy, Writer=peace, peace=peace]"
):
file_path_3 = generate_file(storage_object_1.size)
@ -197,12 +191,13 @@ class Test_http_headers(ClusterTestBase):
headers=headers,
error_pattern=error_pattern,
)
with allure.step("Delete container and verify container deletion"):
with reporter.step("Delete container and verify container deletion"):
delete_container(
wallet=self.wallet,
cid=storage_object_1.cid,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
await_mode=True,
)
self.tick_epoch()
wait_for_container_deletion(
@ -214,9 +209,7 @@ class Test_http_headers(ClusterTestBase):
assert storage_object_1.cid not in list_containers(
self.wallet, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint
)
with allure.step(
"[Negative] Try to download (wget) object via wget with attributes [peace=peace]"
):
with reporter.step("[Negative] Try to download (wget) object via wget with attributes [peace=peace]"):
request = f"/get/{storage_object_1.cid}/peace/peace"
error_pattern = "404 Not Found"
try_to_get_object_via_passed_request_and_expect_error(

72
pytest_tests/testsuites/services/http_gate/test_http_object.py
View file

@ -2,14 +2,19 @@ import logging
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.s3 import AwsCliClient, S3ClientWrapper
from frostfs_testlib.steps.cli.container import create_container
from frostfs_testlib.steps.cli.object import put_object_to_random_node
from frostfs_testlib.steps.http.http_gate import (
assert_hashes_are_equal,
get_object_by_attr_and_verify_hashes,
get_via_http_gate,
try_to_get_object_via_passed_request_and_expect_error,
verify_object_hash,
)
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.testing.cluster_test_base import ClusterTestBase
from frostfs_testlib.utils.file_utils import generate_file
@ -30,24 +35,24 @@ class Test_http_object(ClusterTestBase):
@allure.title("Put over gRPC, Get over HTTP with attributes (obj_size={object_size})")
def test_object_put_get_attributes(self, object_size: ObjectSize):
"""
Test that object can be put using gRPC interface and get using HTTP.
Test that object can be put using gRPC interface and got using HTTP.
Steps:
1. Create object;
2. Put objects using gRPC (frostfs-cli) with attributes [--attributes chapter1=peace,chapter2=war];
3. Download object using HTTP gate (https://github.com/TrueCloudLab/frostfs-http-gw#downloading);
4. Compare hashes between original and downloaded object;
5. [Negative] Try to the get object with specified attributes and `get` request: [get/$CID/chapter1/peace];
6. Download the object with specified attributes and `get_by_attribute` request: [get_by_attribute/$CID/chapter1/peace];
7. Compare hashes between original and downloaded object;
8. [Negative] Try to the get object via `get_by_attribute` request: [get_by_attribute/$CID/$OID];
1. Create an object;
2. Put object(s) using gRPC (frostfs-cli) with attributes [--attributes chapter1=peace,chapter2=war];
3. Download the object using HTTP gate (https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#downloading);
4. Compare hashes of the original and the downloaded object;
5. [Negative] Try to the get the object with the specified attributes and `get` request: [get/$CID/chapter1/peace];
6. Download the object with the specified attributes and `get_by_attribute` request: [get_by_attribute/$CID/chapter1/peace];
7. Compare hashes of the original and the downloaded object;
8. [Negative] Try to the get the object via `get_by_attribute` request: [get_by_attribute/$CID/$OID];
Expected result:
Hashes must be the same.
"""
with allure.step("Create public container"):
with reporter.step("Create public container"):
cid = create_container(
self.wallet,
shell=self.shell,
@ -69,7 +74,7 @@ class Test_http_object(ClusterTestBase):
key_value1 = obj_key1 + "=" + obj_value1
key_value2 = obj_key2 + "=" + obj_value2
with allure.step("Put objects using gRPC [--attributes chapter1=peace,chapter2=war]"):
with reporter.step("Put objects using gRPC [--attributes chapter1=peace,chapter2=war]"):
oid = put_object_to_random_node(
wallet=self.wallet,
path=file_path,
@ -78,7 +83,7 @@ class Test_http_object(ClusterTestBase):
cluster=self.cluster,
attributes=f"{key_value1},{key_value2}",
)
with allure.step("Get object and verify hashes [ get/$CID/$OID ]"):
with reporter.step("Get object and verify hashes [ get/$CID/$OID ]"):
verify_object_hash(
oid=oid,
file_name=file_path,
@ -89,7 +94,8 @@ class Test_http_object(ClusterTestBase):
endpoint=self.cluster.default_http_gate_endpoint,
http_hostname=self.cluster.default_http_hostname[0],
)
with allure.step("[Negative] try to get object: [get/$CID/chapter1/peace]"):
with reporter.step("[Negative] try to get object: [get/$CID/chapter1/peace]"):
attrs = {obj_key1: obj_value1, obj_key2: obj_value2}
request = f"/get/{cid}/{obj_key1}/{obj_value1}"
expected_err_msg = "Failed to get object via HTTP gate:"
@ -103,9 +109,7 @@ class Test_http_object(ClusterTestBase):
http_hostname=self.cluster.default_http_hostname[0],
)
with allure.step(
"Download the object with attribute [get_by_attribute/$CID/chapter1/peace]"
):
with reporter.step("Download the object with attribute [get_by_attribute/$CID/chapter1/peace]"):
get_object_by_attr_and_verify_hashes(
oid=oid,
file_name=file_path,
@ -114,7 +118,7 @@ class Test_http_object(ClusterTestBase):
endpoint=self.cluster.default_http_gate_endpoint,
http_hostname=self.cluster.default_http_hostname[0],
)
with allure.step("[Negative] try to get object: get_by_attribute/$CID/$OID"):
with reporter.step("[Negative] try to get object: get_by_attribute/$CID/$OID"):
request = f"/get_by_attribute/{cid}/{oid}"
try_to_get_object_via_passed_request_and_expect_error(
cid=cid,
@ -124,3 +128,37 @@ class Test_http_object(ClusterTestBase):
endpoint=self.cluster.default_http_gate_endpoint,
http_hostname=self.cluster.default_http_hostname[0],
)
@allure.title("Put over s3, Get over HTTP with bucket name and key")
@pytest.mark.parametrize("s3_client", [AwsCliClient], indirect=True)
def test_object_put_get_bucketname_key(self, object_size: ObjectSize, s3_client: S3ClientWrapper):
"""
Test that object can be put using s3-gateway interface and got via HTTP with bucket name and object key.
Steps:
1. Create an object;
2. Create a bucket via s3;
3. Put the object via s3;
4. Download the object using HTTP gate with the bucket name and the object key;
5. Compare hashes of the original and the downloaded objects;
Expected result:
Hashes must be the same.
"""
file_path = generate_file(object_size.value)
object_key = s3_helper.object_key_from_file_path(file_path)
bucket = s3_client.create_bucket(acl="public-read-write")
s3_client.put_object(bucket=bucket, filepath=file_path, key=object_key)
obj_s3 = s3_client.get_object(bucket=bucket, key=object_key)
request = f"/get/{bucket}/{object_key}"
obj_http = get_via_http_gate(
cid=None,
oid=None,
endpoint=self.cluster.default_http_gate_endpoint,
http_hostname=self.cluster.default_http_hostname[0],
request_path=request,
)
with reporter.step("Verify hashes"):
assert_hashes_are_equal(file_path, obj_http, obj_s3)

14
pytest_tests/testsuites/services/http_gate/test_http_streaming.py
View file

@ -2,6 +2,7 @@ import logging
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.cli.container import create_container
from frostfs_testlib.steps.http.http_gate import upload_via_http_gate_curl, verify_object_hash
@ -12,10 +13,9 @@ from frostfs_testlib.utils.file_utils import generate_file
logger = logging.getLogger("NeoLogger")
@pytest.mark.sanity
@pytest.mark.skip("Skipped temporarly")
@pytest.mark.http_gate
@pytest.mark.http_put
@pytest.mark.skip("Skipped due to deprecated PUT via http")
class Test_http_streaming(ClusterTestBase):
PLACEMENT_RULE = "REP 2 IN X CBF 1 SELECT 4 FROM * AS X"
@ -32,13 +32,13 @@ class Test_http_streaming(ClusterTestBase):
Steps:
1. Create big object;
2. Put object using curl with pipe (streaming);
3. Download object using HTTP gate (https://github.com/TrueCloudLab/frostfs-http-gw#downloading);
3. Download object using HTTP gate (https://git.frostfs.info/TrueCloudLab/frostfs-http-gw#downloading);
4. Compare hashes between original and downloaded object;
Expected result:
Hashes must be the same.
"""
with allure.step("Create public container and verify container creation"):
with reporter.step("Create public container and verify container creation"):
cid = create_container(
self.wallet,
shell=self.shell,
@ -46,13 +46,11 @@ class Test_http_streaming(ClusterTestBase):
rule=self.PLACEMENT_RULE,
basic_acl=PUBLIC_ACL,
)
with allure.step("Allocate big object"):
with reporter.step("Allocate big object"):
# Generate file
file_path = generate_file(complex_object_size.value)
with allure.step(
"Put objects using curl utility and Get object and verify hashes [ get/$CID/$OID ]"
):
with reporter.step("Put objects using curl utility and Get object and verify hashes [ get/$CID/$OID ]"):
oid = upload_via_http_gate_curl(
cid=cid, filepath=file_path, endpoint=self.cluster.default_http_gate_endpoint
)

105
pytest_tests/testsuites/services/http_gate/test_http_system_header.py
View file

@ -5,14 +5,11 @@ from typing import Optional
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.error_patterns import OBJECT_NOT_FOUND
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.steps.cli.container import create_container
from frostfs_testlib.steps.cli.object import (
get_netmap_netinfo,
get_object_from_random_node,
head_object,
)
from frostfs_testlib.steps.cli.object import get_netmap_netinfo, get_object_from_random_node, head_object
from frostfs_testlib.steps.epoch import get_epoch, wait_for_epochs_align
from frostfs_testlib.steps.http.http_gate import (
attr_into_str_header_curl,
@ -35,10 +32,9 @@ SYSTEM_EXPIRATION_TIMESTAMP = "System-Expiration-Timestamp"
SYSTEM_EXPIRATION_RFC3339 = "System-Expiration-RFC3339"
@pytest.mark.sanity
@pytest.mark.skip("Skipped temporarly")
@pytest.mark.http_gate
@pytest.mark.http_put
@pytest.mark.skip("Skipped due to deprecated PUT via http")
class Test_http_system_header(ClusterTestBase):
PLACEMENT_RULE = "REP 2 IN X CBF 1 SELECT 2 FROM * AS X"
@ -76,9 +72,7 @@ class Test_http_system_header(ClusterTestBase):
return f"{mins}m"
@allure.title("Return future timestamp after N epochs are passed")
def epoch_count_into_timestamp(
self, epoch_duration: int, epoch: int, rfc3339: Optional[bool] = False
) -> str:
def epoch_count_into_timestamp(self, epoch_duration: int, epoch: int, rfc3339: Optional[bool] = False) -> str:
current_datetime = datetime.datetime.utcnow()
epoch_count_in_seconds = epoch_duration * epoch
future_datetime = current_datetime + datetime.timedelta(seconds=epoch_count_in_seconds)
@ -96,9 +90,7 @@ class Test_http_system_header(ClusterTestBase):
return False
return True
@allure.title(
f"Validate that only {EXPIRATION_EPOCH_HEADER} exists in header and other headers are abesent"
)
@allure.title(f"Validate that only {EXPIRATION_EPOCH_HEADER} exists in header and other headers are abesent")
def validation_for_http_header_attr(self, head_info: dict, expected_epoch: int) -> None:
# check that __SYSTEM__EXPIRATION_EPOCH attribute has corresponding epoch
assert self.check_key_value_presented_header(
@ -146,13 +138,9 @@ class Test_http_system_header(ClusterTestBase):
@allure.title("[NEGATIVE] Put object with expired epoch")
def test_unable_put_expired_epoch(self, user_container: str, simple_object_size: ObjectSize):
headers = attr_into_str_header_curl(
{"System-Expiration-Epoch": str(get_epoch(self.shell, self.cluster) - 1)}
)
headers = attr_into_str_header_curl({"System-Expiration-Epoch": str(get_epoch(self.shell, self.cluster) - 1)})
file_path = generate_file(simple_object_size.value)
with allure.step(
"Put object using HTTP with attribute Expiration-Epoch where epoch is expired"
):
with reporter.step("Put object using HTTP with attribute Expiration-Epoch where epoch is expired"):
upload_via_http_gate_curl(
cid=user_container,
filepath=file_path,
@ -162,12 +150,10 @@ class Test_http_system_header(ClusterTestBase):
)
@allure.title("[NEGATIVE] Put object with negative System-Expiration-Duration")
def test_unable_put_negative_duration(
self, user_container: str, simple_object_size: ObjectSize
):
def test_unable_put_negative_duration(self, user_container: str, simple_object_size: ObjectSize):
headers = attr_into_str_header_curl({"System-Expiration-Duration": "-1h"})
file_path = generate_file(simple_object_size.value)
with allure.step(
with reporter.step(
"Put object using HTTP with attribute System-Expiration-Duration where duration is negative"
):
upload_via_http_gate_curl(
@ -179,12 +165,10 @@ class Test_http_system_header(ClusterTestBase):
)
@allure.title("[NEGATIVE] Put object with System-Expiration-Timestamp value in the past")
def test_unable_put_expired_timestamp(
self, user_container: str, simple_object_size: ObjectSize
):
def test_unable_put_expired_timestamp(self, user_container: str, simple_object_size: ObjectSize):
headers = attr_into_str_header_curl({"System-Expiration-Timestamp": "1635075727"})
file_path = generate_file(simple_object_size.value)
with allure.step(
with reporter.step(
"Put object using HTTP with attribute System-Expiration-Timestamp where duration is in the past"
):
upload_via_http_gate_curl(
@ -210,10 +194,7 @@ class Test_http_system_header(ClusterTestBase):
)
@allure.title("Priority of attributes epoch>duration (obj_size={object_size})")
@pytest.mark.skip("Temp disable for v0.37")
def test_http_attr_priority_epoch_duration(
self, user_container: str, object_size: ObjectSize, epoch_duration: int
):
def test_http_attr_priority_epoch_duration(self, user_container: str, object_size: ObjectSize, epoch_duration: int):
self.tick_epoch()
epoch_count = 1
expected_epoch = get_epoch(self.shell, self.cluster) + epoch_count
@ -222,21 +203,21 @@ class Test_http_system_header(ClusterTestBase):
)
attributes = {SYSTEM_EXPIRATION_EPOCH: expected_epoch, SYSTEM_EXPIRATION_DURATION: "1m"}
file_path = generate_file(object_size.value)
with allure.step(
with reporter.step(
f"Put objects using HTTP with attributes and head command should display {EXPIRATION_EPOCH_HEADER}: {expected_epoch} attr"
):
oid, head_info = self.oid_header_info_for_object(
file_path=file_path, attributes=attributes, user_container=user_container
)
self.validation_for_http_header_attr(head_info=head_info, expected_epoch=expected_epoch)
with allure.step("Check that object becomes unavailable when epoch is expired"):
with reporter.step("Check that object becomes unavailable when epoch is expired"):
for _ in range(0, epoch_count + 1):
self.tick_epoch()
assert (
get_epoch(self.shell, self.cluster) == expected_epoch + 1
), f"Epochs should be equal: {get_epoch(self.shell, self.cluster)} != {expected_epoch + 1}"
with allure.step("Check object deleted because it expires-on epoch"):
with reporter.step("Check object deleted because it expires-on epoch"):
wait_for_epochs_align(self.shell, self.cluster)
try_to_get_object_and_expect_error(
cid=user_container,
@ -247,15 +228,10 @@ class Test_http_system_header(ClusterTestBase):
)
# check that object is not available via grpc
with pytest.raises(Exception, match=OBJECT_NOT_FOUND):
get_object_from_random_node(
self.wallet, user_container, oid, self.shell, self.cluster
)
get_object_from_random_node(self.wallet, user_container, oid, self.shell, self.cluster)
@allure.title("Priority of attributes duration>timestamp (obj_size={object_size})")
@pytest.mark.skip("Temp disable for v0.37")
def test_http_attr_priority_dur_timestamp(
self, user_container: str, object_size: ObjectSize, epoch_duration: int
):
def test_http_attr_priority_dur_timestamp(self, user_container: str, object_size: ObjectSize, epoch_duration: int):
self.tick_epoch()
epoch_count = 2
expected_epoch = get_epoch(self.shell, self.cluster) + epoch_count
@ -263,29 +239,25 @@ class Test_http_system_header(ClusterTestBase):
f"epoch duration={epoch_duration}, current_epoch= {get_epoch(self.shell, self.cluster)} expected_epoch {expected_epoch}"
)
attributes = {
SYSTEM_EXPIRATION_DURATION: self.epoch_count_into_mins(
epoch_duration=epoch_duration, epoch=2
),
SYSTEM_EXPIRATION_TIMESTAMP: self.epoch_count_into_timestamp(
epoch_duration=epoch_duration, epoch=1
),
SYSTEM_EXPIRATION_DURATION: self.epoch_count_into_mins(epoch_duration=epoch_duration, epoch=2),
SYSTEM_EXPIRATION_TIMESTAMP: self.epoch_count_into_timestamp(epoch_duration=epoch_duration, epoch=1),
}
file_path = generate_file(object_size.value)
with allure.step(
with reporter.step(
f"Put objects using HTTP with attributes and head command should display {EXPIRATION_EPOCH_HEADER}: {expected_epoch} attr"
):
oid, head_info = self.oid_header_info_for_object(
file_path=file_path, attributes=attributes, user_container=user_container
)
self.validation_for_http_header_attr(head_info=head_info, expected_epoch=expected_epoch)
with allure.step("Check that object becomes unavailable when epoch is expired"):
with reporter.step("Check that object becomes unavailable when epoch is expired"):
for _ in range(0, epoch_count + 1):
self.tick_epoch()
assert (
get_epoch(self.shell, self.cluster) == expected_epoch + 1
), f"Epochs should be equal: {get_epoch(self.shell, self.cluster)} != {expected_epoch + 1}"
with allure.step("Check object deleted because it expires-on epoch"):
with reporter.step("Check object deleted because it expires-on epoch"):
wait_for_epochs_align(self.shell, self.cluster)
try_to_get_object_and_expect_error(
cid=user_container,
@ -296,15 +268,10 @@ class Test_http_system_header(ClusterTestBase):
)
# check that object is not available via grpc
with pytest.raises(Exception, match=OBJECT_NOT_FOUND):
get_object_from_random_node(
self.wallet, user_container, oid, self.shell, self.cluster
)
get_object_from_random_node(self.wallet, user_container, oid, self.shell, self.cluster)
@allure.title("Priority of attributes timestamp>Expiration-RFC (obj_size={object_size})")
@pytest.mark.skip("Temp disable for v0.37")
def test_http_attr_priority_timestamp_rfc(
self, user_container: str, object_size: ObjectSize, epoch_duration: int
):
def test_http_attr_priority_timestamp_rfc(self, user_container: str, object_size: ObjectSize, epoch_duration: int):
self.tick_epoch()
epoch_count = 2
expected_epoch = get_epoch(self.shell, self.cluster) + epoch_count
@ -312,29 +279,27 @@ class Test_http_system_header(ClusterTestBase):
f"epoch duration={epoch_duration}, current_epoch= {get_epoch(self.shell, self.cluster)} expected_epoch {expected_epoch}"
)
attributes = {
SYSTEM_EXPIRATION_TIMESTAMP: self.epoch_count_into_timestamp(
epoch_duration=epoch_duration, epoch=2
),
SYSTEM_EXPIRATION_TIMESTAMP: self.epoch_count_into_timestamp(epoch_duration=epoch_duration, epoch=2),
SYSTEM_EXPIRATION_RFC3339: self.epoch_count_into_timestamp(
epoch_duration=epoch_duration, epoch=1, rfc3339=True
),
}
file_path = generate_file(object_size.value)
with allure.step(
with reporter.step(
f"Put objects using HTTP with attributes and head command should display {EXPIRATION_EPOCH_HEADER}: {expected_epoch} attr"
):
oid, head_info = self.oid_header_info_for_object(
file_path=file_path, attributes=attributes, user_container=user_container
)
self.validation_for_http_header_attr(head_info=head_info, expected_epoch=expected_epoch)
with allure.step("Check that object becomes unavailable when epoch is expired"):
with reporter.step("Check that object becomes unavailable when epoch is expired"):
for _ in range(0, epoch_count + 1):
self.tick_epoch()
assert (
get_epoch(self.shell, self.cluster) == expected_epoch + 1
), f"Epochs should be equal: {get_epoch(self.shell, self.cluster)} != {expected_epoch + 1}"
with allure.step("Check object deleted because it expires-on epoch"):
with reporter.step("Check object deleted because it expires-on epoch"):
wait_for_epochs_align(self.shell, self.cluster)
try_to_get_object_and_expect_error(
cid=user_container,
@ -345,9 +310,7 @@ class Test_http_system_header(ClusterTestBase):
)
# check that object is not available via grpc
with pytest.raises(Exception, match=OBJECT_NOT_FOUND):
get_object_from_random_node(
self.wallet, user_container, oid, self.shell, self.cluster
)
get_object_from_random_node(self.wallet, user_container, oid, self.shell, self.cluster)
@allure.title("Object should be deleted when expiration passed (obj_size={object_size})")
@pytest.mark.parametrize(
@ -371,7 +334,7 @@ class Test_http_system_header(ClusterTestBase):
)
}
file_path = generate_file(object_size.value)
with allure.step(
with reporter.step(
f"Put objects using HTTP with attributes and head command should display {EXPIRATION_EPOCH_HEADER}: {expected_epoch} attr"
):
oid, head_info = self.oid_header_info_for_object(
@ -380,7 +343,7 @@ class Test_http_system_header(ClusterTestBase):
user_container=user_container,
)
self.validation_for_http_header_attr(head_info=head_info, expected_epoch=expected_epoch)
with allure.step("Check that object becomes unavailable when epoch is expired"):
with reporter.step("Check that object becomes unavailable when epoch is expired"):
for _ in range(0, epoch_count + 1):
self.tick_epoch()
# check that {EXPIRATION_EXPIRATION_RFC} absents in header output
@ -388,7 +351,7 @@ class Test_http_system_header(ClusterTestBase):
get_epoch(self.shell, self.cluster) == expected_epoch + 1
), f"Epochs should be equal: {get_epoch(self.shell, self.cluster)} != {expected_epoch + 1}"
with allure.step("Check object deleted because it expires-on epoch"):
with reporter.step("Check object deleted because it expires-on epoch"):
wait_for_epochs_align(self.shell, self.cluster)
try_to_get_object_and_expect_error(
cid=user_container,
@ -399,6 +362,4 @@ class Test_http_system_header(ClusterTestBase):
)
# check that object is not available via grpc
with pytest.raises(Exception, match=OBJECT_NOT_FOUND):
get_object_from_random_node(
self.wallet, user_container, oid, self.shell, self.cluster
)
get_object_from_random_node(self.wallet, user_container, oid, self.shell, self.cluster)

28
pytest_tests/testsuites/services/s3_gate/test_s3_ACL.py
View file

@ -1,41 +1,37 @@
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.utils.file_utils import generate_file
@pytest.mark.sanity
@pytest.mark.acl
@pytest.mark.s3_gate
class TestS3GateACL:
@allure.title("Object ACL (s3_client={s3_client})")
@pytest.mark.parametrize("s3_client", [AwsCliClient], indirect=True)
def test_s3_object_ACL(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_object_ACL(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
with allure.step("Put object into bucket, Check ACL is empty"):
with reporter.step("Put object into bucket, Check ACL is empty"):
s3_client.put_object(bucket, file_path)
obj_acl = s3_client.get_object_acl(bucket, file_name)
assert obj_acl == [], f"Expected ACL is empty, got {obj_acl}"
with allure.step("Put object ACL = public-read"):
with reporter.step("Put object ACL = public-read"):
s3_client.put_object_acl(bucket, file_name, "public-read")
obj_acl = s3_client.get_object_acl(bucket, file_name)
s3_helper.assert_s3_acl(acl_grants=obj_acl, permitted_users="AllUsers")
with allure.step("Put object ACL = private"):
with reporter.step("Put object ACL = private"):
s3_client.put_object_acl(bucket, file_name, "private")
obj_acl = s3_client.get_object_acl(bucket, file_name)
s3_helper.assert_s3_acl(acl_grants=obj_acl, permitted_users="CanonicalUser")
with allure.step(
"Put object with grant-read uri=http://acs.amazonaws.com/groups/global/AllUsers"
):
with reporter.step("Put object with grant-read uri=http://acs.amazonaws.com/groups/global/AllUsers"):
s3_client.put_object_acl(
bucket,
file_name,
@ -47,21 +43,17 @@ class TestS3GateACL:
@allure.title("Bucket ACL (s3_client={s3_client})")
@pytest.mark.parametrize("s3_client", [AwsCliClient, Boto3ClientWrapper], indirect=True)
def test_s3_bucket_ACL(self, s3_client: S3ClientWrapper):
with allure.step("Create bucket with ACL = public-read-write"):
bucket = s3_client.create_bucket(
object_lock_enabled_for_bucket=True, acl="public-read-write"
)
with reporter.step("Create bucket with ACL = public-read-write"):
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=True, acl="public-read-write")
bucket_acl = s3_client.get_bucket_acl(bucket)
s3_helper.assert_s3_acl(acl_grants=bucket_acl, permitted_users="AllUsers")
with allure.step("Change bucket ACL to private"):
with reporter.step("Change bucket ACL to private"):
s3_client.put_bucket_acl(bucket, acl="private")
bucket_acl = s3_client.get_bucket_acl(bucket)
s3_helper.assert_s3_acl(acl_grants=bucket_acl, permitted_users="CanonicalUser")
with allure.step(
"Change bucket acl to --grant-write uri=http://acs.amazonaws.com/groups/global/AllUsers"
):
with reporter.step("Change bucket acl to --grant-write uri=http://acs.amazonaws.com/groups/global/AllUsers"):
s3_client.put_bucket_acl(
bucket,
grant_write="uri=http://acs.amazonaws.com/groups/global/AllUsers",

59
pytest_tests/testsuites/services/s3_gate/test_s3_bucket.py
View file

@ -2,54 +2,43 @@ from datetime import datetime, timedelta
import allure
import pytest
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper
from frostfs_testlib import reporter
from frostfs_testlib.s3 import S3ClientWrapper
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.utils.file_utils import generate_file
def pytest_generate_tests(metafunc: pytest.Metafunc):
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize("s3_client", [AwsCliClient, Boto3ClientWrapper], indirect=True)
@pytest.mark.sanity
@pytest.mark.s3_gate
@pytest.mark.s3_gate_bucket
class TestS3GateBucket:
@allure.title("Create Bucket with different ACL (s3_client={s3_client})")
def test_s3_create_bucket_with_ACL(self, s3_client: S3ClientWrapper):
with allure.step("Create bucket with ACL private"):
with reporter.step("Create bucket with ACL private"):
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=True, acl="private")
bucket_acl = s3_client.get_bucket_acl(bucket)
s3_helper.assert_s3_acl(acl_grants=bucket_acl, permitted_users="CanonicalUser")
with allure.step("Create bucket with ACL = public-read"):
bucket_1 = s3_client.create_bucket(
object_lock_enabled_for_bucket=True, acl="public-read"
)
with reporter.step("Create bucket with ACL = public-read"):
bucket_1 = s3_client.create_bucket(object_lock_enabled_for_bucket=True, acl="public-read")
bucket_acl_1 = s3_client.get_bucket_acl(bucket_1)
s3_helper.assert_s3_acl(acl_grants=bucket_acl_1, permitted_users="AllUsers")
with allure.step("Create bucket with ACL public-read-write"):
bucket_2 = s3_client.create_bucket(
object_lock_enabled_for_bucket=True, acl="public-read-write"
)
with reporter.step("Create bucket with ACL public-read-write"):
bucket_2 = s3_client.create_bucket(object_lock_enabled_for_bucket=True, acl="public-read-write")
bucket_acl_2 = s3_client.get_bucket_acl(bucket_2)
s3_helper.assert_s3_acl(acl_grants=bucket_acl_2, permitted_users="AllUsers")
with allure.step("Create bucket with ACL = authenticated-read"):
bucket_3 = s3_client.create_bucket(
object_lock_enabled_for_bucket=True, acl="authenticated-read"
)
with reporter.step("Create bucket with ACL = authenticated-read"):
bucket_3 = s3_client.create_bucket(object_lock_enabled_for_bucket=True, acl="authenticated-read")
bucket_acl_3 = s3_client.get_bucket_acl(bucket_3)
s3_helper.assert_s3_acl(acl_grants=bucket_acl_3, permitted_users="AllUsers")
@allure.title("Create Bucket with different ACL by grant (s3_client={s3_client})")
def test_s3_create_bucket_with_grands(self, s3_client: S3ClientWrapper):
with allure.step("Create bucket with --grant-read"):
with reporter.step("Create bucket with --grant-read"):
bucket = s3_client.create_bucket(
object_lock_enabled_for_bucket=True,
grant_read="uri=http://acs.amazonaws.com/groups/global/AllUsers",
@ -57,7 +46,7 @@ class TestS3GateBucket:
bucket_acl = s3_client.get_bucket_acl(bucket)
s3_helper.assert_s3_acl(acl_grants=bucket_acl, permitted_users="AllUsers")
with allure.step("Create bucket with --grant-wtite"):
with reporter.step("Create bucket with --grant-wtite"):
bucket_1 = s3_client.create_bucket(
object_lock_enabled_for_bucket=True,
grant_write="uri=http://acs.amazonaws.com/groups/global/AllUsers",
@ -65,7 +54,7 @@ class TestS3GateBucket:
bucket_acl_1 = s3_client.get_bucket_acl(bucket_1)
s3_helper.assert_s3_acl(acl_grants=bucket_acl_1, permitted_users="AllUsers")
with allure.step("Create bucket with --grant-full-control"):
with reporter.step("Create bucket with --grant-full-control"):
bucket_2 = s3_client.create_bucket(
object_lock_enabled_for_bucket=True,
grant_full_control="uri=http://acs.amazonaws.com/groups/global/AllUsers",
@ -74,18 +63,14 @@ class TestS3GateBucket:
s3_helper.assert_s3_acl(acl_grants=bucket_acl_2, permitted_users="AllUsers")
@allure.title("Create bucket with object lock (s3_client={s3_client})")
def test_s3_bucket_object_lock(
self, s3_client: S3ClientWrapper, simple_object_size: ObjectSize
):
def test_s3_bucket_object_lock(self, s3_client: S3ClientWrapper, simple_object_size: ObjectSize):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
with allure.step("Create bucket with --no-object-lock-enabled-for-bucket"):
with reporter.step("Create bucket with --no-object-lock-enabled-for-bucket"):
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=False)
date_obj = datetime.utcnow() + timedelta(days=1)
with pytest.raises(
Exception, match=r".*Object Lock configuration does not exist for this bucket.*"
):
with pytest.raises(Exception, match=r".*Object Lock configuration does not exist for this bucket.*"):
# An error occurred (ObjectLockConfigurationNotFoundError) when calling the PutObject operation (reached max retries: 0):
# Object Lock configuration does not exist for this bucket
s3_client.put_object(
@ -94,7 +79,7 @@ class TestS3GateBucket:
object_lock_mode="COMPLIANCE",
object_lock_retain_until_date=date_obj.strftime("%Y-%m-%dT%H:%M:%S"),
)
with allure.step("Create bucket with --object-lock-enabled-for-bucket"):
with reporter.step("Create bucket with --object-lock-enabled-for-bucket"):
bucket_1 = s3_client.create_bucket(object_lock_enabled_for_bucket=True)
date_obj_1 = datetime.utcnow() + timedelta(days=1)
s3_client.put_object(
@ -104,9 +89,7 @@ class TestS3GateBucket:
object_lock_retain_until_date=date_obj_1.strftime("%Y-%m-%dT%H:%M:%S"),
object_lock_legal_hold_status="ON",
)
s3_helper.assert_object_lock_mode(
s3_client, bucket_1, file_name, "COMPLIANCE", date_obj_1, "ON"
)
s3_helper.assert_object_lock_mode(s3_client, bucket_1, file_name, "COMPLIANCE", date_obj_1, "ON")
@allure.title("Delete bucket (s3_client={s3_client})")
def test_s3_delete_bucket(self, s3_client: S3ClientWrapper, simple_object_size: ObjectSize):
@ -116,21 +99,21 @@ class TestS3GateBucket:
file_name_2 = s3_helper.object_key_from_file_path(file_path_2)
bucket = s3_client.create_bucket()
with allure.step("Put two objects into bucket"):
with reporter.step("Put two objects into bucket"):
s3_client.put_object(bucket, file_path_1)
s3_client.put_object(bucket, file_path_2)
s3_helper.check_objects_in_bucket(s3_client, bucket, [file_name_1, file_name_2])
with allure.step("Try to delete not empty bucket and get error"):
with reporter.step("Try to delete not empty bucket and get error"):
with pytest.raises(Exception, match=r".*The bucket you tried to delete is not empty.*"):
s3_client.delete_bucket(bucket)
with allure.step("Delete object in bucket"):
with reporter.step("Delete object in bucket"):
s3_client.delete_object(bucket, file_name_1)
s3_client.delete_object(bucket, file_name_2)
s3_helper.check_objects_in_bucket(s3_client, bucket, [])
with allure.step("Delete empty bucket"):
with reporter.step("Delete empty bucket"):
s3_client.delete_bucket(bucket)
with pytest.raises(Exception, match=r".*Not Found.*"):
s3_client.head_bucket(bucket)

216
pytest_tests/testsuites/services/s3_gate/test_s3_gate.py
View file

@ -4,6 +4,7 @@ from random import choice, choices
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.common import ASSETS_DIR
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper, VersioningStatus
from frostfs_testlib.shell import Shell
@ -22,14 +23,7 @@ from frostfs_testlib.utils.file_utils import (
logger = logging.getLogger("NeoLogger")
def pytest_generate_tests(metafunc: pytest.Metafunc):
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize("s3_client", [AwsCliClient, Boto3ClientWrapper], indirect=True)
@allure.link(
"https://github.com/TrueCloudLab/frostfs-s3-gw#frostfs-s3-gw", name="frostfs-s3-gateway"
)
@allure.link("https://git.frostfs.info/TrueCloudLab/frostfs-s3-gw#frostfs-s3-gw", name="frostfs-s3-gateway")
@pytest.mark.sanity
@pytest.mark.s3_gate
@pytest.mark.s3_gate_base
@ -49,45 +43,43 @@ class TestS3Gate:
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
with allure.step("Create buckets"):
with reporter.step("Create buckets"):
bucket_1 = s3_client.create_bucket(object_lock_enabled_for_bucket=True)
s3_helper.set_bucket_versioning(s3_client, bucket_1, VersioningStatus.ENABLED)
bucket_2 = s3_client.create_bucket()
with allure.step("Check buckets are presented in the system"):
with reporter.step("Check buckets are presented in the system"):
buckets = s3_client.list_buckets()
assert bucket_1 in buckets, f"Expected bucket {bucket_1} is in the list"
assert bucket_2 in buckets, f"Expected bucket {bucket_2} is in the list"
with allure.step("Bucket must be empty"):
with reporter.step("Bucket must be empty"):
for bucket in (bucket_1, bucket_2):
objects_list = s3_client.list_objects(bucket)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
with allure.step("Check buckets are visible with S3 head command"):
with reporter.step("Check buckets are visible with S3 head command"):
s3_client.head_bucket(bucket_1)
s3_client.head_bucket(bucket_2)
with allure.step("Check we can put/list object with S3 commands"):
with reporter.step("Check we can put/list object with S3 commands"):
version_id = s3_client.put_object(bucket_1, file_path)
s3_client.head_object(bucket_1, file_name)
bucket_objects = s3_client.list_objects(bucket_1)
assert (
file_name in bucket_objects
), f"Expected file {file_name} in objects list {bucket_objects}"
assert file_name in bucket_objects, f"Expected file {file_name} in objects list {bucket_objects}"
with allure.step("Try to delete not empty bucket and get error"):
with reporter.step("Try to delete not empty bucket and get error"):
with pytest.raises(Exception, match=r".*The bucket you tried to delete is not empty.*"):
s3_client.delete_bucket(bucket_1)
s3_client.head_bucket(bucket_1)
with allure.step(f"Delete empty bucket {bucket_2}"):
with reporter.step(f"Delete empty bucket {bucket_2}"):
s3_client.delete_bucket(bucket_2)
tick_epoch(client_shell, cluster)
with allure.step(f"Check bucket {bucket_2} deleted"):
with reporter.step(f"Check bucket {bucket_2} deleted"):
with pytest.raises(Exception, match=r".*Not Found.*"):
s3_client.head_bucket(bucket_2)
@ -95,15 +87,15 @@ class TestS3Gate:
assert bucket_1 in buckets, f"Expected bucket {bucket_1} is in the list"
assert bucket_2 not in buckets, f"Expected bucket {bucket_2} is not in the list"
with allure.step(f"Delete object from {bucket_1}"):
with reporter.step(f"Delete object from {bucket_1}"):
s3_client.delete_object(bucket_1, file_name, version_id)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, expected_objects=[])
with allure.step(f"Delete bucket {bucket_1}"):
with reporter.step(f"Delete bucket {bucket_1}"):
s3_client.delete_bucket(bucket_1)
tick_epoch(client_shell, cluster)
with allure.step(f"Check bucket {bucket_1} deleted"):
with reporter.step(f"Check bucket {bucket_1} deleted"):
with pytest.raises(Exception, match=r".*Not Found.*"):
s3_client.head_bucket(bucket_1)
@ -128,7 +120,7 @@ class TestS3Gate:
bucket_1, bucket_2 = two_buckets
for bucket in (bucket_1, bucket_2):
with allure.step("Bucket must be empty"):
with reporter.step("Bucket must be empty"):
objects_list = s3_client.list_objects(bucket)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
@ -136,18 +128,15 @@ class TestS3Gate:
s3_client.head_object(bucket, file_name)
bucket_objects = s3_client.list_objects(bucket)
assert (
file_name in bucket_objects
), f"Expected file {file_name} in objects list {bucket_objects}"
assert file_name in bucket_objects, f"Expected file {file_name} in objects list {bucket_objects}"
with allure.step("Check object's attributes"):
with reporter.step("Check object's attributes"):
for attrs in (["ETag"], ["ObjectSize", "StorageClass"]):
s3_client.get_object_attributes(bucket, file_name, attrs)
@allure.title("Sync directory (s3_client={s3_client})")
def test_s3_sync_dir(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
@pytest.mark.parametrize("s3_client", [AwsCliClient], indirect=True)
def test_s3_sync_dir(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
"""
Test checks sync directory with AWS CLI utility.
"""
@ -155,21 +144,16 @@ class TestS3Gate:
file_path_2 = os.path.join(os.getcwd(), ASSETS_DIR, "test_sync", "test_file_2")
key_to_path = {"test_file_1": file_path_1, "test_file_2": file_path_2}
if not isinstance(s3_client, AwsCliClient):
pytest.skip("This test is not supported with boto3 client")
generate_file_with_content(simple_object_size.value, file_path=file_path_1)
generate_file_with_content(simple_object_size.value, file_path=file_path_2)
s3_client.sync(bucket=bucket, dir_path=os.path.dirname(file_path_1))
with allure.step("Check objects are synced"):
with reporter.step("Check objects are synced"):
objects = s3_client.list_objects(bucket)
with allure.step("Check these are the same objects"):
assert set(key_to_path.keys()) == set(
objects
), f"Expected all objects saved. Got {objects}"
with reporter.step("Check these are the same objects"):
assert set(key_to_path.keys()) == set(objects), f"Expected all objects saved. Got {objects}"
for obj_key in objects:
got_object = s3_client.get_object(bucket, obj_key)
assert get_file_hash(got_object) == get_file_hash(
@ -177,76 +161,60 @@ class TestS3Gate:
), "Expected hashes are the same"
@allure.title("Object versioning (s3_client={s3_client})")
def test_s3_api_versioning(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_api_versioning(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
"""
Test checks basic versioning functionality for S3 bucket.
"""
version_1_content = "Version 1"
version_2_content = "Version 2"
file_name_simple = generate_file_with_content(
simple_object_size.value, content=version_1_content
)
file_name_simple = generate_file_with_content(simple_object_size.value, content=version_1_content)
obj_key = os.path.basename(file_name_simple)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
version_id_1 = s3_client.put_object(bucket, file_name_simple)
generate_file_with_content(
simple_object_size.value, file_path=file_name_simple, content=version_2_content
)
generate_file_with_content(simple_object_size.value, file_path=file_name_simple, content=version_2_content)
version_id_2 = s3_client.put_object(bucket, file_name_simple)
with allure.step("Check bucket shows all versions"):
with reporter.step("Check bucket shows all versions"):
versions = s3_client.list_objects_versions(bucket)
obj_versions = {
version.get("VersionId") for version in versions if version.get("Key") == obj_key
}
obj_versions = {version.get("VersionId") for version in versions if version.get("Key") == obj_key}
assert obj_versions == {
version_id_1,
version_id_2,
}, f"Expected object has versions: {version_id_1, version_id_2}"
with allure.step("Show information about particular version"):
with reporter.step("Show information about particular version"):
for version_id in (version_id_1, version_id_2):
response = s3_client.head_object(bucket, obj_key, version_id=version_id)
assert "LastModified" in response, "Expected LastModified field"
assert "ETag" in response, "Expected ETag field"
assert (
response.get("VersionId") == version_id
), f"Expected VersionId is {version_id}"
assert response.get("VersionId") == version_id, f"Expected VersionId is {version_id}"
assert response.get("ContentLength") != 0, "Expected ContentLength is not zero"
with allure.step("Check object's attributes"):
with reporter.step("Check object's attributes"):
for version_id in (version_id_1, version_id_2):
got_attrs = s3_client.get_object_attributes(
bucket, obj_key, ["ETag"], version_id=version_id
)
got_attrs = s3_client.get_object_attributes(bucket, obj_key, ["ETag"], version_id=version_id)
if got_attrs:
assert (
got_attrs.get("VersionId") == version_id
), f"Expected VersionId is {version_id}"
assert got_attrs.get("VersionId") == version_id, f"Expected VersionId is {version_id}"
with allure.step("Delete object and check it was deleted"):
with reporter.step("Delete object and check it was deleted"):
response = s3_client.delete_object(bucket, obj_key)
version_id_delete = response.get("VersionId")
with pytest.raises(Exception, match=r".*Not Found.*"):
s3_client.head_object(bucket, obj_key)
with allure.step("Get content for all versions and check it is correct"):
with reporter.step("Get content for all versions and check it is correct"):
for version, content in (
(version_id_2, version_2_content),
(version_id_1, version_1_content),
):
file_name = s3_client.get_object(bucket, obj_key, version_id=version)
got_content = get_file_content(file_name)
assert (
got_content == content
), f"Expected object content is\n{content}\nGot\n{got_content}"
assert got_content == content, f"Expected object content is\n{content}\nGot\n{got_content}"
with allure.step("Restore previous object version"):
with reporter.step("Restore previous object version"):
s3_client.delete_object(bucket, obj_key, version_id=version_id_delete)
file_name = s3_client.get_object(bucket, obj_key)
@ -257,17 +225,13 @@ class TestS3Gate:
@pytest.mark.s3_gate_multipart
@allure.title("Object Multipart API (s3_client={s3_client})")
def test_s3_api_multipart(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_api_multipart(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
"""
Test checks S3 Multipart API (Create multipart upload/Abort multipart upload/List multipart upload/
Upload part/List parts/Complete multipart upload).
"""
parts_count = 3
file_name_large = generate_file(
simple_object_size.value * 1024 * 6 * parts_count
) # 5Mb - min part
file_name_large = generate_file(simple_object_size.value * 1024 * 6 * parts_count) # 5Mb - min part
object_key = s3_helper.object_key_from_file_path(file_name_large)
part_files = split_file(file_name_large, parts_count)
parts = []
@ -275,39 +239,33 @@ class TestS3Gate:
uploads = s3_client.list_multipart_uploads(bucket)
assert not uploads, f"Expected there is no uploads in bucket {bucket}"
with allure.step("Create and abort multipart upload"):
with reporter.step("Create and abort multipart upload"):
upload_id = s3_client.create_multipart_upload(bucket, object_key)
uploads = s3_client.list_multipart_uploads(bucket)
assert uploads, f"Expected there one upload in bucket {bucket}"
assert (
uploads[0].get("Key") == object_key
), f"Expected correct key {object_key} in upload {uploads}"
assert (
uploads[0].get("UploadId") == upload_id
), f"Expected correct UploadId {upload_id} in upload {uploads}"
assert uploads[0].get("Key") == object_key, f"Expected correct key {object_key} in upload {uploads}"
assert uploads[0].get("UploadId") == upload_id, f"Expected correct UploadId {upload_id} in upload {uploads}"
s3_client.abort_multipart_upload(bucket, object_key, upload_id)
uploads = s3_client.list_multipart_uploads(bucket)
assert not uploads, f"Expected there is no uploads in bucket {bucket}"
with allure.step("Create new multipart upload and upload several parts"):
with reporter.step("Create new multipart upload and upload several parts"):
upload_id = s3_client.create_multipart_upload(bucket, object_key)
for part_id, file_path in enumerate(part_files, start=1):
etag = s3_client.upload_part(bucket, object_key, upload_id, part_id, file_path)
parts.append((part_id, etag))
with allure.step("Check all parts are visible in bucket"):
with reporter.step("Check all parts are visible in bucket"):
got_parts = s3_client.list_parts(bucket, object_key, upload_id)
assert len(got_parts) == len(
part_files
), f"Expected {parts_count} parts, got\n{got_parts}"
assert len(got_parts) == len(part_files), f"Expected {parts_count} parts, got\n{got_parts}"
s3_client.complete_multipart_upload(bucket, object_key, upload_id, parts)
uploads = s3_client.list_multipart_uploads(bucket)
assert not uploads, f"Expected there is no uploads in bucket {bucket}"
with allure.step("Check we can get whole object from bucket"):
with reporter.step("Check we can get whole object from bucket"):
got_object = s3_client.get_object(bucket, object_key)
assert get_file_hash(got_object) == get_file_hash(file_name_large)
@ -327,9 +285,7 @@ class TestS3Gate:
s3_helper.check_tags_by_bucket(s3_client, bucket, [])
@allure.title("Object tagging API (s3_client={s3_client})")
def test_s3_api_object_tagging(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_api_object_tagging(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
"""
Test checks S3 Object tagging API (Put tag/Get tag/Update tag).
"""
@ -378,12 +334,12 @@ class TestS3Gate:
bucket_1, bucket_2 = two_buckets
with allure.step(f"Generate {max_obj_count} files"):
with reporter.step(f"Generate {max_obj_count} files"):
for _ in range(max_obj_count):
file_paths.append(generate_file(choice(obj_sizes).value))
for bucket in (bucket_1, bucket_2):
with allure.step(f"Bucket {bucket} must be empty as it just created"):
with reporter.step(f"Bucket {bucket} must be empty as it just created"):
objects_list = s3_client.list_objects_v2(bucket)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
@ -391,18 +347,18 @@ class TestS3Gate:
s3_client.put_object(bucket, file_path)
put_objects.append(s3_helper.object_key_from_file_path(file_path))
with allure.step(f"Check all objects put in bucket {bucket} successfully"):
with reporter.step(f"Check all objects put in bucket {bucket} successfully"):
bucket_objects = s3_client.list_objects_v2(bucket)
assert set(put_objects) == set(
bucket_objects
), f"Expected all objects {put_objects} in objects list {bucket_objects}"
with allure.step("Delete some objects from bucket_1 one by one"):
with reporter.step("Delete some objects from bucket_1 one by one"):
objects_to_delete_b1 = choices(put_objects, k=max_delete_objects)
for obj in objects_to_delete_b1:
s3_client.delete_object(bucket_1, obj)
with allure.step("Check deleted objects are not visible in bucket bucket_1"):
with reporter.step("Check deleted objects are not visible in bucket bucket_1"):
bucket_objects = s3_client.list_objects_v2(bucket_1)
assert set(put_objects).difference(set(objects_to_delete_b1)) == set(
bucket_objects
@ -411,11 +367,11 @@ class TestS3Gate:
with pytest.raises(Exception, match="The specified key does not exist"):
s3_client.get_object(bucket_1, object_key)
with allure.step("Delete some objects from bucket_2 at once"):
with reporter.step("Delete some objects from bucket_2 at once"):
objects_to_delete_b2 = choices(put_objects, k=max_delete_objects)
s3_client.delete_objects(bucket_2, objects_to_delete_b2)
with allure.step("Check deleted objects are not visible in bucket bucket_2"):
with reporter.step("Check deleted objects are not visible in bucket bucket_2"):
objects_list = s3_client.list_objects_v2(bucket_2)
assert set(put_objects).difference(set(objects_to_delete_b2)) == set(
objects_list
@ -442,27 +398,25 @@ class TestS3Gate:
file_name_large = s3_helper.object_key_from_file_path(file_path_large)
bucket_objects = [file_name_simple, file_name_large]
with allure.step("Bucket must be empty"):
with reporter.step("Bucket must be empty"):
objects_list = s3_client.list_objects(bucket)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
with allure.step("Put objects into bucket"):
with reporter.step("Put objects into bucket"):
for file_path in (file_path_simple, file_path_large):
s3_client.put_object(bucket, file_path)
with allure.step("Copy one object into the same bucket"):
with reporter.step("Copy one object into the same bucket"):
copy_obj_path = s3_client.copy_object(bucket, file_name_simple)
bucket_objects.append(copy_obj_path)
s3_helper.check_objects_in_bucket(s3_client, bucket, bucket_objects)
with allure.step("Check copied object has the same content"):
with reporter.step("Check copied object has the same content"):
got_copied_file = s3_client.get_object(bucket, copy_obj_path)
assert get_file_hash(file_path_simple) == get_file_hash(
got_copied_file
), "Hashes must be the same"
assert get_file_hash(file_path_simple) == get_file_hash(got_copied_file), "Hashes must be the same"
with allure.step("Delete one object from bucket"):
with reporter.step("Delete one object from bucket"):
s3_client.delete_object(bucket, file_name_simple)
bucket_objects.remove(file_name_simple)
@ -493,56 +447,46 @@ class TestS3Gate:
bucket_1, bucket_2 = two_buckets
with allure.step("Buckets must be empty"):
with reporter.step("Buckets must be empty"):
for bucket in (bucket_1, bucket_2):
objects_list = s3_client.list_objects(bucket)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
with allure.step("Put objects into one bucket"):
with reporter.step("Put objects into one bucket"):
for file_path in (file_path_simple, file_path_large):
s3_client.put_object(bucket_1, file_path)
with allure.step("Copy object from first bucket into second"):
with reporter.step("Copy object from first bucket into second"):
copy_obj_path_b2 = s3_client.copy_object(bucket_1, file_name_large, bucket=bucket_2)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, expected_objects=bucket_1_objects)
s3_helper.check_objects_in_bucket(s3_client, bucket_2, expected_objects=[copy_obj_path_b2])
with allure.step("Check copied object has the same content"):
with reporter.step("Check copied object has the same content"):
got_copied_file_b2 = s3_client.get_object(bucket_2, copy_obj_path_b2)
assert get_file_hash(file_path_large) == get_file_hash(
got_copied_file_b2
), "Hashes must be the same"
assert get_file_hash(file_path_large) == get_file_hash(got_copied_file_b2), "Hashes must be the same"
with allure.step("Delete one object from first bucket"):
with reporter.step("Delete one object from first bucket"):
s3_client.delete_object(bucket_1, file_name_simple)
bucket_1_objects.remove(file_name_simple)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, expected_objects=bucket_1_objects)
s3_helper.check_objects_in_bucket(s3_client, bucket_2, expected_objects=[copy_obj_path_b2])
with allure.step("Delete one object from second bucket and check it is empty"):
with reporter.step("Delete one object from second bucket and check it is empty"):
s3_client.delete_object(bucket_2, copy_obj_path_b2)
s3_helper.check_objects_in_bucket(s3_client, bucket_2, expected_objects=[])
def check_object_attributes(
self, s3_client: S3ClientWrapper, bucket: str, object_key: str, parts_count: int
):
def check_object_attributes(self, s3_client: S3ClientWrapper, bucket: str, object_key: str, parts_count: int):
if not isinstance(s3_client, AwsCliClient):
logger.warning("Attributes check is not supported for boto3 implementation")
return
with allure.step("Check object's attributes"):
obj_parts = s3_client.get_object_attributes(
bucket, object_key, ["ObjectParts"], full_output=False
)
assert (
obj_parts.get("TotalPartsCount") == parts_count
), f"Expected TotalPartsCount is {parts_count}"
assert (
len(obj_parts.get("Parts")) == parts_count
), f"Expected Parts cunt is {parts_count}"
with reporter.step("Check object's attributes"):
obj_parts = s3_client.get_object_attributes(bucket, object_key, ["ObjectParts"], full_output=False)
assert obj_parts.get("TotalPartsCount") == parts_count, f"Expected TotalPartsCount is {parts_count}"
assert len(obj_parts.get("Parts")) == parts_count, f"Expected Parts cunt is {parts_count}"
with allure.step("Check object's attribute max-parts"):
with reporter.step("Check object's attribute max-parts"):
max_parts = 2
obj_parts = s3_client.get_object_attributes(
bucket,
@ -551,15 +495,11 @@ class TestS3Gate:
max_parts=max_parts,
full_output=False,
)
assert (
obj_parts.get("TotalPartsCount") == parts_count
), f"Expected TotalPartsCount is {parts_count}"
assert obj_parts.get("TotalPartsCount") == parts_count, f"Expected TotalPartsCount is {parts_count}"
assert obj_parts.get("MaxParts") == max_parts, f"Expected MaxParts is {parts_count}"
assert (
len(obj_parts.get("Parts")) == max_parts
), f"Expected Parts count is {parts_count}"
assert len(obj_parts.get("Parts")) == max_parts, f"Expected Parts count is {parts_count}"
with allure.step("Check object's attribute part-number-marker"):
with reporter.step("Check object's attribute part-number-marker"):
part_number_marker = 3
obj_parts = s3_client.get_object_attributes(
bucket,
@ -568,9 +508,7 @@ class TestS3Gate:
part_number=part_number_marker,
full_output=False,
)
assert (
obj_parts.get("TotalPartsCount") == parts_count
), f"Expected TotalPartsCount is {parts_count}"
assert obj_parts.get("TotalPartsCount") == parts_count, f"Expected TotalPartsCount is {parts_count}"
assert (
obj_parts.get("PartNumberMarker") == part_number_marker
), f"Expected PartNumberMarker is {part_number_marker}"

111
pytest_tests/testsuites/services/s3_gate/test_s3_locking.py
View file

@ -3,35 +3,26 @@ from datetime import datetime, timedelta
import allure
import pytest
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper
from frostfs_testlib import reporter
from frostfs_testlib.s3 import S3ClientWrapper
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.utils.file_utils import generate_file, generate_file_with_content
def pytest_generate_tests(metafunc: pytest.Metafunc):
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize("s3_client", [AwsCliClient, Boto3ClientWrapper], indirect=True)
@pytest.mark.sanity
@pytest.mark.s3_gate
@pytest.mark.s3_gate_locking
@pytest.mark.parametrize("version_id", [None, "second"])
class TestS3GateLocking:
@allure.title(
"Retention period and legal lock on object (version_id={version_id}, s3_client={s3_client})"
)
def test_s3_object_locking(
self, s3_client: S3ClientWrapper, version_id: str, simple_object_size: ObjectSize
):
@allure.title("Retention period and legal lock on object (version_id={version_id}, s3_client={s3_client})")
def test_s3_object_locking(self, s3_client: S3ClientWrapper, version_id: str, simple_object_size: ObjectSize):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
retention_period = 2
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=True)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
s3_client.put_object(bucket, file_path)
file_name_1 = generate_file_with_content(simple_object_size.value, file_path=file_path)
version_id_2 = s3_client.put_object(bucket, file_name_1)
@ -39,36 +30,30 @@ class TestS3GateLocking:
if version_id:
version_id = version_id_2
with allure.step(f"Put retention period {retention_period}min to object {file_name}"):
with reporter.step(f"Put retention period {retention_period}min to object {file_name}"):
date_obj = datetime.utcnow() + timedelta(minutes=retention_period)
retention = {
"Mode": "COMPLIANCE",
"RetainUntilDate": date_obj,
}
s3_client.put_object_retention(bucket, file_name, retention, version_id)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "COMPLIANCE", date_obj, "OFF"
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "COMPLIANCE", date_obj, "OFF")
with allure.step(f"Put legal hold to object {file_name}"):
with reporter.step(f"Put legal hold to object {file_name}"):
s3_client.put_object_legal_hold(bucket, file_name, "ON", version_id)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "COMPLIANCE", date_obj, "ON"
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "COMPLIANCE", date_obj, "ON")
with allure.step("Fail with deleting object with legal hold and retention period"):
with reporter.step("Fail with deleting object with legal hold and retention period"):
if version_id:
with pytest.raises(Exception):
# An error occurred (AccessDenied) when calling the DeleteObject operation (reached max retries: 0): Access Denied.
s3_client.delete_object(bucket, file_name, version_id)
with allure.step("Check retention period is no longer set on the uploaded object"):
with reporter.step("Check retention period is no longer set on the uploaded object"):
time.sleep((retention_period + 1) * 60)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "COMPLIANCE", date_obj, "ON"
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "COMPLIANCE", date_obj, "ON")
with allure.step("Fail with deleting object with legal hold and retention period"):
with reporter.step("Fail with deleting object with legal hold and retention period"):
if version_id:
with pytest.raises(Exception):
# An error occurred (AccessDenied) when calling the DeleteObject operation (reached max retries: 0): Access Denied.
@ -76,12 +61,8 @@ class TestS3GateLocking:
else:
s3_client.delete_object(bucket, file_name, version_id)
@allure.title(
"Impossible to change retention mode COMPLIANCE (version_id={version_id}, s3_client={s3_client})"
)
def test_s3_mode_compliance(
self, s3_client: S3ClientWrapper, version_id: str, simple_object_size: ObjectSize
):
@allure.title("Impossible to change retention mode COMPLIANCE (version_id={version_id}, s3_client={s3_client})")
def test_s3_mode_compliance(self, s3_client: S3ClientWrapper, version_id: str, simple_object_size: ObjectSize):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
retention_period = 2
@ -89,26 +70,22 @@ class TestS3GateLocking:
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=True)
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
obj_version = s3_client.put_object(bucket, file_path)
if version_id:
version_id = obj_version
s3_helper.check_objects_in_bucket(s3_client, bucket, [file_name])
with allure.step(f"Put retention period {retention_period}min to object {file_name}"):
with reporter.step(f"Put retention period {retention_period}min to object {file_name}"):
date_obj = datetime.utcnow() + timedelta(minutes=retention_period)
retention = {
"Mode": "COMPLIANCE",
"RetainUntilDate": date_obj,
}
s3_client.put_object_retention(bucket, file_name, retention, version_id)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "COMPLIANCE", date_obj, "OFF"
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "COMPLIANCE", date_obj, "OFF")
with allure.step(
f"Try to change retention period {retention_period_1}min to object {file_name}"
):
with reporter.step(f"Try to change retention period {retention_period_1}min to object {file_name}"):
date_obj = datetime.utcnow() + timedelta(minutes=retention_period_1)
retention = {
"Mode": "COMPLIANCE",
@ -117,12 +94,8 @@ class TestS3GateLocking:
with pytest.raises(Exception):
s3_client.put_object_retention(bucket, file_name, retention, version_id)
@allure.title(
"Change retention mode GOVERNANCE (version_id={version_id}, s3_client={s3_client})"
)
def test_s3_mode_governance(
self, s3_client: S3ClientWrapper, version_id: str, simple_object_size: ObjectSize
):
@allure.title("Change retention mode GOVERNANCE (version_id={version_id}, s3_client={s3_client})")
def test_s3_mode_governance(self, s3_client: S3ClientWrapper, version_id: str, simple_object_size: ObjectSize):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
retention_period = 3
@ -131,26 +104,22 @@ class TestS3GateLocking:
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=True)
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
obj_version = s3_client.put_object(bucket, file_path)
if version_id:
version_id = obj_version
s3_helper.check_objects_in_bucket(s3_client, bucket, [file_name])
with allure.step(f"Put retention period {retention_period}min to object {file_name}"):
with reporter.step(f"Put retention period {retention_period}min to object {file_name}"):
date_obj = datetime.utcnow() + timedelta(minutes=retention_period)
retention = {
"Mode": "GOVERNANCE",
"RetainUntilDate": date_obj,
}
s3_client.put_object_retention(bucket, file_name, retention, version_id)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "GOVERNANCE", date_obj, "OFF"
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "GOVERNANCE", date_obj, "OFF")
with allure.step(
f"Try to change retention period {retention_period_1}min to object {file_name}"
):
with reporter.step(f"Try to change retention period {retention_period_1}min to object {file_name}"):
date_obj = datetime.utcnow() + timedelta(minutes=retention_period_1)
retention = {
"Mode": "GOVERNANCE",
@ -159,9 +128,7 @@ class TestS3GateLocking:
with pytest.raises(Exception):
s3_client.put_object_retention(bucket, file_name, retention, version_id)
with allure.step(
f"Try to change retention period {retention_period_1}min to object {file_name}"
):
with reporter.step(f"Try to change retention period {retention_period_1}min to object {file_name}"):
date_obj = datetime.utcnow() + timedelta(minutes=retention_period_1)
retention = {
"Mode": "GOVERNANCE",
@ -170,35 +137,31 @@ class TestS3GateLocking:
with pytest.raises(Exception):
s3_client.put_object_retention(bucket, file_name, retention, version_id)
with allure.step(f"Put new retention period {retention_period_2}min to object {file_name}"):
with reporter.step(f"Put new retention period {retention_period_2}min to object {file_name}"):
date_obj = datetime.utcnow() + timedelta(minutes=retention_period_2)
retention = {
"Mode": "GOVERNANCE",
"RetainUntilDate": date_obj,
}
s3_client.put_object_retention(bucket, file_name, retention, version_id, True)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "GOVERNANCE", date_obj, "OFF"
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "GOVERNANCE", date_obj, "OFF")
@allure.title(
"[NEGATIVE] Lock object in bucket with disabled locking (version_id={version_id}, s3_client={s3_client})"
)
def test_s3_legal_hold(
self, s3_client: S3ClientWrapper, version_id: str, simple_object_size: ObjectSize
):
def test_s3_legal_hold(self, s3_client: S3ClientWrapper, version_id: str, simple_object_size: ObjectSize):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=False)
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
obj_version = s3_client.put_object(bucket, file_path)
if version_id:
version_id = obj_version
s3_helper.check_objects_in_bucket(s3_client, bucket, [file_name])
with allure.step(f"Put legal hold to object {file_name}"):
with reporter.step(f"Put legal hold to object {file_name}"):
with pytest.raises(Exception):
s3_client.put_object_legal_hold(bucket, file_name, "ON", version_id)
@ -213,20 +176,18 @@ class TestS3GateLockingBucket:
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=True)
with allure.step("PutObjectLockConfiguration with ObjectLockEnabled=False"):
with reporter.step("PutObjectLockConfiguration with ObjectLockEnabled=False"):
s3_client.put_object_lock_configuration(bucket, configuration)
with allure.step("PutObjectLockConfiguration with ObjectLockEnabled=True"):
with reporter.step("PutObjectLockConfiguration with ObjectLockEnabled=True"):
configuration["ObjectLockEnabled"] = "Enabled"
s3_client.put_object_lock_configuration(bucket, configuration)
with allure.step("GetObjectLockConfiguration"):
with reporter.step("GetObjectLockConfiguration"):
config = s3_client.get_object_lock_configuration(bucket)
configuration["Rule"]["DefaultRetention"]["Years"] = 0
assert config == configuration, f"Configurations must be equal {configuration}"
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
s3_client.put_object(bucket, file_path)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "COMPLIANCE", None, "OFF", 1
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "COMPLIANCE", None, "OFF", 1)

71
pytest_tests/testsuites/services/s3_gate/test_s3_multipart.py
View file

@ -1,6 +1,7 @@
import allure
import pytest
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper, VersioningStatus
from frostfs_testlib import reporter
from frostfs_testlib.s3 import S3ClientWrapper, VersioningStatus
from frostfs_testlib.steps.cli.container import list_objects, search_container_by_name
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
@ -10,18 +11,10 @@ from frostfs_testlib.utils.file_utils import generate_file, get_file_hash, split
PART_SIZE = 5 * 1024 * 1024
def pytest_generate_tests(metafunc: pytest.Metafunc):
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize("s3_client", [AwsCliClient, Boto3ClientWrapper], indirect=True)
@pytest.mark.sanity
@pytest.mark.s3_gate
@pytest.mark.s3_gate_multipart
class TestS3GateMultipart(ClusterTestBase):
NO_SUCH_UPLOAD = (
"The upload ID may be invalid, or the upload may have been aborted or completed."
)
NO_SUCH_UPLOAD = "The upload ID may be invalid, or the upload may have been aborted or completed."
@allure.title("Object Multipart API (s3_client={s3_client})")
@pytest.mark.parametrize("versioning_status", [VersioningStatus.ENABLED], indirect=True)
@ -32,7 +25,7 @@ class TestS3GateMultipart(ClusterTestBase):
part_files = split_file(file_name_large, parts_count)
parts = []
with allure.step("Upload first part"):
with reporter.step("Upload first part"):
upload_id = s3_client.create_multipart_upload(bucket, object_key)
uploads = s3_client.list_multipart_uploads(bucket)
etag = s3_client.upload_part(bucket, object_key, upload_id, 1, part_files[0])
@ -40,21 +33,19 @@ class TestS3GateMultipart(ClusterTestBase):
got_parts = s3_client.list_parts(bucket, object_key, upload_id)
assert len(got_parts) == 1, f"Expected {1} parts, got\n{got_parts}"
with allure.step("Upload last parts"):
with reporter.step("Upload last parts"):
for part_id, file_path in enumerate(part_files[1:], start=2):
etag = s3_client.upload_part(bucket, object_key, upload_id, part_id, file_path)
parts.append((part_id, etag))
got_parts = s3_client.list_parts(bucket, object_key, upload_id)
s3_client.complete_multipart_upload(bucket, object_key, upload_id, parts)
assert len(got_parts) == len(
part_files
), f"Expected {parts_count} parts, got\n{got_parts}"
assert len(got_parts) == len(part_files), f"Expected {parts_count} parts, got\n{got_parts}"
with allure.step("Check upload list is empty"):
with reporter.step("Check upload list is empty"):
uploads = s3_client.list_multipart_uploads(bucket)
assert not uploads, f"Expected there is no uploads in bucket {bucket}"
with allure.step("Check we can get whole object from bucket"):
with reporter.step("Check we can get whole object from bucket"):
got_object = s3_client.get_object(bucket, object_key)
assert get_file_hash(got_object) == get_file_hash(file_name_large)
@ -74,43 +65,37 @@ class TestS3GateMultipart(ClusterTestBase):
files_count = len(to_upload)
upload_key = "multipart_abort"
with allure.step(f"Get related container_id for bucket '{bucket}'"):
with reporter.step(f"Get related container_id for bucket '{bucket}'"):
container_id = search_container_by_name(
default_wallet, bucket, self.shell, self.cluster.default_rpc_endpoint
)
with allure.step("Create multipart upload"):
with reporter.step("Create multipart upload"):
upload_id = s3_client.create_multipart_upload(bucket, upload_key)
with allure.step(f"Upload {files_count} files to multipart upload"):
with reporter.step(f"Upload {files_count} files to multipart upload"):
for i, file in enumerate(to_upload, 1):
s3_client.upload_part(bucket, upload_key, upload_id, i, file)
with allure.step(f"Check that we have {files_count} files in bucket"):
with reporter.step(f"Check that we have {files_count} files in bucket"):
parts = s3_client.list_parts(bucket, upload_key, upload_id)
assert len(parts) == files_count, f"Expected {files_count} parts, got\n{parts}"
with allure.step(f"Check that we have {files_count} files in container '{container_id}'"):
objects = list_objects(
default_wallet, self.shell, container_id, self.cluster.default_rpc_endpoint
)
assert (
len(objects) == files_count
), f"Expected {files_count} objects in container, got\n{objects}"
with reporter.step(f"Check that we have {files_count} files in container '{container_id}'"):
objects = list_objects(default_wallet, self.shell, container_id, self.cluster.default_rpc_endpoint)
assert len(objects) == files_count, f"Expected {files_count} objects in container, got\n{objects}"
with allure.step("Abort multipart upload"):
with reporter.step("Abort multipart upload"):
s3_client.abort_multipart_upload(bucket, upload_key, upload_id)
uploads = s3_client.list_multipart_uploads(bucket)
assert not uploads, f"Expected no uploads in bucket {bucket}"
with allure.step("Check that we have no files in bucket since upload was aborted"):
with reporter.step("Check that we have no files in bucket since upload was aborted"):
with pytest.raises(Exception, match=self.NO_SUCH_UPLOAD):
s3_client.list_parts(bucket, upload_key, upload_id)
with allure.step("Check that we have no files in container since upload was aborted"):
objects = list_objects(
default_wallet, self.shell, container_id, self.cluster.default_rpc_endpoint
)
with reporter.step("Check that we have no files in container since upload was aborted"):
objects = list_objects(default_wallet, self.shell, container_id, self.cluster.default_rpc_endpoint)
assert len(objects) == 0, f"Expected no objects in container, got\n{objects}"
@allure.title("Upload Part Copy (s3_client={s3_client})")
@ -123,31 +108,27 @@ class TestS3GateMultipart(ClusterTestBase):
parts = []
objs = []
with allure.step(f"Put {parts_count} objects in bucket"):
with reporter.step(f"Put {parts_count} objects in bucket"):
for part in part_files:
s3_client.put_object(bucket, part)
objs.append(s3_helper.object_key_from_file_path(part))
s3_helper.check_objects_in_bucket(s3_client, bucket, objs)
with allure.step("Create multipart upload object"):
with reporter.step("Create multipart upload object"):
upload_id = s3_client.create_multipart_upload(bucket, object_key)
uploads = s3_client.list_multipart_uploads(bucket)
assert uploads, f"Expected there are uploads in bucket {bucket}"
with allure.step("Upload parts to multipart upload"):
with reporter.step("Upload parts to multipart upload"):
for part_id, obj_key in enumerate(objs, start=1):
etag = s3_client.upload_part_copy(
bucket, object_key, upload_id, part_id, f"{bucket}/{obj_key}"
)
etag = s3_client.upload_part_copy(bucket, object_key, upload_id, part_id, f"{bucket}/{obj_key}")
parts.append((part_id, etag))
got_parts = s3_client.list_parts(bucket, object_key, upload_id)
with allure.step("Complete multipart upload"):
with reporter.step("Complete multipart upload"):
s3_client.complete_multipart_upload(bucket, object_key, upload_id, parts)
assert len(got_parts) == len(
part_files
), f"Expected {parts_count} parts, got\n{got_parts}"
assert len(got_parts) == len(part_files), f"Expected {parts_count} parts, got\n{got_parts}"
with allure.step("Check we can get whole object from bucket"):
with reporter.step("Check we can get whole object from bucket"):
got_object = s3_client.get_object(bucket, object_key)
assert get_file_hash(got_object) == get_file_hash(file_name_large)

412
pytest_tests/testsuites/services/s3_gate/test_s3_object.py
View file

@ -7,27 +7,17 @@ from typing import Literal
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.common import ASSETS_DIR, DEFAULT_WALLET_PASS
from frostfs_testlib.resources.error_patterns import S3_MALFORMED_XML_REQUEST
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper, VersioningStatus
from frostfs_testlib.s3 import AwsCliClient, S3ClientWrapper, VersioningStatus
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.testing.test_control import expect_not_raises
from frostfs_testlib.utils import wallet_utils
from frostfs_testlib.utils.file_utils import (
concat_files,
generate_file,
generate_file_with_content,
get_file_hash,
)
from frostfs_testlib.utils.file_utils import concat_files, generate_file, generate_file_with_content, get_file_hash
def pytest_generate_tests(metafunc: pytest.Metafunc):
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize("s3_client", [AwsCliClient, Boto3ClientWrapper], indirect=True)
@pytest.mark.sanity
@pytest.mark.s3_gate
@pytest.mark.s3_gate_object
class TestS3GateObject:
@ -54,10 +44,10 @@ class TestS3GateObject:
objects_list = s3_client.list_objects(bucket_1)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
with allure.step("Put object into one bucket"):
with reporter.step("Put object into one bucket"):
s3_client.put_object(bucket_1, file_path)
with allure.step("Copy one object into the same bucket"):
with reporter.step("Copy one object into the same bucket"):
copy_obj_path = s3_client.copy_object(bucket_1, file_name)
bucket_1_objects.append(copy_obj_path)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, bucket_1_objects)
@ -65,32 +55,22 @@ class TestS3GateObject:
objects_list = s3_client.list_objects(bucket_2)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
with allure.step("Copy object from first bucket into second"):
with reporter.step("Copy object from first bucket into second"):
copy_obj_path_b2 = s3_client.copy_object(bucket_1, file_name, bucket=bucket_2)
s3_helper.check_objects_in_bucket(
s3_client, bucket_1, expected_objects=bucket_1_objects
)
s3_helper.check_objects_in_bucket(
s3_client, bucket_2, expected_objects=[copy_obj_path_b2]
)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, expected_objects=bucket_1_objects)
s3_helper.check_objects_in_bucket(s3_client, bucket_2, expected_objects=[copy_obj_path_b2])
with allure.step("Check copied object has the same content"):
with reporter.step("Check copied object has the same content"):
got_copied_file_b2 = s3_client.get_object(bucket_2, copy_obj_path_b2)
assert get_file_hash(file_path) == get_file_hash(
got_copied_file_b2
), "Hashes must be the same"
assert get_file_hash(file_path) == get_file_hash(got_copied_file_b2), "Hashes must be the same"
with allure.step("Delete one object from first bucket"):
with reporter.step("Delete one object from first bucket"):
s3_client.delete_object(bucket_1, file_name)
bucket_1_objects.remove(file_name)
s3_helper.check_objects_in_bucket(
s3_client, bucket_1, expected_objects=bucket_1_objects
)
s3_helper.check_objects_in_bucket(
s3_client, bucket_2, expected_objects=[copy_obj_path_b2]
)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, expected_objects=bucket_1_objects)
s3_helper.check_objects_in_bucket(s3_client, bucket_2, expected_objects=[copy_obj_path_b2])
with allure.step("Copy one object into the same bucket"):
with reporter.step("Copy one object into the same bucket"):
with pytest.raises(Exception):
s3_client.copy_object(bucket_1, file_name)
@ -102,70 +82,56 @@ class TestS3GateObject:
simple_object_size: ObjectSize,
):
version_1_content = "Version 1"
file_name_simple = generate_file_with_content(
simple_object_size.value, content=version_1_content
)
file_name_simple = generate_file_with_content(simple_object_size.value, content=version_1_content)
obj_key = os.path.basename(file_name_simple)
bucket_1, bucket_2 = two_buckets
s3_helper.set_bucket_versioning(s3_client, bucket_1, VersioningStatus.ENABLED)
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
s3_client.put_object(bucket_1, file_name_simple)
bucket_1_objects = [obj_key]
s3_helper.check_objects_in_bucket(s3_client, bucket_1, [obj_key])
with allure.step("Copy one object into the same bucket"):
with reporter.step("Copy one object into the same bucket"):
copy_obj_path = s3_client.copy_object(bucket_1, obj_key)
bucket_1_objects.append(copy_obj_path)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, bucket_1_objects)
s3_helper.set_bucket_versioning(s3_client, bucket_2, VersioningStatus.ENABLED)
with allure.step("Copy object from first bucket into second"):
with reporter.step("Copy object from first bucket into second"):
copy_obj_path_b2 = s3_client.copy_object(bucket_1, obj_key, bucket=bucket_2)
s3_helper.check_objects_in_bucket(
s3_client, bucket_1, expected_objects=bucket_1_objects
)
s3_helper.check_objects_in_bucket(
s3_client, bucket_2, expected_objects=[copy_obj_path_b2]
)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, expected_objects=bucket_1_objects)
s3_helper.check_objects_in_bucket(s3_client, bucket_2, expected_objects=[copy_obj_path_b2])
with allure.step("Delete one object from first bucket and check object in bucket"):
with reporter.step("Delete one object from first bucket and check object in bucket"):
s3_client.delete_object(bucket_1, obj_key)
bucket_1_objects.remove(obj_key)
s3_helper.check_objects_in_bucket(
s3_client, bucket_1, expected_objects=bucket_1_objects
)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, expected_objects=bucket_1_objects)
with allure.step("Copy one object into the same bucket"):
with reporter.step("Copy one object into the same bucket"):
with pytest.raises(Exception):
s3_client.copy_object(bucket_1, obj_key)
@allure.title("Copy with acl (s3_client={s3_client})")
def test_s3_copy_acl(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_copy_acl(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
version_1_content = "Version 1"
file_name_simple = generate_file_with_content(
simple_object_size.value, content=version_1_content
)
file_name_simple = generate_file_with_content(simple_object_size.value, content=version_1_content)
obj_key = os.path.basename(file_name_simple)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
s3_client.put_object(bucket, file_name_simple)
s3_helper.check_objects_in_bucket(s3_client, bucket, [obj_key])
with allure.step("Copy object and check acl attribute"):
with reporter.step("Copy object and check acl attribute"):
copy_obj_path = s3_client.copy_object(bucket, obj_key, acl="public-read-write")
obj_acl = s3_client.get_object_acl(bucket, copy_obj_path)
s3_helper.assert_s3_acl(acl_grants=obj_acl, permitted_users="CanonicalUser")
@allure.title("Copy object with metadata (s3_client={s3_client})")
def test_s3_copy_metadate(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_copy_metadate(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
object_metadata = {f"{uuid.uuid4()}": f"{uuid.uuid4()}"}
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
@ -173,29 +139,25 @@ class TestS3GateObject:
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
s3_client.put_object(bucket, file_path, metadata=object_metadata)
bucket_1_objects = [file_name]
s3_helper.check_objects_in_bucket(s3_client, bucket, bucket_1_objects)
with allure.step("Copy one object"):
with reporter.step("Copy one object"):
copy_obj_path = s3_client.copy_object(bucket, file_name)
bucket_1_objects.append(copy_obj_path)
s3_helper.check_objects_in_bucket(s3_client, bucket, bucket_1_objects)
obj_head = s3_client.head_object(bucket, copy_obj_path)
assert (
obj_head.get("Metadata") == object_metadata
), f"Metadata must be {object_metadata}"
assert obj_head.get("Metadata") == object_metadata, f"Metadata must be {object_metadata}"
with allure.step("Copy one object with metadata"):
with reporter.step("Copy one object with metadata"):
copy_obj_path = s3_client.copy_object(bucket, file_name, metadata_directive="COPY")
bucket_1_objects.append(copy_obj_path)
obj_head = s3_client.head_object(bucket, copy_obj_path)
assert (
obj_head.get("Metadata") == object_metadata
), f"Metadata must be {object_metadata}"
assert obj_head.get("Metadata") == object_metadata, f"Metadata must be {object_metadata}"
with allure.step("Copy one object with new metadata"):
with reporter.step("Copy one object with new metadata"):
object_metadata_1 = {f"{uuid.uuid4()}": f"{uuid.uuid4()}"}
copy_obj_path = s3_client.copy_object(
bucket,
@ -205,14 +167,10 @@ class TestS3GateObject:
)
bucket_1_objects.append(copy_obj_path)
obj_head = s3_client.head_object(bucket, copy_obj_path)
assert (
obj_head.get("Metadata") == object_metadata_1
), f"Metadata must be {object_metadata_1}"
assert obj_head.get("Metadata") == object_metadata_1, f"Metadata must be {object_metadata_1}"
@allure.title("Copy object with tagging (s3_client={s3_client})")
def test_s3_copy_tagging(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_copy_tagging(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
object_tagging = [(f"{uuid.uuid4()}", f"{uuid.uuid4()}")]
file_path = generate_file(simple_object_size.value)
file_name_simple = s3_helper.object_key_from_file_path(file_path)
@ -220,13 +178,13 @@ class TestS3GateObject:
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
s3_client.put_object(bucket, file_path)
s3_client.put_object_tagging(bucket, file_name_simple, tags=object_tagging)
bucket_1_objects = [file_name_simple]
s3_helper.check_objects_in_bucket(s3_client, bucket, bucket_1_objects)
with allure.step("Copy one object without tag"):
with reporter.step("Copy one object without tag"):
copy_obj_path = s3_client.copy_object(bucket, file_name_simple)
got_tags = s3_client.get_object_tagging(bucket, copy_obj_path)
assert got_tags, f"Expected tags, got {got_tags}"
@ -234,17 +192,15 @@ class TestS3GateObject:
for tag in expected_tags:
assert tag in got_tags, f"Expected tag {tag} in {got_tags}"
with allure.step("Copy one object with tag"):
copy_obj_path_1 = s3_client.copy_object(
bucket, file_name_simple, tagging_directive="COPY"
)
with reporter.step("Copy one object with tag"):
copy_obj_path_1 = s3_client.copy_object(bucket, file_name_simple, tagging_directive="COPY")
got_tags = s3_client.get_object_tagging(bucket, copy_obj_path_1)
assert got_tags, f"Expected tags, got {got_tags}"
expected_tags = [{"Key": key, "Value": value} for key, value in object_tagging]
for tag in expected_tags:
assert tag in got_tags, f"Expected tag {tag} in {got_tags}"
with allure.step("Copy one object with new tag"):
with reporter.step("Copy one object with new tag"):
tag_key = "tag1"
tag_value = uuid.uuid4()
new_tag = f"{tag_key}={tag_value}"
@ -270,75 +226,63 @@ class TestS3GateObject:
):
version_1_content = "Version 1"
version_2_content = "Version 2"
file_name_simple = generate_file_with_content(
simple_object_size.value, content=version_1_content
)
file_name_simple = generate_file_with_content(simple_object_size.value, content=version_1_content)
obj_key = os.path.basename(file_name_simple)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
version_id_1 = s3_client.put_object(bucket, file_name_simple)
file_name_1 = generate_file_with_content(
simple_object_size.value, file_path=file_name_simple, content=version_2_content
)
version_id_2 = s3_client.put_object(bucket, file_name_1)
with allure.step("Check bucket shows all versions"):
with reporter.step("Check bucket shows all versions"):
versions = s3_client.list_objects_versions(bucket)
obj_versions = {
version.get("VersionId") for version in versions if version.get("Key") == obj_key
}
obj_versions = {version.get("VersionId") for version in versions if version.get("Key") == obj_key}
assert obj_versions == {
version_id_1,
version_id_2,
}, f"Object should have versions: {version_id_1, version_id_2}"
with allure.step("Delete 1 version of object"):
with reporter.step("Delete 1 version of object"):
delete_obj = s3_client.delete_object(bucket, obj_key, version_id=version_id_1)
versions = s3_client.list_objects_versions(bucket)
obj_versions = {
version.get("VersionId") for version in versions if version.get("Key") == obj_key
}
obj_versions = {version.get("VersionId") for version in versions if version.get("Key") == obj_key}
assert obj_versions == {version_id_2}, f"Object should have versions: {version_id_2}"
assert "DeleteMarker" not in delete_obj.keys(), "Delete markers should not be created"
with allure.step("Delete second version of object"):
with reporter.step("Delete second version of object"):
delete_obj = s3_client.delete_object(bucket, obj_key, version_id=version_id_2)
versions = s3_client.list_objects_versions(bucket)
obj_versions = {
version.get("VersionId") for version in versions if version.get("Key") == obj_key
}
obj_versions = {version.get("VersionId") for version in versions if version.get("Key") == obj_key}
assert not obj_versions, "Expected object not found"
assert "DeleteMarker" not in delete_obj.keys(), "Delete markers should not be created"
with allure.step("Put new object into bucket"):
with reporter.step("Put new object into bucket"):
file_name_simple = generate_file(complex_object_size.value)
obj_key = os.path.basename(file_name_simple)
s3_client.put_object(bucket, file_name_simple)
with allure.step("Delete last object"):
with reporter.step("Delete last object"):
delete_obj = s3_client.delete_object(bucket, obj_key)
versions = s3_client.list_objects_versions(bucket, True)
assert versions.get("DeleteMarkers", None), "Expected delete Marker"
assert "DeleteMarker" in delete_obj.keys(), "Expected delete Marker"
@allure.title("Bulk delete version of object (s3_client={s3_client})")
def test_s3_bulk_delete_versioning(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_bulk_delete_versioning(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
version_1_content = "Version 1"
version_2_content = "Version 2"
version_3_content = "Version 3"
version_4_content = "Version 4"
file_name_1 = generate_file_with_content(
simple_object_size.value, content=version_1_content
)
file_name_1 = generate_file_with_content(simple_object_size.value, content=version_1_content)
obj_key = os.path.basename(file_name_1)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
version_id_1 = s3_client.put_object(bucket, file_name_1)
file_name_2 = generate_file_with_content(
simple_object_size.value, file_path=file_name_1, content=version_2_content
@ -354,66 +298,48 @@ class TestS3GateObject:
version_id_4 = s3_client.put_object(bucket, file_name_4)
version_ids = {version_id_1, version_id_2, version_id_3, version_id_4}
with allure.step("Check bucket shows all versions"):
with reporter.step("Check bucket shows all versions"):
versions = s3_client.list_objects_versions(bucket)
obj_versions = {
version.get("VersionId") for version in versions if version.get("Key") == obj_key
}
obj_versions = {version.get("VersionId") for version in versions if version.get("Key") == obj_key}
assert obj_versions == version_ids, f"Object should have versions: {version_ids}"
with allure.step("Delete two objects from bucket one by one"):
version_to_delete_b1 = sample(
[version_id_1, version_id_2, version_id_3, version_id_4], k=2
)
with reporter.step("Delete two objects from bucket one by one"):
version_to_delete_b1 = sample([version_id_1, version_id_2, version_id_3, version_id_4], k=2)
version_to_save = list(set(version_ids) - set(version_to_delete_b1))
for ver in version_to_delete_b1:
s3_client.delete_object(bucket, obj_key, ver)
with allure.step("Check bucket shows all versions"):
with reporter.step("Check bucket shows all versions"):
versions = s3_client.list_objects_versions(bucket)
obj_versions = [
version.get("VersionId") for version in versions if version.get("Key") == obj_key
]
assert (
obj_versions.sort() == version_to_save.sort()
), f"Object should have versions: {version_to_save}"
obj_versions = [version.get("VersionId") for version in versions if version.get("Key") == obj_key]
assert obj_versions.sort() == version_to_save.sort(), f"Object should have versions: {version_to_save}"
@allure.title("Get versions of object (s3_client={s3_client})")
def test_s3_get_versioning(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_get_versioning(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
version_1_content = "Version 1"
version_2_content = "Version 2"
file_name_simple = generate_file_with_content(
simple_object_size.value, content=version_1_content
)
file_name_simple = generate_file_with_content(simple_object_size.value, content=version_1_content)
obj_key = os.path.basename(file_name_simple)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
version_id_1 = s3_client.put_object(bucket, file_name_simple)
file_name_1 = generate_file_with_content(
simple_object_size.value, file_path=file_name_simple, content=version_2_content
)
version_id_2 = s3_client.put_object(bucket, file_name_1)
with allure.step("Get first version of object"):
with reporter.step("Get first version of object"):
object_1 = s3_client.get_object(bucket, obj_key, version_id_1, full_output=True)
assert (
object_1.get("VersionId") == version_id_1
), f"Get object with version {version_id_1}"
assert object_1.get("VersionId") == version_id_1, f"Get object with version {version_id_1}"
with allure.step("Get second version of object"):
with reporter.step("Get second version of object"):
object_2 = s3_client.get_object(bucket, obj_key, version_id_2, full_output=True)
assert (
object_2.get("VersionId") == version_id_2
), f"Get object with version {version_id_2}"
assert object_2.get("VersionId") == version_id_2, f"Get object with version {version_id_2}"
with allure.step("Get object"):
with reporter.step("Get object"):
object_3 = s3_client.get_object(bucket, obj_key, full_output=True)
assert (
object_3.get("VersionId") == version_id_2
), f"Get object with version {version_id_2}"
assert object_3.get("VersionId") == version_id_2, f"Get object with version {version_id_2}"
@allure.title("Get range (s3_client={s3_client})")
def test_s3_get_range(
@ -427,12 +353,12 @@ class TestS3GateObject:
file_name = s3_helper.object_key_from_file_path(file_path)
file_hash = get_file_hash(file_path)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
version_id_1 = s3_client.put_object(bucket, file_path)
file_name_1 = generate_file_with_content(simple_object_size.value, file_path=file_path)
version_id_2 = s3_client.put_object(bucket, file_name_1)
with allure.step("Get first version of object"):
with reporter.step("Get first version of object"):
object_1_part_1 = s3_client.get_object(
bucket,
file_name,
@ -460,7 +386,7 @@ class TestS3GateObject:
con_file = concat_files([object_1_part_1, object_1_part_2, object_1_part_3])
assert get_file_hash(con_file) == file_hash, "Hashes must be the same"
with allure.step("Get second version of object"):
with reporter.step("Get second version of object"):
object_2_part_1 = s3_client.get_object(
bucket,
file_name,
@ -483,11 +409,9 @@ class TestS3GateObject:
object_range=[2 * int(simple_object_size.value / 3) + 1, simple_object_size.value],
)
con_file_1 = concat_files([object_2_part_1, object_2_part_2, object_2_part_3])
assert get_file_hash(con_file_1) == get_file_hash(
file_name_1
), "Hashes must be the same"
assert get_file_hash(con_file_1) == get_file_hash(file_name_1), "Hashes must be the same"
with allure.step("Get object"):
with reporter.step("Get object"):
object_3_part_1 = s3_client.get_object(
bucket, file_name, object_range=[0, int(simple_object_size.value / 3)]
)
@ -527,7 +451,7 @@ class TestS3GateObject:
objects_in_bucket = []
objects_count = 3
with allure.step(f"Put {objects_count} into bucket"):
with reporter.step(f"Put {objects_count} into bucket"):
for _ in range(objects_count):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
@ -536,15 +460,15 @@ class TestS3GateObject:
# Extend deletion list to 1001 elements with same keys for test speed
objects_to_delete = self.copy_extend_list(objects_in_bucket, 1001)
with allure.step("Send delete request with 1001 objects and expect error"):
with reporter.step("Send delete request with 1001 objects and expect error"):
with pytest.raises(Exception, match=S3_MALFORMED_XML_REQUEST):
s3_client.delete_objects(bucket, objects_to_delete)
with allure.step("Send delete request with 1000 objects without error"):
with reporter.step("Send delete request with 1000 objects without error"):
with expect_not_raises():
s3_client.delete_objects(bucket, objects_to_delete[:1000])
@allure.title("Copy object with metadata (s3_client={s3_client})")
@allure.title("Object head is unloaded with the correct version (s3_client={s3_client})")
@pytest.mark.smoke
def test_s3_head_object(
self,
@ -558,36 +482,28 @@ class TestS3GateObject:
file_name = s3_helper.object_key_from_file_path(file_path)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
version_id_1 = s3_client.put_object(bucket, file_path, metadata=object_metadata)
file_name_1 = generate_file_with_content(simple_object_size.value, file_path=file_path)
version_id_2 = s3_client.put_object(bucket, file_name_1)
with allure.step("Get head of first version of object"):
with reporter.step("Get head of first version of object"):
response = s3_client.head_object(bucket, file_name)
assert "LastModified" in response, "Expected LastModified field"
assert "ETag" in response, "Expected ETag field"
assert response.get("Metadata") == {}, "Expected Metadata empty"
assert (
response.get("VersionId") == version_id_2
), f"Expected VersionId is {version_id_2}"
assert response.get("VersionId") == version_id_2, f"Expected VersionId is {version_id_2}"
assert response.get("ContentLength") != 0, "Expected ContentLength is not zero"
with allure.step("Get head ob first version of object"):
with reporter.step("Get head ob first version of object"):
response = s3_client.head_object(bucket, file_name, version_id=version_id_1)
assert "LastModified" in response, "Expected LastModified field"
assert "ETag" in response, "Expected ETag field"
assert (
response.get("Metadata") == object_metadata
), f"Expected Metadata is {object_metadata}"
assert (
response.get("VersionId") == version_id_1
), f"Expected VersionId is {version_id_1}"
assert response.get("Metadata") == object_metadata, f"Expected Metadata is {object_metadata}"
assert response.get("VersionId") == version_id_1, f"Expected VersionId is {version_id_1}"
assert response.get("ContentLength") != 0, "Expected ContentLength is not zero"
@allure.title(
"List of objects with version (method_version={list_type}, s3_client={s3_client})"
)
@allure.title("List of objects with version (method_version={list_type}, s3_client={s3_client})")
@pytest.mark.parametrize("list_type", ["v1", "v2"])
def test_s3_list_object(
self,
@ -602,11 +518,11 @@ class TestS3GateObject:
file_name_2 = s3_helper.object_key_from_file_path(file_path_2)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
s3_client.put_object(bucket, file_path_1)
s3_client.put_object(bucket, file_path_2)
with allure.step("Get list of object"):
with reporter.step("Get list of object"):
if list_type == "v1":
list_obj = s3_client.list_objects(bucket)
elif list_type == "v2":
@ -616,7 +532,7 @@ class TestS3GateObject:
list_obj.sort() == [file_name, file_name_2].sort()
), f"bucket should have object key {file_name, file_name_2}"
with allure.step("Delete object"):
with reporter.step("Delete object"):
delete_obj = s3_client.delete_object(bucket, file_name)
if list_type == "v1":
list_obj_1 = s3_client.list_objects(bucket, full_output=True)
@ -624,9 +540,7 @@ class TestS3GateObject:
list_obj_1 = s3_client.list_objects_v2(bucket, full_output=True)
contents = list_obj_1.get("Contents", [])
assert len(contents) == 1, "bucket should have only 1 object"
assert (
contents[0].get("Key") == file_name_2
), f"bucket should have object key {file_name_2}"
assert contents[0].get("Key") == file_name_2, f"bucket should have object key {file_name_2}"
assert "DeleteMarker" in delete_obj.keys(), "Expected delete Marker"
@allure.title("Put object (s3_client={s3_client})")
@ -649,28 +563,22 @@ class TestS3GateObject:
tag_2 = f"{tag_key_2}={tag_value_2}"
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.SUSPENDED)
with allure.step("Put first object into bucket"):
with reporter.step("Put first object into bucket"):
s3_client.put_object(bucket, file_path_1, metadata=object_1_metadata, tagging=tag_1)
obj_head = s3_client.head_object(bucket, file_name)
assert obj_head.get("Metadata") == object_1_metadata, "Metadata must be the same"
got_tags = s3_client.get_object_tagging(bucket, file_name)
assert got_tags, f"Expected tags, got {got_tags}"
assert got_tags == [
{"Key": tag_key_1, "Value": str(tag_value_1)}
], "Tags must be the same"
assert got_tags == [{"Key": tag_key_1, "Value": str(tag_value_1)}], "Tags must be the same"
with allure.step("Rewrite file into bucket"):
file_path_2 = generate_file_with_content(
simple_object_size.value, file_path=file_path_1
)
with reporter.step("Rewrite file into bucket"):
file_path_2 = generate_file_with_content(simple_object_size.value, file_path=file_path_1)
s3_client.put_object(bucket, file_path_2, metadata=object_2_metadata, tagging=tag_2)
obj_head = s3_client.head_object(bucket, file_name)
assert obj_head.get("Metadata") == object_2_metadata, "Metadata must be the same"
got_tags_1 = s3_client.get_object_tagging(bucket, file_name)
assert got_tags_1, f"Expected tags, got {got_tags_1}"
assert got_tags_1 == [
{"Key": tag_key_2, "Value": str(tag_value_2)}
], "Tags must be the same"
assert got_tags_1 == [{"Key": tag_key_2, "Value": str(tag_value_2)}], "Tags must be the same"
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
@ -682,29 +590,19 @@ class TestS3GateObject:
tag_value_3 = uuid.uuid4()
tag_3 = f"{tag_key_3}={tag_value_3}"
with allure.step("Put third object into bucket"):
version_id_1 = s3_client.put_object(
bucket, file_path_3, metadata=object_3_metadata, tagging=tag_3
)
with reporter.step("Put third object into bucket"):
version_id_1 = s3_client.put_object(bucket, file_path_3, metadata=object_3_metadata, tagging=tag_3)
obj_head_3 = s3_client.head_object(bucket, file_name_3)
assert obj_head_3.get("Metadata") == object_3_metadata, "Matadata must be the same"
got_tags_3 = s3_client.get_object_tagging(bucket, file_name_3)
assert got_tags_3, f"Expected tags, got {got_tags_3}"
assert got_tags_3 == [
{"Key": tag_key_3, "Value": str(tag_value_3)}
], "Tags must be the same"
assert got_tags_3 == [{"Key": tag_key_3, "Value": str(tag_value_3)}], "Tags must be the same"
with allure.step("Put new version of file into bucket"):
file_path_4 = generate_file_with_content(
simple_object_size.value, file_path=file_path_3
)
with reporter.step("Put new version of file into bucket"):
file_path_4 = generate_file_with_content(simple_object_size.value, file_path=file_path_3)
version_id_2 = s3_client.put_object(bucket, file_path_4)
versions = s3_client.list_objects_versions(bucket)
obj_versions = {
version.get("VersionId")
for version in versions
if version.get("Key") == file_name_3
}
obj_versions = {version.get("VersionId") for version in versions if version.get("Key") == file_name_3}
assert obj_versions == {
version_id_1,
version_id_2,
@ -712,28 +610,22 @@ class TestS3GateObject:
got_tags_4 = s3_client.get_object_tagging(bucket, file_name_3)
assert not got_tags_4, "No tags expected"
with allure.step("Get object"):
with reporter.step("Get object"):
object_3 = s3_client.get_object(bucket, file_name_3, full_output=True)
assert (
object_3.get("VersionId") == version_id_2
), f"get object with version {version_id_2}"
assert object_3.get("VersionId") == version_id_2, f"get object with version {version_id_2}"
object_3 = s3_client.get_object(bucket, file_name_3)
assert get_file_hash(file_path_4) == get_file_hash(object_3), "Hashes must be the same"
with allure.step("Get first version of object"):
with reporter.step("Get first version of object"):
object_4 = s3_client.get_object(bucket, file_name_3, version_id_1, full_output=True)
assert (
object_4.get("VersionId") == version_id_1
), f"get object with version {version_id_1}"
assert object_4.get("VersionId") == version_id_1, f"get object with version {version_id_1}"
object_4 = s3_client.get_object(bucket, file_name_3, version_id_1)
assert file_hash == get_file_hash(object_4), "Hashes must be the same"
obj_head_3 = s3_client.head_object(bucket, file_name_3, version_id_1)
assert obj_head_3.get("Metadata") == object_3_metadata, "Metadata must be the same"
got_tags_3 = s3_client.get_object_tagging(bucket, file_name_3, version_id_1)
assert got_tags_3, f"Expected tags, got {got_tags_3}"
assert got_tags_3 == [
{"Key": tag_key_3, "Value": str(tag_value_3)}
], "Tags must be the same"
assert got_tags_3 == [{"Key": tag_key_3, "Value": str(tag_value_3)}], "Tags must be the same"
@allure.title("Put object with ACL (versioning={bucket_versioning}, s3_client={s3_client})")
@pytest.mark.parametrize("bucket_versioning", ["ENABLED", "SUSPENDED"])
@ -754,37 +646,31 @@ class TestS3GateObject:
status = VersioningStatus.SUSPENDED
s3_helper.set_bucket_versioning(s3_client, bucket, status)
with allure.step("Put object with acl private"):
with reporter.step("Put object with acl private"):
s3_client.put_object(bucket, file_path_1, acl="private")
obj_acl = s3_client.get_object_acl(bucket, file_name)
s3_helper.assert_s3_acl(acl_grants=obj_acl, permitted_users="CanonicalUser")
object_1 = s3_client.get_object(bucket, file_name)
assert get_file_hash(file_path_1) == get_file_hash(object_1), "Hashes must be the same"
with allure.step("Put object with acl public-read"):
file_path_2 = generate_file_with_content(
simple_object_size.value, file_path=file_path_1
)
with reporter.step("Put object with acl public-read"):
file_path_2 = generate_file_with_content(simple_object_size.value, file_path=file_path_1)
s3_client.put_object(bucket, file_path_2, acl="public-read")
obj_acl = s3_client.get_object_acl(bucket, file_name)
s3_helper.assert_s3_acl(acl_grants=obj_acl, permitted_users="AllUsers")
object_2 = s3_client.get_object(bucket, file_name)
assert get_file_hash(file_path_2) == get_file_hash(object_2), "Hashes must be the same"
with allure.step("Put object with acl public-read-write"):
file_path_3 = generate_file_with_content(
simple_object_size.value, file_path=file_path_1
)
with reporter.step("Put object with acl public-read-write"):
file_path_3 = generate_file_with_content(simple_object_size.value, file_path=file_path_1)
s3_client.put_object(bucket, file_path_3, acl="public-read-write")
obj_acl = s3_client.get_object_acl(bucket, file_name)
s3_helper.assert_s3_acl(acl_grants=obj_acl, permitted_users="AllUsers")
object_3 = s3_client.get_object(bucket, file_name)
assert get_file_hash(file_path_3) == get_file_hash(object_3), "Hashes must be the same"
with allure.step("Put object with acl authenticated-read"):
file_path_4 = generate_file_with_content(
simple_object_size.value, file_path=file_path_1
)
with reporter.step("Put object with acl authenticated-read"):
file_path_4 = generate_file_with_content(simple_object_size.value, file_path=file_path_1)
s3_client.put_object(bucket, file_path_4, acl="authenticated-read")
obj_acl = s3_client.get_object_acl(bucket, file_name)
s3_helper.assert_s3_acl(acl_grants=obj_acl, permitted_users="AllUsers")
@ -794,7 +680,7 @@ class TestS3GateObject:
file_path_5 = generate_file(complex_object_size.value)
file_name_5 = s3_helper.object_key_from_file_path(file_path_5)
with allure.step("Put object with --grant-full-control id=mycanonicaluserid"):
with reporter.step("Put object with --grant-full-control id=mycanonicaluserid"):
generate_file_with_content(simple_object_size.value, file_path=file_path_5)
s3_client.put_object(
bucket,
@ -806,9 +692,7 @@ class TestS3GateObject:
object_5 = s3_client.get_object(bucket, file_name_5)
assert get_file_hash(file_path_5) == get_file_hash(object_5), "Hashes must be the same"
with allure.step(
"Put object with --grant-read uri=http://acs.amazonaws.com/groups/global/AllUsers"
):
with reporter.step("Put object with --grant-read uri=http://acs.amazonaws.com/groups/global/AllUsers"):
generate_file_with_content(simple_object_size.value, file_path=file_path_5)
s3_client.put_object(
bucket,
@ -833,9 +717,7 @@ class TestS3GateObject:
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=True)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step(
"Put object with lock-mode GOVERNANCE lock-retain-until-date +1day, lock-legal-hold-status"
):
with reporter.step("Put object with lock-mode GOVERNANCE lock-retain-until-date +1day, lock-legal-hold-status"):
date_obj = datetime.utcnow() + timedelta(days=1)
s3_client.put_object(
bucket,
@ -844,11 +726,9 @@ class TestS3GateObject:
object_lock_retain_until_date=date_obj.strftime("%Y-%m-%dT%H:%M:%S"),
object_lock_legal_hold_status="OFF",
)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "GOVERNANCE", date_obj, "OFF"
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "GOVERNANCE", date_obj, "OFF")
with allure.step(
with reporter.step(
"Put new version of object with [--object-lock-mode COMPLIANCE] и [--object-lock-retain-until-date +3days]"
):
date_obj = datetime.utcnow() + timedelta(days=2)
@ -859,11 +739,9 @@ class TestS3GateObject:
object_lock_mode="COMPLIANCE",
object_lock_retain_until_date=date_obj,
)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "COMPLIANCE", date_obj, "OFF"
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "COMPLIANCE", date_obj, "OFF")
with allure.step(
with reporter.step(
"Put new version of object with [--object-lock-mode COMPLIANCE] и [--object-lock-retain-until-date +2days]"
):
date_obj = datetime.utcnow() + timedelta(days=3)
@ -875,11 +753,9 @@ class TestS3GateObject:
object_lock_retain_until_date=date_obj,
object_lock_legal_hold_status="ON",
)
s3_helper.assert_object_lock_mode(
s3_client, bucket, file_name, "COMPLIANCE", date_obj, "ON"
)
s3_helper.assert_object_lock_mode(s3_client, bucket, file_name, "COMPLIANCE", date_obj, "ON")
with allure.step("Put object with lock-mode"):
with reporter.step("Put object with lock-mode"):
with pytest.raises(
Exception,
match=r".*must both be supplied*",
@ -887,7 +763,7 @@ class TestS3GateObject:
# x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied
s3_client.put_object(bucket, file_path_1, object_lock_mode="COMPLIANCE")
with allure.step("Put object with lock-mode and past date"):
with reporter.step("Put object with lock-mode and past date"):
date_obj = datetime.utcnow() - timedelta(days=3)
with pytest.raises(
Exception,
@ -902,6 +778,7 @@ class TestS3GateObject:
)
@allure.title("Sync directory (sync_type={sync_type}, s3_client={s3_client})")
@pytest.mark.parametrize("s3_client", [AwsCliClient], indirect=True)
@pytest.mark.parametrize("sync_type", ["sync", "cp"])
def test_s3_sync_dir(
self,
@ -915,9 +792,6 @@ class TestS3GateObject:
object_metadata = {f"{uuid.uuid4()}": f"{uuid.uuid4()}"}
key_to_path = {"test_file_1": file_path_1, "test_file_2": file_path_2}
if not isinstance(s3_client, AwsCliClient):
pytest.skip("This test is not supported with boto3 client")
generate_file_with_content(simple_object_size.value, file_path=file_path_1)
generate_file_with_content(simple_object_size.value, file_path=file_path_2)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
@ -937,22 +811,18 @@ class TestS3GateObject:
metadata=object_metadata,
)
with allure.step("Check objects are synced"):
with reporter.step("Check objects are synced"):
objects = s3_client.list_objects(bucket)
assert set(key_to_path.keys()) == set(
objects
), f"Expected all abjects saved. Got {objects}"
assert set(key_to_path.keys()) == set(objects), f"Expected all abjects saved. Got {objects}"
with allure.step("Check these are the same objects"):
with reporter.step("Check these are the same objects"):
for obj_key in objects:
got_object = s3_client.get_object(bucket, obj_key)
assert get_file_hash(got_object) == get_file_hash(
key_to_path.get(obj_key)
), "Expected hashes are the same"
obj_head = s3_client.head_object(bucket, obj_key)
assert (
obj_head.get("Metadata") == object_metadata
), f"Metadata of object is {object_metadata}"
assert obj_head.get("Metadata") == object_metadata, f"Metadata of object is {object_metadata}"
# Uncomment after https://github.com/nspcc-dev/neofs-s3-gw/issues/685 is solved
# obj_acl = s3_client.get_object_acl(bucket, obj_key)
# s3_helper.assert_s3_acl(acl_grants = obj_acl, permitted_users = "AllUsers")
@ -972,7 +842,7 @@ class TestS3GateObject:
objects_list = s3_client.list_objects(bucket)
assert not objects_list, f"Expected empty bucket, got {objects_list}"
with allure.step("Put object"):
with reporter.step("Put object"):
s3_client.put_object(bucket, file_path_1)
s3_helper.check_objects_in_bucket(s3_client, bucket, [file_name])
@ -981,12 +851,12 @@ class TestS3GateObject:
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
objects_list = s3_client.list_objects_versions(bucket)
with allure.step("Check that bucket is empty"):
with reporter.step("Check that bucket is empty"):
assert not objects_list, f"Expected empty bucket, got {objects_list}"
obj_key = "fake_object_key"
with allure.step("Delete non-existing object"):
with reporter.step("Delete non-existing object"):
delete_obj = s3_client.delete_object(bucket, obj_key)
# there should be no objects or delete markers in the bucket
assert "DeleteMarker" not in delete_obj.keys(), "Delete markers should not be created"
@ -994,37 +864,31 @@ class TestS3GateObject:
assert not objects_list, f"Expected empty bucket, got {objects_list}"
@allure.title("Delete the same object twice (s3_client={s3_client})")
def test_s3_delete_twice(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_delete_twice(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
objects_list = s3_client.list_objects(bucket)
with allure.step("Check that bucket is empty"):
with reporter.step("Check that bucket is empty"):
assert not objects_list, f"Expected empty bucket, got {objects_list}"
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
with allure.step("Put object into one bucket"):
with reporter.step("Put object into one bucket"):
s3_client.put_object(bucket, file_path)
with allure.step("Delete the object from the bucket"):
with reporter.step("Delete the object from the bucket"):
delete_object = s3_client.delete_object(bucket, file_name)
versions = s3_client.list_objects_versions(bucket)
obj_versions = {
version.get("VersionId") for version in versions if version.get("Key") == file_name
}
obj_versions = {version.get("VersionId") for version in versions if version.get("Key") == file_name}
assert obj_versions, f"Object versions were not found {objects_list}"
assert "DeleteMarker" in delete_object.keys(), "Delete markers not found"
with allure.step("Delete the object from the bucket again"):
with reporter.step("Delete the object from the bucket again"):
delete_object_2nd_attempt = s3_client.delete_object(bucket, file_name)
versions_2nd_attempt = s3_client.list_objects_versions(bucket)
assert (
delete_object.keys() == delete_object_2nd_attempt.keys()
), "Delete markers are not the same"
assert delete_object.keys() == delete_object_2nd_attempt.keys(), "Delete markers are not the same"
# check that nothing was changed
# checking here not VersionId only, but all data (for example LastModified)
assert versions == versions_2nd_attempt, "Versions are not the same"

49
pytest_tests/testsuites/services/s3_gate/test_s3_policy.py
View file

@ -2,7 +2,8 @@ import os
import allure
import pytest
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper, VersioningStatus
from frostfs_testlib import reporter
from frostfs_testlib.s3 import S3ClientWrapper, VersioningStatus
from frostfs_testlib.steps.cli.container import search_container_by_name
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.steps.storage_policy import get_simple_object_copies
@ -12,29 +13,17 @@ from frostfs_testlib.testing.test_control import expect_not_raises
from frostfs_testlib.utils.file_utils import generate_file
def pytest_generate_tests(metafunc: pytest.Metafunc):
policy = f"{os.getcwd()}/pytest_tests/resources/files/policy.json"
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize(
"s3_client, s3_policy",
[(AwsCliClient, policy), (Boto3ClientWrapper, policy)],
indirect=True,
ids=["aws cli", "boto3"],
)
@pytest.mark.s3_gate
@pytest.mark.parametrize("s3_policy", ["pytest_tests/resources/files/policy.json"], indirect=True)
class TestS3GatePolicy(ClusterTestBase):
@allure.title("Bucket creation with retention policy applied (s3_client={s3_client})")
def test_s3_bucket_location(
self, default_wallet: str, s3_client: S3ClientWrapper, simple_object_size: ObjectSize
):
def test_s3_bucket_location(self, default_wallet: str, s3_client: S3ClientWrapper, simple_object_size: ObjectSize):
file_path_1 = generate_file(simple_object_size.value)
file_name_1 = s3_helper.object_key_from_file_path(file_path_1)
file_path_2 = generate_file(simple_object_size.value)
file_name_2 = s3_helper.object_key_from_file_path(file_path_2)
with allure.step("Create two buckets with different bucket configuration"):
with reporter.step("Create two buckets with different bucket configuration"):
bucket_1 = s3_client.create_bucket(location_constraint="complex")
s3_helper.set_bucket_versioning(s3_client, bucket_1, VersioningStatus.ENABLED)
bucket_2 = s3_client.create_bucket(location_constraint="rep-3")
@ -44,24 +33,24 @@ class TestS3GatePolicy(ClusterTestBase):
bucket_1 in list_buckets and bucket_2 in list_buckets
), f"Expected two buckets {bucket_1, bucket_2}, got {list_buckets}"
with allure.step("Check head buckets"):
with reporter.step("Check head buckets"):
with expect_not_raises():
s3_client.head_bucket(bucket_1)
s3_client.head_bucket(bucket_2)
with allure.step("Put objects into buckets"):
with reporter.step("Put objects into buckets"):
version_id_1 = s3_client.put_object(bucket_1, file_path_1)
version_id_2 = s3_client.put_object(bucket_2, file_path_2)
s3_helper.check_objects_in_bucket(s3_client, bucket_1, [file_name_1])
s3_helper.check_objects_in_bucket(s3_client, bucket_2, [file_name_2])
with allure.step("Check bucket location"):
with reporter.step("Check bucket location"):
bucket_loc_1 = s3_client.get_bucket_location(bucket_1)
bucket_loc_2 = s3_client.get_bucket_location(bucket_2)
assert bucket_loc_1 == "complex"
assert bucket_loc_2 == "rep-3"
with allure.step("Check object policy"):
with reporter.step("Check object policy"):
cid_1 = search_container_by_name(
default_wallet,
bucket_1,
@ -93,20 +82,20 @@ class TestS3GatePolicy(ClusterTestBase):
@allure.title("Bucket with unexisting location constraint (s3_client={s3_client})")
def test_s3_bucket_wrong_location(self, s3_client: S3ClientWrapper):
with allure.step("Create bucket with unenxisting location constraint policy"):
with reporter.step("Create bucket with unenxisting location constraint policy"):
with pytest.raises(Exception):
s3_client.create_bucket(location_constraint="UNEXISTING LOCATION CONSTRAINT")
@allure.title("Bucket policy (s3_client={s3_client})")
def test_s3_bucket_policy(self, s3_client: S3ClientWrapper):
with allure.step("Create bucket with default policy"):
with reporter.step("Create bucket with default policy"):
bucket = s3_client.create_bucket()
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("GetBucketPolicy"):
with reporter.step("GetBucketPolicy"):
s3_client.get_bucket_policy(bucket)
with allure.step("Put new policy"):
with reporter.step("Put new policy"):
custom_policy = f"file://{os.getcwd()}/pytest_tests/resources/files/bucket_policy.json"
custom_policy = {
"Version": "2008-10-17",
@ -123,20 +112,20 @@ class TestS3GatePolicy(ClusterTestBase):
}
s3_client.put_bucket_policy(bucket, custom_policy)
with allure.step("GetBucketPolicy"):
with reporter.step("GetBucketPolicy"):
policy_1 = s3_client.get_bucket_policy(bucket)
print(policy_1)
@allure.title("Bucket CORS (s3_client={s3_client})")
def test_s3_cors(self, s3_client: S3ClientWrapper):
with allure.step("Create bucket without cors"):
with reporter.step("Create bucket without cors"):
bucket = s3_client.create_bucket()
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with pytest.raises(Exception):
bucket_cors = s3_client.get_bucket_cors(bucket)
with allure.step("Put bucket cors"):
with reporter.step("Put bucket cors"):
cors = {
"CORSRules": [
{
@ -156,11 +145,9 @@ class TestS3GatePolicy(ClusterTestBase):
}
s3_client.put_bucket_cors(bucket, cors)
bucket_cors = s3_client.get_bucket_cors(bucket)
assert bucket_cors == cors.get(
"CORSRules"
), f"Expected CORSRules must be {cors.get('CORSRules')}"
assert bucket_cors == cors.get("CORSRules"), f"Expected CORSRules must be {cors.get('CORSRules')}"
with allure.step("delete bucket cors"):
with reporter.step("delete bucket cors"):
s3_client.delete_bucket_cors(bucket)
with pytest.raises(Exception):

45
pytest_tests/testsuites/services/s3_gate/test_s3_tagging.py
View file

@ -4,18 +4,13 @@ from typing import Tuple
import allure
import pytest
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper
from frostfs_testlib import reporter
from frostfs_testlib.s3 import S3ClientWrapper
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.utils.file_utils import generate_file
def pytest_generate_tests(metafunc: pytest.Metafunc):
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize("s3_client", [AwsCliClient, Boto3ClientWrapper], indirect=True)
@pytest.mark.sanity
@pytest.mark.s3_gate
@pytest.mark.s3_gate_tagging
class TestS3GateTagging:
@ -29,84 +24,80 @@ class TestS3GateTagging:
return tags
@allure.title("Object tagging (s3_client={s3_client})")
def test_s3_object_tagging(
self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize
):
def test_s3_object_tagging(self, s3_client: S3ClientWrapper, bucket: str, simple_object_size: ObjectSize):
file_path = generate_file(simple_object_size.value)
file_name = s3_helper.object_key_from_file_path(file_path)
with allure.step("Put with 3 tags object into bucket"):
with reporter.step("Put with 3 tags object into bucket"):
tag_1 = "Tag1=Value1"
s3_client.put_object(bucket, file_path, tagging=tag_1)
got_tags = s3_client.get_object_tagging(bucket, file_name)
assert got_tags, f"Expected tags, got {got_tags}"
assert got_tags == [{"Key": "Tag1", "Value": "Value1"}], "Tags must be the same"
with allure.step("Put 10 new tags for object"):
with reporter.step("Put 10 new tags for object"):
tags_2 = self.create_tags(10)
s3_client.put_object_tagging(bucket, file_name, tags=tags_2)
s3_helper.check_tags_by_object(
s3_client, bucket, file_name, tags_2, [("Tag1", "Value1")]
)
s3_helper.check_tags_by_object(s3_client, bucket, file_name, tags_2, [("Tag1", "Value1")])
with allure.step("Put 10 extra new tags for object"):
with reporter.step("Put 10 extra new tags for object"):
tags_3 = self.create_tags(10)
s3_client.put_object_tagging(bucket, file_name, tags=tags_3)
s3_helper.check_tags_by_object(s3_client, bucket, file_name, tags_3, tags_2)
with allure.step("Copy one object with tag"):
with reporter.step("Copy one object with tag"):
copy_obj_path_1 = s3_client.copy_object(bucket, file_name, tagging_directive="COPY")
s3_helper.check_tags_by_object(s3_client, bucket, copy_obj_path_1, tags_3, tags_2)
with allure.step("Put 11 new tags to object and expect an error"):
with reporter.step("Put 11 new tags to object and expect an error"):
tags_4 = self.create_tags(11)
with pytest.raises(Exception, match=r".*Object tags cannot be greater than 10*"):
# An error occurred (BadRequest) when calling the PutObjectTagging operation: Object tags cannot be greater than 10
s3_client.put_object_tagging(bucket, file_name, tags=tags_4)
with allure.step("Put empty tag"):
with reporter.step("Put empty tag"):
tags_5 = []
s3_client.put_object_tagging(bucket, file_name, tags=tags_5)
s3_helper.check_tags_by_object(s3_client, bucket, file_name, [])
with allure.step("Put 10 object tags"):
with reporter.step("Put 10 object tags"):
tags_6 = self.create_tags(10)
s3_client.put_object_tagging(bucket, file_name, tags=tags_6)
s3_helper.check_tags_by_object(s3_client, bucket, file_name, tags_6)
with allure.step("Delete tags by delete-object-tagging"):
with reporter.step("Delete tags by delete-object-tagging"):
s3_client.delete_object_tagging(bucket, file_name)
s3_helper.check_tags_by_object(s3_client, bucket, file_name, [])
@allure.title("Bucket tagging (s3_client={s3_client})")
def test_s3_bucket_tagging(self, s3_client: S3ClientWrapper, bucket: str):
with allure.step("Put 10 bucket tags"):
with reporter.step("Put 10 bucket tags"):
tags_1 = self.create_tags(10)
s3_client.put_bucket_tagging(bucket, tags_1)
s3_helper.check_tags_by_bucket(s3_client, bucket, tags_1)
with allure.step("Put new 10 bucket tags"):
with reporter.step("Put new 10 bucket tags"):
tags_2 = self.create_tags(10)
s3_client.put_bucket_tagging(bucket, tags_2)
s3_helper.check_tags_by_bucket(s3_client, bucket, tags_2, tags_1)
with allure.step("Put 11 new tags to bucket and expect an error"):
with reporter.step("Put 11 new tags to bucket and expect an error"):
tags_3 = self.create_tags(11)
with pytest.raises(Exception, match=r".*Object tags cannot be greater than 10.*"):
# An error occurred (BadRequest) when calling the PutBucketTagging operation (reached max retries: 0): Object tags cannot be greater than 10
s3_client.put_bucket_tagging(bucket, tags_3)
with allure.step("Put empty tag"):
with reporter.step("Put empty tag"):
tags_4 = []
s3_client.put_bucket_tagging(bucket, tags_4)
s3_helper.check_tags_by_bucket(s3_client, bucket, tags_4)
with allure.step("Put new 10 bucket tags"):
with reporter.step("Put new 10 bucket tags"):
tags_5 = self.create_tags(10)
s3_client.put_bucket_tagging(bucket, tags_5)
s3_helper.check_tags_by_bucket(s3_client, bucket, tags_5, tags_2)
with allure.step("Delete tags by delete-bucket-tagging"):
with reporter.step("Delete tags by delete-bucket-tagging"):
s3_client.delete_bucket_tagging(bucket)
s3_helper.check_tags_by_bucket(s3_client, bucket, [])

53
pytest_tests/testsuites/services/s3_gate/test_s3_versioning.py
View file

@ -1,19 +1,12 @@
import os
import allure
import pytest
from frostfs_testlib.s3 import AwsCliClient, Boto3ClientWrapper, S3ClientWrapper, VersioningStatus
from frostfs_testlib import reporter
from frostfs_testlib.s3 import S3ClientWrapper, VersioningStatus
from frostfs_testlib.steps.s3 import s3_helper
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
from frostfs_testlib.utils.file_utils import generate_file, generate_file_with_content
def pytest_generate_tests(metafunc: pytest.Metafunc):
if "s3_client" in metafunc.fixturenames:
metafunc.parametrize("s3_client", [AwsCliClient, Boto3ClientWrapper], indirect=True)
@pytest.mark.sanity
@pytest.mark.s3_gate
@pytest.mark.s3_gate_versioning
class TestS3GateVersioning:
@ -31,21 +24,13 @@ class TestS3GateVersioning:
bucket = s3_client.create_bucket(object_lock_enabled_for_bucket=False)
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.SUSPENDED)
with allure.step("Put object into bucket"):
with reporter.step("Put object into bucket"):
s3_client.put_object(bucket, file_path)
objects_list = s3_client.list_objects(bucket)
assert (
objects_list == bucket_objects
), f"Expected list with single objects in bucket, got {objects_list}"
assert objects_list == bucket_objects, f"Expected list with single objects in bucket, got {objects_list}"
object_version = s3_client.list_objects_versions(bucket)
actual_version = [
version.get("VersionId")
for version in object_version
if version.get("Key") == file_name
]
assert actual_version == [
"null"
], f"Expected version is null in list-object-versions, got {object_version}"
actual_version = [version.get("VersionId") for version in object_version if version.get("Key") == file_name]
assert actual_version == ["null"], f"Expected version is null in list-object-versions, got {object_version}"
object_0 = s3_client.head_object(bucket, file_name)
assert (
object_0.get("VersionId") == "null"
@ -53,34 +38,26 @@ class TestS3GateVersioning:
s3_helper.set_bucket_versioning(s3_client, bucket, VersioningStatus.ENABLED)
with allure.step("Put several versions of object into bucket"):
with reporter.step("Put several versions of object into bucket"):
version_id_1 = s3_client.put_object(bucket, file_path)
file_name_1 = generate_file_with_content(simple_object_size.value, file_path=file_path)
version_id_2 = s3_client.put_object(bucket, file_name_1)
with allure.step("Check bucket shows all versions"):
with reporter.step("Check bucket shows all versions"):
versions = s3_client.list_objects_versions(bucket)
obj_versions = [
version.get("VersionId") for version in versions if version.get("Key") == file_name
]
obj_versions = [version.get("VersionId") for version in versions if version.get("Key") == file_name]
assert (
obj_versions.sort() == [version_id_1, version_id_2, "null"].sort()
), f"Expected object has versions: {version_id_1, version_id_2, 'null'}"
with allure.step("Get object"):
with reporter.step("Get object"):
object_1 = s3_client.get_object(bucket, file_name, full_output=True)
assert (
object_1.get("VersionId") == version_id_2
), f"Get object with version {version_id_2}"
assert object_1.get("VersionId") == version_id_2, f"Get object with version {version_id_2}"
with allure.step("Get first version of object"):
with reporter.step("Get first version of object"):
object_2 = s3_client.get_object(bucket, file_name, version_id_1, full_output=True)
assert (
object_2.get("VersionId") == version_id_1
), f"Get object with version {version_id_1}"
assert object_2.get("VersionId") == version_id_1, f"Get object with version {version_id_1}"
with allure.step("Get second version of object"):
with reporter.step("Get second version of object"):
object_3 = s3_client.get_object(bucket, file_name, version_id_2, full_output=True)
assert (
object_3.get("VersionId") == version_id_2
), f"Get object with version {version_id_2}"
assert object_3.get("VersionId") == version_id_2, f"Get object with version {version_id_2}"

25
pytest_tests/testsuites/services/test_binaries.py
View file

@ -1,11 +1,12 @@
import logging
import os
from http import HTTPStatus
from re import match, fullmatch
from re import fullmatch, match
import allure
import pytest
import requests
from frostfs_testlib import reporter
from frostfs_testlib.hosting import Hosting
from frostfs_testlib.resources.common import ASSETS_DIR
from frostfs_testlib.utils.env_utils import read_env_properties, save_env_properties
@ -14,14 +15,14 @@ from pytest import FixtureRequest
logger = logging.getLogger("NeoLogger")
@allure.title("Check binaries versions")
@pytest.mark.sanity
@pytest.mark.check_binaries
def test_binaries_versions(request: FixtureRequest, hosting: Hosting):
"""
Compare binaries versions from external source (url) and deployed on servers.
"""
with allure.step("Get binaries versions from servers"):
with reporter.step("Get binaries versions from servers"):
got_versions = get_remote_binaries_versions(hosting)
environment_dir = request.config.getoption("--alluredir") or ASSETS_DIR
@ -32,14 +33,16 @@ def test_binaries_versions(request: FixtureRequest, hosting: Hosting):
exeptions = []
additional_env_properties = {}
for binary, version in got_versions.items():
if not fullmatch(r"^\d+\.\d+\.\d+(-.*)?(?<!dirty)", version):
exeptions.append(f"{binary}: Actual version doesn't conform to format '0.0.0-000-aaaaaaa': {version}")
for binary_name, binary in got_versions.items():
version = binary["version"]
requires_check = binary["check"]
if requires_check and not fullmatch(r"^\d+\.\d+\.\d+(-.*)?(?<!dirty)", version):
exeptions.append(f"{binary_name}: Actual version doesn't conform to format '0.0.0-000-aaaaaaa': {version}")
# If some binary was not listed in the env properties file, let's add it
# so that we have full information about versions in allure report
if env_properties and binary not in env_properties:
additional_env_properties[binary] = version
if env_properties and binary_name not in env_properties:
additional_env_properties[binary_name] = version
if env_properties and additional_env_properties:
save_env_properties(env_file, additional_env_properties)
@ -50,15 +53,13 @@ def test_binaries_versions(request: FixtureRequest, hosting: Hosting):
raise AssertionError(f"Found binaries with unexpected versions:\n{msg}")
@allure.step("Download versions info from {url}")
@reporter.step("Download versions info from {url}")
def download_versions_info(url: str) -> dict:
binaries_to_version = {}
response = requests.get(url)
assert (
response.status_code == HTTPStatus.OK
), f"Got {response.status_code} code. Content {response.json()}"
assert response.status_code == HTTPStatus.OK, f"Got {response.status_code} code. Content {response.json()}"
content = response.text
assert content, f"Expected file with content, got {response}"

21
pytest_tests/testsuites/session_token/test_object_session_token.py
View file

@ -2,6 +2,7 @@ import random
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.common import DEFAULT_WALLET_PASS
from frostfs_testlib.resources.error_patterns import SESSION_NOT_FOUND
from frostfs_testlib.steps.cli.container import create_container
@ -31,16 +32,14 @@ class TestDynamicObjectSession(ClusterTestBase):
with a session token
"""
with allure.step("Init wallet"):
with reporter.step("Init wallet"):
wallet = default_wallet
address = wallet_utils.get_last_address_from_wallet(wallet, "")
with allure.step("Nodes Settlements"):
session_token_node, container_node, non_container_node = random.sample(
self.cluster.storage_nodes, 3
)
with reporter.step("Nodes Settlements"):
session_token_node, container_node, non_container_node = random.sample(self.cluster.storage_nodes, 3)
with allure.step("Create Session Token"):
with reporter.step("Create Session Token"):
session_token = create_session_token(
shell=self.shell,
owner=address,
@ -49,7 +48,7 @@ class TestDynamicObjectSession(ClusterTestBase):
rpc_endpoint=session_token_node.get_rpc_endpoint(),
)
with allure.step("Create Private Container"):
with reporter.step("Create Private Container"):
un_locode = container_node.get_un_locode()
locode = "SPB" if un_locode == "RU LED" else un_locode.split()[1]
placement_policy = (
@ -64,7 +63,7 @@ class TestDynamicObjectSession(ClusterTestBase):
rule=placement_policy,
)
with allure.step("Put Objects"):
with reporter.step("Put Objects"):
file_path = generate_file(object_size.value)
oid = put_object_to_random_node(
wallet=wallet,
@ -81,7 +80,7 @@ class TestDynamicObjectSession(ClusterTestBase):
cluster=self.cluster,
)
with allure.step("Node not in container but granted a session token"):
with reporter.step("Node not in container but granted a session token"):
put_object(
wallet=wallet,
path=file_path,
@ -99,7 +98,7 @@ class TestDynamicObjectSession(ClusterTestBase):
session=session_token,
)
with allure.step("Node in container and not granted a session token"):
with reporter.step("Node in container and not granted a session token"):
with pytest.raises(Exception, match=SESSION_NOT_FOUND):
put_object(
wallet=wallet,
@ -119,7 +118,7 @@ class TestDynamicObjectSession(ClusterTestBase):
session=session_token,
)
with allure.step("Node not in container and not granted a session token"):
with reporter.step("Node not in container and not granted a session token"):
with pytest.raises(Exception, match=SESSION_NOT_FOUND):
put_object(
wallet=wallet,

73
pytest_tests/testsuites/session_token/test_static_object_session_token.py
View file

@ -2,6 +2,7 @@ import logging
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.error_patterns import (
EXPIRED_SESSION_TOKEN,
MALFORMED_REQUEST,
@ -48,15 +49,9 @@ RANGE_OFFSET_FOR_COMPLEX_OBJECT = 200
@pytest.fixture(scope="module")
def storage_containers(
owner_wallet: WalletInfo, client_shell: Shell, cluster: Cluster
) -> list[str]:
cid = create_container(
owner_wallet.path, shell=client_shell, endpoint=cluster.default_rpc_endpoint
)
other_cid = create_container(
owner_wallet.path, shell=client_shell, endpoint=cluster.default_rpc_endpoint
)
def storage_containers(owner_wallet: WalletInfo, client_shell: Shell, cluster: Cluster) -> list[str]:
cid = create_container(owner_wallet.path, shell=client_shell, endpoint=cluster.default_rpc_endpoint)
other_cid = create_container(owner_wallet.path, shell=client_shell, endpoint=cluster.default_rpc_endpoint)
yield [cid, other_cid]
@ -75,7 +70,7 @@ def storage_objects(
file_path = generate_file(object_size.value)
storage_objects = []
with allure.step("Put objects"):
with reporter.step("Put objects"):
# upload couple objects
for _ in range(3):
storage_object_id = put_object_to_random_node(
@ -98,10 +93,8 @@ def storage_objects(
delete_objects(storage_objects, client_shell, cluster)
@allure.step("Get ranges for test")
def get_ranges(
storage_object: StorageObjectInfo, max_object_size: int, shell: Shell, endpoint: str
) -> list[str]:
@reporter.step("Get ranges for test")
def get_ranges(storage_object: StorageObjectInfo, max_object_size: int, shell: Shell, endpoint: str) -> list[str]:
"""
Returns ranges to test range/hash methods via static session
"""
@ -112,8 +105,7 @@ def get_ranges(
return [
"0:10",
f"{object_size-10}:10",
f"{max_object_size - RANGE_OFFSET_FOR_COMPLEX_OBJECT}:"
f"{RANGE_OFFSET_FOR_COMPLEX_OBJECT * 2}",
f"{max_object_size - RANGE_OFFSET_FOR_COMPLEX_OBJECT}:" f"{RANGE_OFFSET_FOR_COMPLEX_OBJECT * 2}",
]
else:
return ["0:10", f"{object_size-10}:10"]
@ -148,9 +140,7 @@ def static_sessions(
@pytest.mark.static_session
class TestObjectStaticSession(ClusterTestBase):
@allure.title(
"Read operations with static session (method={method_under_test.__name__}, obj_size={object_size})"
)
@allure.title("Read operations with static session (method={method_under_test.__name__}, obj_size={object_size})")
@pytest.mark.parametrize(
"method_under_test,verb",
[
@ -181,13 +171,12 @@ class TestObjectStaticSession(ClusterTestBase):
session=static_sessions[verb],
)
@allure.title(
"Range operations with static session (method={method_under_test.__name__}, obj_size={object_size})"
)
@allure.title("Range operations with static session (method={method_under_test.__name__}, obj_size={object_size})")
@pytest.mark.parametrize(
"method_under_test,verb",
[(get_range, ObjectVerb.RANGE), (get_range_hash, ObjectVerb.RANGEHASH)],
)
@pytest.mark.sanity
def test_static_session_range(
self,
user_wallet: WalletInfo,
@ -201,12 +190,10 @@ class TestObjectStaticSession(ClusterTestBase):
Validate static session with range operations
"""
storage_object = storage_objects[0]
ranges_to_test = get_ranges(
storage_object, max_object_size, self.shell, self.cluster.default_rpc_endpoint
)
ranges_to_test = get_ranges(storage_object, max_object_size, self.shell, self.cluster.default_rpc_endpoint)
for range_to_test in ranges_to_test:
with allure.step(f"Check range {range_to_test}"):
with reporter.step(f"Check range {range_to_test}"):
with expect_not_raises():
method_under_test(
user_wallet.path,
@ -241,9 +228,7 @@ class TestObjectStaticSession(ClusterTestBase):
)
assert sorted(expected_object_ids) == sorted(actual_object_ids)
@allure.title(
"[NEGATIVE] Static session with object id not in session (obj_size={object_size})"
)
@allure.title("[NEGATIVE] Static session with object id not in session (obj_size={object_size})")
def test_static_session_unrelated_object(
self,
user_wallet: WalletInfo,
@ -307,9 +292,7 @@ class TestObjectStaticSession(ClusterTestBase):
session=static_sessions[ObjectVerb.HEAD],
)
@allure.title(
"[NEGATIVE] Static session with container id not in session (obj_size={object_size})"
)
@allure.title("[NEGATIVE] Static session with container id not in session (obj_size={object_size})")
def test_static_session_unrelated_container(
self,
user_wallet: WalletInfo,
@ -450,7 +433,7 @@ class TestObjectStaticSession(ClusterTestBase):
object_id = storage_objects[0].oid
expiration = Lifetime(epoch + 1, epoch, epoch)
with allure.step("Create session token"):
with reporter.step("Create session token"):
token_expire_at_next_epoch = get_object_signed_token(
owner_wallet,
user_wallet,
@ -462,7 +445,7 @@ class TestObjectStaticSession(ClusterTestBase):
expiration,
)
with allure.step("Object should be available with session token after token creation"):
with reporter.step("Object should be available with session token after token creation"):
with expect_not_raises():
head_object(
user_wallet.path,
@ -473,9 +456,7 @@ class TestObjectStaticSession(ClusterTestBase):
session=token_expire_at_next_epoch,
)
with allure.step(
"Object should be available at last epoch before session token expiration"
):
with reporter.step("Object should be available at last epoch before session token expiration"):
self.tick_epoch()
with expect_not_raises():
head_object(
@ -487,7 +468,7 @@ class TestObjectStaticSession(ClusterTestBase):
session=token_expire_at_next_epoch,
)
with allure.step("Object should NOT be available after session token expiration epoch"):
with reporter.step("Object should NOT be available after session token expiration epoch"):
self.tick_epoch()
with pytest.raises(Exception, match=EXPIRED_SESSION_TOKEN):
head_object(
@ -499,6 +480,7 @@ class TestObjectStaticSession(ClusterTestBase):
session=token_expire_at_next_epoch,
)
@pytest.mark.sanity
@allure.title("Static session which is valid since next epoch (obj_size={object_size})")
def test_static_session_start_at_next(
self,
@ -517,7 +499,7 @@ class TestObjectStaticSession(ClusterTestBase):
object_id = storage_objects[0].oid
expiration = Lifetime(epoch + 2, epoch + 1, epoch)
with allure.step("Create session token"):
with reporter.step("Create session token"):
token_start_at_next_epoch = get_object_signed_token(
owner_wallet,
user_wallet,
@ -529,7 +511,7 @@ class TestObjectStaticSession(ClusterTestBase):
expiration,
)
with allure.step("Object should NOT be available with session token after token creation"):
with reporter.step("Object should NOT be available with session token after token creation"):
with pytest.raises(Exception, match=MALFORMED_REQUEST):
head_object(
user_wallet.path,
@ -540,9 +522,7 @@ class TestObjectStaticSession(ClusterTestBase):
session=token_start_at_next_epoch,
)
with allure.step(
"Object should be available with session token starting from token nbf epoch"
):
with reporter.step("Object should be available with session token starting from token nbf epoch"):
self.tick_epoch()
with expect_not_raises():
head_object(
@ -554,9 +534,7 @@ class TestObjectStaticSession(ClusterTestBase):
session=token_start_at_next_epoch,
)
with allure.step(
"Object should be available at last epoch before session token expiration"
):
with reporter.step("Object should be available at last epoch before session token expiration"):
self.tick_epoch()
with expect_not_raises():
head_object(
@ -568,7 +546,7 @@ class TestObjectStaticSession(ClusterTestBase):
session=token_start_at_next_epoch,
)
with allure.step("Object should NOT be available after session token expiration epoch"):
with reporter.step("Object should NOT be available after session token expiration epoch"):
self.tick_epoch()
with pytest.raises(Exception, match=EXPIRED_SESSION_TOKEN):
head_object(
@ -640,6 +618,7 @@ class TestObjectStaticSession(ClusterTestBase):
session=static_sessions[ObjectVerb.DELETE],
)
@pytest.mark.sanity
@allure.title("Put verb is restricted for static session (obj_size={object_size})")
def test_static_session_put_verb(
self,

37
pytest_tests/testsuites/session_token/test_static_session_token_container.py
View file

@ -1,14 +1,9 @@
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.resources.wellknown_acl import PUBLIC_ACL
from frostfs_testlib.shell import Shell
from frostfs_testlib.steps.acl import create_eacl, set_eacl, wait_for_cache_expired
from frostfs_testlib.steps.cli.container import (
create_container,
delete_container,
get_container,
list_containers,
)
from frostfs_testlib.steps.cli.container import create_container, delete_container, get_container, list_containers
from frostfs_testlib.steps.session_token import ContainerVerb, get_container_signed_token
from frostfs_testlib.storage.dataclasses.acl import EACLAccess, EACLOperation, EACLRole, EACLRule
from frostfs_testlib.storage.dataclasses.object_size import ObjectSize
@ -33,9 +28,7 @@ class TestSessionTokenContainer(ClusterTestBase):
Returns dict with static session token file paths for all verbs with default lifetime
"""
return {
verb: get_container_signed_token(
owner_wallet, user_wallet, verb, client_shell, temp_directory
)
verb: get_container_signed_token(owner_wallet, user_wallet, verb, client_shell, temp_directory)
for verb in ContainerVerb
}
@ -48,7 +41,7 @@ class TestSessionTokenContainer(ClusterTestBase):
"""
Validate static session with create operation
"""
with allure.step("Create container with static session token"):
with reporter.step("Create container with static session token"):
cid = create_container(
user_wallet.path,
session_token=static_sessions[ContainerVerb.CREATE],
@ -65,9 +58,7 @@ class TestSessionTokenContainer(ClusterTestBase):
assert cid not in list_containers(
user_wallet.path, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint
)
assert cid in list_containers(
owner_wallet.path, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint
)
assert cid in list_containers(owner_wallet.path, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint)
def test_static_session_token_container_create_with_other_verb(
self,
@ -77,7 +68,7 @@ class TestSessionTokenContainer(ClusterTestBase):
"""
Validate static session without create operation
"""
with allure.step("Try create container with static session token without PUT rule"):
with reporter.step("Try create container with static session token without PUT rule"):
for verb in [verb for verb in ContainerVerb if verb != ContainerVerb.CREATE]:
with pytest.raises(Exception):
create_container(
@ -96,7 +87,7 @@ class TestSessionTokenContainer(ClusterTestBase):
"""
Validate static session with create operation for other wallet
"""
with allure.step("Try create container with static session token without PUT rule"):
with reporter.step("Try create container with static session token without PUT rule"):
with pytest.raises(Exception):
create_container(
stranger_wallet.path,
@ -115,14 +106,14 @@ class TestSessionTokenContainer(ClusterTestBase):
"""
Validate static session with delete operation
"""
with allure.step("Create container"):
with reporter.step("Create container"):
cid = create_container(
owner_wallet.path,
shell=self.shell,
endpoint=self.cluster.default_rpc_endpoint,
wait_for_creation=False,
)
with allure.step("Delete container with static session token"):
with reporter.step("Delete container with static session token"):
delete_container(
wallet=user_wallet.path,
cid=cid,
@ -136,6 +127,7 @@ class TestSessionTokenContainer(ClusterTestBase):
owner_wallet.path, shell=self.shell, endpoint=self.cluster.default_rpc_endpoint
)
@pytest.mark.sanity
def test_static_session_token_container_set_eacl(
self,
owner_wallet: WalletInfo,
@ -147,7 +139,7 @@ class TestSessionTokenContainer(ClusterTestBase):
"""
Validate static session with set eacl operation
"""
with allure.step("Create container"):
with reporter.step("Create container"):
cid = create_container(
owner_wallet.path,
basic_acl=PUBLIC_ACL,
@ -157,11 +149,8 @@ class TestSessionTokenContainer(ClusterTestBase):
file_path = generate_file(simple_object_size.value)
assert can_put_object(stranger_wallet.path, cid, file_path, self.shell, self.cluster)
with allure.step("Deny all operations for other via eACL"):
eacl_deny = [
EACLRule(access=EACLAccess.DENY, role=EACLRole.OTHERS, operation=op)
for op in EACLOperation
]
with reporter.step("Deny all operations for other via eACL"):
eacl_deny = [EACLRule(access=EACLAccess.DENY, role=EACLRole.OTHERS, operation=op) for op in EACLOperation]
set_eacl(
user_wallet.path,
cid,

13
pytest_tests/testsuites/shard/test_control_shard.py
View file

@ -66,8 +66,7 @@ class Shard:
blobstor_count = Shard._get_blobstor_count_from_section(config_object, shard_id)
blobstors = [
Blobstor.from_config_object(config_object, shard_id, blobstor_id)
for blobstor_id in range(blobstor_count)
Blobstor.from_config_object(config_object, shard_id, blobstor_id) for blobstor_id in range(blobstor_count)
]
write_cache_enabled = config_object.as_bool(f"{var_prefix}_WRITECACHE_ENABLED")
@ -81,15 +80,10 @@ class Shard:
@staticmethod
def from_object(shard):
metabase = shard["metabase"]["path"] if "path" in shard["metabase"] else shard["metabase"]
writecache = (
shard["writecache"]["path"] if "path" in shard["writecache"] else shard["writecache"]
)
writecache = shard["writecache"]["path"] if "path" in shard["writecache"] else shard["writecache"]
return Shard(
blobstor=[
Blobstor(path=blobstor["path"], path_type=blobstor["type"])
for blobstor in shard["blobstor"]
],
blobstor=[Blobstor(path=blobstor["path"], path_type=blobstor["type"]) for blobstor in shard["blobstor"]],
metabase=metabase,
writecache=writecache,
)
@ -111,7 +105,6 @@ def shards_from_env(contents: str) -> list[Shard]:
return [Shard.from_config_object(configObj, shard_id) for shard_id in range(num_shards)]
@pytest.mark.sanity
@pytest.mark.shard
class TestControlShard:
@staticmethod

14
pytest_tests/testsuites/special/test_logs.py
View file

@ -4,6 +4,7 @@ from datetime import datetime
import allure
import pytest
from frostfs_testlib import reporter
from frostfs_testlib.hosting import Host
from frostfs_testlib.testing.cluster_test_base import Cluster
from frostfs_testlib.testing.parallel import parallel
@ -34,10 +35,11 @@ class TestLogs:
logs_dir = os.path.join(temp_directory, "logs")
os.makedirs(logs_dir)
# Using \b here because 'oom' and 'panic' can sometimes be found in OID or CID
issues_regex = r"\bpanic\b|\boom\b|too many|insufficient funds|insufficient amount of gas"
issues_regex = r"\bpanic\b|\boom\b|too many|insufficient funds|insufficient amount of gas|wallet passwd|secret key|access key"
exclude_filter = r"too many requests"
futures = parallel(
self._collect_logs_on_host, cluster.hosts, logs_dir, issues_regex, session_start_time, end_time
self._collect_logs_on_host, cluster.hosts, logs_dir, issues_regex, session_start_time, end_time, exclude_filter
)
hosts_with_problems = [
@ -50,9 +52,9 @@ class TestLogs:
not hosts_with_problems
), f"The following hosts contains contain critical errors in system logs: {', '.join(hosts_with_problems)}"
def _collect_logs_on_host(self, host: Host, logs_dir: str, regex: str, since: datetime, until: datetime):
with allure.step(f"Get logs from {host.config.address}"):
logs = host.get_filtered_logs(regex, since, until)
def _collect_logs_on_host(self, host: Host, logs_dir: str, regex: str, since: datetime, until: datetime, exclude_filter: str):
with reporter.step(f"Get logs from {host.config.address}"):
logs = host.get_filtered_logs(regex, since, until, exclude_filter)
if not logs:
return None
@ -66,4 +68,4 @@ class TestLogs:
# Zip all files and attach to Allure because it is more convenient to download a single
# zip with all logs rather than mess with individual logs files per service or node
logs_zip_file_path = shutil.make_archive(logs_dir, "zip", logs_dir)
allure.attach.file(logs_zip_file_path, name="logs.zip", extension="zip")
reporter.attach(logs_zip_file_path, "logs.zip")