amaks/frostfs-testlib
1
0
Fork 0
forked from TrueCloudLab/frostfs-testlib
Code Pull requests Activity

[#5] Remove testlib config file support

Browse source 
In order to make library as flexible as possible we will try to use
configuration methods similar to function `logging.dictConfig` from the
standard library. So, we won't support configuration file
`.neofs-testlib.yaml`, but will allow users to call `configure` method
that will load plugins and initialize library components.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>
This commit is contained in:
Vladimir Domnich 2022-10-06 14:17:19 +04:00 committed by Vladimir
parent c5ff64b3fd
commit 834ddede36
11 changed files with 248 additions and 135 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

53
README.md
View file

@ -8,44 +8,44 @@ $ pip install neofs-testlib
```
## Configuration
Library components can be configured explicitly via code or implicitly via configuration file that supports plugin-based extensions.
By default testlib uses configuration from file `.neofs-testlib.yaml` that must be located next to the process entry point. Path to the file can be customized via environment variable `NEOFS_TESTLIB_CONFIG`. Config file should have either YAML or JSON format.
Some library components support configuration that allows dynamic loading of extensions via plugins. Configuration of such components is described in this section.
### Reporter Configuration
Currently only reporter component can be configured. Function `set_reporter` assigns current reporter that should be used in the library:
Reporter is a singleton component that is used by the library to store test artifacts.
Reporter sends artifacts to handlers that are responsible for actual storing in particular system. By default reporter is initialized without any handlers and won't take any actions to store the artifacts. To add handlers directly via code you can use method `register_handler`:
```python
from neofs_testlib.reporter import AllureReporter, set_reporter
from neofs_testlib.reporter import AllureHandler, get_reporter
reporter = AllureReporter()
set_reporter(reporter)
get_reporter().register_handler(AllureHandler())
```
Assignment of reporter must happen before any testlib modules were imported. Otherwise, testlib code will bind to default dummy reporter. It is not convenient to call utility functions at specific time, so alternative approach is to set reporter in configuration file. To do that, please, specify name of reporter plugin in configuration parameter `reporter`:
```yaml
reporter: allure
```
This registration should happen early at the test session, because any artifacts produced before handler is registered won't be stored anywhere.
Testlib provides two built-in reporters: `allure` and `dummy`. However, you can use any custom reporter via [plugins](#plugins).
Alternative approach for registering handlers is to use method `configure`. It is similar to method [dictConfig](https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig) in a sense that it receives a config structure that describes handlers that should be registered in the reporter. Each handler is defined by it's plugin name; for example, to register the built-in Allure handler, we can use the following config:
```python
get_reporter().configure({ "handlers": [{"plugin_name": "allure"}] })
```
## Plugins
Testlib uses [entrypoint specification](https://docs.python.org/3/library/importlib.metadata.html) for plugins. Testlib supports the following entrypoint groups for plugins:
- `neofs.testlib.reporter` - group for reporter plugins. Plugin should be a class that implements interface `neofs_testlib.reporter.interfaces.Reporter`.
- `neofs.testlib.reporter` - group for reporter handler plugins. Plugin should be a class that implements interface `neofs_testlib.reporter.interfaces.ReporterHandler`.
### Example reporter plugin
In this example we will consider two Python projects:
- Project "my_neofs_plugins" where we will build a plugin that extends testlib functionality.
- Project "my_neofs_tests" that uses "neofs_testlib" and "my_neofs_plugins" to build some tests.
Let's say we want to implement some custom reporter that can be used as a plugin for testlib. Pseudo-code of implementation can look like that:
Let's say we want to implement some custom reporter handler that can be used as a plugin for testlib. Pseudo-code of implementation can look like that:
```python
# my_neofs_plugins/src/x/y/z/custom_reporter.py
# File my_neofs_plugins/src/foo/bar/custom_handler.py
from contextlib import AbstractContextManager
from neofs_testlib.reporter.interfaces import Reporter
from neofs_testlib.reporter import ReporterHandler
class CustomReporter(Reporter):
class CustomHandler(ReporterHandler):
def step(self, name: str) -> AbstractContextManager:
... some implementation ...
@ -53,20 +53,23 @@ class CustomReporter(Reporter):
... some implementation ...
```
Then in `pyproject.toml` of "my_neofs_plugins" we should register entrypoint for this plugin. Entrypoint must belong to the group `neofs.testlib.reporter`:
Then in the file `pyproject.toml` of "my_neofs_plugins" we should register entrypoint for this plugin. Entrypoint must belong to the group `neofs.testlib.reporter`:
```yaml
# my_neofs_plugins/pyproject.toml
# File my_neofs_plugins/pyproject.toml
[project.entry-points."neofs.testlib.reporter"]
my_custom_reporter = "x.y.z.custom_reporter:CustomReporter"
my_custom_handler = "foo.bar.custom_handler:CustomHandler"
```
Finally, to use this reporter in our test project "my_neofs_tests", we should specify its entrypoint name in testlib config:
```yaml
# my_neofs_tests/pyproject.toml
reporter: my_custom_reporter
Finally, to use this handler in our test project "my_neofs_tests", we should configure reporter with name of the handler plugin:
```python
# File my_neofs_tests/src/conftest.py
from neofs_testlib.reporter import get_reporter
get_reporter().configure({ "handlers": [{"plugin_name": "my_custom_handler"}] })
```
Detailed information on registering entrypoints can be found at [setuptools docs](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
Detailed information about registering entrypoints can be found at [setuptools docs](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
## Library structure
The library provides the following primary components:

4
pyproject.toml
View file

@ -20,7 +20,6 @@ dependencies = [
"importlib_metadata>=5.0; python_version < '3.10'",
"paramiko>=2.10.3",
"pexpect>=4.8.0",
"pyyaml>=6.0",
]
requires-python = ">=3.9"
@ -31,8 +30,7 @@ dev = ["black", "bumpver", "isort", "pre-commit"]
Homepage = "https://github.com/nspcc-dev/neofs-testlib"
[project.entry-points."neofs.testlib.reporter"]
allure = "neofs_testlib.reporter.allure_reporter:AllureReporter"
dummy = "neofs_testlib.reporter.dummy_reporter:DummyReporter"
allure = "neofs_testlib.reporter.allure_handler:AllureHandler"
[tool.isort]
profile = "black"

1
requirements.txt
View file

@ -2,7 +2,6 @@ allure-python-commons==2.9.45
importlib_metadata==5.0.0
paramiko==2.10.3
pexpect==4.8.0
pyyaml==6.0
# Dev dependencies
black==22.8.0

59
src/neofs_testlib/__init__.py
View file

@ -1,60 +1 @@
import json
import os
import sys
from typing import Any, Optional
import yaml
from neofs_testlib.reporter import set_reporter
if sys.version_info < (3, 10):
from importlib_metadata import entry_points
else:
from importlib.metadata import entry_points
__version__ = "0.1.0"
def __read_config() -> dict[str, Any]:
"""
Loads configuration of library from default file .neofs-testlib.yaml or from
the file configured via environment variable NEOFS_TESTLIB_CONFIG.
"""
file_path = os.getenv("NEOFS_TESTLIB_CONFIG", ".neofs-testlib.yaml")
if os.path.exists(file_path):
_, extension = os.path.splitext(file_path)
if extension == ".yaml":
with open(file_path, "r") as file:
return yaml.full_load(file)
if extension == ".json":
with open(file_path, "r") as file:
return json.load(file)
return {}
def __load_plugin(group: str, name: Optional[str]) -> Any:
"""
Loads plugin using entry point specification.
"""
if not name:
return None
plugins = entry_points(group=group)
if name not in plugins.names:
return None
plugin = plugins[name]
return plugin.load()
def __init_lib():
"""
Initializes singleton components in the library.
"""
config = __read_config()
reporter = __load_plugin("neofs.testlib.reporter", config.get("reporter"))
if reporter:
set_reporter(reporter)
__init_lib()

25
src/neofs_testlib/plugins/__init__.py Normal file
View file

@ -0,0 +1,25 @@
import sys
from typing import Any
if sys.version_info < (3, 10):
# On Python prior 3.10 we need to use backport of entry points
from importlib_metadata import entry_points
else:
from importlib.metadata import entry_points
def load_plugin(plugin_group: str, name: str) -> Any:
"""Loads plugin using entry point specification.
Args:
plugin_group: Name of plugin group that contains the plugin.
name: Name of the plugin in the group.
Returns:
Plugin class if the plugin was found; otherwise returns None.
"""
plugins = entry_points(group=plugin_group)
if name not in plugins.names:
return None
plugin = plugins[name]
return plugin.load()

29
src/neofs_testlib/reporter/__init__.py
View file

@ -1,24 +1,17 @@
from neofs_testlib.reporter.allure_reporter import AllureReporter
from neofs_testlib.reporter.dummy_reporter import DummyReporter
from neofs_testlib.reporter.interfaces import Reporter
from neofs_testlib.reporter.allure_handler import AllureHandler
from neofs_testlib.reporter.interfaces import ReporterHandler
from neofs_testlib.reporter.reporter import Reporter
__reporter = DummyReporter()
__reporter = Reporter()
def get_reporter() -> Reporter:
"""
Returns reporter that library should use for storing artifacts.
"""Returns reporter that the library should use for storing artifacts.
Reporter is a singleton instance that can be configured with multiple handlers that store
artifacts in various systems. Most common use case is to use single handler.
Returns:
Singleton reporter instance.
"""
return __reporter
def set_reporter(reporter: Reporter) -> None:
"""
Assigns specified reporter for storing test artifacts produced by the library.
This function must be called before any testlib modules are imported.
Recommended way to assign reporter is via configuration file; please, refer to
testlib documentation for details.
"""
global __reporter
__reporter = reporter

14
src/neofs_testlib/reporter/allure_reporter.py → src/neofs_testlib/reporter/allure_handler.py
View file

@ -6,13 +6,11 @@ from typing import Any
import allure
from allure import attachment_type
from neofs_testlib.reporter.interfaces import Reporter
from neofs_testlib.reporter.interfaces import ReporterHandler
class AllureReporter(Reporter):
"""
Implements storing of test artifacts in Allure report.
"""
class AllureHandler(ReporterHandler):
"""Handler that stores test artifacts in Allure report."""
def step(self, name: str) -> AbstractContextManager:
name = shorten(name, width=70, placeholder="...")
@ -25,9 +23,9 @@ class AllureReporter(Reporter):
allure.attach(body, attachment_name, attachment_type)
def _resolve_attachment_type(self, extension: str) -> attachment_type:
"""
Try to find matching Allure attachment type by extension. If no match was found,
default to TXT format.
"""Try to find matching Allure attachment type by extension.
If no match was found, default to TXT format.
"""
extension = extension.lower()
return next(

19
src/neofs_testlib/reporter/dummy_reporter.py
View file

@ -1,19 +0,0 @@
from contextlib import AbstractContextManager, contextmanager
from typing import Any
from neofs_testlib.reporter.interfaces import Reporter
@contextmanager
def _dummy_step():
yield
class DummyReporter(Reporter):
"""Dummy implementation of reporter, does not store artifacts anywhere."""
def step(self, name: str) -> AbstractContextManager:
return _dummy_step()
def attach(self, content: Any, file_name: str) -> None:
pass

4
src/neofs_testlib/reporter/interfaces.py
View file

@ -3,8 +3,8 @@ from contextlib import AbstractContextManager
from typing import Any
class Reporter(ABC):
"""Interface that supports storage of test artifacts in some reporting tool."""
class ReporterHandler(ABC):
"""Interface of handler that stores test artifacts in some reporting tool."""
@abstractmethod
def step(self, name: str) -> AbstractContextManager:

102
src/neofs_testlib/reporter/reporter.py Normal file
View file

@ -0,0 +1,102 @@
from contextlib import AbstractContextManager, contextmanager
from types import TracebackType
from typing import Any, Optional
from neofs_testlib.plugins import load_plugin
from neofs_testlib.reporter.interfaces import ReporterHandler
@contextmanager
def _empty_step():
yield
class Reporter:
"""Root reporter that sends artifacts to handlers."""
handlers: list[ReporterHandler]
def __init__(self) -> None:
super().__init__()
self.handlers = []
def register_handler(self, handler: ReporterHandler) -> None:
"""Register a new handler for the reporter.
Args:
handler: Handler instance to add to the reporter.
"""
self.handlers.append(handler)
def configure(self, config: dict[str, Any]) -> None:
"""Configure handlers in the reporter from specified config.
All existing handlers will be removed from the reporter.
Args:
config: dictionary with reporter configuration.
"""
# Reset current configuration
self.handlers = []
# Setup handlers from the specified config
handler_configs = config.get("handlers", [])
for handler_config in handler_configs:
handler_class = load_plugin("neofs.testlib.reporter", handler_config["plugin_name"])
self.register_handler(handler_class())
def step(self, name: str) -> AbstractContextManager:
"""Register a new step in test execution.
Args:
name: Name of the step.
Returns:
Step context.
"""
if not self.handlers:
return _empty_step()
step_contexts = [handler.step(name) for handler in self.handlers]
return AggregateContextManager(step_contexts)
def attach(self, content: Any, file_name: str) -> None:
"""Attach specified content with given file name to the test report.
Args:
content: Content to attach. If content value is not a string, it will be
converted to a string.
file_name: File name of attachment.
"""
for handler in self.handlers:
handler.attach(content, file_name)
class AggregateContextManager(AbstractContextManager):
"""Aggregates multiple context managers in a single context."""
contexts: list[AbstractContextManager]
def __init__(self, contexts: list[AbstractContextManager]) -> None:
super().__init__()
self.contexts = contexts
def __enter__(self):
for context in self.contexts:
context.__enter__()
return self
def __exit__(
self,
exc_type: Optional[type[BaseException]],
exc_value: Optional[BaseException],
traceback: Optional[TracebackType],
) -> Optional[bool]:
suppress_decisions = []
for context in self.contexts:
suppress_decision = context.__exit__(exc_type, exc_value, traceback)
suppress_decisions.append(suppress_decision)
# If all context agreed to suppress exception, then suppress it;
# otherwise return None to reraise
return True if all(suppress_decisions) else None

73
tests/test_reporter.py Normal file
View file

@ -0,0 +1,73 @@
from contextlib import AbstractContextManager
from types import TracebackType
from typing import Optional
from unittest import TestCase
from unittest.mock import MagicMock
from neofs_testlib.reporter import Reporter
class TestLocalShellInteractive(TestCase):
def setUp(self):
self.reporter = Reporter()
def test_handler_step_is_invoked(self):
handler = MagicMock()
self.reporter.register_handler(handler)
with self.reporter.step("test_step"):
pass
handler.step.assert_called_once_with("test_step")
def test_two_handler_steps_are_invoked(self):
handler1 = MagicMock()
handler2 = MagicMock()
self.reporter.register_handler(handler1)
self.reporter.register_handler(handler2)
with self.reporter.step("test_step"):
pass
handler1.step.assert_called_once_with("test_step")
handler2.step.assert_called_once_with("test_step")
def test_handlers_can_suppress_exception(self):
handler1 = MagicMock()
handler1.step = MagicMock(return_value=StubContext(suppress_exception=True))
handler2 = MagicMock()
handler2.step = MagicMock(return_value=StubContext(suppress_exception=True))
self.reporter.register_handler(handler1)
self.reporter.register_handler(handler2)
with self.reporter.step("test_step"):
raise ValueError("Test exception")
def test_handler_can_override_exception_suppression(self):
handler1 = MagicMock()
handler1.step = MagicMock(return_value=StubContext(suppress_exception=True))
handler2 = MagicMock()
handler2.step = MagicMock(return_value=StubContext(suppress_exception=False))
self.reporter.register_handler(handler1)
self.reporter.register_handler(handler2)
with self.assertRaises(ValueError):
with self.reporter.step("test_step"):
raise ValueError("Test exception")
class StubContext(AbstractContextManager):
def __init__(self, suppress_exception: bool) -> None:
super().__init__()
self.suppress_exception = suppress_exception
def __exit__(
self,
exc_type: Optional[type[BaseException]],
exc_value: Optional[BaseException],
traceback: Optional[TracebackType],
) -> Optional[bool]:
return self.suppress_exception