[#5] Remove testlib config file support

In order to make library as flexible as possible we will try to use
configuration methods similar to function `logging.dictConfig` from the
standard library. So, we won't support configuration file
`.neofs-testlib.yaml`, but will allow users to call `configure` method
that will load plugins and initialize library components.

Signed-off-by: Vladimir Domnich <v.domnich@yadro.com>

README.md

@@ -8,44 +8,44 @@ $ pip install neofs-testlib
 ```
 
 ## Configuration
-Library components can be configured explicitly via code or implicitly via configuration file that supports plugin-based extensions.
+Some library components support configuration that allows dynamic loading of extensions via plugins. Configuration of such components is described in this section.
-
-By default testlib uses configuration from file `.neofs-testlib.yaml` that must be located next to the process entry point. Path to the file can be customized via environment variable `NEOFS_TESTLIB_CONFIG`. Config file should have either YAML or JSON format.
 
 ### Reporter Configuration
-Currently only reporter component can be configured. Function `set_reporter` assigns current reporter that should be used in the library:
+Reporter is a singleton component that is used by the library to store test artifacts.
+
+Reporter sends artifacts to handlers that are responsible for actual storing in particular system. By default reporter is initialized without any handlers and won't take any actions to store the artifacts. To add handlers directly via code you can use method `register_handler`:
 
 ```python
-from neofs_testlib.reporter import AllureReporter, set_reporter
+from neofs_testlib.reporter import AllureHandler, get_reporter
 
-reporter = AllureReporter()
+get_reporter().register_handler(AllureHandler())
-set_reporter(reporter)
 ```
 
-Assignment of reporter must happen before any testlib modules were imported. Otherwise, testlib code will bind to default dummy reporter. It is not convenient to call utility functions at specific time, so alternative approach is to set reporter in configuration file. To do that, please, specify name of reporter plugin in configuration parameter `reporter`:
+This registration should happen early at the test session, because any artifacts produced before handler is registered won't be stored anywhere.
-```yaml
-reporter: allure
-```
 
-Testlib provides two built-in reporters: `allure` and `dummy`. However, you can use any custom reporter via [plugins](#plugins).
+Alternative approach for registering handlers is to use method `configure`. It is similar to method [dictConfig](https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig) in a sense that it receives a config structure that describes handlers that should be registered in the reporter. Each handler is defined by it's plugin name; for example, to register the built-in Allure handler, we can use the following config:
+
+```python
+get_reporter().configure({ "handlers": [{"plugin_name": "allure"}] })
+```
 
 ## Plugins
 Testlib uses [entrypoint specification](https://docs.python.org/3/library/importlib.metadata.html) for plugins. Testlib supports the following entrypoint groups for plugins:
- - `neofs.testlib.reporter` - group for reporter plugins. Plugin should be a class that implements interface `neofs_testlib.reporter.interfaces.Reporter`.
+ - `neofs.testlib.reporter` - group for reporter handler plugins. Plugin should be a class that implements interface `neofs_testlib.reporter.interfaces.ReporterHandler`.
 
 ### Example reporter plugin
 In this example we will consider two Python projects:
  - Project "my_neofs_plugins" where we will build a plugin that extends testlib functionality.
  - Project "my_neofs_tests" that uses "neofs_testlib" and "my_neofs_plugins" to build some tests.
 
-Let's say we want to implement some custom reporter that can be used as a plugin for testlib. Pseudo-code of implementation can look like that:
+Let's say we want to implement some custom reporter handler that can be used as a plugin for testlib. Pseudo-code of implementation can look like that:
 ```python
-# my_neofs_plugins/src/x/y/z/custom_reporter.py
+# File my_neofs_plugins/src/foo/bar/custom_handler.py
 from contextlib import AbstractContextManager
-from neofs_testlib.reporter.interfaces import Reporter
+from neofs_testlib.reporter import ReporterHandler
 
 
-class CustomReporter(Reporter):
+class CustomHandler(ReporterHandler):
     def step(self, name: str) -> AbstractContextManager:
         ... some implementation ...
 
@@ -53,20 +53,23 @@ class CustomReporter(Reporter):
         ... some implementation ...
 ```
 
-Then in `pyproject.toml` of "my_neofs_plugins" we should register entrypoint for this plugin. Entrypoint must belong to the group `neofs.testlib.reporter`:
+Then in the file `pyproject.toml` of "my_neofs_plugins" we should register entrypoint for this plugin. Entrypoint must belong to the group `neofs.testlib.reporter`:
 ```yaml
-# my_neofs_plugins/pyproject.toml
+# File my_neofs_plugins/pyproject.toml
 [project.entry-points."neofs.testlib.reporter"]
-my_custom_reporter = "x.y.z.custom_reporter:CustomReporter"
+my_custom_handler = "foo.bar.custom_handler:CustomHandler"
 ```
 
-Finally, to use this reporter in our test project "my_neofs_tests", we should specify its entrypoint name in testlib config:
+Finally, to use this handler in our test project "my_neofs_tests", we should configure reporter with name of the handler plugin:
-```yaml
+
-# my_neofs_tests/pyproject.toml
+```python
-reporter: my_custom_reporter
+# File my_neofs_tests/src/conftest.py
+from neofs_testlib.reporter import get_reporter
+
+get_reporter().configure({ "handlers": [{"plugin_name": "my_custom_handler"}] })
 ```
 
-Detailed information on registering entrypoints can be found at [setuptools docs](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
+Detailed information about registering entrypoints can be found at [setuptools docs](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
 
 ## Library structure
 The library provides the following primary components:

pyproject.toml

@@ -20,7 +20,6 @@ dependencies = [
     "importlib_metadata>=5.0; python_version < '3.10'",
     "paramiko>=2.10.3",
     "pexpect>=4.8.0",
-    "pyyaml>=6.0",
 ]
 requires-python = ">=3.9"
 
@@ -31,8 +30,7 @@ dev = ["black", "bumpver", "isort", "pre-commit"]
 Homepage = "https://github.com/nspcc-dev/neofs-testlib"
 
 [project.entry-points."neofs.testlib.reporter"]
-allure = "neofs_testlib.reporter.allure_reporter:AllureReporter"
+allure = "neofs_testlib.reporter.allure_handler:AllureHandler"
-dummy = "neofs_testlib.reporter.dummy_reporter:DummyReporter"
 
 [tool.isort]
 profile = "black"

requirements.txt

@@ -2,7 +2,6 @@ allure-python-commons==2.9.45
 importlib_metadata==5.0.0
 paramiko==2.10.3
 pexpect==4.8.0
-pyyaml==6.0
 
 # Dev dependencies
 black==22.8.0

src/neofs_testlib/__init__.py

@@ -1,60 +1 @@
-import json
-import os
-import sys
-from typing import Any, Optional
-
-import yaml
-
-from neofs_testlib.reporter import set_reporter
-
-if sys.version_info < (3, 10):
-    from importlib_metadata import entry_points
-else:
-    from importlib.metadata import entry_points
-
-
 __version__ = "0.1.0"
-
-
-def __read_config() -> dict[str, Any]:
-    """
-    Loads configuration of library from default file .neofs-testlib.yaml or from
-    the file configured via environment variable NEOFS_TESTLIB_CONFIG.
-    """
-    file_path = os.getenv("NEOFS_TESTLIB_CONFIG", ".neofs-testlib.yaml")
-    if os.path.exists(file_path):
-        _, extension = os.path.splitext(file_path)
-        if extension == ".yaml":
-            with open(file_path, "r") as file:
-                return yaml.full_load(file)
-        if extension == ".json":
-            with open(file_path, "r") as file:
-                return json.load(file)
-    return {}
-
-
-def __load_plugin(group: str, name: Optional[str]) -> Any:
-    """
-    Loads plugin using entry point specification.
-    """
-    if not name:
-        return None
-    plugins = entry_points(group=group)
-    if name not in plugins.names:
-        return None
-    plugin = plugins[name]
-    return plugin.load()
-
-
-def __init_lib():
-    """
-    Initializes singleton components in the library.
-    """
-    config = __read_config()
-
-    reporter = __load_plugin("neofs.testlib.reporter", config.get("reporter"))
-    if reporter:
-        set_reporter(reporter)
-
-
-__init_lib()

src/neofs_testlib/plugins/__init__.py

@@ -0,0 +1,25 @@
+import sys
+from typing import Any
+
+if sys.version_info < (3, 10):
+    # On Python prior 3.10 we need to use backport of entry points
+    from importlib_metadata import entry_points
+else:
+    from importlib.metadata import entry_points
+
+
+def load_plugin(plugin_group: str, name: str) -> Any:
+    """Loads plugin using entry point specification.
+
+    Args:
+        plugin_group: Name of plugin group that contains the plugin.
+        name: Name of the plugin in the group.
+
+    Returns:
+        Plugin class if the plugin was found; otherwise returns None.
+    """
+    plugins = entry_points(group=plugin_group)
+    if name not in plugins.names:
+        return None
+    plugin = plugins[name]
+    return plugin.load()

src/neofs_testlib/reporter/__init__.py

@@ -1,24 +1,17 @@
-from neofs_testlib.reporter.allure_reporter import AllureReporter
+from neofs_testlib.reporter.allure_handler import AllureHandler
-from neofs_testlib.reporter.dummy_reporter import DummyReporter
+from neofs_testlib.reporter.interfaces import ReporterHandler
-from neofs_testlib.reporter.interfaces import Reporter
+from neofs_testlib.reporter.reporter import Reporter
 
-__reporter = DummyReporter()
+__reporter = Reporter()
 
 
 def get_reporter() -> Reporter:
-    """
+    """Returns reporter that the library should use for storing artifacts.
-    Returns reporter that library should use for storing artifacts.
+
+    Reporter is a singleton instance that can be configured with multiple handlers that store
+    artifacts in various systems. Most common use case is to use single handler.
+
+    Returns:
+        Singleton reporter instance.
     """
     return __reporter
-
-
-def set_reporter(reporter: Reporter) -> None:
-    """
-    Assigns specified reporter for storing test artifacts produced by the library.
-
-    This function must be called before any testlib modules are imported.
-    Recommended way to assign reporter is via configuration file; please, refer to
-    testlib documentation for details.
-    """
-    global __reporter
-    __reporter = reporter

src/neofs_testlib/reporter/allure_handler.py

@@ -6,13 +6,11 @@ from typing import Any
 import allure
 from allure import attachment_type
 
-from neofs_testlib.reporter.interfaces import Reporter
+from neofs_testlib.reporter.interfaces import ReporterHandler
 
 
-class AllureReporter(Reporter):
+class AllureHandler(ReporterHandler):
-    """
+    """Handler that stores test artifacts in Allure report."""
-    Implements storing of test artifacts in Allure report.
-    """
 
     def step(self, name: str) -> AbstractContextManager:
         name = shorten(name, width=70, placeholder="...")
@@ -25,9 +23,9 @@ class AllureReporter(Reporter):
         allure.attach(body, attachment_name, attachment_type)
 
     def _resolve_attachment_type(self, extension: str) -> attachment_type:
-        """
+        """Try to find matching Allure attachment type by extension.
-        Try to find matching Allure attachment type by extension. If no match was found,
+
-        default to TXT format.
+        If no match was found, default to TXT format.
         """
         extension = extension.lower()
         return next(

src/neofs_testlib/reporter/dummy_reporter.py

@@ -1,19 +0,0 @@
-from contextlib import AbstractContextManager, contextmanager
-from typing import Any
-
-from neofs_testlib.reporter.interfaces import Reporter
-
-
-@contextmanager
-def _dummy_step():
-    yield
-
-
-class DummyReporter(Reporter):
-    """Dummy implementation of reporter, does not store artifacts anywhere."""
-
-    def step(self, name: str) -> AbstractContextManager:
-        return _dummy_step()
-
-    def attach(self, content: Any, file_name: str) -> None:
-        pass

src/neofs_testlib/reporter/interfaces.py

@@ -3,8 +3,8 @@ from contextlib import AbstractContextManager
 from typing import Any
 
 
-class Reporter(ABC):
+class ReporterHandler(ABC):
-    """Interface that supports storage of test artifacts in some reporting tool."""
+    """Interface of handler that stores test artifacts in some reporting tool."""
 
     @abstractmethod
     def step(self, name: str) -> AbstractContextManager:

src/neofs_testlib/reporter/reporter.py

@@ -0,0 +1,102 @@
+from contextlib import AbstractContextManager, contextmanager
+from types import TracebackType
+from typing import Any, Optional
+
+from neofs_testlib.plugins import load_plugin
+from neofs_testlib.reporter.interfaces import ReporterHandler
+
+
+@contextmanager
+def _empty_step():
+    yield
+
+
+class Reporter:
+    """Root reporter that sends artifacts to handlers."""
+
+    handlers: list[ReporterHandler]
+
+    def __init__(self) -> None:
+        super().__init__()
+        self.handlers = []
+
+    def register_handler(self, handler: ReporterHandler) -> None:
+        """Register a new handler for the reporter.
+
+        Args:
+            handler: Handler instance to add to the reporter.
+        """
+        self.handlers.append(handler)
+
+    def configure(self, config: dict[str, Any]) -> None:
+        """Configure handlers in the reporter from specified config.
+
+        All existing handlers will be removed from the reporter.
+
+        Args:
+            config: dictionary with reporter configuration.
+        """
+        # Reset current configuration
+        self.handlers = []
+
+        # Setup handlers from the specified config
+        handler_configs = config.get("handlers", [])
+        for handler_config in handler_configs:
+            handler_class = load_plugin("neofs.testlib.reporter", handler_config["plugin_name"])
+            self.register_handler(handler_class())
+
+    def step(self, name: str) -> AbstractContextManager:
+        """Register a new step in test execution.
+
+        Args:
+            name: Name of the step.
+
+        Returns:
+            Step context.
+        """
+        if not self.handlers:
+            return _empty_step()
+
+        step_contexts = [handler.step(name) for handler in self.handlers]
+        return AggregateContextManager(step_contexts)
+
+    def attach(self, content: Any, file_name: str) -> None:
+        """Attach specified content with given file name to the test report.
+
+        Args:
+            content: Content to attach. If content value is not a string, it will be
+                converted to a string.
+            file_name: File name of attachment.
+        """
+        for handler in self.handlers:
+            handler.attach(content, file_name)
+
+
+class AggregateContextManager(AbstractContextManager):
+    """Aggregates multiple context managers in a single context."""
+
+    contexts: list[AbstractContextManager]
+
+    def __init__(self, contexts: list[AbstractContextManager]) -> None:
+        super().__init__()
+        self.contexts = contexts
+
+    def __enter__(self):
+        for context in self.contexts:
+            context.__enter__()
+        return self
+
+    def __exit__(
+        self,
+        exc_type: Optional[type[BaseException]],
+        exc_value: Optional[BaseException],
+        traceback: Optional[TracebackType],
+    ) -> Optional[bool]:
+        suppress_decisions = []
+        for context in self.contexts:
+            suppress_decision = context.__exit__(exc_type, exc_value, traceback)
+            suppress_decisions.append(suppress_decision)
+
+        # If all context agreed to suppress exception, then suppress it;
+        # otherwise return None to reraise
+        return True if all(suppress_decisions) else None

tests/test_reporter.py

@@ -0,0 +1,73 @@
+from contextlib import AbstractContextManager
+from types import TracebackType
+from typing import Optional
+from unittest import TestCase
+from unittest.mock import MagicMock
+
+from neofs_testlib.reporter import Reporter
+
+
+class TestLocalShellInteractive(TestCase):
+    def setUp(self):
+        self.reporter = Reporter()
+
+    def test_handler_step_is_invoked(self):
+        handler = MagicMock()
+        self.reporter.register_handler(handler)
+
+        with self.reporter.step("test_step"):
+            pass
+
+        handler.step.assert_called_once_with("test_step")
+
+    def test_two_handler_steps_are_invoked(self):
+        handler1 = MagicMock()
+        handler2 = MagicMock()
+
+        self.reporter.register_handler(handler1)
+        self.reporter.register_handler(handler2)
+
+        with self.reporter.step("test_step"):
+            pass
+
+        handler1.step.assert_called_once_with("test_step")
+        handler2.step.assert_called_once_with("test_step")
+
+    def test_handlers_can_suppress_exception(self):
+        handler1 = MagicMock()
+        handler1.step = MagicMock(return_value=StubContext(suppress_exception=True))
+        handler2 = MagicMock()
+        handler2.step = MagicMock(return_value=StubContext(suppress_exception=True))
+
+        self.reporter.register_handler(handler1)
+        self.reporter.register_handler(handler2)
+
+        with self.reporter.step("test_step"):
+            raise ValueError("Test exception")
+
+    def test_handler_can_override_exception_suppression(self):
+        handler1 = MagicMock()
+        handler1.step = MagicMock(return_value=StubContext(suppress_exception=True))
+        handler2 = MagicMock()
+        handler2.step = MagicMock(return_value=StubContext(suppress_exception=False))
+
+        self.reporter.register_handler(handler1)
+        self.reporter.register_handler(handler2)
+
+        with self.assertRaises(ValueError):
+            with self.reporter.step("test_step"):
+                raise ValueError("Test exception")
+
+
+class StubContext(AbstractContextManager):
+    def __init__(self, suppress_exception: bool) -> None:
+        super().__init__()
+        self.suppress_exception = suppress_exception
+
+    def __exit__(
+        self,
+        exc_type: Optional[type[BaseException]],
+        exc_value: Optional[BaseException],
+        traceback: Optional[TracebackType],
+    ) -> Optional[bool]:
+        return self.suppress_exception