Kirill Sosnovskikh
cd41eb5d03
The following flags have been added: - `--plugins` - patterns that plugins must match. - `--files` - paths to files with fixtures. Signed-off-by: Kirill Sosnovskikh <k.sosnovskikh@yadro.com>
115 lines
No EOL
3.4 KiB
Markdown
115 lines
No EOL
3.4 KiB
Markdown
# allure-validator
|
|
|
|
Linter for checking uniqueness and correctness of `allure.title` in the entire test database.
|
|
|
|
## Installation
|
|
|
|
```shell
|
|
pip install git+https://git.frostfs.info/TrueCloudLab/allure-validator.git@master
|
|
```
|
|
|
|
## Usage
|
|
|
|
To start, simply specify the folder with the test database that needs to be checked:
|
|
|
|
```shell
|
|
allure-validator <folder> [OPTIONS]
|
|
```
|
|
|
|
Additionally, `allure-validator` can be added to pre-commit hooks:
|
|
```yaml
|
|
# .pre-commit-config.yaml
|
|
repos:
|
|
- repo: https://git.frostfs.info/TrueCloudLab/allure-validator
|
|
rev: latest
|
|
hooks:
|
|
- id: allure-validator
|
|
args: ["pytest_tests/"]
|
|
pass_filenames: false
|
|
```
|
|
|
|
### Ignore mechanism
|
|
|
|
The linter can be made to ignore functions/classes. To do this, just write a comment `# noqa: allure-validator` to the right of the function/class name. For example:
|
|
|
|
```python
|
|
@pytest.mark.parametrize("test_param", [1, 2])
|
|
def test_ignored_function(test_param): # noqa: allure-validator
|
|
...
|
|
|
|
|
|
@pytest.mark.parametrize("class_param", [1, 2])
|
|
class TestIngoredClass: # noqa: allure-validator
|
|
def test_1(self, class_param):
|
|
...
|
|
|
|
def test_2(self, class_param):
|
|
...
|
|
```
|
|
|
|
> Note: this may also miss fixtures, so use with caution.
|
|
|
|
### Options
|
|
|
|
The linter supports the following flags:
|
|
- `--plugins` - regex patterns that plugins must match (paths are compiled according to the template: `plugin / src / plugin`).
|
|
- `--files` - paths to files with fixtures.
|
|
|
|
Configuration options can also be provided using special command line arguments, for example:
|
|
```shell
|
|
allure-validator pytest_tests/ --plugins some[-_]plugin --files etc/workspace/some_plugin/src/some_plugin/fixtures.py
|
|
```
|
|
|
|
## Work example
|
|
|
|
Let's run the linter for some [sample tests](https://git.frostfs.info/Kiriruso/allure-validator/src/branch/master/tests):
|
|
|
|
```shell
|
|
NOT UNIQUE TITLE: Not unique test
|
|
In the following tests:
|
|
tests/test_validator.py:20:0: title is not unique by `test_not_unique_2`
|
|
tests/test_validator.py:119:4: title is not unique by `test_not_unique_3`
|
|
tests/test_validator.py:15:0: title is not unique by `test_not_unique_1`
|
|
|
|
NOT UNIQUE TITLE: Test title
|
|
In the following tests:
|
|
tests/test_validator.py:30:0: title is not unique by `test_title_not_unique_2`
|
|
tests/test_validator.py:25:0: title is not unique by `test_title_not_unique_1`
|
|
|
|
tests/test_validator.py:5:0: EMPTY TITLE: Missing title in `test_empty_title`
|
|
tests/test_validator.py:10:0: EMPTY TITLE: Missing title in `test_empty_title_2`
|
|
tests/test_validator.py:20:0: MISSING PARAMS: Parameters are missing from title: object_size
|
|
tests/test_validator.py:62:0: MISSING PARAMS: Parameters are missing from title: param_two OR param_three
|
|
tests/test_validator.py:115:4: MISSING PARAMS: Parameters are missing from title: root_fixture
|
|
tests/test_validator.py:119:4: MISSING PARAMS: Parameters are missing from title: param_class_1
|
|
```
|
|
|
|
## Undetectable cases
|
|
|
|
Since the linter works with AST python code, any dynamic titles will not be found. For example:
|
|
|
|
```python
|
|
import allure
|
|
|
|
|
|
def test_example():
|
|
data = 10
|
|
allure.dynamic.title(f"Dynamic title {data}") # Undetected title
|
|
...
|
|
|
|
```
|
|
|
|
Dynamic fixtures are only supported if they are declared in `pytest_generate_tests`:
|
|
|
|
```python
|
|
def pytest_generate_tests(metafunc):
|
|
...
|
|
|
|
metafunc.parametrize("fixture1, fixture2", [(1, 2), (3, 4)]) # OK
|
|
|
|
...
|
|
|
|
if some_condition:
|
|
metafunc.parametrize("fixture3", *some_value_generator) # OK
|
|
|
|
``` |