forked from TrueCloudLab/allure-validator
Kirill Sosnovskikh
d050a94272
- Added ability to pass any parameters to `params` keyword in `pytest.fixture`, except:
1) `params=<callable object>`, but `params=[<callable object>, ...]` is allowed
2) `params=[*<any collection>]`
- Analysis of `pytest_generate_tests` is almost equivalent to regular fixtures, except:
1) `metafunc.parametrize(*<any collection>, **<any dict>)`
2) `metafunc.parametrize("any", <callable object>)`
3) `metafunc.parametrize("any1, any2", [(<callable object>, [*<any collection>])]`
Signed-off-by: Kirill Sosnovskikh <k.sosnovskikh@yadro.com>
115 lines
No EOL
3.4 KiB
Markdown
115 lines
No EOL
3.4 KiB
Markdown
# allure-validator
|
|
|
|
Linter for checking uniqueness and correctness of `allure.title` in the entire test database.
|
|
|
|
## Installation
|
|
|
|
```shell
|
|
pip install git+https://git.frostfs.info/TrueCloudLab/allure-validator.git@master
|
|
```
|
|
|
|
## Usage
|
|
|
|
To start, simply specify the folder with the test database that needs to be checked:
|
|
|
|
```shell
|
|
allure-validator <folder> [OPTIONS]
|
|
```
|
|
|
|
Additionally, `allure-validator` can be added to pre-commit hooks:
|
|
```yaml
|
|
# .pre-commit-config.yaml
|
|
repos:
|
|
- repo: https://git.frostfs.info/TrueCloudLab/allure-validator
|
|
rev: latest
|
|
hooks:
|
|
- id: allure-validator
|
|
args: ["pytest_tests/"]
|
|
pass_filenames: false
|
|
```
|
|
|
|
### Ignore mechanism
|
|
|
|
The linter can be made to ignore functions/classes. To do this, just write a comment `# noqa: allure-validator` to the right of the function/class name. For example:
|
|
|
|
```python
|
|
@pytest.mark.parametrize("test_param", [1, 2])
|
|
def test_ignored_function(test_param): # noqa: allure-validator
|
|
...
|
|
|
|
|
|
@pytest.mark.parametrize("class_param", [1, 2])
|
|
class TestIngoredClass: # noqa: allure-validator
|
|
def test_1(self, class_param):
|
|
...
|
|
|
|
def test_2(self, class_param):
|
|
...
|
|
```
|
|
|
|
> Note: this may also miss fixtures, so use with caution.
|
|
|
|
### Options
|
|
|
|
The linter supports the following flags:
|
|
- `--plugins` - regex patterns that plugins must match (paths are compiled according to the template: `plugin_name / src / plugin_name`).
|
|
- `--files` - paths to files with fixtures.
|
|
|
|
Configuration options can also be provided using special command line arguments, for example:
|
|
```shell
|
|
allure-validator <tests folder> --plugins <plugin regex>, ... --files <path to file with fixtures>, ...
|
|
```
|
|
|
|
## Work example
|
|
|
|
Let's run the linter for some [sample tests](https://git.frostfs.info/Kiriruso/allure-validator/src/branch/master/tests):
|
|
|
|
```shell
|
|
NOT UNIQUE TITLE: Not unique test
|
|
In the following tests:
|
|
tests/test_validator.py:20:0: title is not unique by `test_not_unique_2`
|
|
tests/test_validator.py:119:4: title is not unique by `test_not_unique_3`
|
|
tests/test_validator.py:15:0: title is not unique by `test_not_unique_1`
|
|
|
|
NOT UNIQUE TITLE: Test title
|
|
In the following tests:
|
|
tests/test_validator.py:30:0: title is not unique by `test_title_not_unique_2`
|
|
tests/test_validator.py:25:0: title is not unique by `test_title_not_unique_1`
|
|
|
|
tests/test_validator.py:5:0: EMPTY TITLE: Missing title in `test_empty_title`
|
|
tests/test_validator.py:10:0: EMPTY TITLE: Missing title in `test_empty_title_2`
|
|
tests/test_validator.py:20:0: MISSING PARAMS: Parameters are missing from title: object_size
|
|
tests/test_validator.py:62:0: MISSING PARAMS: Parameters are missing from title: param_two OR param_three
|
|
tests/test_validator.py:115:4: MISSING PARAMS: Parameters are missing from title: root_fixture
|
|
tests/test_validator.py:119:4: MISSING PARAMS: Parameters are missing from title: param_class_1
|
|
```
|
|
|
|
## Undetectable cases
|
|
|
|
Since the linter works with AST python code, any dynamic titles will not be found. For example:
|
|
|
|
```python
|
|
import allure
|
|
|
|
|
|
def test_example():
|
|
data = 10
|
|
allure.dynamic.title(f"Dynamic title {data}") # Undetected title
|
|
...
|
|
|
|
```
|
|
|
|
Dynamic fixtures are only supported if they are declared in `pytest_generate_tests`:
|
|
|
|
```python
|
|
def pytest_generate_tests(metafunc):
|
|
...
|
|
|
|
metafunc.parametrize("fixture1, fixture2", [(1, 2), (3, 4)]) # OK
|
|
|
|
...
|
|
|
|
if some_condition:
|
|
metafunc.parametrize("fixture3", *some_value_generator) # OK
|
|
|
|
``` |