forked from TrueCloudLab/allure-validator
104 lines
3 KiB
Markdown
104 lines
3 KiB
Markdown
|
# allure-validator
|
||
|
|
|
||
|
|
Linter for checking uniqueness and correctness of `allure.title` in the entire test database.
|
||
|
|
|
||
|
|
## Installation
|
||
|
|
|
||
|
|
```shell
|
||
|
|
pip install git+https://git.frostfs.info/TrueCloudLab/allure-validator.git@master
|
||
|
|
```
|
||
|
|
|
||
|
|
## Usage
|
||
|
|
|
||
|
|
To start, simply specify the folder with the test database that needs to be checked:
|
||
|
|
|
||
|
|
```shell
|
||
|
|
allure-validator <folder>
|
||
|
|
```
|
||
|
|
|
||
|
|
Additionally, `allure-validator` can be added to pre-commit hooks:
|
||
|
|
```yaml
|
||
|
|
# .pre-commit-config.yaml
|
||
|
|
repos:
|
||
|
|
- repo: local
|
||
|
|
hooks:
|
||
|
|
- id: allure-validator
|
||
|
|
name: allure-validator
|
||
|
|
entry: allure-validator
|
||
|
|
language: system
|
||
|
|
args: ["pytest_tests/"] # folder with tests
|
||
|
|
pass_filenames: false
|
||
|
|
types: [python]
|
||
|
|
```
|
||
|
|
|
||
|
|
### Ignore mechanism
|
||
|
|
|
||
|
|
The linter can be made to ignore functions/classes. To do this, just write a comment `# noqa: allure-validator` to the right of the function/class name. For example:
|
||
|
|
|
||
|
|
```python
|
||
|
|
@pytest.mark.parametrize("test_param", [1, 2])
|
||
|
|
def test_ignored_function(test_param): # noqa: allure-validator
|
||
|
|
...
|
||
|
|
|
||
|
|
|
||
|
|
@pytest.mark.parametrize("class_param", [1, 2])
|
||
|
|
class TestIngoredClass: # noqa: allure-validator
|
||
|
|
def test_1(self, class_param):
|
||
|
|
...
|
||
|
|
|
||
|
|
def test_2(self, class_param):
|
||
|
|
...
|
||
|
|
```
|
||
|
|
|
||
|
|
> Note: this may also miss fixtures, so use with caution.
|
||
|
|
|
||
|
|
## Work example
|
||
|
|
|
||
|
|
Let's run the linter for some [sample tests](https://git.frostfs.info/Kiriruso/allure-validator/src/branch/master/tests):
|
||
|
|
|
||
|
|
```shell
|
||
|
|
NOT UNIQUE TITLE: Not unique test
|
||
|
|
In the following tests:
|
||
|
|
tests/test_validator.py:20:0: title is not unique by `test_not_unique_2`
|
||
|
|
tests/test_validator.py:119:4: title is not unique by `test_not_unique_3`
|
||
|
|
tests/test_validator.py:15:0: title is not unique by `test_not_unique_1`
|
||
|
|
|
||
|
|
NOT UNIQUE TITLE: Test title
|
||
|
|
In the following tests:
|
||
|
|
tests/test_validator.py:30:0: title is not unique by `test_title_not_unique_2`
|
||
|
|
tests/test_validator.py:25:0: title is not unique by `test_title_not_unique_1`
|
||
|
|
|
||
|
|
tests/test_validator.py:5:0: EMPTY TITLE: Missing title in `test_empty_title`
|
||
|
|
tests/test_validator.py:10:0: EMPTY TITLE: Missing title in `test_empty_title_2`
|
||
|
|
tests/test_validator.py:20:0: MISSING PARAMS: Parameters are missing from title: object_size
|
||
|
|
tests/test_validator.py:62:0: MISSING PARAMS: Parameters are missing from title: param_two OR param_three
|
||
|
|
tests/test_validator.py:115:4: MISSING PARAMS: Parameters are missing from title: root_fixture
|
||
|
|
tests/test_validator.py:119:4: MISSING PARAMS: Parameters are missing from title: param_class_1
|
||
|
|
```
|
||
|
|
|
||
|
|
## Undetectable cases
|
||
|
|
|
||
|
|
Since the linter works with AST python code, any dynamic titles will not be found. For example:
|
||
|
|
|
||
|
|
```python
|
||
|
|
import allure
|
||
|
|
|
||
|
|
|
||
|
|
def test_example():
|
||
|
|
data = 10
|
||
|
|
allure.dynamic.title(f"Dynamic title {data}") # Undetected title
|
||
|
|
...
|
||
|
|
|
||
|
|
```
|
||
|
|
|
||
|
|
Dynamic fixtures are only supported if they are declared in `pytest_generate_tests`:
|
||
|
|
|
||
|
|
```python
|
||
|
|
def pytest_generate_tests(metafunc):
|
||
|
|
...
|
||
|
|
metafunc.parametrize("fixture1, fixture2", [(1, 2), (3, 4)]) # OK
|
||
|
|
...
|
||
|
|
|
||
|
|
```
|
||
|
|
|
||
|
|
Plugins are not supported (**temporarily**), so fixtures and tests from there cannot be found.
|