allure-validator/README.md

# allure-validator

Linter for checking uniqueness and correctness of `allure.title` in the entire test database.

## Installation

```shell
pip install git+https://git.frostfs.info/TrueCloudLab/allure-validator.git@master
```

## Usage

To start, simply specify the folder with the test database that needs to be checked:

```shell
allure-validator <folder> [OPTIONS]
```

Additionally, `allure-validator` can be added to pre-commit hooks:
```yaml
# .pre-commit-config.yaml
repos:
  - repo: https://git.frostfs.info/TrueCloudLab/allure-validator
    rev: latest
    hooks:
      - id: allure-validator
        args: ["pytest_tests/"]
        pass_filenames: false
```

### Ignore mechanism

The linter can be made to ignore functions/classes. To do this, just write a comment `# noqa: allure-validator` to the right of the function/class name. For example:

```python
@pytest.mark.parametrize("test_param", [1, 2])
def test_ignored_function(test_param):  # noqa: allure-validator
    ...


@pytest.mark.parametrize("class_param", [1, 2])
class TestIngoredClass:  # noqa: allure-validator
    def test_1(self, class_param):
        ...

    def test_2(self, class_param):
        ...
```

> Note: this may also miss fixtures, so use with caution.

### Options

The linter supports the following flags:
- `--plugins` - regex patterns that plugins must match (paths are compiled according to the template: `plugin / src / plugin`).
- `--files` - paths to files with fixtures.

Configuration options can also be provided using special command line arguments, for example:
```shell
allure-validator pytest_tests/ --plugins some[-_]plugin --files etc/workspace/some_plugin/src/some_plugin/fixtures.py
```

## Work example

Let's run the linter for some [sample tests](https://git.frostfs.info/Kiriruso/allure-validator/src/branch/master/tests):

```shell
NOT UNIQUE TITLE: Not unique test
In the following tests:
        tests/test_validator.py:20:0: title is not unique by `test_not_unique_2`
        tests/test_validator.py:119:4: title is not unique by `test_not_unique_3`
        tests/test_validator.py:15:0: title is not unique by `test_not_unique_1`

NOT UNIQUE TITLE: Test title
In the following tests:
        tests/test_validator.py:30:0: title is not unique by `test_title_not_unique_2`
        tests/test_validator.py:25:0: title is not unique by `test_title_not_unique_1`

tests/test_validator.py:5:0: EMPTY TITLE: Missing title in `test_empty_title`
tests/test_validator.py:10:0: EMPTY TITLE: Missing title in `test_empty_title_2`
tests/test_validator.py:20:0: MISSING PARAMS: Parameters are missing from title: object_size
tests/test_validator.py:62:0: MISSING PARAMS: Parameters are missing from title: param_two OR param_three
tests/test_validator.py:115:4: MISSING PARAMS: Parameters are missing from title: root_fixture
tests/test_validator.py:119:4: MISSING PARAMS: Parameters are missing from title: param_class_1
```

## Undetectable cases

Since the linter works with AST python code, any dynamic titles will not be found. For example:

```python
import allure


def test_example():
    data = 10
    allure.dynamic.title(f"Dynamic title {data}")  # Undetected title
    ...

```

Dynamic fixtures are only supported if they are declared in `pytest_generate_tests`:

```python
def pytest_generate_tests(metafunc):
    ...

    metafunc.parametrize("fixture1, fixture2", [(1, 2), (3, 4)])  # OK
    
    ...
    
    if some_condition:
        metafunc.parametrize("fixture3", *some_value_generator)  # OK

```