allure-validator

Linter for checking uniqueness and correctness of allure.title in the entire test database.

Installation

pip install git+https://git.frostfs.info/TrueCloudLab/allure-validator.git@master

Usage

To start, simply specify the folder with the test database that needs to be checked:

allure-validator <folder>

Additionally, allure-validator can be added to pre-commit hooks:

# .pre-commit-config.yaml
repos:
  - repo: https://git.frostfs.info/TrueCloudLab/allure-validator
    rev:  1.0.0
    hooks:
      - id: allure-validator
        args: ["pytest_tests/"]
        pass_filenames: false

Ignore mechanism

The linter can be made to ignore functions/classes. To do this, just write a comment # noqa: allure-validator to the right of the function/class name. For example:

@pytest.mark.parametrize("test_param", [1, 2])
def test_ignored_function(test_param):  # noqa: allure-validator
    ...


@pytest.mark.parametrize("class_param", [1, 2])
class TestIngoredClass:  # noqa: allure-validator
    def test_1(self, class_param):
        ...

    def test_2(self, class_param):
        ...

Note: this may also miss fixtures, so use with caution.

Work example

Let's run the linter for some sample tests:

NOT UNIQUE TITLE: Not unique test
In the following tests:
        tests/test_validator.py:20:0: title is not unique by `test_not_unique_2`
        tests/test_validator.py:119:4: title is not unique by `test_not_unique_3`
        tests/test_validator.py:15:0: title is not unique by `test_not_unique_1`

NOT UNIQUE TITLE: Test title
In the following tests:
        tests/test_validator.py:30:0: title is not unique by `test_title_not_unique_2`
        tests/test_validator.py:25:0: title is not unique by `test_title_not_unique_1`

tests/test_validator.py:5:0: EMPTY TITLE: Missing title in `test_empty_title`
tests/test_validator.py:10:0: EMPTY TITLE: Missing title in `test_empty_title_2`
tests/test_validator.py:20:0: MISSING PARAMS: Parameters are missing from title: object_size
tests/test_validator.py:62:0: MISSING PARAMS: Parameters are missing from title: param_two OR param_three
tests/test_validator.py:115:4: MISSING PARAMS: Parameters are missing from title: root_fixture
tests/test_validator.py:119:4: MISSING PARAMS: Parameters are missing from title: param_class_1

Undetectable cases

Since the linter works with AST python code, any dynamic titles will not be found. For example:

import allure


def test_example():
    data = 10
    allure.dynamic.title(f"Dynamic title {data}")  # Undetected title
    ...

Dynamic fixtures are only supported if they are declared in pytest_generate_tests:

def pytest_generate_tests(metafunc):
    ...
    metafunc.parametrize("fixture1, fixture2", [(1, 2), (3, 4)])  # OK
    ...

Plugins are not supported (temporarily), so fixtures and tests from there cannot be found.