Software documentation

This document describes the Quality-time software. It is aimed at Quality-time developers and maintainers.

Component overview

Quality-time consists of seven Docker components, as depicted above. Each oval is a Docker component.

Bespoke components

There are four bespoke components:

• A frontend, serving the user interface. The frontend is written in JavaScript using ReactJS and Semantic UI React.

• An API-server serving the API for the user interface. The API-server is written in Python using Bottle as web framework.

• A collector to collect the measurements from the sources. The collector is written in Python using aiohttp as HTTP client library.

• A notifier to notify users about events such as metrics turning red. The notifier is written in Python.

Source code that is shared between the Python components lives in the shared data model and shared code components. These are not run-time components. The code of these components is shared at build time, when the Docker images are created. The data model is used by all Python components, i.e. the API-server, the collector, and the notifier. The shared code is used by the API-server.

Standard components

Quality-time uses three standard components:

• A proxy, routing traffic from and to the user’s browser. The proxy is based on Nginx.

• A database, for storing reports and measurements. The database is based on MongoDB.

• A renderer, to export reports to PDF. The renderer is based on Puppeteer.

In addition, unless forward authentication is used, an LDAP server is expected to be available to authenticate users.

Test components

For testing purposes there a few additional components:

• A web server serving test data.

• A test LDAP server.

• A tool to administer users in the LDAP server (phpldapadmin).

• A tool to view and edit the database contents (mongo-express).

Frontend

The frontend contains the React frontend code. This component was bootstrapped using Create React App.

Health check

As a health check, the favicon is downloaded.

Environment variables

The frontend uses the following environment variables:

Name	Default value	Description
FRONTEND_PORT	5000	The port the frontend listens on.

API-server

API

The API of the API-server is versioned. The version is not changed when backwards compatible changes are made, such as the addition of new endpoints. The public API is documented separately.

Health check

The Dockerfile contains a health check that uses curl to retrieve an end-point (api/health) from the API-server. Officially, this end-point does not exist, but since the server returns an empty JSON file for non-existing endpoints it works for checking the health of the API-server.

Environment variables

The API-server uses the following environment variables:

Name	Default value	Description
API_SERVER_PORT	5001	Port of the API-server.
API_SERVER_LOG_LEVEL	WARNING	Log level. Allowed values are DEBUG, INFO, WARNING, ERROR, and CRITICAL.
DATABASE_URL	mongodb://root:root@database:27017	Mongo database connection URL.
LDAP_URL	ldap://ldap:389	Comma-separated list of LDAP connection URL(s).
LDAP_ROOT_DN	dc=example,dc=org	LDAP root distinguished name.
LDAP_LOOKUP_USER_DN	cn=admin,dc=example,dc=org	LDAP lookup user distinguished name.
LDAP_LOOKUP_USER_PASSWORD	admin	LDAP lookup user password.
LDAP_SEARCH_FILTER	(|(uid=$$username)(cn=$$username))	LDAP search filter. With this default search filter, users can use either their LDAP canonical name (cn) or their LDAP user id to login. The $username variable is filled by Quality-time at run time with the username that the user enters in the login dialog box.
LOAD_EXAMPLE_REPORTS	True	Whether or not to import example reports in the database on start up.
FORWARD_AUTH_ENABLED	False	Whether or not to enable forward authentication.
FORWARD_AUTH_HEADER	X-Forwarded-User	Header to use for getting the username if forward authentication is turned on.
USER_SESSION_DURATION	120	Duration of user session in number of hours.

Collector

The collector is responsible for collecting measurement data from sources. It wakes up periodically and retrieves a list of all metrics from the database. For each metric, the collector gets the measurement data from each of its sources and stores a new measurement to the database.

If a metric has been recently measured and its parameters haven’t been changed, the collector skips the metric.

By default, the collector measures metrics whose configuration hasn’t been changed every 15 minutes and sleeps 60 seconds in between measurements. This can be changed using the environment variables listed below.

Health check

Every time the collector wakes up, it writes the current date and time in ISO format to the health check file. This date and time is read by the Docker health check (see the Dockerfile). If the written date and time are too long ago, the collector container is considered to be unhealthy.

Environment variables

The collector uses the following environment variables:

Name	Default value	Description
COLLECTOR_LOG_LEVEL	WARNING	Log level. Allowed values are DEBUG, INFO, WARNING, CRITICAL, and ERROR.
COLLECTOR_SLEEP_DURATION	20	The maximum amount of time (in seconds) that the collector sleeps between collecting measurements.
COLLECTOR_MEASUREMENT_FREQUENCY	900	The amount of time (in seconds) after which a metric should be measured again.
COLLECTOR_MEASUREMENT_LIMIT	30	The maximum number of metrics that the collector measures each time it wakes up. If more metrics need to be measured, they will be measured the next time the collector wakes up.
COLLECTOR_MEASUREMENT_TIMEOUT	120	The amount of time (in seconds) after which a source connection should timeout.
DATABASE_URL	mongodb://root:root@database:27017	Mongo database connection URL.
HEALTH_CHECK_FILE	/home/collector/health_check.txt	Path to the file used for health check.
HTTP(S)_PROXY		Proxy to use by the collector.

See also

See the aiohttp documentation for more information on proxy support.

Notifier

The notifier is responsible for notifying users about significant events, such as metrics turning red. It wakes up periodically and asks the server for all reports. For each report, the notifier determines whether notification destinations have been configured, and whether events happened that need to be notified.

Health check

Every time the notifier wakes up, it writes the current date and time in ISO format to the health check file. This date and time is read by the Docker health check (see the Dockerfile). If the written date and time are too long ago, the notifier container is considered to be unhealthy.

Environment variables

The notifier uses the following environment variables:

Name	Default value	Description
DATABASE_URL	mongodb://root:root@database:27017	Mongo database connection URL.
HEALTH_CHECK_FILE	/home/notifier/health_check.txt	Path to the file used for health check.
NOTIFIER_LOG_LEVEL	WARNING	Log level. Allowed values are DEBUG, INFO, WARNING, ERROR, and CRITICAL.
NOTIFIER_SLEEP_DURATION	60	The amount of time (in seconds) that the notifier sleeps between sending notifications.

Shared code

The shared code component contains code and resources shared between the servers and the collector and notifier components. This includes the example reports, a shared data model, and code to initialize the servers, access the database, and provide endpoints.

Example reports

The example reports are imported when a server is started and the database doesn’t contain any sample reports yet. Turn off the loading of example report by setting LOAD_EXAMPLE_REPORTS to False. See the sections on configuration of the servers below.

Shared data model

The shared data model contains the data model that is shared between all Python components.

Data model

The data model package describes the domain model used by the application. It allows for a frontend that doesn’t need to know about specific metrics and sources. When a server component starts up, it checks whether the data model has changed, and if so, imports it into the database.

The data model package consists of a meta model and the data model itself. The data model consists of four major parts:

• Scales

• Metrics

• Sources

• Subjects

The meta model uses Pydantic to specify the components and attributes of the data model.

Scales

The scales part of the data model defines the scales that can be used to measure metrics. At the time of writing these include an absolute count scale, a percentage scale, and a version number scale.

Each metric defines the scales it supports.

Metrics

The metrics part of the data model is a dictionary with all supported metric types. The keys are the metric type names. The values are objects describing the metric type. All metric types have the following meta model:

class Metric(DescribedModel):
    """Base model for metrics."""

    scales: list[str] = Field(["count"], min_length=1)
    default_scale: str = "count"
    unit: Unit = Unit.NONE
    addition: Addition = Addition.SUM
    direction: Direction = Direction.FEWER_IS_BETTER
    target: str = "0"
    near_target: str = "10"
    sources: list[str] = Field(..., min_items=1)
    tags: list[Tag] = []
    rationale: str = ""  # Answers the question "Why measure this metric?", included in documentation and UI
    rationale_urls: list[str] = []
    explanation: str | None = ""  # Optional explanation of concepts in text format, included in documentation and UI
    explanation_urls: list[str] = []
    documentation: str | None = ""  # Optional documentation in Markdown format, only included in the documentation

The name is the default name of metrics of this type. The description describes what the metric measures. These are part of the DescribedModel.

The scales list shows which scales the metric supports and the default_scale specifies which scale is the default scale.

The unit is the default unit of the metric, e.g. lines of code, security warnings, or in the above example, complex units.

The addition determines how values from multiple sources are combined: possible values are max, min, and sum.

The direction specifies whether smaller measurement values are better or worse.

The target is the default target value for the metric. The near_target is when the metric becomes red. Values between target and near_target are yellow.

The list of sources contains the keys of source types that support this metric type.

Finally, tags are strings used to group related metrics.

Users with sufficient rights can override the default name, unit, and target of metrics via the user interface.

Sources

The sources part of the data model is a dictionary that describes all supported source types. The keys are the source type names. The values are objects describing the source type. All source types have the following meta model:

class Source(DescribedModel):
    """The source model extends the base model with source parameters and measurement entities."""

    url: HttpsUrl | None = None
    documentation: dict[str, str] | None = None  # Documentation in Markdown format
    configuration: dict[str, Configuration] = {}
    parameters: dict[str, Parameter]
    entities: dict[str, Entity] = {}
    issue_tracker: bool | None = False

The name is the default name of sources of this type. The description gives some background information on the source type. These are part of the DescribedModel.

The url links to a landing page describing the source type.

Configuration

In cases where Quality-time needs information about sources that doesn’t need to be parameterizable, Configurations can be added to the source. A configuration consists of a name (via NamedModel), a list of metrics to which the configuration applies, and a value:

class Configuration(NamedModel):
    """Configuration for specific metrics."""

    metrics: list[str] = Field(..., min_items=1)
    value: list[str] = Field(..., min_items=1)

Parameters

The parameters describe the parameters that need to be entered by the user to configure the source:

class Parameter(NamedModel):
    """Source parameter model."""

    short_name: str | None = None
    help: str | None = None
    help_url: HttpsUrl | None = None
    type: ParameterType
    placeholder: str | None = None
    mandatory: bool = False
    default_value: str | list[str] = ""
    unit: str | None = None
    metrics: list[str] = Field(..., min_items=1)
    values: list[str] | None = None
    api_values: dict[str, str] | None = None
    validate_on: list[str] | None = None

Each parameter has a name (via NamedModel) and a short_name used as label in the user interface. The parameter can have a help string or a help_url (but not both).

The type specifies the type of the parameter and the widget used to get user input. Possible values are amongst others string, password, integer, and multiple_choice.

The placeholder contains text to display in case of multiple choice parameters. For example, in the case of a multiple choice ‘severities’ parameter with possible values of ‘low’, ‘medium’, ‘high’, the placeholder can be set to ‘all severities’ to indicate that by default all severities will be measured.

The mandatory field indicates whether the parameter is mandatory.

The default_value specifies the default value. In case of multiple choice parameters this should be a, possibly empty, list of values.

The unit indicates the unit of the parameter. If the type is integer the unit and min_value need to be specified.

For each parameter, a list of metrics must be given for which the parameter is applicable. This is needed because not every metric needs the same parameters.

If the type is multiple_choice the possible values need to be specified. Also, an api_values mapping can specify how the values map to the values used in the API of the source.

The validate_on field specifies that the parameter needs to be validated when the parameters in the list change. This can be used to specify that e.g. a URL parameter must be validated when the user changes the password parameter.

Entities

Measurement entities are the things that are counted or measured to get the measurement value. For example, the measurement entities of the ‘violations’ metric are the individual violations. Sometimes, the measurement entities are not interesting enough to show, e.g. when measuring the size in terms of lines of code. In other cases, groups of entities are shown, for example test runs as entities for the ‘tests’ metric.

The Entity and EntityAttribute meta models look as follows:

class Entity(BaseModel):
    """Measurement entity (violation, warning, etc.)."""

    name: str = Field(..., regex=r"[a-z]+")
    name_plural: str | None = None
    attributes: list[EntityAttribute]
    measured_attribute: str | None = None


class EntityAttribute(NamedModel):
    """Attributes of measurement entities."""

    key: str | None = None
    help: str | None = None
    url: str | None = None  # Which key to use to get the URL for this attribute
    color: dict[str, Color] | None = None
    type: EntityAttributeType | None = None
    alignment: EntityAttributeAligment | None = None  # If not given, the aligment is based on the attribute type
    pre: bool | None = None  # Should the attribute be formatted using <pre></pre>? Defaults to False
    visible: bool | None = None  # Should this attribute be visible in the UI? Defaults to True

Each entity contains the name (both singular and plural) of the entities and a list of attributes.

The attributes are shown as columns in the front end. Each attribute/column consists of a name (via NamedModel), which is used as column header, and a key, used to get the data from the database.

An attribute/column can have a key url to specify which field contains the URL to be used in the column. In theory, each column can link to a different URL this way.

To specify the data type of the attribute/column, use the type field. If no type is specified, string is assumed and no special formatting is applied. Other types supported at the moment are date, datetime, float, integer, and status. When using date or datetime, the column should be an ISO-formatted date or date-time string and Date.toLocaleDateString() or Date.toLocaleString() is used to format the date or date-time.

Values can be mapped to colors using the optional color field with a column-value-to-color mapping as value. Possible colors are positive (green), negative (red), warning (yellow) and active (grey). These correspond to the possible states of table rows in Semantic UI React.

Users can mark entities as false positive to ignore them. By default, Quality-time subtracts one from the metric value for each ignored entity. However, this would be incorrect if an entity represents a value greater than one, for example when the metric is the amount of ready user story points and each entity is a user story. In that case Quality-time can use an attribute of the entity to subtract from the value. The entity field measured_attribute determines which attribute to use.

In most cases, the measured attribute is one of the attributes. In other cases, the measured attribute may depend on the parameters selected by the user. For example, when measuring ‘tests’ using Azure DevOps as source, the test results (failed/passed/etc.) selected by the user influence how many tests Quality-time has to subtract from the total if the user decides to ignore a test run. To accommodate this, it is possible to add an attribute that is not shown by the front end, but is used as measured attribute, by marking the attribute as not visible.

Of course, the collector needs to compute the extra attribute and add it to the measurement entities.

Subjects

The subjects part of the data model is an object where the keys are the subject types and the values are objects describing the subject. The Subject meta model looks as follows:

class Subject(DescribedModel):
    """Base model for subjects."""

    metrics: list[str] = Field(..., min_items=1)

The name is the default name of the subject. The description describes the subject type. Both fields are part of DescribedModel.

The list of metrics contains the metrics that make the most sense for the subject type, and is used for filtering the list of metrics in the dropdown menu of the buttons for moving and copying metrics.

Proxy

The proxy routes traffic from and to the user’s browser. Quality-time uses Nginx, but this can be replaced by another proxy if so desired.

The proxy Dockerfile adds the Quality-time configuration to the Nginx image.

Environment variables

The proxy uses the following environment variables:

Name	Default value	Description
FRONTEND_HOST	frontend	The host name of the frontend.
FRONTEND_PORT	5000	The port the frontend listens on.
API_SERVER_HOST	api_server	The hostname of the API-server.
API_SERVER_PORT	5001	The port the API-server listens on.

Database

The database component consists of a Mongo database to store reports and measurements.

The proxy Dockerfile wraps the MongoDB image in a Quality-time image so the MongoDB version number can be changed when needed.

Quality-time stores its data in a Mongo database using the following collections: datamodels, measurements, reports, reports_overviews, and sessions.

The two server components are the only components that directly interacts with the database.

Data models, reports, and reports overviews are temporal objects. Every time a new version of the data model is loaded or the user edits a report or the reports overview, an updated copy of the object (a “document” in Mongo-parlance) is added to the collection. Since each copy has a timestamp, this enables the API-server to retrieve the documents as they were at a specific moment in time and provide time-travel functionality.

Environment variables

The database uses the following environment variables:

Name	Default value	Description
MONGO_INITDB_ROOT_USERNAME	root	The MongoDB root username.
MONGO_INITDB_ROOT_PASSWORD	root	The MongoDB root password.

Renderer

The renderer component is used to export quality reports to PDF. Quality-time uses puppeteer.

The renderer Dockerfile wraps puppeteer with a small API that uses puppeteer to convert a report URL into a PDF file.

Health check

The Dockerfile contains a health check that uses curl to retrieve an API (api/health) from the renderer API server.

Environment variables

The renderer uses the following environment variables:

Name	Default value	Description
PROXY_HOST	www	Hostname of the proxy. The renderer uses this to access the reports that need to be exported to PDF.
PROXY_PORT	80	Port of the proxy. The renderer uses this to access the reports that need to be exported to PDF.
PROXY_PROTOCOL	http	Protocol of the proxy. The renderer uses this to access the reports that need to be exported to PDF.
LC_ALL		Set the date format in the PDF export. For example, to get DD-MM-YYYY use: en_GB.UTF-8.
TZ		Make the PDF export use the correct timezone. For example, to get Central European Time use: Europe/Amsterdam.

Test data

This component contains test data for the example reports. The Docker image is published as ictu/quality-time_testdata on Docker Hub.

Running the test data component

The test data component is started as part of the docker composition for development, see the developer manual.

To serve the test data locally, you can also start a web server from a console, for example:

python3 -m http.server

Adding test data

Add the example file(s) to the test data reports and update one or more of the example reports in the shared code component.

Acknowledgements

• cobertura.xml was copied from https://github.com/Bachmann1234/diff_cover/blob/main/diff_cover/tests/fixtures/dotnet_coverage.xml.

• testng-results.xml was copied from https://github.com/richie-b/AtnApiTest/blob/master/test-output/testng-results.xml.

Test LDAP server

A test LDAP server with test users is included for development and testing purposes. An admin interface (phpldapadmin) is included to administer users in this LDAP server.

LDAP users

The LDAP database has two users:

User	Email address	Username	Password
Jane Doe	janedoe@example.org	jadoe	secret
John Doe	johndoe@example.org	jodoe	secret