Reference manual

This is an overview of all subjects that Quality-time can measure, all metrics that Quality-time can use to measure subjects, and all sources that Quality-time can use to collect data from to measure the metrics. For each supported combination of metric and source, the parameters that can be used to configure the source are listed.

Subjects

This is an overview of all the subjects that Quality-time can measure. For each subject, the metrics that can be used to measure the subject are listed.

CI-environment

A continuous integration environment.

Supporting metrics

• Failed CI-jobs

• Job runs within time period

• Merge requests

• Performancetest duration

• CI-pipeline duration

• Software version

• Source up-to-dateness

• Source version

• Unmerged branches

• Unused CI-jobs

Process

A software development and/or maintenance process.

Supporting metrics

• Average issue lead time

• Change failure rate

• Issues

• Manual test duration

• Manual test execution

• Merge requests

• Source up-to-dateness

• Time remaining

• Unmerged branches

• User story points

• Velocity

• Sentiment

Quality report

A software quality report.

Supporting metrics

• Metrics

• Missing metrics

Software

A custom software application or component.

Supporting metrics

• Average issue lead time

• Change failure rate

• Commented out code

• Complex units

• Dependencies

• Duplicated lines

• Issues

• Job runs within time period

• Size (LOC)

• Long units

• Manual test duration

• Manual test execution

• Many parameters

• Merge requests

• Performancetest duration

• Performancetest stability

• CI-pipeline duration

• Violation remediation effort

• Scalability

• Security warnings

• Slow transactions

• Software version

• Source up-to-dateness

• Source version

• Suppressed violations

• Test cases

• Tests

• Todo and fixme comments

• Test branch coverage

• Test line coverage

• Unmerged branches

• User story points

• Violations

Metrics

This is an overview of all the metrics that Quality-time can use to measure subjects. For each metric, the default target, the supported scales, and the default tags are given. In addition, the sources that can be used to collect data from to measure the metric are listed.

Average issue lead time

The average lead time for changes completed in a certain time period.

Why measure average issue lead time? The shorter the lead time for changes, the sooner the new features can be used. Also, the shorter the lead time for changes, the fewer changes are in progress at the same time. Less context switching is needed and the risk of interfering changes is reduced.

Default target

≦ 10 days

Scales

count

Default tags

process efficiency

Supported subjects

• Process

• Software

Supporting sources

• Azure DevOps Server

• Jira

CI-pipeline duration

The duration of a CI-pipeline.

Why measure cI-pipeline duration? CI-pipelines that take too much time may signal something wrong with the build.

Default target

≦ 10 minutes

Scales

count

Default tags

ci

Supported subjects

• CI-environment

• Software

Supporting sources

• Manual number

• GitLab

Change failure rate

The percentage of deployments causing a failure in production.

Why measure change failure rate? The change failure rate is an indicator of the DevOps effectiveness of a team.

How to configure change failure rate? Because this metric is a “rate”, it needs a numerator (the number of failed deployments in the look back period) and a denominator (the number of deployments in the look back period). The metric can be configured with a combination of Jira, and either Jenkins or GitLab CI as source. Jira is used as source for the number of failures (numerator). Either Jenkins or GitLab CI is used as source for the number of deployments (denominator).

When configuring Azure DevOps as source for the metric it is used both for number of failures (numerator) and for number of deployments (denominator). Azure DevOps needs to be added as source only once.

Default target

≦ 0% of the failed deployments

Scales

percentage

Default tags

process efficiency

Supported subjects

• Process

• Software

Supporting sources

• Azure DevOps Server

• Jenkins

• Jira

• GitLab

Commented out code

The number of blocks of commented out lines of code.

Why measure commented out code? Code should not be commented out because it bloats the sources and may confuse the reader as to why the code is still there, making the source code harder to understand and maintain. Unused code should be deleted. It can be retrieved from the version control system if needed.

See also

1. https://rules.sonarsource.com/python/RSPEC-125

2. https://kentcdodds.com/blog/please-dont-commit-commented-out-code

Default target

≦ 0 blocks

Scales

count

Default tags

maintainability

Supported subjects

• Software

Supporting sources

• Manual number

• SonarQube

Complex units

The number of units (classes, functions, methods, files) that are too complex.

Why measure complex units? Complex code makes software harder to test and harder to maintain. Complex code is harder to test because there are more execution paths that need to be tested. Complex code is harder to maintain because it is harder to understand and analyze.

See also

1. https://www.softwareimprovementgroup.com/wp-content/uploads/SIG-TUViT-Evaluation-Criteria-Trusted-Product-Maintainability-Guidance-for-producers.pdf

2. https://blog.codacy.com/an-in-depth-explanation-of-code-complexity/

3. https://blog.sonarsource.com/cognitive-complexity-because-testability-understandability

Default target

≦ 0 complex units

Scales

count (default)

percentage

Default tags

maintainability

testability

Supported subjects

• Software

Supporting sources

• Manual number

• SonarQube

Dependencies

The number of (outdated) dependencies.

Why measure dependencies? Dependencies that are out of date can be considered a form of technical debt. On the one hand, not upgrading a dependency postpones the work of testing the new version. And, if the new version of a dependency has backwards-incompatible changes, it also postpones making adaptations to cater for those changes. On the other hand, upgrading the dependency may fix bugs and vulnerabilities, and unlock new features. Measuring the number of outdated dependencies provides insight into the size of this backlog.

Default target

≦ 0 dependencies

Scales

count (default)

percentage

Default tags

maintainability

Supported subjects

• Software

Supporting sources

• Composer

• Dependency-Track

• Manual number

• npm

• OWASP Dependency-Check

• pip

Duplicated lines

The number of lines that are duplicated.

Why measure duplicated lines? Duplicate code makes software larger and thus potentially harder to maintain. Also, if the duplicated code contains bugs, they need to be fixed in multiple locations.

See also

1. https://www.softwareimprovementgroup.com/wp-content/uploads/SIG-TUViT-Evaluation-Criteria-Trusted-Product-Maintainability-Guidance-for-producers.pdf

Default target

≦ 0 lines

Scales

count (default)

percentage

Default tags

maintainability

Supported subjects

• Software

Supporting sources

• Manual number

• SonarQube

Failed CI-jobs

The number of continuous integration jobs or pipelines that have failed.

Why measure failed CI-jobs? Although it is to be expected that CI-jobs or pipelines will fail from time to time, a significant number of failed CI-jobs or pipelines over a longer period of time could be a sign that the continuous integration process is not functioning properly. Also, having many failed CI-jobs or pipelines makes it hard to see that additional jobs or pipelines start failing.

Default target

≦ 0 CI-jobs

Scales

count

Default tags

ci

Supported subjects

• CI-environment

Supporting sources

• Azure DevOps Server

• Jenkins

• GitLab

• Manual number

Issues

The number of issues.

Why measure issues? What exactly issues are, depends on what is available in the source. The issues metric can for example be used to count the number of open bug reports, the number of ready user stories, or the number of overdue customer service requests. For sources that support a query language, the issues to be counted can be specified using the query language of the source.

Default target

≦ 0 issues

Scales

count

Supported subjects

• Process

• Software

Supporting sources

• Azure DevOps Server

• Jira

• Manual number

• Trello

Job runs within time period

The number of job runs within a specified time period.

Why measure job runs within time period? Frequent deployments are associated with smaller, less risky deployments and with lower lead times for new features.

Default target

≧ 30 CI-job runs

Scales

count

Default tags

ci

Supported subjects

• CI-environment

• Software

Supporting sources

• Azure DevOps Server

• GitLab

• Jenkins

• Manual number

Long units

The number of units (functions, methods, files) that are too long.

Why measure long units? Long units are deemed harder to maintain.

See also

1. https://www.softwareimprovementgroup.com/wp-content/uploads/SIG-TUViT-Evaluation-Criteria-Trusted-Product-Maintainability-Guidance-for-producers.pdf

Default target

≦ 0 long units

Scales

count (default)

percentage

Default tags

maintainability

Supported subjects

• Software

Supporting sources

• Manual number

• SonarQube

Manual test duration

The duration of the manual test in minutes.

Why measure manual test duration? Preferably, all regression tests are automated. When this is not feasible, it is good to know how much time it takes to execute the manual tests, since they need to be executed before every release.

Default target

≦ 0 minutes

Scales

count

Default tags

test quality

Supported subjects

• Process

• Software

Supporting sources

• Jira

• Manual number

Manual test execution

Measure the number of manual test cases that have not been tested on time.

Why measure manual test execution? Preferably, all regression tests are automated. When this is not feasible, it is good to know whether the manual regression tests have been executed recently.

Default target

≦ 0 manual test cases

Scales

count

Default tags

test quality

Supported subjects

• Process

• Software

Supporting sources

• Jira

• Manual number

Many parameters

The number of units (functions, methods, procedures) that have too many parameters.

Why measure many parameters? Units with many parameters are deemed harder to maintain.

See also

1. https://www.softwareimprovementgroup.com/wp-content/uploads/SIG-TUViT-Evaluation-Criteria-Trusted-Product-Maintainability-Guidance-for-producers.pdf

Default target

≦ 0 units with too many parameters

Scales

count (default)

percentage

Default tags

maintainability

Supported subjects

• Software

Supporting sources

• Manual number

• SonarQube

Merge requests

The number of merge requests.

Why measure merge requests? Merge requests need to be reviewed and approved. This metric allows for measuring the number of merge requests without the required approvals.

How to configure merge requests? In itself, the number of merge requests is not indicative of software quality. However, by setting the parameter “Minimum number of upvotes”, the metric can report on merge requests that have fewer than the minimum number of upvotes. The parameter “Merge request state” can be used to exclude closed merge requests, for example. The parameter “Target branches to include” can be used to further limit the merge requests to only count merge requests that target specific branches, for example the “develop” branch.

Default target

≦ 0 merge requests

Scales

count (default)

percentage

Default tags

ci

Supported subjects

• CI-environment

• Process

• Software

Supporting sources

• Azure DevOps Server

• GitLab

• Manual number

Metrics

The number of metrics from one or more quality reports, with specific states and/or tags.

Why measure metrics? Use this metric to monitor other quality reports. For example, count the number of metrics that don’t meet their target value, or count the number of metrics that have been marked as technical debt for more than two months.

How to configure metrics? After adding Quality-time as a source to a “Metrics”-metric, one can configure which statuses to count and which metrics to consider by filtering on report names or identifiers, on metric types, on source types, and on tags.

Note

If the “Metrics” metric is itself part of the set of metrics it counts, a peculiar situation may occur: when you’ve configured the “Metrics” to count red metrics and its target is not met, the metric itself will become red and thus be counted as well. For example, if the target is at most five red metrics, and the number of red metrics increases from five to six, the “Metrics” value will go from five to seven. You can prevent this by making sure the “Metrics” metric is not in the set of counted metrics, for example by putting it in a different report and only count metrics in the other report(s).

Default target

≦ 0 metrics

Scales

count (default)

percentage

Supported subjects

• Quality report

Supporting sources

• Manual number

• Quality-time

Missing metrics

The number of metrics that can be added to each report, but have not been added yet.

Why measure missing metrics? Provide an overview of metrics still to be added to the quality report. If metrics will not be added, a reason can be documented.

Default target

≦ 0 missing metrics

Scales

count (default)

percentage

Supported subjects

• Quality report

Supporting sources

• Manual number

• Quality-time

Performancetest duration

The duration of the performancetest in minutes.

Why measure performancetest duration? Performance tests, especially endurance tests, may need to run for a minimum duration to give relevant results.

Default target

≧ 30 minutes

Scales

count

Default tags

performance

Supported subjects

• CI-environment

• Software

Supporting sources

• Gatling

• JMeter CSV

• Manual number

• Performancetest-runner

Performancetest stability

The duration of the performancetest at which throughput or error count increases.

Why measure performancetest stability? When testing endurance, the throughput and error count should remain stable for the complete duration of the performancetest. If throughput or error count starts to increase during the performancetest, this may indicate memory leaks or other resource problems.

Default target

≧ 100% of the minutes

Scales

percentage

Default tags

performance

Supported subjects

• Software

Supporting sources

• Manual number

• Performancetest-runner

Scalability

The number of virtual users (or percentage of the maximum number of virtual users) at which ramp-up of throughput breaks.

Why measure scalability? When stress testing, the load on the system-under-test has to increase sufficiently to detect the point at which the system breaks, as indicated by increasing throughput or error counts. If this breakpoint is not detected, the load has not been increased enough.

Default target

≧ 75 virtual users

Scales

count (default)

percentage

Default tags

performance

Supported subjects

• Software

Supporting sources

• Manual number

• Performancetest-runner

Security warnings

The number of security warnings about the software.

Why measure security warnings? Monitor security warnings about the software, its source code, dependencies, or infrastructure so vulnerabilities can be fixed before they end up in production.

See also

1. What is the difference between violations and warnings?

Default target

≦ 0 security warnings

Scales

count

Default tags

security

Supported subjects

• Software

Supporting sources

• Anchore

• Anchore Jenkins plugin

• Bandit

• Cargo Audit

• Checkmarx CxSAST

• Dependency-Track

• JSON file with security warnings

• Harbor

• Harbor JSON

• Manual number

• OpenVAS

• OWASP Dependency-Check

• OWASP ZAP

• Pyupio Safety

• SARIF

• Snyk

• SonarQube

• Trivy JSON

Sentiment

How are the team members feeling?

Why measure sentiment? Satisfaction is how fulfilled developers feel with their work, team, tools, or culture; well-being is how healthy and happy they are, and how their work impacts it. Measuring satisfaction and well-being can be beneficial for understanding productivity and perhaps even for predicting it. For example, productivity and satisfaction are correlated, and it is possible that satisfaction could serve as a leading indicator for productivity; a decline in satisfaction and engagement could signal upcoming burnout and reduced productivity.

See also

1. https://queue.acm.org/detail.cfm?id=3454124

Default target

≧ 10

Scales

count

Supported subjects

• Process

Supporting sources

• Manual number

Size (LOC)

The size of the software in lines of code.

Why measure size (LOC)? The size of software is correlated with the effort it takes to maintain it. Lines of code is one of the most widely used metrics to measure size of software.

See also

1. https://www.softwareimprovementgroup.com/wp-content/uploads/SIG-TUViT-Evaluation-Criteria-Trusted-Product-Maintainability-Guidance-for-producers.pdf

How to configure size (LOC)? To track the absolute size of the software, set the scale of the metric to ‘count’ and add one or more sources.

To track the relative size of sources, set the scale of the metric to ‘percentage’ and add cloc as source. Make sure to generate cloc JSON reports with the --by-file option so that filenames are included in the JSON reports. Use the ‘files to include’ field to select the sources that should be compared to the total amount of code. For example, to measure the relative amount of test code, use the regular expression .*test.*.

Note

Unfortunately, SonarQube can only used to measure the absolute size of the software. SonarQube does not report the sizes of files with test code for all programming languages. Hence it cannot be used to measure the relative size of test code.

Default target

≦ 30000 lines

Scales

count (default)

percentage

Default tags

maintainability

Supported subjects

• Software

Supporting sources

• cloc

• Manual number

• SonarQube

Slow transactions

The number of transactions slower than their target response time.

Why measure slow transactions? Transations slower than their target response time indicate performance problems that need attention.

Default target

≦ 0 transactions

Scales

count

Default tags

performance

Supported subjects

• Software

Supporting sources

• Gatling

• Manual number

• JMeter CSV

• JMeter JSON

• Performancetest-runner

Software version

The version number of the software as analyzed by the source.

Why measure software version? Monitor that the version of the software is at least a specific version or get notified when the software version becomes higher than a specific version.

More information Quality-time uses the packaging library (1) to parse version numbers. The packaging library expects version numbers to comply with PEP-440 (2). PEP is an abbrevation for Python Enhancement Proposal, but this PEP is primarily a standard for version numbers. PEP-440 encompasses most of the semantic versioning scheme (3) so version numbers that follow semantic versioning are usually parsed correctly.

See also

1. https://pypi.org/project/packaging/

2. https://peps.python.org/pep-0440/

3. https://semver.org

Default target

≧ 1.0

Scales

version_number

Default tags

ci

Supported subjects

• CI-environment

• Software

Supporting sources

• Performancetest-runner

• SonarQube

Source up-to-dateness

The number of days since the source was last updated.

Why measure source up-to-dateness? If the information provided by sources is outdated, so will be the metrics in Quality-time. Hence it is important to monitor that sources are up-to-date.

Default target

≦ 3 days

Scales

count

Default tags

ci

Supported subjects

• CI-environment

• Process

• Software

Supporting sources

• Anchore

• Anchore Jenkins plugin

• Axe-core

• Azure DevOps Server

• Bandit

• Calendar date

• Cobertura

• Cobertura Jenkins plugin

• Checkmarx CxSAST

• Dependency-Track

• Gatling

• GitLab

• JaCoCo

• JaCoCo Jenkins plugin

• Jenkins

• Jenkins test report

• JMeter CSV

• JUnit XML report

• NCover

• Robot Framework

• OpenVAS

• OWASP Dependency-Check

• OWASP ZAP

• Performancetest-runner

• Quality-time

• Robot Framework Jenkins plugin

• SonarQube

• TestNG

• Trello

Source version

The version number of the source.

Why measure source version? Monitor that the version of a source is at least a specific version or get notified when the version becomes higher than a specific version.

How to configure source version? If you decide with your development team that you are only interested in the major and minor updates of tools and want to ignore any patch updates, then the following settings can be helpful.

Tip

Leave the metric direction default, i.e. a higher version number is better. Set the metric target to the first two digits of the latest available software version. So if the latest is 3.9.4, set it to 3.9. Set the metric near target to only the first digit, so in this example that will be 3.

The result will be that when there is a minor update, this metric will turn yellow. So, suppose the version of your tool is at version 2.12.15 and the latest version available is 2.13.2, then this metric will turn yellow. If there is only a patch update, the metric will stay green. If there is a major update, the metric will turn red. So when we have 2.9.3 and the version available is 3.9.1, then the metric will turn red.

More information Quality-time uses the packaging library (1) to parse version numbers. The packaging library expects version numbers to comply with PEP-440 (2). PEP is an abbrevation for Python Enhancement Proposal, but this PEP is primarily a standard for version numbers. PEP-440 encompasses most of the semantic versioning scheme (3) so version numbers that follow semantic versioning are usually parsed correctly.

See also

1. https://pypi.org/project/packaging/

2. https://peps.python.org/pep-0440/

3. https://semver.org

Default target

≧ 1.0

Scales

version_number

Default tags

ci

Supported subjects

• CI-environment

• Software

Supporting sources

• Axe-core

• cloc

• Cobertura

• Checkmarx CxSAST

• Dependency-Track

• Gatling

• GitLab

• Jenkins

• Jira

• OpenVAS

• OWASP Dependency-Check

• OWASP ZAP

• Quality-time

• Robot Framework

• SonarQube

Suppressed violations

The number of violations suppressed in the source.

Why measure suppressed violations? Some tools allow for suppression of violations. Having the number of suppressed violations violations visible in Quality-time allows for a double check of the suppressions.

Default target

≦ 0 suppressed violations

Scales

count (default)

percentage

Default tags

maintainability

Supported subjects

• Software

Supporting sources

• Manual number

• SonarQube

Test branch coverage

The number of code branches not covered by tests.

Why measure test branch coverage? Code branches not covered by tests may contain bugs and signal incomplete tests.

See also

1. https://martinfowler.com/bliki/TestCoverage.html

Default target

≦ 0 uncovered branches

Scales

count (default)

percentage

Default tags

test quality

Supported subjects

• Software

Supporting sources

• Cobertura

• Cobertura Jenkins plugin

• JaCoCo

• JaCoCo Jenkins plugin

• Manual number

• NCover

• SonarQube

Test cases

The number of test cases.

Why measure test cases? Track the test results of test cases so there is traceability from the test cases, defined in Jira, to the test results in test reports produced by tools such as Robot Framework or Junit.

How to configure test cases? The test cases metric reports on the number of test cases, and their test results. The test case metric is different than other metrics because it combines data from two types of sources: it needs one or more sources for the test cases, and one or more sources for the test results. The test case metric then matches the test results with the test cases.

Currently, only Jira is supported as source for the test cases. JUnit, TestNG, and Robot Framework are supported as source for the test results. So, to configure the test cases metric, you need to add at least one Jira source and one JUnit, TestNG, Robot Framework source. In addition, to allow the test case metric to match test cases from Jira with test results from the JUnit, TestNG, or Robot Framework XML files, the test results should mention Jira issue keys in their title or description.

Suppose you have configured Jira with the query: project = "My Project" and type = "Logical Test Case" and this results in these test cases:

Key	Summary
MP-1	Test case 1
MP-2	Test case 2
MP-3	Test case 3
MP-4	Test case 4

Also suppose your JUnit XML has the following test results:

<testsuite tests="5" errors="0" failures="1" skipped="1">
<testcase name="MP-1; step 1">
    <failure />
</testcase>
<testcase name="MP-1; step 2">
    <skipped />
</testcase>
<testcase name="MP-2">
    <skipped />
</testcase>
<testcase name="MP-3; step 1"/>
<testcase name="MP-3; step 2"/>
</testsuite>

The test case metric will combine the JUnit XML file with the test cases from Jira and report one failed, one skipped, one passed, and one untested test case:

Key	Summary	Test result
MP-1	Test case 1	failed
MP-2	Test case 2	skipped
MP-3	Test case 3	passed
MP-4	Test case 4	untested

If multiple test results in the JUnit, TestNG, or Robot Framework XML file map to one Jira test case (as with MP-1 and MP-3 above), the ‘worst’ test result is reported. Possible test results from worst to best are: errored, failed, skipped, and passed. Test cases not found in the test results are listed as untested (as with MP-4 above).

Default target

≧ 0 test cases

Scales

count (default)

percentage

Default tags

test quality

Supported subjects

• Software

Supporting sources

• Jenkins test report

• Jira

• JUnit XML report

• Manual number

• Robot Framework

• TestNG

Test line coverage

The number of lines of code not covered by tests.

Why measure test line coverage? Code lines not covered by tests may contain bugs and signal incomplete tests.

See also

1. https://martinfowler.com/bliki/TestCoverage.html

Default target

≦ 0 uncovered lines

Scales

count (default)

percentage

Default tags

test quality

Supported subjects

• Software

Supporting sources

• Cobertura

• Cobertura Jenkins plugin

• JaCoCo

• JaCoCo Jenkins plugin

• Manual number

• NCover

• SonarQube

Tests

The number of tests.

Why measure tests? Keep track of the total number of tests or the number of tests with different states, for example failed or errored.

Default target

≧ 0 tests

Scales

count (default)

percentage

Default tags

test quality

Supported subjects

• Software

Supporting sources

• Azure DevOps Server

• Gatling

• Jenkins test report

• JMeter CSV

• JMeter JSON

• JUnit XML report

• Manual number

• Performancetest-runner

• Robot Framework

• Robot Framework Jenkins plugin

• SonarQube

• TestNG

Time remaining

The number of days remaining until a date in the future.

Why measure time remaining? Keep track of the time remaining until for example a release date, the end date of a policy, or the next team building retreat.

Default target

≧ 28 days

Scales

count

Supported subjects

• Process

Supporting sources

• Calendar date

Todo and fixme comments

The number of todo and fixme comments in source code.

Why measure todo and fixme comments? Code should not contain traces of unfinished work. The presence of todo and fixme comments may be indicative of technical debt or hidden defects.

See also

1. https://rules.sonarsource.com/python/RSPEC-1135/

2. https://cwe.mitre.org/data/definitions/546

Default target

≦ 0 todo and fixme comments

Scales

count

Default tags

maintainability

Supported subjects

• Software

Supporting sources

• Manual number

• SonarQube

Unmerged branches

The number of branches that have not been merged to the default branch.

Why measure unmerged branches? It is strange if branches have had no activity for a while and have not been merged to the default branch. Maybe commits have been cherry picked, or maybe the work has been postponed, but it also sometimes happen that someone simply forgets to merge the branch.

How to configure unmerged branches? To change how soon Quality-time should consider branches to be inactive, use the parameter “Number of days since last commit after which to consider branches inactive”.

What exactly is the default branch is configured in GitLab or Azure DevOps. If you want to use a different branch as default branch, you need to configure this in the source, see the documentation for GitLab or Azure DevOps.

Default target

≦ 0 branches

Scales

count

Default tags

ci

Supported subjects

• CI-environment

• Process

• Software

Supporting sources

• Azure DevOps Server

• GitLab

• Manual number

Unused CI-jobs

The number of continuous integration jobs that are unused.

Why measure unused CI-jobs? Removing unused, obsolete CI-jobs helps to keep a clear overview of the relevant CI-jobs.

Default target

≦ 0 CI-jobs

Scales

count

Default tags

ci

Supported subjects

• CI-environment

Supporting sources

• Azure DevOps Server

• GitLab

• Jenkins

• Manual number

User story points

The total number of points of a selection of user stories.

Why measure user story points? Keep track of the number of user story points so the team has sufficient ‘ready’ stories to plan the next sprint.

Default target

≧ 100 user story points

Scales

count

Default tags

process efficiency

Supported subjects

• Process

• Software

Supporting sources

• Azure DevOps Server

• Jira

• Manual number

Velocity

The average number of user story points delivered in recent sprints.

Why measure velocity? Keep track of the velocity so the team knows how many story points need at least be ‘ready’ to plan the next sprint

Default target

≧ 20 user story points per sprint

Scales

count

Default tags

process efficiency

Supported subjects

• Process

Supporting sources

• Jira

• Manual number

Violation remediation effort

The amount of effort it takes to remediate violations.

Why measure violation remediation effort? To better plan the work to remediate violations, it is helpful to have an estimate of the amount of effort it takes to remediate them.

See also

1. https://docs.sonarqube.org/latest/user-guide/metric-definitions/

Default target

≦ 60 minutes

Scales

count

Default tags

maintainability

Supported subjects

• Software

Supporting sources

• Manual number

• SonarQube

Violations

The number of violations of rules in the software.

Why measure violations? Depending on what kind of rules or guideliens the source checks, violations may pose a risk for different quality characteristics of the software. For example, the more programming rules are violated, the harder it may be to understand and maintain the software (1, 2). And the more accessibility guidelines are violated, the harder it may be for users to use the software (3).

See also

1. https://docs.sonarqube.org/latest/user-guide/rules/

2. https://martinfowler.com/bliki/CodeSmell.html

3. https://www.w3.org/standards/webdesign/accessibility

4. What is the difference between violations and warnings?

Default target

≦ 0 violations

Scales

count

Supported subjects

• Software

Supporting sources

• Axe CSV

• Axe-core

• Axe HTML reporter

• Manual number

• OJAudit

• SARIF

• SonarQube

Sources

This is an overview of all the sources that Quality-time can use to measure metrics. For each source, the metrics that the source can measure are listed. Also, a link to the source’s own documentation is provided.

Anchore

Anchore image scan analysis report in JSON format.

Supported metrics

• Security warnings

• Source up-to-dateness

See also

https://docs.anchore.com/current/docs/using/integration/ci_cd/inline_scanning/

Anchore Jenkins plugin

A Jenkins job with an Anchore report produced by the Anchore Jenkins plugin.

Supported metrics

• Security warnings

• Source up-to-dateness

See also

https://plugins.jenkins.io/anchore-container-scanner/

Axe CSV

An Axe accessibility report in CSV format.

Supported metrics

• Violations

See also

https://github.com/ICTU/axe-reports

Axe HTML reporter

Creates an HTML report from the axe-core library AxeResults object.

Supported metrics

• Violations

See also

https://www.npmjs.com/package/axe-html-reporter

Axe-core

Axe is an accessibility testing engine for websites and other HTML-based user interfaces.

Supported metrics

• Source up-to-dateness

• Source version

• Violations

See also

https://github.com/dequelabs/axe-core

Azure DevOps Server

Azure DevOps Server (formerly known as Team Foundation Server) by Microsoft provides source code management, reporting, requirements management, project management, automated builds, testing and release management.

Supported metrics

• Average issue lead time

• Change failure rate

• Failed CI-jobs

• Issues

• Job runs within time period

• Merge requests

• Source up-to-dateness

• Tests

• Unmerged branches

• Unused CI-jobs

• User story points

See also

https://azure.microsoft.com/en-us/services/devops/server/

Bandit

Bandit is a tool designed to find common security issues in Python code.

Supported metrics

• Security warnings

• Source up-to-dateness

See also

https://github.com/PyCQA/bandit

Calendar date

Specify a specific date in the past or the future. Can be used to, for example, warn when it is time for the next security test.

Supported metrics

• Source up-to-dateness

• Time remaining

Cargo Audit

Cargo Audit is a linter for Rust Cargo.lock files for crates.

Supported metrics

• Security warnings

See also

https://docs.rs/cargo-audit/latest/cargo_audit/

Checkmarx CxSAST

Static analysis software to identify security vulnerabilities in both custom code and open source components.

Supported metrics

• Security warnings

• Source up-to-dateness

• Source version

See also

https://checkmarx.com/glossary/static-application-security-testing-sast/

Cobertura

Cobertura is a free Java tool that calculates the percentage of code accessed by tests.

Supported metrics

• Source up-to-dateness

• Source version

• Test branch coverage

• Test line coverage

See also

https://cobertura.github.io/cobertura/

Cobertura Jenkins plugin

Jenkins plugin for Cobertura, a free Java tool that calculates the percentage of code accessed by tests.

Supported metrics

• Source up-to-dateness

• Test branch coverage

• Test line coverage

See also

https://plugins.jenkins.io/cobertura/

Composer

A Dependency Manager for PHP.

Supported metrics

• Dependencies

See also

https://getcomposer.org/

Dependency-Track

Dependency-Track is a component analysis platform that allows organizations to identify and reduce risk in the software supply chain.

Supported metrics

• Dependencies

• Security warnings

• Source up-to-dateness

• Source version

See also

https://dependencytrack.org/

Gatling

Gatling is an open-source load testing solution, designed for continuous load testing and development pipeline integration.

Supported metrics

• Performancetest duration

• Slow transactions

• Source up-to-dateness

• Source version

• Tests

See also

https://gatling.io/

GitLab

GitLab provides Git-repositories, wiki’s, issue-tracking and continuous integration/continuous deployment pipelines.

Note

Some metric sources are documents in JSON, XML, CSV, or HTML format. Examples include JUnit XML reports, JaCoCo XML reports and Axe CSV reports. Usually, you add a JUnit (or JaCoCo, or Axe…) source and then simply configure the same URL that you use to access the document via the browser. Unfortunately, this does not work if the document is stored in GitLab. In that case, you still use the JUnit (or JaCoCo, or Axe…) source, but provide a GitLab API URL as URL. Depending on where the document is stored in GitLab, there are two scenarios; the source is a build artifact of a GitLab CI pipeline, or the source is stored in a GitLab repository:

1. When the metric source is a build artifact of a GitLab CI pipeline, use URLs of the following format:

   https://<gitlab-server>/api/v4/projects/<project-id>/jobs/artifacts/<branch>/raw/<path>/<to>/<file-name>?job=<job-name>

   The project id can be found under the project’s general settings.

   If the repository is private, you also need to enter an personal access token with the scope read_api in the private token field.

2. When the metric source is a file stored in a GitLab repository, use URLs of the following format:

   https://<gitlab-server>/api/v4/projects/<project-id>/repository/files/<file-path-with-slashes-%2F-encoded>/raw?ref=<branch>

   The project id can be found under the project’s general settings.

   If the repository is private, you also need to enter an personal access token with the scope read_repository in the private token field.

Supported metrics

• Change failure rate

• Failed CI-jobs

• Job runs within time period

• Merge requests

• CI-pipeline duration

• Source up-to-dateness

• Source version

• Unmerged branches

• Unused CI-jobs

See also

https://about.gitlab.com/

Harbor

Harbor is an open source registry that secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted.

Supported metrics

• Security warnings

See also

https://goharbor.io/

Harbor JSON

Harbor is an open source registry that secures artifacts with policies and role-based access control, ensures images are scanned and free from vulnerabilities, and signs images as trusted. Use Harbor JSON as source for accessing vulnerability reports downloaded from the Harbor API in JSON format.

Supported metrics

• Security warnings

See also

https://goharbor.io/

JMeter CSV

Apache JMeter application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance.

Supported metrics

• Performancetest duration

• Slow transactions

• Source up-to-dateness

• Tests

See also

https://jmeter.apache.org/

JMeter JSON

Apache JMeter application is open source software, a 100% pure Java application designed to load test functional behavior and measure performance.

Supported metrics

• Slow transactions

• Tests

See also

https://jmeter.apache.org/

JSON file with security warnings

A generic vulnerability report with security warnings in JSON format.

In some cases, there are security vulnerabilities not found by automated tools. Quality-time has the ability to parse security warnings from JSON files with a generic format.

The JSON format consists of an object with one key vulnerabilities. The value should be a list of vulnerabilities. Each vulnerability is an object with three keys: title, description, and severity. The title and description values should be strings. The severity is also a string and can be either low, medium, or high.

Example generic JSON file:

{
    "vulnerabilities": [
        {
            "title": "ISO27001:2013 A9 Insufficient Access Control",
            "description": "The Application does not enforce Two-Factor Authentication and therefore does not satisfy                 security best practices.",
            "severity": "high"
        },
        {
            "title": "Threat Model Finding: Uploading Malicious Files",
            "description": "An attacker can upload malicious files with low privileges that can perform direct API                 calls and perform unwanted mutations or see unauthorized information.",
            "severity": "medium"
        }
    ]
}

Supported metrics

• Security warnings

JUnit XML report

Test reports in the JUnit XML format.

Supported metrics

• Source up-to-dateness

• Test cases

• Tests

See also

https://junit.org/junit5/

JaCoCo

JaCoCo is an open-source tool for measuring and reporting Java code coverage.

Supported metrics

• Source up-to-dateness

• Test branch coverage

• Test line coverage

See also

https://www.eclemma.org/jacoco/

JaCoCo Jenkins plugin

A Jenkins job with a JaCoCo coverage report produced by the JaCoCo Jenkins plugin.

Supported metrics

• Source up-to-dateness

• Test branch coverage

• Test line coverage

See also

https://plugins.jenkins.io/jacoco

Jenkins

Jenkins is an open source continuous integration/continuous deployment server.

Note

Some metric sources are documents in JSON, XML, CSV, or HTML format. Examples include JUnit XML reports, JaCoCo XML reports and Axe CSV reports. Usually, you add a JUnit (or JaCoCo, or Axe…) source and then simply configure the same URL that you use to access the document via the browser. If the document is stored in Jenkins and Quality-time needs to be authorized to access resources in Jenkins, there are two options:

1. Configure a Jenkins user and password. The username of the Jenkins user needs to be entered in the “Username” field and the password in the “Password” field.

2. Configure a Jenkins user and a private token of that user. The username of the Jenkins user needs to be entered in the “Username” field and the private token in the “Password” field.

Supported metrics

• Change failure rate

• Failed CI-jobs

• Job runs within time period

• Source up-to-dateness

• Source version

• Unused CI-jobs

See also

https://www.jenkins.io/

Jenkins test report

A Jenkins job with test results.

Supported metrics

• Source up-to-dateness

• Test cases

• Tests

See also

https://plugins.jenkins.io/junit

Jira

Jira is a proprietary issue tracker developed by Atlassian supporting bug tracking and agile project management.

Supported metrics

• Average issue lead time

• Change failure rate

• Issues

• Manual test duration

• Manual test execution

• Source version

• Test cases

• User story points

• Velocity

See also

https://www.atlassian.com/software/jira

Manual number

A number entered manually by a Quality-time user.

The manual number source supports all metric types that take a number as value. Because users have to keep the value up to date by hand, this source is only meant to be used as a temporary solution for when no automated source is available yet. For example, when the results of a security audit are only available in a PDF-report, a ‘security warnings’ metric can be added with the number of findings as manual number source.

Supported metrics

• Commented out code

• Complex units

• Dependencies

• Duplicated lines

• Failed CI-jobs

• Issues

• Job runs within time period

• Size (LOC)

• Long units

• Manual test duration

• Manual test execution

• Many parameters

• Merge requests

• Metrics

• Missing metrics

• Performancetest duration

• Performancetest stability

• CI-pipeline duration

• Violation remediation effort

• Scalability

• Security warnings

• Sentiment

• Slow transactions

• Suppressed violations

• Test cases

• Tests

• Todo and fixme comments

• Test branch coverage

• Test line coverage

• Unmerged branches

• Unused CI-jobs

• User story points

• Velocity

• Violations

NCover

A .NET code coverage solution.

Supported metrics

• Source up-to-dateness

• Test branch coverage

• Test line coverage

See also

https://www.ncover.com/

OJAudit

An Oracle JDeveloper program to audit Java code against JDeveloper’s audit rules.

Supported metrics

• Violations

See also

https://www.oracle.com/application-development/technologies/jdeveloper.html

OWASP Dependency-Check

OWASP Dependency-Check is a utility that identifies project dependencies and checks if there are any known, publicly disclosed, vulnerabilities.

Supported metrics

• Dependencies

• Security warnings

• Source up-to-dateness

• Source version

See also

https://owasp.org/www-project-dependency-check/

OWASP ZAP

The OWASP Zed Attack Proxy (ZAP) can help automatically find security vulnerabilities in web applications while the application is being developed and tested.

Supported metrics

• Security warnings

• Source up-to-dateness

• Source version

See also

https://owasp.org/www-project-zap/

OpenVAS

OpenVAS (Open Vulnerability Assessment System) is a software framework of several services and tools offering vulnerability scanning and vulnerability management.

Supported metrics

• Security warnings

• Source up-to-dateness

• Source version

See also

https://www.openvas.org/

Performancetest-runner

An open source tool to run performancetests and create performancetest reports.

Supported metrics

• Performancetest duration

• Performancetest stability

• Scalability

• Slow transactions

• Software version

• Source up-to-dateness

• Tests

See also

https://github.com/ICTU/performancetest-runner

Pyupio Safety

Safety checks Python dependencies for known security vulnerabilities.

Supported metrics

• Security warnings

See also

https://github.com/pyupio/safety

Quality-time

Quality report software for software development and maintenance.

Supported metrics

• Metrics

• Missing metrics

• Source up-to-dateness

• Source version

See also

https://github.com/ICTU/quality-time

Robot Framework

Robot Framework is a generic open source automation framework for acceptance testing, acceptance test driven development, and robotic process automation.

Supported metrics

• Source up-to-dateness

• Source version

• Test cases

• Tests

See also

https://robotframework.org/

Robot Framework Jenkins plugin

A Jenkins plugin for Robot Framework, a generic open source automation framework for acceptance testing, acceptance test driven development, and robotic process automation.

Supported metrics

• Source up-to-dateness

• Tests

See also

https://plugins.jenkins.io/robot/

SARIF

A Static Analysis Results Interchange Format (SARIF) vulnerability report in JSON format.

Supported metrics

• Security warnings

• Violations

See also

https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=sarif

Snyk

Snyk vulnerability report in JSON format.

Supported metrics

• Security warnings

See also

https://docs.snyk.io/products/snyk-code/cli-for-snyk-code/working-with-the-snyk-code-cli-results

SonarQube

SonarQube is an open-source platform for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs, code smells, and security vulnerabilities on 20+ programming languages.

Supported metrics

• Commented out code

• Complex units

• Duplicated lines

• Size (LOC)

• Long units

• Many parameters

• Violation remediation effort

• Security warnings

• Software version

• Source up-to-dateness

• Source version

• Suppressed violations

• Tests

• Todo and fixme comments

• Test branch coverage

• Test line coverage

• Violations

See also

https://www.sonarqube.org/

TestNG

Test reports in the TestNG XML format.

Supported metrics

• Source up-to-dateness

• Test cases

• Tests

See also

https://testng.org/

Trello

Trello is a collaboration tool that organizes projects into boards.

Supported metrics

• Issues

• Source up-to-dateness

See also

https://trello.com/

Trivy JSON

A Trivy vulnerability report in JSON format.

Supported metrics

• Security warnings

See also

https://aquasecurity.github.io/trivy/latest/docs/configuration/reporting/#json

cloc

cloc is an open-source tool for counting blank lines, comment lines, and physical lines of source code in many programming languages.

Supported metrics

• Size (LOC)

• Source version

See also

https://github.com/AlDanial/cloc

npm

npm is a package manager for the JavaScript programming language.

Supported metrics

• Dependencies

See also

https://docs.npmjs.com/

pip

pip is the package installer for Python. You can use pip to install packages from the Python Package Index and other indexes.

Supported metrics

• Dependencies

See also

https://pip.pypa.io/en/stable/

Metric-source combinations

This is an overview of all supported combinations of metrics and sources. For each combination of metric and source, the mandatory and optional parameters are listed that can be used to configure the source to measure the metric. If Quality-time needs to make certain assumptions about the source, for example which SonarQube rules to use to count long methods, then these assumptions are listed under ‘configurations’.

Average issue lead time from Azure DevOps Server

Azure DevOps Server can be used to measure average issue lead time.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Issue query in WIQL (Work Item Query Language). This should only contain the WHERE clause of a WIQL query, as the selected fields are static. For example, use the following clause to hide issues marked as done: “[System.State] <> ‘Done’”. See https://docs.microsoft.com/en-us/azure/devops/boards/queries/wiql-syntax?view=azure-devops.

• Number of days to look back for work items. Work items are selected if they are completed and have been updated within the number of days configured. The default value is: 90.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

Average issue lead time from Jira

Jira can be used to measure average issue lead time.

Mandatory parameters

• Issue query in JQL (Jira Query Language).

  See also

  https://support.atlassian.com/jira-software-cloud/docs/use-advanced-search-with-jira-query-language-jql/

• URL. URL of the Jira instance, with port if necessary. For example, ‘https://jira.example.org’.

Optional parameters

• Number of days to look back for selecting issues. Issues are selected if they are completed and have been updated within the number of days configured. The default value is: 90.

• Password for basic authentication.

• Private token.

  See also

  https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

• Username for basic authentication.

Change failure rate from Azure DevOps Server

Azure DevOps Server can be used to measure change failure rate.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Issue query in WIQL (Work Item Query Language). This should only contain the WHERE clause of a WIQL query, as the selected fields are static. For example, use the following clause to hide issues marked as done: “[System.State] <> ‘Done’”. See https://docs.microsoft.com/en-us/azure/devops/boards/queries/wiql-syntax?view=azure-devops.

• Number of days to look back for selecting pipeline runs. The default value is: 90.

• Number of days to look back for work items. Work items are selected if they are completed and have been updated within the number of days configured. The default value is: 90.

• Pipelines to ignore (regular expressions or pipeline names). Pipelines to ignore can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Pipelines to include (regular expressions or pipeline names). Pipelines to include can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

Change failure rate from Jenkins

Jenkins can be used to measure change failure rate.

Mandatory parameters

• URL. URL of the Jenkins instance, with port if necessary, but without path. For example, ‘https://jenkins.example.org’.

Optional parameters

• Jobs to ignore (regular expressions or job names). Jobs to ignore can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Jobs to include (regular expressions or job names). Jobs to include can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Number of days to look back for selecting job builds. The default value is: 90.

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Change failure rate from Jira

Jira can be used to measure change failure rate.

Mandatory parameters

• Issue query in JQL (Jira Query Language).

  See also

  https://support.atlassian.com/jira-software-cloud/docs/use-advanced-search-with-jira-query-language-jql/

• URL. URL of the Jira instance, with port if necessary. For example, ‘https://jira.example.org’.

Optional parameters

• Number of days to look back for selecting issues. Issues are selected if they are completed and have been updated within the number of days configured. The default value is: 90.

• Password for basic authentication.

• Private token.

  See also

  https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

• Username for basic authentication.

Change failure rate from GitLab

GitLab can be used to measure change failure rate.

Mandatory parameters

• GitLab instance URL. URL of the GitLab instance, with port if necessary, but without path. For example, ‘https://gitlab.com’.

• Project (name with namespace or id).

  See also

  https://docs.gitlab.com/ee/user/project/

Optional parameters

• Branches and tags to ignore (regular expressions, branch names or tag names).

  See also

  https://docs.gitlab.com/ee/user/project/repository/branches/

• Jobs to ignore (regular expressions or job names). Jobs to ignore can be specified by job name or by regular expression.

• Number of days to look back for selecting pipeline jobs. The default value is: 90.

• Private token (with read_api scope).

  See also

  https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Commented out code from Manual number

Manual number can be used to measure commented out code.

Mandatory parameters

• Number. The default value is: 0.

Commented out code from SonarQube

SonarQube can be used to measure commented out code.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Configurations:

• Rules used to detect commented out code:

  • abap:S125

  • c:S125

  • cpp:S125

  • csharpsquid:S125

  • flex:CommentedCode

  • java:S125

  • javascript:S125

  • kotlin:S125

  • objc:S125

  • php:S125

  • plsql:S125

  • python:S125

  • scala:S125

  • swift:S125

  • tsql:S125

  • typescript:S125

  • Web:AvoidCommentedOutCodeCheck

  • xml:S125

Complex units from Manual number

Manual number can be used to measure complex units.

Mandatory parameters

• Number. The default value is: 0.

Complex units from SonarQube

SonarQube can be used to measure complex units.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Configurations:

• Rules used to detect complex units:

  • abap:S1541

  • abap:S3776

  • c:S1541

  • c:S3776

  • cpp:S1541

  • cpp:S3776

  • csharpsquid:S1541

  • csharpsquid:S3776

  • flex:FunctionComplexity

  • go:S3776

  • java:S1541

  • java:S3776

  • javascript:S1541

  • javascript:S3776

  • kotlin:S3776

  • objc:S1541

  • objc:S3776

  • php:S1541

  • php:S3776

  • plsql:PlSql.FunctionAndProcedureComplexity

  • python:FunctionComplexity

  • python:S3776

  • ruby:S3776

  • scala:S3776

  • swift:S1541

  • swift:S3776

  • typescript:S1541

  • typescript:S3776

  • vbnet:S1541

  • vbnet:S3776

Dependencies from Composer

Composer can be used to measure dependencies.

Mandatory parameters

• URL to a Composer ‘outdated’ report in JSON format or to a zip with Composer ‘outdated’ reports in JSON format.

  See also

  https://getcomposer.org/doc/03-cli.md#outdated

Optional parameters

• Latest version statuses. Limit which latest version statuses to show. The status ‘safe update possible’ means that based on semantic versioning the update should be backwards compatible. This parameter is multiple choice. Possible statuses are: safe update possible, unknown, up-to-date, update possible. The default value is: all statuses.

• Password for basic authentication.

• Private token.

• URL to a Composer ‘outdated’ report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Composer ‘outdated’ report in JSON format.

• Username for basic authentication.

Dependencies from Dependency-Track

Dependency-Track can be used to measure dependencies.

Mandatory parameters

• URL of the Dependency-Track API. URL of the Dependency-Track API, with port if necessary, but without path. For example, ‘https://api.dependencytrack.example.org’.

Optional parameters

• API key.

  See also

  https://docs.dependencytrack.org/integrations/rest-api/

• Latest version statuses. Limit which latest version statuses to show. This parameter is multiple choice. Possible statuses are: unknown, up-to-date, update possible. The default value is: all statuses.

• URL of the Dependency-Track instance. URL of the Dependency-Track instance, with port if necessary, but without path. For example, ‘https://www.dependencytrack.example.org’. If provided, users clicking the source URL will visit this URL instead of the Dependency-Track API.

Dependencies from Manual number

Manual number can be used to measure dependencies.

Mandatory parameters

• Number. The default value is: 0.

Dependencies from npm

npm can be used to measure dependencies.

Tip

To generate the list of outdated dependencies, run:

npm outdated --all --long --json > npm-outdated.json

To run npm outdated with Docker, use:

docker run --rm -v "$SRC":/work -w /work node:lts npm outdated --all --long --json > outdated.json"

Mandatory parameters

• URL to a npm ‘outdated’ report in JSON format or to a zip with npm ‘outdated’ reports in JSON format.

  See also

  https://docs.npmjs.com/cli-commands/outdated.html

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a npm ‘outdated’ report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the npm ‘outdated’ report in JSON format.

• Username for basic authentication.

Dependencies from OWASP Dependency-Check

OWASP Dependency-Check can be used to measure dependencies.

Mandatory parameters

• URL to an OWASP Dependency-Check report in XML format or to a zip with OWASP Dependency-Check reports in XML format.

Optional parameters

• Parts of file paths to ignore (regular expressions). Parts of file paths to ignore can be specified by regular expression. The parts of file paths that match one or more of the regular expressions are removed. If, after applying the regular expressions, multiple warnings are the same only one is reported.

• Password for basic authentication.

• Private token.

• URL to an OWASP Dependency-Check report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OWASP Dependency-Check report in XML format.

• Username for basic authentication.

Dependencies from pip

pip can be used to measure dependencies.

Tip

To generate the list of outdated packages in JSON format, run:

python -m pip list --outdated --format=json > pip-outdated.json

Mandatory parameters

• URL to a pip ‘outdated’ report in JSON format or to a zip with pip ‘outdated’ reports in JSON format.

  See also

  https://pip.pypa.io/en/stable/cli/pip_list/

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a pip ‘outdated’ report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the pip ‘outdated’ report in JSON format.

• Username for basic authentication.

Duplicated lines from Manual number

Manual number can be used to measure duplicated lines.

Mandatory parameters

• Number. The default value is: 0.

Duplicated lines from SonarQube

SonarQube can be used to measure duplicated lines.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Failed CI-jobs from Azure DevOps Server

Azure DevOps Server can be used to measure failed ci-jobs.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Failure types. Limit which failure types to count as failed. This parameter is multiple choice. Possible failure types are: canceled, failed, no result, partially succeeded. The default value is: all failure types.

• Pipelines to ignore (regular expressions or pipeline names). Pipelines to ignore can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Pipelines to include (regular expressions or pipeline names). Pipelines to include can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

Failed CI-jobs from Jenkins

Jenkins can be used to measure failed ci-jobs.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL. URL of the Jenkins instance, with port if necessary, but without path. For example, ‘https://jenkins.example.org’.

Optional parameters

• Failure types. Limit which failure types to count as failed. This parameter is multiple choice. Possible failure types are: Aborted, Failure, Not built, Unstable. The default value is: all failure types.

• Jobs to ignore (regular expressions or job names). Jobs to ignore can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Jobs to include (regular expressions or job names). Jobs to include can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Failed CI-jobs from GitLab

GitLab can be used to measure failed ci-jobs.

Mandatory parameters

• GitLab instance URL. URL of the GitLab instance, with port if necessary, but without path. For example, ‘https://gitlab.com’.

• Project (name with namespace or id).

  See also

  https://docs.gitlab.com/ee/user/project/

Optional parameters

• Branches and tags to ignore (regular expressions, branch names or tag names).

  See also

  https://docs.gitlab.com/ee/user/project/repository/branches/

• Failure types. Limit which failure types to count as failed. This parameter is multiple choice. Possible failure types are: canceled, failed, skipped. The default value is: all failure types.

• Jobs to ignore (regular expressions or job names). Jobs to ignore can be specified by job name or by regular expression.

• Number of days to look back for selecting pipeline jobs. The default value is: 90.

• Private token (with read_api scope).

  See also

  https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Failed CI-jobs from Manual number

Manual number can be used to measure failed ci-jobs.

Mandatory parameters

• Number. The default value is: 0.

Issues from Azure DevOps Server

Azure DevOps Server can be used to measure issues.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Issue query in WIQL (Work Item Query Language). This should only contain the WHERE clause of a WIQL query, as the selected fields are static. For example, use the following clause to hide issues marked as done: “[System.State] <> ‘Done’”. See https://docs.microsoft.com/en-us/azure/devops/boards/queries/wiql-syntax?view=azure-devops.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

Issues from Jira

Jira can be used to measure issues.

Mandatory parameters

• Issue query in JQL (Jira Query Language).

  See also

  https://support.atlassian.com/jira-software-cloud/docs/use-advanced-search-with-jira-query-language-jql/

• URL. URL of the Jira instance, with port if necessary. For example, ‘https://jira.example.org’.

Optional parameters

• Password for basic authentication.

• Private token.

  See also

  https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

• Username for basic authentication.

Issues from Manual number

Manual number can be used to measure issues.

Mandatory parameters

• Number. The default value is: 0.

Issues from Trello

Trello can be used to measure issues.

Mandatory parameters

• Board (title or id).

  See also

  https://trello.com/1/members/me/boards?fields=name

• URL. The default value is: https://trello.com.

Optional parameters

• API key.

  See also

  https://trello.com/app-key

• Cards to count. This parameter is multiple choice. Possible cards are: inactive, overdue. The default value is: all cards.

• Lists to ignore (title or id).

• Number of days without activity after which to consider cards inactive. The default value is: 30.

• Token.

  See also

  https://trello.com/app-key

Job runs within time period from Azure DevOps Server

Azure DevOps Server can be used to measure job runs within time period.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Number of days to look back for selecting pipeline runs. The default value is: 90.

• Pipelines to ignore (regular expressions or pipeline names). Pipelines to ignore can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Pipelines to include (regular expressions or pipeline names). Pipelines to include can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

Job runs within time period from GitLab

GitLab can be used to measure job runs within time period.

Mandatory parameters

• GitLab instance URL. URL of the GitLab instance, with port if necessary, but without path. For example, ‘https://gitlab.com’.

• Project (name with namespace or id).

  See also

  https://docs.gitlab.com/ee/user/project/

Optional parameters

• Branches and tags to ignore (regular expressions, branch names or tag names).

  See also

  https://docs.gitlab.com/ee/user/project/repository/branches/

• Jobs to ignore (regular expressions or job names). Jobs to ignore can be specified by job name or by regular expression.

• Number of days to look back for selecting pipeline jobs. The default value is: 90.

• Private token (with read_api scope).

  See also

  https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Job runs within time period from Jenkins

Jenkins can be used to measure job runs within time period.

Mandatory parameters

• URL. URL of the Jenkins instance, with port if necessary, but without path. For example, ‘https://jenkins.example.org’.

Optional parameters

• Jobs to ignore (regular expressions or job names). Jobs to ignore can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Jobs to include (regular expressions or job names). Jobs to include can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Number of days to look back for selecting job builds. The default value is: 90.

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Job runs within time period from Manual number

Manual number can be used to measure job runs within time period.

Mandatory parameters

• Number. The default value is: 0.

Size (LOC) from cloc

cloc can be used to measure size (loc).

Mandatory parameters

• URL to a cloc report in JSON format or to a zip with cloc reports in JSON format.

Optional parameters

• Files to include (regular expressions or file names). Note that filtering files only works when the cloc report is generated with the –by-file option.

• Languages to ignore (regular expressions or language names).

  See also

  https://github.com/AlDanial/cloc#recognized-languages-

• Password for basic authentication.

• Private token.

• URL to a cloc report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the cloc report in JSON format.

• Username for basic authentication.

Size (LOC) from Manual number

Manual number can be used to measure size (loc).

Mandatory parameters

• Number. The default value is: 0.

Size (LOC) from SonarQube

SonarQube can be used to measure size (loc).

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Languages to ignore (regular expressions or language names).

  See also

  https://docs.sonarqube.org/latest/analysis/languages/overview/

• Lines to count. Either count all lines including lines with comments or only count lines with code, excluding comments. Note: it’s possible to ignore specific languages only when counting lines with code. This is a SonarQube limitation. This parameter is single choice. Possible lines to count are: all lines, lines with code. The default value is: lines with code.

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Long units from Manual number

Manual number can be used to measure long units.

Mandatory parameters

• Number. The default value is: 0.

Long units from SonarQube

SonarQube can be used to measure long units.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Configurations:

• Rules used to detect long units:

  • abap:S104

  • c:S104

  • c:S1151

  • c:S138

  • cpp:S104

  • cpp:S1151

  • cpp:S1188

  • cpp:S138

  • cpp:S6184

  • csharpsquid:S104

  • csharpsquid:S1151

  • csharpsquid:S138

  • flex:S1151

  • flex:S138

  • go:S104

  • go:S1151

  • go:S138

  • java:S104

  • java:S1151

  • java:S1188

  • java:S138

  • java:S2972

  • java:S5612

  • javascript:S104

  • javascript:S138

  • kotlin:S104

  • kotlin:S1151

  • kotlin:S138

  • kotlin:S5612

  • objc:S104

  • objc:S1151

  • objc:S138

  • php:S104

  • php:S1151

  • php:S138

  • php:S2042

  • plsql:S104

  • plsql:S1151

  • python:S104

  • python:S138

  • ruby:S104

  • ruby:S1151

  • ruby:S138

  • scala:S104

  • scala:S1151

  • scala:S138

  • swift:S104

  • swift:S1151

  • swift:S1188

  • swift:S138

  • swift:S2042

  • tsql:S104

  • tsql:S1151

  • tsql:S138

  • typescript:S104

  • typescript:S138

  • vbnet:S104

  • vbnet:S1151

  • vbnet:S138

  • Web:FileLengthCheck

  • Web:LongJavaScriptCheck

Manual test duration from Jira

Jira can be used to measure manual test duration.

Mandatory parameters

• Issue query in JQL (Jira Query Language).

  See also

  https://support.atlassian.com/jira-software-cloud/docs/use-advanced-search-with-jira-query-language-jql/

• Manual test duration field (name or id).

  See also

  https://confluence.atlassian.com/jirakb/how-to-find-id-for-custom-field-s-744522503.html

• URL. URL of the Jira instance, with port if necessary. For example, ‘https://jira.example.org’.

Optional parameters

• Password for basic authentication.

• Private token.

  See also

  https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

• Username for basic authentication.

Manual test duration from Manual number

Manual number can be used to measure manual test duration.

Mandatory parameters

• Number. The default value is: 0.

Manual test execution from Jira

Jira can be used to measure manual test execution.

Mandatory parameters

• Default expected manual test execution frequency (days). Specify how often the manual tests should be executed. For example, if the sprint length is three weeks, manual tests should be executed at least once every 21 days. The default value is: 21.

• Issue query in JQL (Jira Query Language).

  See also

  https://support.atlassian.com/jira-software-cloud/docs/use-advanced-search-with-jira-query-language-jql/

• URL. URL of the Jira instance, with port if necessary. For example, ‘https://jira.example.org’.

Optional parameters

• Manual test execution frequency field (name or id).

  See also

  https://confluence.atlassian.com/jirakb/how-to-find-id-for-custom-field-s-744522503.html

• Password for basic authentication.

• Private token.

  See also

  https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

• Username for basic authentication.

Manual test execution from Manual number

Manual number can be used to measure manual test execution.

Mandatory parameters

• Number. The default value is: 0.

Many parameters from Manual number

Manual number can be used to measure many parameters.

Mandatory parameters

• Number. The default value is: 0.

Many parameters from SonarQube

SonarQube can be used to measure many parameters.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Configurations:

• Rules used to detect units with many parameters:

  • c:S107

  • cpp:S107

  • csharpsquid:S107

  • csharpsquid:S2436

  • flex:S107

  • go:S107

  • java:S107

  • javascript:S107

  • kotlin:S107

  • objc:S107

  • php:S107

  • python:S107

  • ruby:S107

  • scala:S107

  • swift:S107

  • tsql:S107

  • typescript:S107

  • vbnet:S107

Merge requests from Azure DevOps Server

Azure DevOps Server can be used to measure merge requests.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Merge request states. Limit which merge request states to count. This parameter is multiple choice. Possible states are: abandoned, active, completed, not set. The default value is: all states.

• Minimum number of upvotes. Only count merge requests with fewer than the minimum number of upvotes. The default value is: 0.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

• Repository (name or id).

• Target branches to include (regular expressions or branch names).

  See also

  https://docs.microsoft.com/en-us/azure/devops/repos/git/manage-your-branches?view=azure-devops

Merge requests from GitLab

GitLab can be used to measure merge requests.

Mandatory parameters

• GitLab instance URL. URL of the GitLab instance, with port if necessary, but without path. For example, ‘https://gitlab.com’.

• Project (name with namespace or id).

  See also

  https://docs.gitlab.com/ee/user/project/

Optional parameters

• Approval states to include (requires GitLab Premium). This parameter is multiple choice. Possible approval states are: approved, not approved, unknown. The default value is: all approval states.

  See also

  https://docs.gitlab.com/ee/user/project/merge_requests/approvals/

• Merge request states. Limit which merge request states to count. This parameter is multiple choice. Possible states are: closed, locked, merged, opened. The default value is: all states.

• Minimum number of upvotes. Only count merge requests with fewer than the minimum number of upvotes. The default value is: 0.

• Private token (with read_api scope).

  See also

  https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

• Target branches to include (regular expressions or branch names).

  See also

  https://docs.gitlab.com/ee/user/project/repository/branches/

Merge requests from Manual number

Manual number can be used to measure merge requests.

Mandatory parameters

• Number. The default value is: 0.

Metrics from Manual number

Manual number can be used to measure metrics.

Mandatory parameters

• Number. The default value is: 0.

Metrics from Quality-time

Quality-time can be used to measure metrics.

Mandatory parameters

• Quality-time URL. URL of the Quality-time instance, with port if necessary, but without path. For example, ‘https://quality-time.example.org’.

Optional parameters

• Metric statuses. This parameter is multiple choice. Possible metric statuses are: informative (blue), near target met (yellow), target met (green), target not met (red), technical debt target met (grey), unknown (white). The default value is: all metric statuses.

• Metric types. If provided, only count metrics with the selected metric types. This parameter is multiple choice. Possible metric types are: Average issue lead time, CI-pipeline duration, Change failure rate, Commented out code, Complex units, Dependencies, Duplicated lines, Failed CI-jobs, Issues, Job runs within time period, Long units, Manual test duration, Manual test execution, Many parameters, Merge requests, Metrics, Missing metrics, Performancetest duration, Performancetest stability, Scalability, Security warnings, Sentiment, Size (LOC), Slow transactions, Software version, Source up-to-dateness, Source version, Suppressed violations, Test branch coverage, Test cases, Test line coverage, Tests, Time remaining, Todo and fixme comments, Unmerged branches, Unused CI-jobs, User story points, Velocity, Violation remediation effort, Violations. The default value is: all metric types.

• Minimum metric status duration. Only count metrics whose status has not changed for the given number of days. The default value is: 0.

• Report names or identifiers.

• Source types. If provided, only count metrics with one or more sources of the selected source types. This parameter is multiple choice. Possible source types are: Anchore Jenkins plugin, Anchore, Axe CSV, Axe HTML reporter, Axe-core, Azure DevOps Server, Bandit, Calendar date, Cargo Audit, Checkmarx CxSAST, Cobertura Jenkins plugin, Cobertura, Composer, Dependency-Track, Gatling, GitLab, Harbor JSON, Harbor, JMeter CSV, JMeter JSON, JSON file with security warnings, JUnit XML report, JaCoCo Jenkins plugin, JaCoCo, Jenkins test report, Jenkins, Jira, Manual number, NCover, OJAudit, OWASP Dependency-Check, OWASP ZAP, OpenVAS, Performancetest-runner, Pyupio Safety, Quality-time, Robot Framework Jenkins plugin, Robot Framework, SARIF, Snyk, SonarQube, TestNG, Trello, Trivy JSON, cloc, npm, pip. The default value is: all source types.

• Tags. If provided, only count metrics with one ore more of the given tags.

Missing metrics from Manual number

Manual number can be used to measure missing metrics.

Mandatory parameters

• Number. The default value is: 0.

Missing metrics from Quality-time

Quality-time can be used to measure missing metrics.

Mandatory parameters

• Quality-time URL. URL of the Quality-time instance, with port if necessary, but without path. For example, ‘https://quality-time.example.org’.

Optional parameters

• Report names or identifiers.

• Subjects to ignore (subject names or identifiers). The Quality-time missing metrics collector will ignore metrics that are missing in the list of subjects to ignore.

Performancetest duration from Gatling

Gatling can be used to measure performancetest duration.

Mandatory parameters

• URL to a Gatling report in HTML format or to a zip with Gatling reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Performancetest duration from JMeter CSV

JMeter CSV can be used to measure performancetest duration.

Mandatory parameters

• URL to a JMeter report in CSV format or to a zip with JMeter reports in CSV format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a JMeter report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JMeter report in CSV format.

• Username for basic authentication.

Performancetest duration from Manual number

Manual number can be used to measure performancetest duration.

Mandatory parameters

• Number. The default value is: 0.

Performancetest duration from Performancetest-runner

Performancetest-runner can be used to measure performancetest duration.

Mandatory parameters

• URL to a Performancetest-runner report in HTML format or to a zip with Performancetest-runner reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Performancetest stability from Manual number

Manual number can be used to measure performancetest stability.

Mandatory parameters

• Number. The default value is: 0.

Performancetest stability from Performancetest-runner

Performancetest-runner can be used to measure performancetest stability.

Mandatory parameters

• URL to a Performancetest-runner report in HTML format or to a zip with Performancetest-runner reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

CI-pipeline duration from Manual number

Manual number can be used to measure ci-pipeline duration.

Mandatory parameters

• Number. The default value is: 0.

CI-pipeline duration from GitLab

GitLab can be used to measure ci-pipeline duration.

Mandatory parameters

• GitLab instance URL. URL of the GitLab instance, with port if necessary, but without path. For example, ‘https://gitlab.com’.

• Project (name with namespace or id).

  See also

  https://docs.gitlab.com/ee/user/project/

Optional parameters

• Branches (regular expressions or branch names).

  See also

  https://docs.gitlab.com/ee/user/project/repository/branches/

• Number of days to look back for selecting pipelines. The default value is: 7.

• Pipeline statuses to include. This parameter is multiple choice. Possible pipeline statuses are: canceled, created, failed, manual, pending, preparing, running, scheduled, skipped, success, waiting for resource. The default value is: all pipeline statuses.

• Pipeline triggers to include. This parameter is multiple choice. Possible pipeline triggers are: api, chat, external pull request event, external, merge request event, ondemand DAST scan, ondemand DAST validation, parent pipeline, pipeline, push, schedule, trigger, web-IDE, web. The default value is: all pipeline triggers.

• Private token (with read_api scope).

  See also

  https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Violation remediation effort from Manual number

Manual number can be used to measure violation remediation effort.

Mandatory parameters

• Number. The default value is: 0.

Violation remediation effort from SonarQube

SonarQube can be used to measure violation remediation effort.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

• Types of effort. This parameter is multiple choice. Possible effort types are: effort to fix all bug issues, effort to fix all code smells, effort to fix all vulnerabilities. The default value is: all effort types.

  See also

  https://docs.sonarqube.org/latest/user-guide/metric-definitions/

Scalability from Manual number

Manual number can be used to measure scalability.

Mandatory parameters

• Number. The default value is: 0.

Scalability from Performancetest-runner

Performancetest-runner can be used to measure scalability.

Mandatory parameters

• URL to a Performancetest-runner report in HTML format or to a zip with Performancetest-runner reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Security warnings from Anchore

Anchore can be used to measure security warnings.

Mandatory parameters

• URL to an Anchore vulnerability report in JSON format or to a zip with Anchore vulnerability reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: Critical, High, Low, Medium, Negligible, Unknown. The default value is: all severities.

• URL to an Anchore vulnerability report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Anchore vulnerability report in JSON format.

• Username for basic authentication.

Security warnings from Anchore Jenkins plugin

Anchore Jenkins plugin can be used to measure security warnings.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with an Anchore report generated by the Anchore plugin. For example, ‘https://jenkins.example.org/job/anchore’ or https://jenkins.example.org/job/anchore/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: Critical, High, Low, Medium, Negligible, Unknown. The default value is: all severities.

• Username for basic authentication.

Security warnings from Bandit

Bandit can be used to measure security warnings.

Mandatory parameters

• URL to a Bandit report in JSON format or to a zip with Bandit reports in JSON format.

Optional parameters

• Confidence levels. If provided, only count security warnings with the selected confidence levels. This parameter is multiple choice. Possible confidence levels are: high, low, medium. The default value is: all confidence levels.

• Password for basic authentication.

• Private token.

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: high, low, medium. The default value is: all severities.

• URL to a Bandit report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Bandit report in JSON format.

• Username for basic authentication.

Security warnings from Cargo Audit

Cargo Audit can be used to measure security warnings.

Mandatory parameters

• URL to a Cargo Audit report in JSON format or to a zip with Cargo Audit reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Cargo Audit report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Cargo Audit report in JSON format.

• Username for basic authentication.

• Warning types. If provided, only count security warnings with the selected warning types. This parameter is multiple choice. Possible warning types are: unsound, vulnerability, yanked. The default value is: all warning types.

Security warnings from Checkmarx CxSAST

Checkmarx CxSAST can be used to measure security warnings.

Mandatory parameters

• Password for basic authentication.

• Project (name or id).

• URL. URL of the Checkmarx instance, with port if necessary, but without path. For example ‘https://checkmarx.example.org’.

• Username for basic authentication.

Optional parameters

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: high, info, low, medium. The default value is: all severities.

Security warnings from Dependency-Track

Dependency-Track can be used to measure security warnings.

Mandatory parameters

• URL of the Dependency-Track API. URL of the Dependency-Track API, with port if necessary, but without path. For example, ‘https://api.dependencytrack.example.org’.

Optional parameters

• API key.

  See also

  https://docs.dependencytrack.org/integrations/rest-api/

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: Critical, High, Info, Low, Medium, Unassigned. The default value is: all severities.

• URL of the Dependency-Track instance. URL of the Dependency-Track instance, with port if necessary, but without path. For example, ‘https://www.dependencytrack.example.org’. If provided, users clicking the source URL will visit this URL instead of the Dependency-Track API.

Security warnings from JSON file with security warnings

JSON file with security warnings can be used to measure security warnings.

Mandatory parameters

• URL to a generic vulnerability report in JSON format or to a zip with generic vulnerability reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: high, low, medium. The default value is: all severities.

• URL to a generic vulnerability report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the generic vulnerability report in JSON format.

• Username for basic authentication.

Security warnings from Harbor

Harbor can be used to measure security warnings.

Mandatory parameters

• URL. URL of the Harbor instance, with port if necessary, but without path. For example ‘https://demo.goharbor.io’.

Optional parameters

• Password for basic authentication.

• Projects to ignore (regular expressions or project names). Projects to ignore can be specified by project name or by regular expression.

• Projects to include (regular expressions or project names). Projects to include can be specified by project name or by regular expression.

• Repositories to ignore (regular expressions or repository names). Repositories to ignore can be specified by repository name or by regular expression.

• Repositories to include (regular expressions or repository names). Repositories to include can be specified by repository name or by regular expression.

• Username for basic authentication.

Security warnings from Harbor JSON

Harbor JSON can be used to measure security warnings.

Mandatory parameters

• URL to a Harbor vulnerability report in JSON format or to a zip with Harbor vulnerability reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: critical, high, low, medium. The default value is: all severities.

• URL to a Harbor vulnerability report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Harbor vulnerability report in JSON format.

• Username for basic authentication.

Security warnings from Manual number

Manual number can be used to measure security warnings.

Mandatory parameters

• Number. The default value is: 0.

Security warnings from OpenVAS

OpenVAS can be used to measure security warnings.

Mandatory parameters

• URL to an OpenVAS report in XML format or to a zip with OpenVAS reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: high, log, low, medium. The default value is: all severities.

• URL to an OpenVAS report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OpenVAS report in XML format.

• Username for basic authentication.

Security warnings from OWASP Dependency-Check

OWASP Dependency-Check can be used to measure security warnings.

Mandatory parameters

• URL to an OWASP Dependency-Check report in XML format or to a zip with OWASP Dependency-Check reports in XML format.

Optional parameters

• Parts of file paths to ignore (regular expressions). Parts of file paths to ignore can be specified by regular expression. The parts of file paths that match one or more of the regular expressions are removed. If, after applying the regular expressions, multiple warnings are the same only one is reported.

• Password for basic authentication.

• Private token.

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: critical, high, low, medium, moderate. The default value is: all severities.

• URL to an OWASP Dependency-Check report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OWASP Dependency-Check report in XML format.

• Username for basic authentication.

Security warnings from OWASP ZAP

OWASP ZAP can be used to measure security warnings.

Mandatory parameters

• URL to an OWASP ZAP report in XML format or to a zip with OWASP ZAP reports in XML format.

Optional parameters

• Count alert types or alert instances. Determine whether to count each alert type in the OWASP ZAP report as a security warning or to count each alert instance (URL). This parameter is single choice. Possible count alert types or instances setting are: alert instances, alert types. The default value is: alert instances.

• Parts of URLs to ignore (regular expressions). Parts of URLs to ignore can be specified by regular expression. The parts of URLs that match one or more of the regular expressions are removed. If, after applying the regular expressions, multiple warnings are the same only one is reported.

• Password for basic authentication.

• Private token.

• Risks. If provided, only count security warnings with the selected risks. This parameter is multiple choice. Possible risks are: high, informational, low, medium. The default value is: all risks.

• URL to an OWASP ZAP report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OWASP ZAP report in XML format.

• Username for basic authentication.

Security warnings from Pyupio Safety

Pyupio Safety can be used to measure security warnings.

Mandatory parameters

• URL to a Safety report in JSON format or to a zip with Safety reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Safety report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Safety report in JSON format.

• Username for basic authentication.

Security warnings from SARIF

SARIF can be used to measure security warnings.

Mandatory parameters

• URL to a SARIF vulnerability report in JSON format or to a zip with SARIF vulnerability reports in JSON format.

Optional parameters

• Levels. If provided, only count entities with the selected levels. This parameter is multiple choice. Possible levels are: error, none, note, warning. The default value is: all levels.

• Password for basic authentication.

• Private token.

• URL to a SARIF vulnerability report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the SARIF vulnerability report in JSON format.

• Username for basic authentication.

Security warnings from Snyk

Snyk can be used to measure security warnings.

Mandatory parameters

• URL to a Snyk vulnerability report in JSON format or to a zip with Snyk vulnerability reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• Severities. If provided, only count security warnings with the selected severities. This parameter is multiple choice. Possible severities are: high, low, medium. The default value is: all severities.

• URL to a Snyk vulnerability report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Snyk vulnerability report in JSON format.

• Username for basic authentication.

Security warnings from SonarQube

SonarQube can be used to measure security warnings.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

• Security hotspot review priorities. This parameter is multiple choice. Possible review priorities are: high, low, medium. The default value is: all review priorities.

  See also

  https://docs.sonarqube.org/latest/user-guide/security-hotspots/

• Security hotspot statuses. This parameter is multiple choice. Possible hotspot statuses are: acknowledged, fixed, safe, to review. The default value is: acknowledged, to review.

  See also

  https://docs.sonarqube.org/latest/user-guide/security-hotspots/

• Security issue types (measuring security hotspots requires SonarQube 8.2 or newer). This parameter is multiple choice. Possible types are: security_hotspot, vulnerability. The default value is: vulnerability.

  See also

  https://docs.sonarqube.org/latest/user-guide/rules/

• Severities. This parameter is multiple choice. Possible severities are: blocker, critical, info, major, minor. The default value is: all severities.

  See also

  https://docs.sonarqube.org/latest/user-guide/issues/

• Tags to include.

  See also

  https://docs.sonarsource.com/sonarqube/latest/user-guide/issues/

Security warnings from Trivy JSON

Trivy JSON can be used to measure security warnings.

Mandatory parameters

• URL to a Trivy vulnerability report in JSON format or to a zip with Trivy vulnerability reports in JSON format.

Optional parameters

• Levels. If provided, only count security warnings with the selected levels. This parameter is multiple choice. Possible levels are: critical, high, low, medium, unknown. The default value is: all levels.

• Password for basic authentication.

• Private token.

• URL to a Trivy vulnerability report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Trivy vulnerability report in JSON format.

• Username for basic authentication.

Sentiment from Manual number

Manual number can be used to measure sentiment.

Mandatory parameters

• Number. The default value is: 0.

Slow transactions from Gatling

Gatling can be used to measure slow transactions.

Mandatory parameters

• URL to a Gatling report in HTML format or to a zip with Gatling reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Response time type to evaluate against the target response time. Which response time type to compare with the target response time to determine slow transactions. This parameter is single choice. Possible response time types to evaluate are: 50th percentile, 75th percentile, 95th percentile, 99th percentile, maximum, mean, minimum. The default value is: 95th percentile.

• Target response time. The response times of the transactions should be less than or equal to the target response time. The default value is: 1000.

• Transaction-specific target response times (regular expressions or transaction names:target response time). Transactions-specific target responses times (in milliseconds) can be specified by transaction name or by regular expression, separated from the target response time by a colon, e.g.: ‘/api/v?/search/.*:1500’.

• Transactions to ignore (regular expressions or transaction names). Transactions to ignore can be specified by transaction name or by regular expression.

• Transactions to include (regular expressions or transaction names). Transactions to include can be specified by transaction name or by regular expression.

• Username for basic authentication.

Slow transactions from Manual number

Manual number can be used to measure slow transactions.

Mandatory parameters

• Number. The default value is: 0.

Slow transactions from JMeter CSV

JMeter CSV can be used to measure slow transactions.

Mandatory parameters

• URL to a JMeter report in CSV format or to a zip with JMeter reports in CSV format.

Optional parameters

• Password for basic authentication.

• Private token.

• Response time type to evaluate against the target response time. Which response time type to compare with the target response time to determine slow transactions. This parameter is single choice. Possible response time types to evaluate are: 90th percentile, 95th percentile, 99th percentile, maximum, mean, median, minimum. The default value is: 90th percentile.

• Target response time. The response times of the transactions should be less than or equal to the target response time. The default value is: 1000.

• Transaction-specific target response times (regular expressions or transaction names:target response time). Transactions-specific target responses times (in milliseconds) can be specified by transaction name or by regular expression, separated from the target response time by a colon, e.g.: ‘/api/v?/search/.*:1500’.

• Transactions to ignore (regular expressions or transaction names). Transactions to ignore can be specified by transaction name or by regular expression.

• Transactions to include (regular expressions or transaction names). Transactions to include can be specified by transaction name or by regular expression.

• URL to a JMeter report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JMeter report in CSV format.

• Username for basic authentication.

Slow transactions from JMeter JSON

JMeter JSON can be used to measure slow transactions.

Mandatory parameters

• URL to a JMeter report in JSON format or to a zip with JMeter reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• Response time type to evaluate against the target response time. Which response time type to compare with the target response time to determine slow transactions. This parameter is single choice. Possible response time types to evaluate are: 90th percentile, 95th percentile, 99th percentile, maximum, mean, median, minimum. The default value is: 90th percentile.

• Target response time. The response times of the transactions should be less than or equal to the target response time. The default value is: 1000.

• Transaction-specific target response times (regular expressions or transaction names:target response time). Transactions-specific target responses times (in milliseconds) can be specified by transaction name or by regular expression, separated from the target response time by a colon, e.g.: ‘/api/v?/search/.*:1500’.

• Transactions to ignore (regular expressions or transaction names). Transactions to ignore can be specified by transaction name or by regular expression.

• Transactions to include (regular expressions or transaction names). Transactions to include can be specified by transaction name or by regular expression.

• URL to a JMeter report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JMeter report in JSON format.

• Username for basic authentication.

Slow transactions from Performancetest-runner

Performancetest-runner can be used to measure slow transactions.

Mandatory parameters

• URL to a Performancetest-runner report in HTML format or to a zip with Performancetest-runner reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Thresholds. If provided, only count transactions that surpass the selected thresholds. This parameter is multiple choice. Possible thresholds are: high, warning. The default value is: all thresholds.

• Transactions to ignore (regular expressions or transaction names). Transactions to ignore can be specified by transaction name or by regular expression.

• Transactions to include (regular expressions or transaction names). Transactions to include can be specified by transaction name or by regular expression.

• Username for basic authentication.

Software version from Performancetest-runner

Performancetest-runner can be used to measure software version.

Mandatory parameters

• URL to a Performancetest-runner report in HTML format or to a zip with Performancetest-runner reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Software version from SonarQube

SonarQube can be used to measure software version.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Source up-to-dateness from Anchore

Anchore can be used to measure source up-to-dateness.

Mandatory parameters

• URL to an Anchore details report in JSON format or to a zip with Anchore details reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to an Anchore vulnerability report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Anchore vulnerability report in JSON format.

• Username for basic authentication.

Source up-to-dateness from Anchore Jenkins plugin

Anchore Jenkins plugin can be used to measure source up-to-dateness.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with an Anchore report generated by the Anchore plugin. For example, ‘https://jenkins.example.org/job/anchore’ or https://jenkins.example.org/job/anchore/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Source up-to-dateness from Axe-core

Axe-core can be used to measure source up-to-dateness.

When running Axe-core on a webpage, the run function returns a results object. The results objects may be stored in separate JSON files and served to Quality-time in a zipfile, or the results objects can be combined in one JSON file that contains a list of results objects.

Axe-core adds a timestamp field to each results object. That field is used by Quality-time to determine the up-to-dateness of the report. If there is more than one results object in the JSON file, Quality-time uses the first one it encounters, assuming that all timestamps in one JSON file will be roughly equal.

Tip

When combining results objects, make sure the timestamp field is retained in the JSON.

Mandatory parameters

• URL to an Axe-core report in JSON format or to a zip with Axe-core reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to an Axe-core report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Axe-core report in JSON format.

• Username for basic authentication.

Source up-to-dateness from Azure DevOps Server

Azure DevOps Server can be used to measure source up-to-dateness.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Branch. The default value is: master.

• File or folder path. Use the date and time the path was last changed to determine the up-to-dateness. If no path is specified, the pipeline is used to determine the up-to-dateness.

• Pipelines to ignore (regular expressions or pipeline names). Pipelines to ignore can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Pipelines to include (regular expressions or pipeline names). Pipelines to include can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

• Repository (name or id).

Source up-to-dateness from Bandit

Bandit can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a Bandit report in JSON format or to a zip with Bandit reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Bandit report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Bandit report in JSON format.

• Username for basic authentication.

Source up-to-dateness from Calendar date

Calendar date can be used to measure source up-to-dateness.

Mandatory parameters

• Date. The default value is: 2024-05-01.

Source up-to-dateness from Cobertura

Cobertura can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a Cobertura report in XML format or to a zip with Cobertura reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Cobertura report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Cobertura report in XML format.

• Username for basic authentication.

Source up-to-dateness from Cobertura Jenkins plugin

Cobertura Jenkins plugin can be used to measure source up-to-dateness.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a coverage report generated by the Cobertura plugin. For example, ‘https://jenkins.example.org/job/cobertura’ or https://jenkins.example.org/job/cobertura/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Source up-to-dateness from Checkmarx CxSAST

Checkmarx CxSAST can be used to measure source up-to-dateness.

Mandatory parameters

• Password for basic authentication.

• Project (name or id).

• URL. URL of the Checkmarx instance, with port if necessary, but without path. For example ‘https://checkmarx.example.org’.

• Username for basic authentication.

Source up-to-dateness from Dependency-Track

Dependency-Track can be used to measure source up-to-dateness.

Mandatory parameters

• URL of the Dependency-Track API. URL of the Dependency-Track API, with port if necessary, but without path. For example, ‘https://api.dependencytrack.example.org’.

Optional parameters

• API key.

  See also

  https://docs.dependencytrack.org/integrations/rest-api/

• URL of the Dependency-Track instance. URL of the Dependency-Track instance, with port if necessary, but without path. For example, ‘https://www.dependencytrack.example.org’. If provided, users clicking the source URL will visit this URL instead of the Dependency-Track API.

Source up-to-dateness from Gatling

Gatling can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a Gatling report in HTML format or to a zip with Gatling reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Source up-to-dateness from GitLab

GitLab can be used to measure source up-to-dateness.

Mandatory parameters

• GitLab instance URL. URL of the GitLab instance, with port if necessary, but without path. For example, ‘https://gitlab.com’.

• Project (name with namespace or id).

  See also

  https://docs.gitlab.com/ee/user/project/

Optional parameters

• Branch. The default value is: master.

  See also

  https://docs.gitlab.com/ee/user/project/repository/branches/

• File or folder path. Use the date and time the path was last changed to determine the up-to-dateness. If no path is specified, the pipeline is used to determine the up-to-dateness.

• Number of days to look back for selecting pipeline jobs. The default value is: 90.

• Pipeline statuses to include. This parameter is multiple choice. Possible pipeline statuses are: canceled, created, failed, manual, pending, preparing, running, scheduled, skipped, success, waiting for resource. The default value is: all pipeline statuses.

• Pipeline triggers to include. This parameter is multiple choice. Possible pipeline triggers are: api, chat, external pull request event, external, merge request event, ondemand DAST scan, ondemand DAST validation, parent pipeline, pipeline, push, schedule, trigger, web-IDE, web. The default value is: all pipeline triggers.

• Private token (with read_api scope).

  See also

  https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Source up-to-dateness from JaCoCo

JaCoCo can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a JaCoCo report in XML format or to a zip with JaCoCo reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a JaCoCo report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JaCoCo report in XML format.

• Username for basic authentication.

Source up-to-dateness from JaCoCo Jenkins plugin

JaCoCo Jenkins plugin can be used to measure source up-to-dateness.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a coverage report generated by the JaCoCo plugin. For example, ‘https://jenkins.example.org/job/jacoco’ or https://jenkins.example.org/job/jacoco/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Source up-to-dateness from Jenkins

Jenkins can be used to measure source up-to-dateness.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL. URL of the Jenkins instance, with port if necessary, but without path. For example, ‘https://jenkins.example.org’.

Optional parameters

• Build result types. Limit which build result types to include. This parameter is multiple choice. Possible result types are: Aborted, Failure, Not built, Success, Unstable. The default value is: all result types.

• Jobs to ignore (regular expressions or job names). Jobs to ignore can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Jobs to include (regular expressions or job names). Jobs to include can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Source up-to-dateness from Jenkins test report

Jenkins test report can be used to measure source up-to-dateness.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a test report generated by the JUnit plugin. For example, ‘https://jenkins.example.org/job/test’ or https://jenkins.example.org/job/test/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Source up-to-dateness from JMeter CSV

JMeter CSV can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a JMeter report in CSV format or to a zip with JMeter reports in CSV format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a JMeter report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JMeter report in CSV format.

• Username for basic authentication.

Source up-to-dateness from JUnit XML report

JUnit XML report can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a JUnit report in XML format or to a zip with JUnit reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a JUnit report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JUnit report in XML format.

• Username for basic authentication.

Source up-to-dateness from NCover

NCover can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a NCover report in HTML format or to a zip with NCover reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Source up-to-dateness from Robot Framework

Robot Framework can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a Robot Framework report in XML format or to a zip with Robot Framework reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Robot Framework report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Robot Framework report in XML format.

• Username for basic authentication.

Source up-to-dateness from OpenVAS

OpenVAS can be used to measure source up-to-dateness.

Mandatory parameters

• URL to an OpenVAS report in XML format or to a zip with OpenVAS reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to an OpenVAS report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OpenVAS report in XML format.

• Username for basic authentication.

Source up-to-dateness from OWASP Dependency-Check

OWASP Dependency-Check can be used to measure source up-to-dateness.

Mandatory parameters

• URL to an OWASP Dependency-Check report in XML format or to a zip with OWASP Dependency-Check reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to an OWASP Dependency-Check report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OWASP Dependency-Check report in XML format.

• Username for basic authentication.

Source up-to-dateness from OWASP ZAP

OWASP ZAP can be used to measure source up-to-dateness.

Mandatory parameters

• URL to an OWASP ZAP report in XML format or to a zip with OWASP ZAP reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to an OWASP ZAP report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OWASP ZAP report in XML format.

• Username for basic authentication.

Source up-to-dateness from Performancetest-runner

Performancetest-runner can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a Performancetest-runner report in HTML format or to a zip with Performancetest-runner reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Source up-to-dateness from Quality-time

Quality-time can be used to measure source up-to-dateness.

Mandatory parameters

• Quality-time URL. URL of the Quality-time instance, with port if necessary, but without path. For example, ‘https://quality-time.example.org’.

Optional parameters

• Report names or identifiers.

Source up-to-dateness from Robot Framework Jenkins plugin

Robot Framework Jenkins plugin can be used to measure source up-to-dateness.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a test report generated by the Robot Framework plugin. For example, ‘https://jenkins.example.org/job/robot’ or https://jenkins.example.org/job/robot/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Source up-to-dateness from SonarQube

SonarQube can be used to measure source up-to-dateness.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Source up-to-dateness from TestNG

TestNG can be used to measure source up-to-dateness.

Mandatory parameters

• URL to a TestNG report in XML format or to a zip with TestNG reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a TestNG report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the TestNG report in XML format.

• Username for basic authentication.

Source up-to-dateness from Trello

Trello can be used to measure source up-to-dateness.

Mandatory parameters

• Board (title or id).

  See also

  https://trello.com/1/members/me/boards?fields=name

• URL. The default value is: https://trello.com.

Optional parameters

• API key.

  See also

  https://trello.com/app-key

• Lists to ignore (title or id).

• Token.

  See also

  https://trello.com/app-key

Source version from Axe-core

Axe-core can be used to measure source version.

When running Axe-core on a webpage, the run function returns a results object. The results objects may be stored in separate JSON files and served to Quality-time in a zipfile, or the results objects can be combined in one JSON file that contains a list of results objects.

Axe-core adds a testEngine field to each results object. That field is used by Quality-time to determine the version of Axe-core used to generate the report. If there is more than one results object in the JSON file, Quality-time uses the first one it encounters, assuming that all test engines used in one JSON file will be equal.

Tip

When combining results objects, make sure the testEngine field is retained in the JSON.

Mandatory parameters

• URL to an Axe-core report in JSON format or to a zip with Axe-core reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to an Axe-core report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Axe-core report in JSON format.

• Username for basic authentication.

Source version from cloc

cloc can be used to measure source version.

Mandatory parameters

• URL to a cloc report in JSON format or to a zip with cloc reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a cloc report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the cloc report in JSON format.

• Username for basic authentication.

Source version from Cobertura

Cobertura can be used to measure source version.

Mandatory parameters

• URL to a Cobertura report in XML format or to a zip with Cobertura reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Cobertura report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Cobertura report in XML format.

• Username for basic authentication.

Source version from Checkmarx CxSAST

Checkmarx CxSAST can be used to measure source version.

Mandatory parameters

• Password for basic authentication.

• URL. URL of the Checkmarx instance, with port if necessary, but without path. For example ‘https://checkmarx.example.org’.

• Username for basic authentication.

Source version from Dependency-Track

Dependency-Track can be used to measure source version.

Mandatory parameters

• URL of the Dependency-Track API. URL of the Dependency-Track API, with port if necessary, but without path. For example, ‘https://api.dependencytrack.example.org’.

Optional parameters

• URL of the Dependency-Track instance. URL of the Dependency-Track instance, with port if necessary, but without path. For example, ‘https://www.dependencytrack.example.org’. If provided, users clicking the source URL will visit this URL instead of the Dependency-Track API.

Source version from Gatling

Gatling can be used to measure source version.

Mandatory parameters

• URL to a Gatling report in HTML format or to a zip with Gatling reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Source version from GitLab

GitLab can be used to measure source version.

Mandatory parameters

• GitLab instance URL. URL of the GitLab instance, with port if necessary, but without path. For example, ‘https://gitlab.com’.

Optional parameters

• Private token (with read_api scope).

  See also

  https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Source version from Jenkins

Jenkins can be used to measure source version.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL. URL of the Jenkins instance, with port if necessary, but without path. For example, ‘https://jenkins.example.org’.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Source version from Jira

Jira can be used to measure source version.

Mandatory parameters

• URL. URL of the Jira instance, with port if necessary. For example, ‘https://jira.example.org’.

Optional parameters

• Password for basic authentication.

• Private token.

  See also

  https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

• Username for basic authentication.

Source version from OpenVAS

OpenVAS can be used to measure source version.

Mandatory parameters

• URL to an OpenVAS report in XML format or to a zip with OpenVAS reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to an OpenVAS report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OpenVAS report in XML format.

• Username for basic authentication.

Source version from OWASP Dependency-Check

OWASP Dependency-Check can be used to measure source version.

Mandatory parameters

• URL to an OWASP Dependency-Check report in XML format or to a zip with OWASP Dependency-Check reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to an OWASP Dependency-Check report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OWASP Dependency-Check report in XML format.

• Username for basic authentication.

Source version from OWASP ZAP

OWASP ZAP can be used to measure source version.

Mandatory parameters

• URL to an OWASP ZAP report in XML format or to a zip with OWASP ZAP reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to an OWASP ZAP report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OWASP ZAP report in XML format.

• Username for basic authentication.

Source version from Quality-time

Quality-time can be used to measure source version.

Mandatory parameters

• Quality-time URL. URL of the Quality-time instance, with port if necessary, but without path. For example, ‘https://quality-time.example.org’.

Source version from Robot Framework

Robot Framework can be used to measure source version.

Mandatory parameters

• URL to a Robot Framework report in XML format or to a zip with Robot Framework reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Robot Framework report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Robot Framework report in XML format.

• Username for basic authentication.

Source version from SonarQube

SonarQube can be used to measure source version.

Mandatory parameters

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Suppressed violations from Manual number

Manual number can be used to measure suppressed violations.

Mandatory parameters

• Number. The default value is: 0.

Suppressed violations from SonarQube

SonarQube can be used to measure suppressed violations.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

• Severities. This parameter is multiple choice. Possible severities are: blocker, critical, info, major, minor. The default value is: all severities.

  See also

  https://docs.sonarqube.org/latest/user-guide/issues/

• Types. This parameter is multiple choice. Possible types are: bug, code_smell, vulnerability. The default value is: all types.

  See also

  https://docs.sonarqube.org/latest/user-guide/rules/

Configurations:

• Rules used to detect suppressed violations:

  • c:S1291

  • cpp:S1291

  • csharpsquid:S1309

  • java:NoSonar

  • java:S1309

  • java:S1310

  • java:S1315

  • objc:S1291

  • php:NoSonar

  • plsql:NoSonarCheck

  • python:NoSonar

  • tsql:NoSonar

Test cases from Jenkins test report

Jenkins test report can be used to measure test cases.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a test report generated by the JUnit plugin. For example, ‘https://jenkins.example.org/job/test’ or https://jenkins.example.org/job/test/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Test cases from Jira

Jira can be used to measure test cases.

Mandatory parameters

• Issue query in JQL (Jira Query Language).

  See also

  https://support.atlassian.com/jira-software-cloud/docs/use-advanced-search-with-jira-query-language-jql/

• URL. URL of the Jira instance, with port if necessary. For example, ‘https://jira.example.org’.

Optional parameters

• Password for basic authentication.

• Private token.

  See also

  https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: errored, failed, passed, skipped, untested. The default value is: all test results.

• Username for basic authentication.

Test cases from JUnit XML report

JUnit XML report can be used to measure test cases.

Mandatory parameters

• URL to a JUnit report in XML format or to a zip with JUnit reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a JUnit report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JUnit report in XML format.

• Username for basic authentication.

Test cases from Manual number

Manual number can be used to measure test cases.

Mandatory parameters

• Number. The default value is: 0.

Test cases from Robot Framework

Robot Framework can be used to measure test cases.

Mandatory parameters

• URL to a Robot Framework report in XML format or to a zip with Robot Framework reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Robot Framework report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Robot Framework report in XML format.

• Username for basic authentication.

Test cases from TestNG

TestNG can be used to measure test cases.

Mandatory parameters

• URL to a TestNG report in XML format or to a zip with TestNG reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a TestNG report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the TestNG report in XML format.

• Username for basic authentication.

Tests from Azure DevOps Server

Azure DevOps Server can be used to measure tests.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Names of test runs to include (regular expressions or test run names). Limit which test runs to include by test run name.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

• States of the test runs to include. Limit which test runs to include by test run state. This parameter is multiple choice. Possible test run states are: aborted, completed, in progress, not started. The default value is: all test run states.

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: failed, incomplete, not applicable, passed. The default value is: all test results.

Tests from Gatling

Gatling can be used to measure tests.

Mandatory parameters

• URL to a Gatling report in HTML format or to a zip with Gatling reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: failed, success. The default value is: all test results.

• Transactions to ignore (regular expressions or transaction names). Transactions to ignore can be specified by transaction name or by regular expression.

• Transactions to include (regular expressions or transaction names). Transactions to include can be specified by transaction name or by regular expression.

• Username for basic authentication.

Tests from Jenkins test report

Jenkins test report can be used to measure tests.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a test report generated by the JUnit plugin. For example, ‘https://jenkins.example.org/job/test’ or https://jenkins.example.org/job/test/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: failed, passed, skipped. The default value is: all test results.

• Username for basic authentication.

Tests from JMeter CSV

JMeter CSV can be used to measure tests.

Mandatory parameters

• URL to a JMeter report in CSV format or to a zip with JMeter reports in CSV format.

Optional parameters

• Password for basic authentication.

• Private token.

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: failed, success. The default value is: all test results.

• Transactions to ignore (regular expressions or transaction names). Transactions to ignore can be specified by transaction name or by regular expression.

• Transactions to include (regular expressions or transaction names). Transactions to include can be specified by transaction name or by regular expression.

• URL to a JMeter report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JMeter report in CSV format.

• Username for basic authentication.

Tests from JMeter JSON

JMeter JSON can be used to measure tests.

Mandatory parameters

• URL to a JMeter report in JSON format or to a zip with JMeter reports in JSON format.

Optional parameters

• Password for basic authentication.

• Private token.

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: failed, success. The default value is: all test results.

• Transactions to ignore (regular expressions or transaction names). Transactions to ignore can be specified by transaction name or by regular expression.

• Transactions to include (regular expressions or transaction names). Transactions to include can be specified by transaction name or by regular expression.

• URL to a JMeter report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JMeter report in JSON format.

• Username for basic authentication.

Tests from JUnit XML report

JUnit XML report can be used to measure tests.

Mandatory parameters

• URL to a JUnit report in XML format or to a zip with JUnit reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: errored, failed, passed, skipped. The default value is: all test results.

• URL to a JUnit report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JUnit report in XML format.

• Username for basic authentication.

Tests from Manual number

Manual number can be used to measure tests.

Mandatory parameters

• Number. The default value is: 0.

Tests from Performancetest-runner

Performancetest-runner can be used to measure tests.

Mandatory parameters

• URL to a Performancetest-runner report in HTML format or to a zip with Performancetest-runner reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: failed, success. The default value is: all test results.

• Transactions to ignore (regular expressions or transaction names). Transactions to ignore can be specified by transaction name or by regular expression.

• Transactions to include (regular expressions or transaction names). Transactions to include can be specified by transaction name or by regular expression.

• Username for basic authentication.

Tests from Robot Framework

Robot Framework can be used to measure tests.

Mandatory parameters

• URL to a Robot Framework report in XML format or to a zip with Robot Framework reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: fail, pass, skip. The default value is: all test results.

• URL to a Robot Framework report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Robot Framework report in XML format.

• Username for basic authentication.

Tests from Robot Framework Jenkins plugin

Robot Framework Jenkins plugin can be used to measure tests.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a test report generated by the Robot Framework plugin. For example, ‘https://jenkins.example.org/job/robot’ or https://jenkins.example.org/job/robot/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: fail, pass. The default value is: all test results.

• Username for basic authentication.

Tests from SonarQube

SonarQube can be used to measure tests.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: errored, failed, passed, skipped. The default value is: all test results.

Tests from TestNG

TestNG can be used to measure tests.

Mandatory parameters

• URL to a TestNG report in XML format or to a zip with TestNG reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Test results. Limit which test results to count. Note: depending on which results are selected, the direction of the metric may need to be adapted. For example, when counting passed tests, more is better, but when counting failed tests, fewer is better. This parameter is multiple choice. Possible test results are: failed, ignored, passed, skipped. The default value is: all test results.

• URL to a TestNG report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the TestNG report in XML format.

• Username for basic authentication.

Time remaining from Calendar date

Calendar date can be used to measure time remaining.

Mandatory parameters

• Date. The default value is: 2024-05-01.

Todo and fixme comments from Manual number

Manual number can be used to measure todo and fixme comments.

Mandatory parameters

• Number. The default value is: 0.

Todo and fixme comments from SonarQube

SonarQube can be used to measure todo and fixme comments.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Configurations:

• Rules used to detect todo and fixme comments:

  • c:S1134

  • c:S1135

  • cloudformation:S1135

  • cpp:S1134

  • cpp:S1135

  • csharpsquid:S1134

  • csharpsquid:S1135

  • docker:S1135

  • go:S1134

  • go:S1135

  • java:S1134

  • java:S1135

  • javascript:S1134

  • javascript:S1135

  • kotlin:S1134

  • kotlin:S1135

  • kubernetes:S1135

  • objc:S1134

  • objc:S1135

  • php:S1134

  • php:S1135

  • plsql:S1134

  • plsql:S1135

  • python:S1134

  • python:S1135

  • ruby:S1134

  • ruby:S1135

  • scala:S1134

  • scala:S1135

  • swift:S1134

  • swift:S1135

  • terraform:S1135

  • tsql:S1134

  • tsql:S1135

  • typescript:S1134

  • typescript:S1135

  • vbnet:S1134

  • vbnet:S1135

  • Web:S1134

  • Web:S1135

  • xml:S1134

  • xml:S1135

Test branch coverage from Cobertura

Cobertura can be used to measure test branch coverage.

Mandatory parameters

• URL to a Cobertura report in XML format or to a zip with Cobertura reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Cobertura report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Cobertura report in XML format.

• Username for basic authentication.

Test branch coverage from Cobertura Jenkins plugin

Cobertura Jenkins plugin can be used to measure test branch coverage.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a coverage report generated by the Cobertura plugin. For example, ‘https://jenkins.example.org/job/cobertura’ or https://jenkins.example.org/job/cobertura/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Test branch coverage from JaCoCo

JaCoCo can be used to measure test branch coverage.

Mandatory parameters

• URL to a JaCoCo report in XML format or to a zip with JaCoCo reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a JaCoCo report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JaCoCo report in XML format.

• Username for basic authentication.

Test branch coverage from JaCoCo Jenkins plugin

JaCoCo Jenkins plugin can be used to measure test branch coverage.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a coverage report generated by the JaCoCo plugin. For example, ‘https://jenkins.example.org/job/jacoco’ or https://jenkins.example.org/job/jacoco/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Test branch coverage from Manual number

Manual number can be used to measure test branch coverage.

Mandatory parameters

• Number. The default value is: 0.

Test branch coverage from NCover

NCover can be used to measure test branch coverage.

Mandatory parameters

• URL to a NCover report in HTML format or to a zip with NCover reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Test branch coverage from SonarQube

SonarQube can be used to measure test branch coverage.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Test line coverage from Cobertura

Cobertura can be used to measure test line coverage.

Mandatory parameters

• URL to a Cobertura report in XML format or to a zip with Cobertura reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a Cobertura report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Cobertura report in XML format.

• Username for basic authentication.

Test line coverage from Cobertura Jenkins plugin

Cobertura Jenkins plugin can be used to measure test line coverage.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a coverage report generated by the Cobertura plugin. For example, ‘https://jenkins.example.org/job/cobertura’ or https://jenkins.example.org/job/cobertura/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Test line coverage from JaCoCo

JaCoCo can be used to measure test line coverage.

Mandatory parameters

• URL to a JaCoCo report in XML format or to a zip with JaCoCo reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• URL to a JaCoCo report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the JaCoCo report in XML format.

• Username for basic authentication.

Test line coverage from JaCoCo Jenkins plugin

JaCoCo Jenkins plugin can be used to measure test line coverage.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL to Jenkins job. URL to a Jenkins job with a coverage report generated by the JaCoCo plugin. For example, ‘https://jenkins.example.org/job/jacoco’ or https://jenkins.example.org/job/jacoco/job/main’ in case of a pipeline job.

Optional parameters

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Test line coverage from Manual number

Manual number can be used to measure test line coverage.

Mandatory parameters

• Number. The default value is: 0.

Test line coverage from NCover

NCover can be used to measure test line coverage.

Mandatory parameters

• URL to a NCover report in HTML format or to a zip with NCover reports in HTML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Username for basic authentication.

Test line coverage from SonarQube

SonarQube can be used to measure test line coverage.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

Unmerged branches from Azure DevOps Server

Azure DevOps Server can be used to measure unmerged branches.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Branches to ignore (regular expressions or branch names).

  See also

  https://docs.microsoft.com/en-us/azure/devops/repos/git/manage-your-branches?view=azure-devops

• Number of days since last commit after which to consider branches inactive. The default value is: 7.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

• Repository (name or id).

Unmerged branches from GitLab

GitLab can be used to measure unmerged branches.

Mandatory parameters

• GitLab instance URL. URL of the GitLab instance, with port if necessary, but without path. For example, ‘https://gitlab.com’.

• Project (name with namespace or id).

  See also

  https://docs.gitlab.com/ee/user/project/

Optional parameters

• Branches to ignore (regular expressions or branch names).

  See also

  https://docs.gitlab.com/ee/user/project/repository/branches/

• Number of days since last commit after which to consider branches inactive. The default value is: 7.

• Private token (with read_api scope).

  See also

  https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Unmerged branches from Manual number

Manual number can be used to measure unmerged branches.

Mandatory parameters

• Number. The default value is: 0.

Unused CI-jobs from Azure DevOps Server

Azure DevOps Server can be used to measure unused ci-jobs.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Number of days since last build after which to consider pipelines inactive. The default value is: 21.

• Pipelines to ignore (regular expressions or pipeline names). Pipelines to ignore can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Pipelines to include (regular expressions or pipeline names). Pipelines to include can be specified by pipeline name or by regular expression. Use {folder name}/{pipeline name} for the names of pipelines in folders.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

Unused CI-jobs from GitLab

GitLab can be used to measure unused ci-jobs.

Mandatory parameters

• GitLab instance URL. URL of the GitLab instance, with port if necessary, but without path. For example, ‘https://gitlab.com’.

• Project (name with namespace or id).

  See also

  https://docs.gitlab.com/ee/user/project/

Optional parameters

• Branches and tags to ignore (regular expressions, branch names or tag names).

  See also

  https://docs.gitlab.com/ee/user/project/repository/branches/

• Jobs to ignore (regular expressions or job names). Jobs to ignore can be specified by job name or by regular expression.

• Number of days to look back for selecting pipeline jobs. The default value is: 90.

• Number of days without builds after which to consider CI-jobs unused. The default value is: 90.

• Private token (with read_api scope).

  See also

  https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Unused CI-jobs from Jenkins

Jenkins can be used to measure unused ci-jobs.

To authorize Quality-time for (non-public resources in) Jenkins, you can either use a username and password or a username and API token. Note that, unlike other sources, when using the API token Jenkins also requires the username to which the token belongs.

Mandatory parameters

• URL. URL of the Jenkins instance, with port if necessary, but without path. For example, ‘https://jenkins.example.org’.

Optional parameters

• Jobs to ignore (regular expressions or job names). Jobs to ignore can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Jobs to include (regular expressions or job names). Jobs to include can be specified by job name or by regular expression. Use {parent job name}/{child job name} for the names of nested jobs.

• Number of days without builds after which to consider CI-jobs unused. The default value is: 90.

• Password or API token for basic authentication.

  See also

  https://www.jenkins.io/doc/book/system-administration/authenticating-scripted-clients/

• Username for basic authentication.

Unused CI-jobs from Manual number

Manual number can be used to measure unused ci-jobs.

Mandatory parameters

• Number. The default value is: 0.

User story points from Azure DevOps Server

Azure DevOps Server can be used to measure user story points.

Mandatory parameters

• URL including organization and project. URL of the Azure DevOps instance, with port if necessary, and with organization and project. For example: ‘https://dev.azure.com/{organization}/{project}’.

Optional parameters

• Issue query in WIQL (Work Item Query Language). This should only contain the WHERE clause of a WIQL query, as the selected fields are static. For example, use the following clause to hide issues marked as done: “[System.State] <> ‘Done’”. See https://docs.microsoft.com/en-us/azure/devops/boards/queries/wiql-syntax?view=azure-devops.

• Private token.

  See also

  https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops

User story points from Jira

Jira can be used to measure user story points.

Mandatory parameters

• Issue query in JQL (Jira Query Language).

  See also

  https://support.atlassian.com/jira-software-cloud/docs/use-advanced-search-with-jira-query-language-jql/

• Story points field (name or id). The default value is: Story Points.

  See also

  https://confluence.atlassian.com/jirakb/how-to-find-id-for-custom-field-s-744522503.html

• URL. URL of the Jira instance, with port if necessary. For example, ‘https://jira.example.org’.

Optional parameters

• Password for basic authentication.

• Private token.

  See also

  https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

• Username for basic authentication.

User story points from Manual number

Manual number can be used to measure user story points.

Mandatory parameters

• Number. The default value is: 0.

Velocity from Jira

Jira can be used to measure velocity.

Mandatory parameters

• Board (name or id).

  See also

  https://support.atlassian.com/jira-software-cloud/docs/what-is-a-jira-software-board/

• Number of sprints to base velocity on. Velocity is defined as the number of user story points realized per sprint. By default, velocity is calculated by averaging the number of user story points realized in the three most recently completed sprints. The number of sprints to use can be changed via this parameter. The default value is: 3.

• Type of velocity. Whether to report the number of story points committed to, the number of story points actually completed, or the difference between the two. This parameter is single choice. Possible velocity type are: committed points, completed points minus committed points, completed points. The default value is: completed points.

• URL. URL of the Jira instance, with port if necessary. For example, ‘https://jira.example.org’.

Optional parameters

• Password for basic authentication.

• Private token.

  See also

  https://confluence.atlassian.com/enterprise/using-personal-access-tokens-1026032365.html

• Username for basic authentication.

Velocity from Manual number

Manual number can be used to measure velocity.

Mandatory parameters

• Number. The default value is: 0.

Violations from Axe CSV

Axe CSV can be used to measure violations.

Mandatory parameters

• URL to an Axe report in CSV format or to a zip with Axe reports in CSV format.

Optional parameters

• Elements to exclude (regular expressions). Elements to exclude can be specified by regular expression.

• Elements to include (regular expressions). Elements to include can be specified by regular expression.

• Impact levels. If provided, only count accessibility violations with the selected impact levels. This parameter is multiple choice. Possible impact levels are: critical, minor, moderate, serious. The default value is: all impact levels.

• Password for basic authentication.

• Private token.

• URL to an Axe report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Axe report in CSV format.

• Username for basic authentication.

Violations from Axe-core

Axe-core can be used to measure violations.

When running Axe-core on a webpage, the run function returns a results object. The results objects may be stored in separate JSON files and served to Quality-time in a zipfile, or the results objects can be combined in one JSON file that contains a list of results objects.

Mandatory parameters

• URL to an Axe-core report in JSON format or to a zip with Axe-core reports in JSON format.

Optional parameters

• Elements to exclude (regular expressions). Elements to exclude can be specified by regular expression.

• Elements to include (regular expressions). Elements to include can be specified by regular expression.

• Impact levels. If provided, only count accessibility violations with the selected impact levels. This parameter is multiple choice. Possible impact levels are: critical, minor, moderate, serious. The default value is: all impact levels.

• Password for basic authentication.

• Private token.

• Result types. Limit which result types to count. This parameter is multiple choice. Possible result types are: inapplicable, incomplete, passes, violations. The default value is: violations.

• Tags to ignore (regular expressions or tags). Tags to ignore can be specified by tag or by regular expression.

• Tags to include (regular expressions or tags). Tags to include can be specified by tag or by regular expression.

• URL to an Axe-core report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the Axe-core report in JSON format.

• Username for basic authentication.

Violations from Axe HTML reporter

Axe HTML reporter can be used to measure violations.

Mandatory parameters

• URL to an Axe report in HTML format or to a zip with Axe reports in HTML format.

Optional parameters

• Elements to exclude (regular expressions). Elements to exclude can be specified by regular expression.

• Elements to include (regular expressions). Elements to include can be specified by regular expression.

• Impact levels. If provided, only count accessibility violations with the selected impact levels. This parameter is multiple choice. Possible impact levels are: critical, minor, moderate, serious. The default value is: all impact levels.

• Password for basic authentication.

• Private token.

• Result types. Limit which result types to count. This parameter is multiple choice. Possible result types are: inapplicable, incomplete, passes, violations. The default value is: violations.

• Tags to ignore (regular expressions or tags). Tags to ignore can be specified by tag or by regular expression.

• Tags to include (regular expressions or tags). Tags to include can be specified by tag or by regular expression.

• Username for basic authentication.

Violations from Manual number

Manual number can be used to measure violations.

Mandatory parameters

• Number. The default value is: 0.

Violations from OJAudit

OJAudit can be used to measure violations.

Mandatory parameters

• URL to an OJAudit report in XML format or to a zip with OJAudit reports in XML format.

Optional parameters

• Password for basic authentication.

• Private token.

• Severities. If provided, only count violations with the selected severities. This parameter is multiple choice. Possible severities are: advisory, error, exception, incomplete, warning. The default value is: all severities.

• URL to an OJAudit report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the OJAudit report in XML format.

• Username for basic authentication.

Violations from SARIF

SARIF can be used to measure violations.

Mandatory parameters

• URL to a SARIF vulnerability report in JSON format or to a zip with SARIF vulnerability reports in JSON format.

Optional parameters

• Levels. If provided, only count entities with the selected levels. This parameter is multiple choice. Possible levels are: error, none, note, warning. The default value is: all levels.

• Password for basic authentication.

• Private token.

• URL to a SARIF vulnerability report in a human readable format. If provided, users clicking the source URL will visit this URL instead of the SARIF vulnerability report in JSON format.

• Username for basic authentication.

Violations from SonarQube

SonarQube can be used to measure violations.

Mandatory parameters

• Project key. The project key can be found by opening the project in SonarQube and looking at the bottom of the grey column on the right.

• URL. URL of the SonarQube instance, with port if necessary, but without path. For example, ‘https://sonarcloud.io’.

Optional parameters

• Branch (only supported by commercial SonarQube editions). The default value is: master.

  See also

  https://docs.sonarqube.org/latest/branches/overview/

• Private token.

  See also

  https://docs.sonarqube.org/latest/user-guide/user-token/

• Severities. This parameter is multiple choice. Possible severities are: blocker, critical, info, major, minor. The default value is: all severities.

  See also

  https://docs.sonarqube.org/latest/user-guide/issues/

• Tags to include.

  See also

  https://docs.sonarsource.com/sonarqube/latest/user-guide/issues/

• Types. This parameter is multiple choice. Possible types are: bug, code_smell, vulnerability. The default value is: all types.

  See also

  https://docs.sonarqube.org/latest/user-guide/rules/