Maven plugin

Apache Maven is a common build automation tool used primarily for Java projects. In addition, Maven can be used to build projects written in C#, Ruby, Scala and other languages.

Meliora Testlab’s Maven Plugin is an extension to Maven which makes it possible to publish xUnit-compatible testing results and Robot Framework output to your Testlab project. 

 

Using the plugin in your build

The plugin binary is available from jcenter repository. You should register this repository to your build to make the plugin available:

<repository>
<id>jcenter</id>
<name>jcenter</name>
<url>https://jcenter.bintray.com/</url>
</repository>

 

Then, you should specify the version in your project’s plugin configuration:

<project>
  ...
  <build>
    <!-- To define the plugin version in your parent POM -->
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>fi.meliora</groupId>
          <artifactId>testlab-maven-plugin</artifactId>
          <version>1.1</version>
        </plugin>
        ...
      </plugins>
    </pluginManagement>
    <!-- To use the plugin goals in your POM or parent POM -->
    <plugins>
      <plugin>
        <groupId>fi.meliora</groupId>
        <artifactId>testlab-maven-plugin</artifactId>
        <version>1.1</version>
      </plugin>
      ...
    </plugins>
  </build>
  ...
</project>

For more information, see “Guide to Configuring Plug-ins”

 

Goals

Goals available for this plugin:

 Goal Description 
 testlab:publish Maven plugin Mojo to publish testing results to Meliora Testlab. 

 

testlab:publish 

A goal to publish xUnit-compatible testing results generated by the build to Testlab as a test run.

 

Required parameters
Name Type Since Description
apiKey String - Api key to use in communicating with the Testlab you want to publish the results to. An api key must be set for security on your Company settings at Testlab side.
projectKey String - The key of the Testlab project you want to publish the results to.
resources List - File resource patterns to find the xUnit’s .xml files to push the actual results from.
<resources>
    <resource>
        <directory>target/surefire-reports</directory>
        <includes>
            <include>**/*.xml</include>
        </includes>
        <excludes>
            <exclude>**/junk/**</exclude>
        </excludes>
    </resource>
</resources>
testCaseMappingField String - Title of the Testlab’s custom field to use to map the test cases to unit test methods.
To make it possible to push unit test results to Testlab the unit test class.method name must be mapped to a test case in your Testlab project:
  1. Configure a new “text” custom field to your project at Testlab side. Give the field a human readable title such as “JUnit test”.
  2. Update your test cases at Testlab side and enter proper JUnit test class+method name pairs for your test cases.
  3. Enter “JUnit test” to this parameter in plugin configuration. This way when unit tests in the build fail they will be mapped to test cases with matching “JUnit test” names at Testlab side. When publishing Testlab will peek the set custom field of a test case and if it matches the class.method name of a failed unit test the test case will be marked as failed.
testRun String - The name of the Testlab test run to create or update.
If a test run with the same name already exists in Testlab it will be always replaced with test results when this plugin is run. If it does not it will be created.
 
Optional parameters
Name Type Since Description
addIssues boolean - If true, issues will be added to the Testlab project for failing tests.
assignIssuesToUser String - When set to Testlab’s user id value for an user who has a role in the set project the added issue(s) are automatically assigned to this user.
companyId String - Company ID of the hosted Testlab you want to publish the results to. If your testlab is accessed from mytestlab.melioratestlab.com your Company ID most likely is “mytestlab”.
forcePublish boolean - If set, the results will be always published ignoring the possible -Dtest and -Dit.test parameters.
By default this option is “false”. This means, that if a test or subset of tests are specified by -Dtest or -Dit.test parameters on command line, the (probably partial) results will not be published to Testlab. Set this option to “true” to ignore these properties and always publish the results to Testlab.
ignoreFailedTests boolean - If set, the build will not be marked as FAILURE if any failures are detected. Note that this plugin marks the build as failed only by inspecting the results of the resources configured.
importTestCases boolean - If set, automatically creates test cases when publishing.
importTestCasesRootCategory String - If set, sets the root category path where the test cases are created. By default, “Import”.
mergeAsSingleIssue boolean - If checked added issues are merged to a single issue. Checking this option results always in a maximum of a single issue added per plugin execution.
milestone String - Milestone in Testlab the results and issues are bound to. Milestone must exist in the project. If milestone does not exist the results and issues are added without the milestone binding.
You can enter an identifier of a milestone or a title of a milestone. The identifier is matched first.
onpremiseUrl String - For on-premises installation, a full URL address (such as ‘https://testlab.mycompany.com’) of your Testlab must be set.
  • For on-premise installations on your own server: set onpremiseUrl
  • For hosted Testlab: set Company ID
parameters Map - Test cases in Testlab can include parameters in the content as ${PARAM1} formatted tags. Normally, these tags are set and replaced with values set during the test case execution or execution planning phase. Set this configuration map to send the parameter values for test case parameters.
<parameters>
    <UI>Desktop</UI>
    <BROWSER>Firefox</BROWSER>
</parameters>
reopenExistingIssues boolean - The plugin will create the issues with the title set to the name of the failing test class. When this setting is set the plugin tries to reopen existing issues in the project by matching the issue’s title.
When finding existing issues if
  • a single resolved or closed issue is found it is reopened and updated,
  • a single still non-resolved issue is found it is updated,
  • multiple issues are found and the latest issue is resolved or closed it is reopened and updated,
  • multiple issues are found and the latest issue is still non-resolved it is updated.

Otherwise, a new issue is added.
When reopening an issue version, environment, description and assigned to -fields are updated to match the plugin sent values. When updating a found non-resolved issue only description is updated to plugin sent value.

tags List - Tags to be set for the Testlab test run.
When updating runs, if the plugin sends even one tag all existing tags in the run will be replaced. Leaving this parameter unset leaves the tags at Testlab side untouched.
<tags>
    <tag>maven</tag>
    <tag>automated</tag>
</tags>
testEnvironment String - Test environment in Testlab the results and issues are bound to. If the environment does not exist in Testlab it is added to the project.
testRunComment String - A comment to be set for the Testlab test run to create or update.
Hint: You can use Maven’s properties in all the configuration variables and in the testRunComment field this is especially handy. Use the project.*, settings.* and env.* variables to decorate your configuration as needed. The variables are referenced in typical format such as ${project.version}.
testRunUser String - The “created by” user name of the test run created or updated. Defaults to “Maven”.
version String - Version in Testlab the results and issues are bound to. If version does not exist in Testlab it is added to the project.
format String 1.1 Format of the result resources to publish. “junit” for standard xUnit formatted files and “robot” for Robot Framework’s output files. Defaults to “junit”.
robotCatenateParentKeywords boolean 1.1 If true, when the XML file containing the results is in Robot Framework format, and in the XML keyword has sub keywords, the sub-keywords are catenated to a single step in the result. For example, if the robot result has
    <kw name="Open site">
        <kw name="Open URL">
            <kw name="Navigate browser">
                ...
            </kw>
        </kw>
    </kw>
    ...

.. the test case is added with a single step described as “Open site – Open URL – Navigate browser”. When concatenating, if a step fails it is always included as a step.
If false, each sub keyword will generate a separate step to the result.
This value defaults to true.

 

Parameter details

addIssues:

If true, issues will be added to the Testlab project for failing tests.
  • Type: boolean
  • Required: No

apiKey:

API key to use in communicating with the Testlab you want to publish the results to. An API key must be set for security on your Company settings at Testlab side.
  • Type: java.lang.String
  • Required: Yes

assignIssuesToUser:

When set to Testlab’s user id value for an user who has a role in the set project the added issue(s) are automatically assigned to this user.
  • Type: java.lang.String
  • Required: No

companyId:

Company ID of the hosted Testlab you want to publish the results to. If your testlab is accessed from mytestlab.melioratestlab.com your Company ID most likely is “mytestlab”.
  • Type: java.lang.String
  • Required: No

forcePublish:

If set, the results will be always published ignoring the possible -Dtest and -Dit.test parameters.
By default this option is “false”. This means, that if a test or subset of tests are specified by -Dtest or -Dit.test parameters on command line, the (probably partial) results will not be published to Testlab. Set this option to “true” to ignore these properties and always publish the results to Testlab.
  • Type: boolean
  • Required: No

format:

Format of the result resources to publish. “junit” for standard xUnit formatted files and “robot” for Robot Framework’s output files. Defaults to “junit”.
  • Type: java.lang.String
  • Required: No

robotCatenateParentKeywords:

If true, when the xml file containing the results is in Robot Framework format, and in the xml keyword has sub keywords, the sub keywords are catenated to a single step in the result. For example, if the robot result has
    <kw name="Open site">
        <kw name="Open URL">
            <kw name="Navigate browser">
                ...
            </kw>
        </kw>
    </kw>
    ...

.. the test case is added with a single step described as “Open site – Open URL – Navigate browser”. When concatenating, if a step fails it is always included as a step.
If false, each sub keyword will generate a separate step to the result.
This value defaults to true.

  • Type: boolean
  • Required: No

ignoreFailedTests:

If set, the build will not be marked as FAILURE if any failures are detected. Note that this plugin marks the build as failed only by inspecting the results of the resources configured.
  • Type: boolean
  • Required: No

importTestCases:

If set, automatically creates test cases when publishing.
  • Type: boolean
  • Required: No

importTestCasesRootCategory:

If set sets the root category path where the test cases are created. By default, “Import”.
  • Type: java.lang.String
  • Required: No

mergeAsSingleIssue:

If checked added issues are merged to a single issue. Checking this option results always in a maximum of a single issue added per plugin execution.
  • Type: boolean
  • Required: No

milestone:

The milestone in Testlab the results and issues are bound to. Milestone must exist in the project. If milestone does not exist the results and issues are added without the milestone binding.
You can enter an identifier of a milestone or a title of a milestone. The identifier is matched first.
  • Type: java.lang.String
  • Required: No

onpremiseUrl:

For on-premises installation, a full URL address (such as ‘https://testlab.mycompany.com’) of your Testlab must be set.
  • For on-premise installations on your own server: set onpremiseUrl
  • For hosted Testlab: set Company ID
  • Type: java.lang.String
  • Required: No

parameters:

Test cases in Testlab can include parameters in the content as ${PARAM1} formatted tags. Normally, these tags are set and replaced with values set during the test case execution or execution planning phase. Set this configuration map to send the parameter values for test case parameters.
<parameters>
    <UI>Desktop</UI>
    <BROWSER>Firefox</BROWSER>
</parameters>
  • Type: java.util.Map
  • Required: No

projectKey:

The key of the Testlab project you want to publish the results to.
  • Type: java.lang.String
  • Required: Yes

reopenExistingIssues:

The plugin will create the issues with the title set to the name of the failing test class. When this setting is set the plugin tries to reopen existing issues in the project by matching the issue’s title.
When finding existing issues if
  • a single resolved or closed issue is found it is reopened and updated,
  • a single still non-resolved issue is found it is updated,
  • multiple issues are found and the latest issue is resolved or closed it is reopened and updated,
  • multiple issues are found and the latest issue is still non-resolved it is updated.

Otherwise, a new issue is added.
When reopening an issue version, environment, description and assigned to -fields are updated to match the plugin sent values. When updating a found non-resolved issue only description is updated to plugin sent value.

  • Type: boolean
  • Required: No

resources:

File resource patterns to find the files to push the actual results from. Depending on the format, should be a set of xUnit formatted XML files or for Robot Framework, a single output.xml file.
<resources>
    <resource>
        <directory>target/surefire-reports</directory>
        <includes>
            <include>**/*.xml</include>
        </includes>
        <excludes>
            <exclude>**/junk/**</exclude>
        </excludes>
    </resource>
</resources>
  • Type: java.util.List
  • Required: Yes

tags:

Tags to be set for the Testlab test run.
When updating runs, if the plugin sends even one tag all existing tags in the run will be replaced. Leaving this parameter unset leaves the tags at Testlab side untouched.
<tags>
    <tag>maven</tag>
    <tag>automated</tag>
</tags>
  • Type: java.util.List
  • Required: No

testCaseMappingField:

The title of the Testlab’s custom field to use to map the test cases to unit test methods.
To make it possible to push unit test results to Testlab the unit test class.method name must be mapped to a test case in your Testlab project:
  1. Configure a new “text” custom field to your project at Testlab side. Give the field a human readable title such as “JUnit test”.
  2. Update your test cases at Testlab side and enter proper JUnit test class+method name pairs for your test cases.
  3. Enter “JUnit test” to this parameter in the plugin configuration. This way when unit tests in the build fail they will be mapped to test cases with matching “JUnit test” names at Testlab side. When publishing Testlab will peek the set custom field of a test case and if it matches the class.method name of a failed unit test the test case will be marked as failed.
  • Type: java.lang.String
  • Required: Yes

testEnvironment:

Test environment in Testlab the results and issues are bound to. If the environment does not exist in Testlab it is added to the project.
  • Type: java.lang.String
  • Required: No

testRun:

The name of the Testlab test run to create or update.
If a test run with the same name already exists in Testlab it will be always replaced with test results when this plugin is run. If it does not it will be created.
  • Type: java.lang.String
  • Required: Yes

testRunComment:

A comment to be set for the Testlab test run to create or update.
Hint: You can use Maven’s properties in all the configuration variables and in the testRunComment field this is especially handy. Use the project.*, settings.* and env.* variables to decorate your configuration as needed. The variables are referenced in a typical format such as ${project.version}.
  • Type: java.lang.String
  • Required: No

testRunUser:

The “created by” username of the test run created or updated. Defaults to “Maven”.
  • Type: java.lang.String
  • Required: No

version:

The version in Testlab the results and issues are bound to. If the version does not exist in Testlab it is added to the project.
  • Type: java.lang.String
  • Required: No

 

Usage and examples

The testlab:publish goal provided by this plugin is used to push testing results parsed from xUnit-compatible .xml files to Meliora Testlab as testing results.

Some common points must be taken into account when using this plugin:

  • The results are parsed from xUnit-compatible reports (.xml-files) such as generated by the surefire plugin.
  • You must configure your project in a way that the failed tests won’t halt the execution as FAILURE. Otherwise, this plugin won’t run. For example, with surefire, you can configure maven-surefire-plugin with
    ...
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                    ...
                </configuration>
            </plugin>
            ...
        </plugins>
    ...
  • The goal can be configured with multiple directories and patterns from where the .xml files generated during the build are scanned from. This way, you can publish multiple results with a single invocation.
  • The goal should be run after all the goals generating testing results have run. Typically, publish goal in verify phase might do the trick.
Example: complete configuration

A complete surefire-based configuration block for the plugin is as follows:

...
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <!-- ignore failure here to allow other plugins to run -->
                    <testFailureIgnore>true</testFailureIgnore>
                </configuration>
            </plugin>
            <plugin>
                <groupId>fi.meliora</groupId>
                <artifactId>testlab-maven-plugin</artifactId>
                <version>1.1</version>
                <configuration>
                    <companyId>mycompany</companyId>
                    <apiKey>mytopsecretapikey</apiKey>
                    <!-- or, if you are using Testlab on-premise -->
                    <!-- onpremiseUrl>http://testlab.mycompany.com:8080</onpremiseUrl -->
                    <testCaseMappingField>Automated</testCaseMappingField>
                    <projectKey>TLABDEMO</projectKey>
                    <testRun>Integration tests</testRun>
                    <testRunUser>Maven</testRunUser>
                    <testRunComment>Integration tests run for 1.0-SNAPSHOT at  ${maven.build.timestamp}</testRunComment>
                    <milestone>M1</milestone>
                    <version>1.0-SNAPSHOT</version>
                    <testEnvironment>system-testing</testEnvironment>
                    <tags>
                        <tag>maven</tag>
                        <tag>automated</tag>
                        <tag>integration</tag>
                    </tags>
                    <parameters>
                        <UI>Desktop</UI>
                        <BROWSER>Firefox</BROWSER>
                    </parameters>
                    <addIssues>true</addIssues>
                    <assignIssuesToUser>agentsmith</assignIssuesToUser>
                    <reopenExistingIssues>true</reopenExistingIssues>
                    <mergeAsSingleIssue>true</mergeAsSingleIssue>
<importTestCases>true</importTestCases>
<importTestCasesRootCategory>Automated/Import</importTestCasesRootCategory>
<format>junit</format> <resources> <resource> <directory>target/surefire-reports</directory> <includes> <include>**/*.xml</include> </includes> <!-- excludes> <exclude>**/junk/**</exclude> </excludes --> </resource> </resources>
<forcePublish>false</forcePublish>
<ignoreFailedTests>false</ignoreFailedTests> </configuration> <executions> <execution> <phase>verify</phase> <goals> <goal>publish</goal> </goals> </execution> </executions> </plugin> ... </plugins> </build> ...

The above will run all tests with surefire-plugin and publish the results

  • to hosted Testlab with company identifier mycompany (mycompany.melioratestlabcom),
  • using “mytopsecretapikey” as API key for authentication,
  • by mapping the automated tests to test cases with custom field named “Automated”,
  • to Testlab project TLABDEMO,
  • to test run titled “Integration tests”,
  • by setting the test run’s creator as “Maven”,
  • by setting the test run’s comment field as “Integration tests run for [build version] at [timestamp]”,
  • by targeting milestone “M1” in Testlab,
  • by setting the version field in testrun to match the POM’s version,
  • by setting the environment as “system-testing”,
  • by tagging the test run with tags “maven”, “automated” and “integration”,
  • by setting the test case parameters in Testlab as UI = Desktop and BROWSER = Firefox,
  • with adding issues to Testlab for any failed tests,
  • by assigning all the issues to user “agentsmith”,
  • by reopening existing matching issues in Testlab if found,
  • by merging all testing failures to a single issue,
  • by automatically adding missing test cases (= missing by mapping identifier) to Testlab project,
  • by adding the missing test cases to test category Automated / Import,
  • by posting junit-formatted (xUnit) result files,
  • by looking for the xUnit .xml files from target/surefire-reports path,
  • by honoring the -Dtest and -Dit.test properties so that if either one is set the results are not published,
  • by marking the build as failed if any failures or errors are present in result files configured and
  • running the plugin in publish goal in verify phase.
Example: minimal configuration

A minimal surefire-based configuration block for the plugin is as follows:

...
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                </configuration>
            </plugin>
            <plugin>
                <groupId>fi.meliora</groupId>
                <artifactId>testlab-maven-plugin</artifactId>
                <version>1.1</version>
                <configuration>
                    <companyId>mycompany</companyId>
                    <apiKey>mytopsecretapikey</apiKey>
                    <testCaseMappingField>Automated</testCaseMappingField>
                    <projectKey>TLABDEMO</projectKey>
                    <testRun>Integration tests</testRun>
                    <resources>
                        <resource>
                            <directory>target/surefire-reports</directory>
                            <includes>
                                <include>**/*.xml</include>
                            </includes>
                        </resource>
                    </resources>
                </configuration>
                <executions>
                    <execution>
                        <phase>verify</phase>
                        <goals>
                            <goal>publish</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            ...
        </plugins>
    </build>
...

The above will run all tests with surefire-plugin and publish the results

  • to hosted Testlab with company identifier mycompany (mycompany.melioratestlabcom),
  • using “mytopsecretapikey” as API key for authentication,
  • by mapping the automated tests to test cases with custom field named “Automated”,
  • to Testlab project TLABDEMO,
  • to test run titled “Integration tests”,
  • by looking for the junit / xUnit .xml files from target/surefire-reports path and
  • running the plugin in publish goal in verify phase.
Example: On-premise Testlab
...
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                </configuration>
            </plugin>
            <plugin>
                <groupId>fi.meliora</groupId>
                <artifactId>testlab-maven-plugin</artifactId>
                <version>1.1</version>
                <configuration>
                    <onpremiseUrl>http://testlab.mycompany.com:8080</onpremiseUrl>
                    <apiKey>mytopsecretapikey</apiKey>
                    <testCaseMappingField>Automated</testCaseMappingField>
                    <projectKey>TLABDEMO</projectKey>
                    <testRun>Integration tests</testRun>
                    <resources>
                        <resource>
                            <directory>target/surefire-reports</directory>
                            <includes>
                                <include>**/*.xml</include>
                            </includes>
                        </resource>
                    </resources>
                </configuration>
                <executions>
                    <execution>
                        <phase>verify</phase>
                        <goals>
                            <goal>publish</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            ...
        </plugins>
    </build>
...

The above will run all tests with surefire-plugin and publish the results

  • to Testlab running at http://testlab.mycompany.com:8080,
  • using “mytopsecretapikey” as API key for authentication,
  • by mapping the automated tests to test cases with custom field named “Automated”,
  • to Testlab project TLABDEMO,
  • to test run titled “Integration tests”,
  • by looking for the xUnit .xml files from target/surefire-reports path and
  • running the plugin in publish goal in verify phase.
Example: robot framework publish

A minimal surefire-based configuration block for the plugin to publish Robot Framework’s output is as follows:

...
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <configuration>
                    <testFailureIgnore>true</testFailureIgnore>
                </configuration>
            </plugin>
            <plugin>
                <groupId>fi.meliora</groupId>
                <artifactId>testlab-maven-plugin</artifactId>
                <version>1.1</version>
                <configuration>
                    <companyId>mycompany</companyId>
                    <apiKey>mytopsecretapikey</apiKey>
                    <testCaseMappingField>Automated</testCaseMappingField>
                    <projectKey>TLABDEMO</projectKey>
                    <testRun>Robot tests</testRun>
<format>robot</format> <resources> <resource> <directory>target/robot</directory> <includes> <include>output.xml</include> </includes> </resource> </resources> </configuration> <executions> <execution> <phase>verify</phase> <goals> <goal>publish</goal> </goals> </execution> </executions> </plugin> ... </plugins> </build> ...

The above will run all tests with surefire-plugin and publish the results

  • to hosted Testlab with company identifier mycompany (mycompany.melioratestlabcom),
  • using “mytopsecretapikey” as API key for authentication,
  • by mapping the automated tests to test cases with custom field named “Automated”,
  • to Testlab project TLABDEMO,
  • to test run titled “Integration tests”,
  • by looking for Robot’s output.xml file from target/robot path and
  • running the plugin in publish goal in verify phase.

    

Frequently asked questions

 
In short – what exactly does this do?

This plugin provides support for Apache Maven based builds to publish the automated testing results generated to Testlab.

 

From where exactly do I set this up with?

In pom.xml of your build. Setting up this plugin probably requires basic knowledge on Maven and it’s concepts.

 

We have Maven based jobs in our Jenkins, do I need this plugin?

No. You are probably better off with Testlab’s Jenkins plugin which does the exact same thing but provides a more user-friendly configuration UI in Jenkins.

 

I’m not getting any results in Testlab?

The testing results in xUnit-compatible files generated are mapped against Testlab’s test cases for results. For an in-depth explanation on how this works please see Jenkins plugin’s documentation (How it works, Mapping Testlab’s test cases to Jenkins run tests, …).

If you prefer to automatically create the needed test cases to your Testlab project, set the importTestCases configuration parameter to true. This way, when the testing results are pushed to your project, all tests with no mapped test cases found are automatically created to your project.

It should also be noted that by default if you specify -Dtest or -Dit.test options to pick a subset of tests for execution, the results will not be published. You can set the forcePublish configuration parameter to true to disable this behavior.

 

Surefire/Failsafe plugin fails my build before this plugin runs?

You must configure your test runner plugin to ignore the failures and let the build continue (for example setting testFailureIgnore to true e). By default, this plugin will also fail the build if any failures or errors are reported by the results published.



 
 
Best-of-class cross-browser hosted SaaS quality and test management, testing and issue management tools to improve your quality. Site information.