This post is to help you plan your deployment of Testlab. It gives pointers on things to consider when enrolling Testlab in use, and food for thought when adapting Testlab to your software development process.
Adapting a test management tool to your software process usually takes a bit of planning. Tools are easy to use but to make sure the features – especially analysis features – are taken in use in a best way possible there are some things to plan and decide before jumping into use head on. In this post we will discuss different details of Testlab’s features, way of storing the quality related assets and give pointers on how you should enroll Testlab in use as efficiently as possible.
Even though we speak of Testlab as a software-centric test management tool, you should keep in mind that Testlab is a fully featured quality management suite. This means, that you can use Testlab for specification, verification and analysis of your product quality regardless what your product actually is.
For simplicity’s sake, in this article, we’ll discuss Testlab in the context of development of software projects. This is also natural because Testlab offers out-of-box integrations to commonly used software development tools such as Atlassian JIRA and Confluence.
The easiest way to grasp what Testlab is about is to
1. read through this article.
2. Log into the “Demo” project and browse through the data. When you look around it is good to read through the description of the demo project which gives you insight on the (fictional) software development project the Demo project is about.
3. Give it a go and feel free to add assets, edit assets and execute some tests in your Demo project. This helps you shed light on how different things affect others in Testlab. You can also add a new blank project to your Testlab for sandboxing if you wish to retain your copy of Demo project as it is.
4. Browse through the help manual found in your Testlab. You can also click the question mark icons on each view to jump around the manual in context sensitive manner.
5. Browse through the documentation on-line.
6. Add a new project, some users and evaluate your Testlab in a small real world project of yours. This is by far the most easy way to really experience the benefits of using such a tool and see the structure it brings to your development project.
It helps to understand how the central quality related assets are stored in your tool. First, all your data (excluding your company details) will be stored under a project. You should think of project as a domain which will include all requirements (specification of some kind), test cases for them, executed testing results, issues, etc for a single product, application or a system under testing.
The next diagram shows a simplified diagram about how the different assets relate to each other in Testlab inside a project.
Each project can be split into milestones, which are project reference points that aim to be completed in some point of time. Milestones can be used in various ways to separate the assets inside the project from each other and in addition, using milestones makes it easier to track the progress of testing. For more examples on how milestones can be used please see the 5.2 Milestones section of Testlab’s help manual.
Every good test plan is based on a solid set of requirements. In Testlab, requirements can be anything that can be used to specify, define or qualify features and functionality for the product, service or an application under testing. Testlab is development process agnostic, so you can define your requirements as user stories, use cases, traditional business requirements or something else that fits best to your development process. You can even use requirements to group your test cases to the grouping of your liking if you for some reason prefer to skip using the requirements in specification phase. See the next chapter on more pointers on using requirements.
For the testers to be able to test the system they must have information about what they are testing and most of the time instructions on how something should be tested. For this, test cases are defined to include description, pre-conditions, the actual step by step instructions and expected end results. For the tool to be able to automatically track which functionality each test case is testing they are usually linked to requirements. When a test case is linked to a requirement we say the test case verifies this requirement.
When test cases are executed they are always executed in a test run. A test run records the results for each test case that is included in it and specifies for which version (of the software for example) the testing was executed against and in which environment the testing was executed in.
During testing test runs are recorded with testing results for each test case. Good. But when something is encountered which requires attention later on it should be recorded as an issue. Issues can be seen as tasks which should be handled somehow later on. Issues can be defects encountered (i.e. bugs in software), future development ideas, new features or something else you wish to record for later use. Keep in mind that you can add issues anytime – they don’t have to be results of some specific testing session.
All major testing data assets: requirements, test cases and issues have automatically stored history data. This means you can find out what these assets have been at each point of time.
Is there any data in Testlab which can be shared between Projects ?
Yes there is.
It is common that the quality, coverage, precision and availability of system’s specification varies. Sometimes, you would just like to manage your test cases and the effort of tracking requirements in the tool seems too much (even though importing them is quite easy). All is good, using requirements in Testlab is not required; You can add just your test cases, execute them and manage the found issues.
Just to note as said earlier, the requirements in Testlab don’t have to be traditional business requirements even though the term implies so. Requirements can be anything that fit well to your development process and can be used to specify the qualities and functionality of your system under testing. In agile projects these can be user stories for example. Or better yet, a mixture of different kinds of specification assets.
The type of the requirement is really not important but the link between the requirement and it’s verifying test cases is; From this link Testlab can analyze and report out the coverage of your testing. In other words, you can easily see what functionality has been tested as passing and which parts of the system still have issues with them. From testers’ point of view the advantage is that you always know why are you testing certain thing and if the requirement, or implementation of requirement should change, you can check which test cases you might have to update or run.
We recommend that you consider some middle ground alternatives on how to manage your requirements in Testlab even if you don’t prefer to use Testlab for specification:
Keep in mind, that requirements are added to a tree. The coverage analysis is hierarchical, so all leaf nodes in the tree are summed up when calculating the coverage.
When talking about workflow in Testlab, we mean a way of working with different types of assets in a project. A workflow basicly specifies
Testlab provides two different kinds of workflows out of box for you to assign for your projects. A “Simple” style workflow is shown in the following diagram:
“Simple” asset workflow
“Simple” workflow defines a simple asset workflow for requirements and test cases where assets are first created to “In design” status. When the asset seems ready it is approved as ready. That’s it. If some requirements or test cases get stale they can be deprecated and therefore regarded as removed. Choose this workflow for your projects if you prefer to keep workflow with your test cases and requirements as simple as possible.
If you prefer, that your requirements and/or test cases should be reviewed by a 3rd party (for example your product owner, customer or someone else) you can use the “Default” workflow described in the following diagram:
“With review” asset workflow with review process included
In this workflow, assets are created to “In design” status and when they seem ready, they are marked as ready for review. Persons in appropriate REVIEWER role (see “Roles for users” chapter below for more information) then either reject or approve the asset. You should pick this workflow for your projects if you prefer your assets to be reviewed through the above process.
Keep in mind, that you can customize all workflows for your liking by copying an existing workflow and making the needed changes for it. You can customize permissions on user roles and asset workflows for requirements, test cases and issues separately.
When you sign up for Testlab you will be granted with an user account as an company administrator. This user account is a special account granting you administration rights. Administrators have power to do anything in their projects.
Permission system in Testlab is based on user roles. When creating user accounts you should grant the user an appropriate user role in the projects they will be assigned to. So as said, user roles are project specific so each account can have different user roles in each project.
A role grants a set of permissions and workflow actions to the user in the project. The roles provided by the out of box workflows of Testlab defined the user roles as follows (for more specific details please see 8.2 Default workflows and roles -chapter in Testlab help):
VIEWER-role is for users who should only be able to view and read the existing content in the project. Users in this role can view the data, generate reports and diagrams and export data in CSV format.
This role is useful for users who should only be able to view the content.
REQUIREMENTDESIGNER-role is for users who should be able to design and manage the requirements of the testing project. Designer roles have all permissions of the VIEWER and in addition, permissions to add, edit, delete and import requirements.
This role should be granted for users who are responsible for managing the requirements in the project.
REQUIREMENTREVIEWER-role is for users who should be able to approve and reject existing requirements. Users in this role have all permissions of the VIEWER and in addition they can edit requirements to approve and reject them.
If you are using the Default-workflow with review, this role should be granted for users who are responsible for approving or rejecting the requirements.
TESTDESIGNER-role is for users who should be able to design and manage the test cases of the testing project. This role is similar to the REQUIREMENTDESIGNER role but applies to test case design only.
This role should be granted for users who are responsible for managing the test cases in the project.
TESTREVIEWER-role is for users who should be able to approve and reject existing test cases. This role is similar to the REQUIREMENTREVIEWER role but applies to test case design only.
If you are using the Default-workflow with review, this role should be granted for users who are responsible for approving or rejecting the test cases.
TESTER-role is for users who should be able to plan testing, execute test cases through test runs and manage issues. Users in this role have all permissions of the VIEWER and in addition, permissions to manage test sets and test runs, execute tests, add, edit and import issues.
This role should be granted for all users who should be able to execute test cases (i.e. testers).
ISSUEMANAGER-role is for users who should be able to manage issues and close them. Users in this role have all permissions of the VIEWER and in addition, they can add and edit issues and mark them as closed.
This role should be granted for all users who should be able to review resolved issues and accept them as closed.
ISSUERESOLVER-role is for users who should be able to manage issues excluding the permission to close them. Users in this role have all permissions of the VIEWER and in addition, they can add and edit issues without the permission to mark the issues as closed.
This role should be granted for all users who should be able to handle and resolve issues (i.e. developers, programmers).
PROJECTMANAGER-role is for users who should be able to manage the details of the testing project and user rights for the project and add new users. Users in this role have all permissions of the VIEWER and in addition, they can edit project details, add new users and edit user details, grant project roles for users and import and export data.
This role should be granted for all users who should be able to handle administrative project specific tasks (i.e. project managers).
TESTMANAGER-role is for users who should have full permissions to manage the assets and information included in the testing project. Users in this role have all permissions combined from the above roles. Keep in mind, that TESTMANAGERs don’t have the company management permissions of the company administrator users.
This role should be granted for all users who should be able to handle all tasks in a testing project (i.e. test managers).
When assigning roles for users with provided workflows, a simple strategy is as follows:
The fields you see on the forms when you edit requirements, test cases and issues are a default set of fields visible out of box. If you have some information you would like to store to an additional field, let’s say for example for your issues, you can configure a custom field for it.
In Testlab, custom fields are configured via project details and are always project specific. You can add custom fields for requirements, test cases and issues.
First of all, using milestones is not mandatory. You can start with your project without milestones and if and when you need one, add one.
Milestones are project reference points which make it easier for you to track the progress of testing inside the milestone and in addition, you can target assets to milestones which makes it easier to separate assets from each other inside the project. You should think of milestone as something that gets completed later on. In software projects, these are typically software releases, sprints, or something else that you aim to track your project in.
You can setup your milestones in a way which enables you to control how milestones relate to each other. This is called milestone inheritance and it allows you to set up a milestone to inherit requirements (and test cases) of an another.
Some examples follow.
When no inheritance relations are set the milestones are separated inside the project. In the above example, the project has three milestones: Release 1.0, New features and Billing. Non-targeted requirements illustrates the requirements which have no target milestone set at all. In this example, all three milestones have their own requirements, test cases and other assets. You could think of these milestones inside the project as small sub projects of their own: when reporting the progress and coverage of these milestones gives you a view of only the requirements each milestone is targeted with.
The above example has the same three milestones as the example presented earlier: Release 1.0, New features and Billing. The Release 1.0 is a milestone developing the first release version of the product, New features milestone is developing some new additional features for the product and the Billing milestone is developing billing related functions for it.
The inheritance relations between the relations are set as presented:
You can always edit the inheritance relations of the milestones later on. The edits are non-desctructive in a way, that Testlab will keep all inheritance relations of the milestones intact and when you edit them, the reporting and the views of the project will reflect the changes immediately.
It is good to understand, that a version is an important factor in analysis and reporting. It is often used as a term to group and/or filter the data reported. For example,
As a quick guide, when you are ready to add and kick off your first project in Testlab, technically you need to:
1. Add a new Project to Testlab. You’ll need to decide on a workflow – select “Simple” to keep it simple if you don’t want your requirements and test cases to be reviewed and approved.
2. Add users to Testlab. You’ll need to create an account for each person participating in your project. Note that all users will get their account details sent to them by e-mail – no need to inform your users on account details separately.
3. Grant users roles they need in your project. As said, good practice is that you grant a TESTMANAGER role to someone who is in charge of the testing in the project. If you need to limit the access for rest of the users you might want to grant some other roles for them.
In this article we’ve discussed some note-worthy details of Testlab which should help you get started. If you have any questions, please feel free to contact us and we’ll help you.