Getting started with
This article is to help you plan your deployment of Testlab. It gives pointers on things to consider when enrolling Testlab in use, and food for thought when adapting Testlab to your software development process.
Adapting a test management tool to your software process usually takes a bit of planning. Especially so, when you want to utilize the full power of requirements / specifications management and task management. The tool is easy to use when you decide how it should be used, but how to get to that point? There can be many stakeholders and many possible areas to improve. In this post, we will discuss different details of Testlab’s features, way of storing the quality-related assets, and give pointers on how you should enroll Testlab in use as efficiently as possible.
Familiarizing yourself with Testlab
Even though we speak of Testlab as a software-centric test management tool, you should keep in mind that Testlab is a fully-featured quality management suite. This means, that you can use Testlab for specification, verification, issue management, task management and analysis of your product quality regardless of what your product actually is.
The easiest way to grasp what Testlab is about is to:
- Go through the Hello World. It’s a very fast way to see the most important components of the Testlab user interface.
- Log into the “Demo” project and browse through the data. When you look around it is good to read through the description of the demo project which gives you insight into the (fictional) software development project the Demo project is about.
- When you are logged in Testlab, you can always restart the tour feature that tells you about the user interface elements. The tour starts from the part you are currently seeing, so it is good when you are familiarizing.
- Give it a go and feel free to add assets, edit assets, and execute some tests in your Demo project. This helps you shed light on how different things affect others in Testlab. You can also add a new blank project to your Testlab for sandboxing if you wish to retain your copy of the Demo project as it is. When you tried to tool a bit, it is easier to put rest of reading in to right context.
- read through this article.
- Now you Know a bit more about how Meliora Testlab works. Try it out a bit more. Log on to Demo project or create a sandbox project for you to play around.
- Browse through the help manual found in your Testlab. Chapter 9 has some guides you should read.
- When in Testlab, you can also click the question mark icons on each view to jump around the manual in a context-sensitive manner.
- Browse through the documentation on-line.
- Pick a real project where to try Testlab in. You can start small – you can start with one part of a project to get experience about how Testlab works. Once you learn how it works, you can roll out the usage to larger part and for more people.
Requirements, user stories and other specifications. How to use them in Testlab?
In Testlab terms, formal requirements, user stories and all other design specifications are called requirements. They are just different types of requirements. We recommend using that information when designing test cases and tracking how they are tested. Testlab offers great ways to utilize the information.
First of all, if you haven’t used requirements as a basis for planning your testing, you don’t need to when using Testlab either. We do recommend trying it out at some point, but you can skip straight to test design if you want. If you have your specifications managed in another tool ( Like Atlassian JIRA ), you can integrate your tool to bring your requirements to Testlab automatically. if you have your requirements in spreadsheet format, you can use CSV import to get them to Testlab. Or, if starting a new project, you can start designing and managing your requirements in Testlab. We have all the needed features for easy and efficient requirements specification.
Just to note as said earlier, the requirements in Testlab don’t have to be traditional business requirements even though the term implies so. Requirements can be anything that fits well with your development process and can be used to specify the qualities and functionality of your system under testing. In agile projects, these can be user stories for example. Or better yet, a mixture of different kinds of specification assets.
Why manage the requirements in Testlab in the first place?
- When the one ordering software and the one developing software have the same view about what is to be developed, the project has a much better chance to succeed. Testlab offers all the needed features to efficiently collaborate on what is to be developed.
- Often you need to see the big picture – see how far you are in design, and see what are the areas still to be developed. Testlab collects the situation automatically and you can always report the situation when you want.
- Testlab allows a large group of people to work on your set of requirements simultaneously.
What your testers get out of having requirements in the tool?
- Covering the features with test cases is a lot easier when you have the features listed. The result is a more relevant set of test cases.
- When the requirements change you see what test cases you need to modify.
- When executing test cases you see linked requirements. This helps testers concentrate on what is important.
What the rest of the team benefits?
- Developers get better bug reports when the tester can point directly to why something has been filed as a bug.
- Management gets a lot more accurate view of quality and the status of the project when test results are linked to original requirements. It is easier to estimate if the project is able to meet the deadlines when there is more relevant information.
- It is easier to concentrate the work on the most relevant area during the development when you have well-organized feature specification. This holds true to developers and designers as well.
Some tips and things to consider
- If you have your specification somewhere else outside Testlab you could add requirement “stubs” to your Testlab project which makes a reference to your existing specification. For example, If your specification has the functions of your system with identifiers (which it should anyway) the easiest way would be to add requirements to your Testlab project with the same identifiers set and just leave the descriptions and other fields as blank as you prefer. This way linking the test cases to your (outside) specification is a breeze by just dragging and dropping the test cases and you would get transparency on how your testing is doing from your specification point of view.
- If you don’t have any specification at hand for which to link the test cases to you should think of some kind of categorization for which you would like to see and analyze your testing. You should think about this as a grouping of test cases. For example: Let’s presume you have a UI-centric web application which you would like to execute test cases. You decide, that it would be great that you could see the testing coverage from the user interface point of view. You could then add each page of your web application as a requirement to your Testlab project and link all test cases to these pages. When requirements and test cases are set up this way in Testlab, you could see the testing status and coverage for each page and/or group of pages from test coverage analysis.
- If you are expecting a huge amount of requirements you might want to add custom fields to classify your data more. This way you can create relevant groups easily.
- Keep in mind, that requirements are added to a tree structure. It is a good idea to take advantage of this. If you have a folder level before the actual requirements, it is much easier to handle the requirement mass. The coverage analysis is using the same hierarchy, so all leaf nodes in the tree are summed up when calculating the coverage.
Maintaining your test case library
Testlab stores test cases in the test case repository. When your test case repository structure is well designed, it is easy to maintain, it is easy to find relevant test cases from the repository to be tested, and you can use it for reporting the testing situation.
Making a well-organized test case repository is not hard. Most often the best structure is to store the test cases based on the structure of the system under test. The structure might be based on user interface structure, feature structure, or something else that makes sense to you. You might already have that thought the structure for your requirements, and if so, you can use the same. You can have multiple levels in the test case tree ( just make folders for the structure ), so it’s best to use those.
What you should not do is to make the tree structure based on when the software is being released. So do not create folders named after releases or sprints. This would make it super easy to pick test cases to be tested, right? True that, but there is a big but. In what sprint or release ceases to be any relevant just after that release. Imagine having 20 sprints. You would need to re-run a test case that was designed 18 sprints ago. Or was it 17? It is hard to find test cases from this structure, and it does not help when reporting either. Instead of using a folder for release, in Testlab you can use the field “Testable in”. This way you can find the most recent test cases easily using the field and at the same time have a test case tree structure that stands the test of time.
Sometimes you might need to classify your test case mass further. You can define priority for your test cases, you can define a custom field with selectable values which allow a systematic approach to classifying your test cases. Just keep in mind the these need maintenance as well, so often less is more. Less to maintain, less to explain to now people make more efficient testing process. You can always easily use tags to make groups of test cases if you need.
Deciding on a workflow
In Testlab, customizable workflows are used to define how you can work with your assets ( requirements, test cases and issues ). The workflows define possible statuses, what status transitions are possible, and who can access them. You can pick from predefined workflows or you can build your own.
The easiest way to start using Testlab is to use a predefined workflow. If you are more into a lean way, choose simple. If you want to enforce review processes, choose “With review”.
- Possible statuses for an asset, and
- who (= which user role)
- can do what (= execute an action)
- on what (= a requirement, test case, or an issue in a specific status).
As mentioned, Testlab provides two different kinds of workflows out of the box for you to assign for your projects. A “Simple” style workflow for a test case and a requirement is shown in the following diagram:
“Simple” workflow defines a simple asset workflow for requirements and test cases where assets are first created to “In design” status. When the asset seems ready it is approved as ready. That’s it. If some requirements or test cases get stale they can be deprecated and therefore regarded as removed.
If you prefer that your requirements and/or test cases should be reviewed (for example your product owner, customer or someone else) you can use the “With review” workflow described in the following diagram:
In this workflow, assets are created to “In design” status and when they seem ready, they are marked as ready for review. Persons in the appropriate REVIEWER role (see “Roles for users” chapter below for more information) then either reject or approve the asset. You should pick this workflow for your projects if you prefer your assets to be reviewed through the above process.
Keep in mind, that you can customize all workflows for your liking by copying an existing workflow and making the needed changes for it. You can customize permissions on user roles and asset workflows for requirements, test cases and issues separately.
Roles for users
Testlab has role-based privilege system. You can define what permission each role has, and assign your users to the roles. User roles are project-specific so each account can have different user roles in each project.
Roles Description VIEWER VIEWER-role is for users who should only be able to view and read the existing content in the project. Users in this role can view the data, generate reports and diagrams and export data in CSV format.
This role is useful for users who should only be able to view the content.
REQUIREMENTDESIGNER REQUIREMENTDESIGNER-role is for users who should be able to design and manage the requirements of the testing project. Designer roles have all permissions of the VIEWER and in addition, permissions to add, edit, delete and import requirements.
This role should be granted for users who are responsible for managing the requirements in the project.
REQUIREMENTREVIEWER REQUIREMENTREVIEWER-role is for users who should be able to approve and reject existing requirements. Users in this role have all permissions of the VIEWER and in addition, they can edit requirements to approve and reject them.
If you are using the Default-workflow with a review, this role should be granted for users who are responsible for approving or rejecting the requirements.
TESTDESIGNER TESTDESIGNER-role is for users who should be able to design and manage the test cases of the testing project. This role is similar to the REQUIREMENTDESIGNER role but applies to test case design only.
This role should be granted for users who are responsible for managing the test cases in the project.
TESTREVIEWER TESTREVIEWER-role is for users who should be able to approve and reject existing test cases. This role is similar to the REQUIREMENTREVIEWER role but applies to test case design only.
If you are using the Default-workflow with a review, this role should be granted for users who are responsible for approving or rejecting the test cases.
TESTER TESTER-role is for users who should be able to plan to test, execute test cases through test runs and manage issues. Users in this role have all permissions of the VIEWER and in addition, permissions to manage test sets and test runs, execute tests, add, edit and import issues.
This role should be granted for all users who should be able to execute test cases (i.e. testers).
ISSUEMANAGER ISSUEMANAGER-role is for users who should be able to manage issues and close them. Users in this role have all permissions of the VIEWER and in addition, they can add and edit issues and mark them as closed.
This role should be granted for all users who should be able to review resolved issues and accept them as closed.
ISSUERESOLVER ISSUERESOLVER-role is for users who should be able to manage issues excluding the permission to close them. Users in this role have all permissions of the VIEWER and in addition, they can add and edit issues without permission to mark the issues as closed.
This role should be granted for all users who should be able to handle and resolve issues (i.e. developers, programmers).
PROJECTMANAGER PROJECTMANAGER-role is for users who should be able to manage the details of the testing project and user rights for the project and add new users. Users in this role have all permissions of the VIEWER and in addition, they can edit project details, add new users and edit user details, grant project roles for users and import and export data.
This role should be granted for all users who should be able to handle administrative project-specific tasks (i.e. project managers).
TESTMANAGER TESTMANAGER-role is for users who should have full permissions to manage the assets and information included in the testing project. Users in this role have all permissions combined from the above roles. Keep in mind, that TESTMANAGERs don’t have the company management permissions of the company administrator users.
This role should be granted for all users who should be able to handle all tasks in a testing project (i.e. test managers).
When assigning roles for users with provided workflows, a simple strategy is as follows:
- Give Company admin roles only to those who really need it. They can do everything in every project Testlab.
- Grant TESTMANAGER role to all users who should have permission to do about everything in the project (manage requirements, test cases, and issues, execute tests, manage users, etc.). Typically this might be the test manager leading the testing project.
- Grant REQUIREMENTDESIGNER and/or TESTDESIGNER roles to all users who should have permission to manage the assets.
- If you are using review based workflows grant REQUIREMENTREVIEWER and/or TESTREVIEWER roles to all users who should be permissions to approve or reject the assets.
- All developers who should be able to resolve issues should be granted with ISSUERESOLVER role.
- Grant VIEWER role to all users who should be read-only access to the project’s data.
Customizing fields of assets
The fields you see on the forms when you edit requirements, test cases and issues are a default set of fields visible out of the box. If you have some information you would like to store in an additional field, let’s say for example for your issues, you can configure a custom field for it. You can also hide the unneeded ones.
In Testlab, custom fields are configured via project details and are always project-specific. You can add custom fields for requirements, test cases, test case steps and issues. You can also set some of the fields to be mandatory if you want to be sure they are filled.
Milestones – how do I use them?
Most projects have a way of dividing the project into smaller chunks, for example, releases. The Milestones are Testlab’s way of keeping track of those. Milestones can be further split into sprints.
Generally, it is a lot easier to manage a project when it is split into pieces. Smaller pieces are easier to manage. Most likely your project already has some sort of milestones. If so, just use the projects milestones in Testlab. The same goes for sprints. If your project uses sprints it makes sense to utilize them in Testlab as well. It is possible to use Testlab without touching milestones or sprints, but you probably want. A few of the benefits are:
- If you track when certain features, as requirements, are to be released, you can better manage the feature design. This is especially relevant in a littlebit larger projects.
- You can also track the test design from the requirements. You can always check if you have test cases ready for new features.
- When your test cases have information on what milestone they can first be tested, it is easy to create test runs for new features.
- When you tell Testlab what milestone or sprint you are currently testing, you can create astonishing coverage reports easily. With a few clicks you can for example
- Find out what features you have tested in this milestone, and what not
- Find out what you have tested in last two or more milestones – What are the test cases that you have not run in a long time.
- The defects you find during testing are automatically set to be found in a milestone they were tested in. It is a pretty relevant information when investigating what has happened.
So the milestones can be split into sprints. If your project uses sprints, use them in Testlab as well. This way you use the same language with everyone. It helps you with representing situation as well. You can easily make reports for whole milestone about what is happening in it’s sprints.
How should I use versions and/or environments?
Versions and environments are used to mark what exactly you are testing. In Testlab they are not tied to milestones or sprints.
- a version is most often a version of software under testing. It might be for example a build number.
- An environment is where you test the software in.
- Versions, environments and milestones are separate from each other. Often you will have many software versions released and tested during one milestone, but Testlab does not enforce this. For reporting, all these can be used separately or in combination. You can for example find out what you tested during “Milestone 1” in “System test” environment.
When your test runs contain information about the version and the environment, your information is automatically transferred to a bug report when you create one during testing. This is often very good information in troubleshooting.
How does Testlab store my data?
It helps to understand how the central quality-related assets are stored in your tool. First, all your data (excluding your company details) will be stored under a project. You should think of the project as a domain that will include all requirements (specification of some kind), test cases for them, executed testing results, issues, etc for a single product, application or a system under testing.
The next diagram shows a simplified diagram about how the different assets relate to each other in Testlab inside a project.
Id addition to the picture above, Tasks can be linked to Requirement, test run or an issue.
Each project can be split into milestones, and milestones can be split in to sprints which are project reference points that aim to be completed at some point of time. Milestones can be used in various ways to separate the assets inside the project from each other and in addition, using milestones makes it easier to track the progress of testing. For more examples on how milestones can be used please see the 5.2 Milestones section of Testlab’s help manual.
Every good test plan is based on a solid set of requirements. In Testlab, requirements can be anything that can be used to specify, define or qualify features and functionality for the product, service or an application under testing. Testlab is development process agnostic, so you can define your requirements as user stories, use cases, traditional business requirements or something else that fits best to your development process. You can even use requirements to group your test cases to the grouping of your liking if you for some reason prefer to skip using the requirements in the specification phase. See the next chapter on more pointers on user requirements.
For the testers to be able to test the system they must have information about what they are testing and most of the time instructions on how something should be tested. For this, test cases are defined to include description, pre-conditions, the actual step by step instructions and expected end results. For the tool to be able to automatically track which functionality each test case is testing, they are usually linked to requirements. When a test case is linked to a requirement we say the test case verifies this requirement.
When test cases are executed they are always executed in a test run. A test run records the results for each test case that is included in it and specifies for which version (of the software for example) the testing was executed against and in which environment the testing was executed.
During testing, test runs are recorded with testing results for each test case. Good. But when something is encountered which requires attention later on it should be recorded as an issue. Issues can be seen as tasks that should be handled somehow later on. Issues can be defects encountered (i.e. bugs in software), future development ideas, new features or something else you wish to record for later use. Keep in mind that you can add issues anytime – they don’t have to be results of some specific testing session.
All major testing data assets: requirements, test cases, and issues have automatically stored history data. This means you can find out what these assets have been at each point in time.
Is there any data in Testlab which can be shared between Projects?
Yes, there is.
- Users: Users of your company are of course shared between all your projects.
- Workflows: Workflow schemes can be configured once and shared between projects. When you set up a new project you can select a workflow scheme from a list of your previously configured schemes. A workflow scheme defines the “who can do what on which assets” in a project.
- Notice schemes: Notice schemes can be shared between projects. A notice scheme defines the events which trigger e-mail notifications to your users in a project.
- Reports: Reports can be configured and published to be shared between your projects.
- Announcements: Announcements can be published on your dashboard to all your projects.
How to manage tasks in Testlab
Testlab has a taskboard that can be used to manage tasks that are popular in agile methodologies. Managing tasks this way gives the team better visibility what the team is doing as a whole and helps track the situation, distribute work meaningfully and prioritize the tasks.
There is a good best practices document here, and you should read it first.