Meet Boris Buyer. Boris has a successful web store selling old books in the internet which has been providing a steady income for him and his family. It was christmas season of 2018 when he noticed his sales declining. He decided to call some of his old and faithful customers and asked if there was something he could do to serve them better.
Boris’ web store was old. Yes, you could order books from the store but only way to actually place an order was to send an e-mail message to Boris. His customers were telling him that he really should look into his web store and make the order process better. There were all kinds of requests for new payment methods, delivery options and more and Boris had a feeling he was losing customers to more agile competitors.
This is a description of a DEMO project you’ll get as an example when registering your own Testlab and aims to demonstrate how a small typical software project could be managed with Testlab. Keep in mind that the chosen agile project model is just one kind of project style you could use.
You can find highlight thumbnails from Testlab’s user interface in relevant sections. You can click the thumbnail to open up associated screenshot with an informative caption.
Luckily, Boris has a brother in law named Victor who owns an IT-company specializing in web design. Boris meets up with Victor on holidays and agrees on tentative plan to implement a shopping cart to his web store enabling his customers to place orders more easily.
And so it happens that Victor has few months free for this team right away at the start of January 2019. Boris and Victor agree that it is necessary to implement the basic functionality of the shopping cart as soon as possible. The first milestone of the project has only the basic shopping cart functionality enabling self-service shopping without the clumsy e-mailing. Victor suggests that the current Cash On Delivery payment type available as an only payment option is probably a show stopper for many customers. Boris and Victor agree that the second milestone for the project would be to implement PayPal payment enabling credit card shopping. To make it possible to offer discounts for holiday sales Victor suggests that Boris should consider implementing coupon codes to his web store. Coupons would grant a discount on orders. Boris likes the idea and it is agreed to be implemented as a third milestone in the project.
A rough sketch on the project contents and it’s timetable is agreed:
|Milestone 1 – Develop a basic shopping cart to the web store
|Week 1||Requirement planning and design|
|Week 2 and 3||Implementation / programming and test design|
|Week 4||Acceptance testing for Milestone 1|
|Week 5||Installation to production, Milestone 1 in use for customers|
|Milestone 2 – Add PayPal support
|Week 5||Requirement planning and design|
|Week 6 and 7||Implementation / programming and test design|
|Week 8||Acceptance testing for Milestone 2|
|Week 9||Installation to production, Milestone 2 in use for customers|
|Milestone 3 – Add support for discounts via coupons
|Week 9||Requirement planning and design|
Boris and Victor agree that Victor and his team will do the preliminary design before the turn of the year which enables a snappy start for the project. Victor suggests that for user interface design they employ an old friend who has promised to deliver approved user interface layouts by the end of week 1 of 2019. Victor also knows that the real usability and accessibility for the shopping experience is crucially important. He employs an old colleague who has expertise in usability field to review the final implementation and deliver a report on it.
Victor recommends to Boris that it is wise to use a good tool to document the features, test cases and all the related assets. This makes the future easier for Boris if and when he decides to develop his online ventures further. Boris likes the idea as in addition, this keeps Boris more up to date on what is going on in his project. Naturally, Victor suggests of using Meliora Testlab and Boris agrees.
The following diagram presents a timetable for the project explained above. The upper half of the diagram shows the phases of the project and the bottom half present when at the latest the milestone relevant assets are added to the Testlab project. The timeline is divided to weeks and the purple highlight on timeline presents the moment in time the project data currently is in.
Victor’s trusty team of developers is assigned to this project. The responsibilities for the project are divided as follows:
Victor P. Owner
And as external resources the team will be backed up with
User Interface Designer
Victor’s team is familiar with agile methods and prefers modeling requirements as user stories. As a team they decide to use a hybrid method by modeling the features of the Shopping Cart as user stories and other features as generic requirements.
Victor hosts a design workshop meeting with his team and Boris and they talk through the requirements for the Cart implementation. Victor tells Boris that he will document the requirements to Testlab and Boris should review and approve them later on so that they match the discussed features.
Victor starts working. He knows, that all features planned to be implemented in the first milestone will act as a basis for all further work in Boris’ web store. This way, he decides to create all assets of the first milestone without targeting them explicitly to the first milestone. He goes on, and adds a new milestone Milestone 1 to the project and configures it to inherit all [Non-targeted] assets.
He brings in his expertise on web application design and decides to leave the actual Cart features to later time and start on generic security and usability related requirements. He wants the requirements to be easily tracked later on in the hierarchy and by their identifier so he creates two folders “SEC – Security” and “U – Usability“ to his project. When a requirement is specified as a Folder it means that the requirement just groups sub requirements below it. Victor goes on and adds the needed security and usability related requirements to these folders.
As said, the actual functionality of the Shopping Card is decided to be modeled as agile user stories. Victor decides to create user stories to a hierarchy by the actor. He adds some folder classed requirements as
The sub folder named by the actor will hold the actual user stories for each actor. Victor goes on and adds the user stories to each folder. US1 will hold the user stories of the Customer meaning the user who is shopping in Boris’ online book store. US2 will hold the user stories of the system – in this case the Web store. US3 will hold the user stories of the Seller meaning Boris. Victor documents the user stories gathered earlier to these folders.
When Victor is happy with the requirements and user stories he goes through them once more. He prints out some reports, takes his work home and has look through the requirements before telling Boris to approve them. Victor thinks the requirements are now good enough, assigns requirements to Boris and marks the them as ready for review.
To make sure the team and the buyer are on the same page it was agreed that Boris will review and approve the documented requirements. Boris goes on and finds that Victor has assigned the documented requirements to Boris for review. Boris reads through the requirements, finds the requirements as adequate and approves them as ready. This tells the team that the requirements have been approved by the buyer and the design can go on.
Liana takes on the challenge and starts to design the test cases which will verify the requirements Victor and Boris have planned. First, she creates a proper test category structure for the test cases to make the grouping of test cases easier. She decides to follow the similar actor based category structure as in requirements for all functional tests and add some folders for security and usability related test cases. Liana also informs Pedro to reserve some time for test case review at the end of the week as Liana plans to have the test cases ready as soon as possible.
Liana reads through the requirements and goes through the user interface layouts provided by the user interface designer and starts to type in test cases. With the help of Dean she describes the needed security test cases to fulfill the security requirements defined. Liana knows that the usability will be reviewed by an external expert. For this reason for usability, she just adds a stub test case for usability defined as just to ensure that the report the expert will deliver gives a green light for the cart’s usability. Liana knows that it is not always necessary to design as detailed steps for test cases as possible – just as detailed as needed to ensure that the requirements are verified.
When Liana is happy with her test cases she goes thru them once more. She additionally ensures that all the test cases are bound correctly to the requirements which the test cases are designed to verify. For this Liana just drags and drops the test cases on the requirements for the binding.
Finally Liana assigns test cases to Pedro for review. At the end of the week Pedro logs in to Testlab and as agreed goes through the test cases and reviews and approves them. He uses the Requirements-, Test Case Design- and Test Coverage -sections to see if the test cases are detailed enough to cover and verify Boris’ requirements for the Shopping Cart. Pedro likes what he sees and as he goes thru the requirements he edits them and marks them as Covered. This tells the team that the test cases have been reviewed and Pedro thinks that the test cases are good enough to verify the requirements for Boris’ Shopping Cart.
Pedro knows that Dean has been working day and night to get the first milestone done. He knows that Dean would like that Liana would start the testing on the features already testable as soon as possible. He goes to work and starts thinking of sets of test cases that would be run as a test runs in this milestone.
Execution planning in Testlab is used to plan repeatable test sets and add the test runs to be executed.
Pedro knows that it’s possible to add the needed test runs to Testlab directly by using his personal work set in execution planning. Then again, he knows that he probably should add some repeatable test sets to the system as the project will span multiple milestones which will be tested fully as they are finished. He ponders a bit and ends up adding a “Production deployment tests” titled set of tests which he thinks should list the minimum set of test cases always run to a new version before it should be accepted to be installed to production environment.
Pedro wants to easily find the actual acceptance test sets for each milestone so he goes on and adds a new test set category “Acceptance tests“. To this category he copies the already created production deployment tests as “Milestone 1 Acceptance tests” and adds the few missing test cases to it which are not contained in the original test set, for example he knows that “Validate accessibility” test case is not run for every version because they have agreed for the Usability expert to deliver the usability report for only the first version.
Pedro speaks with Dean and gets informed about the functions Dean thinks will be testable by end of the week. He knows that Liana is also able to pick the test cases to be executed herself but chooses to help Liana in her efforts by adding test run “Milestone 1 Sprint 1 tests” for her to test the Dean’s progress with. Pedro set the starting date for the test run for Jan 11 and expects the tests be run for next monday and assigns the test run to Liana.
Pedro also adds two additional test runs for Milestone 1’s acceptance tests and installation tests. The first to contain all test cases that should pass for the version for it to be accepted as working and the latter one for test cases that should be run to verify that the version works correctly in production environment after the installation.
Dean is eager to know if the functionality he has already implemented works at all. Liana gets notified via Testlab that she has a test run assigned to her to be run. She opens up the test run in Testlab, selects all tests in it and starts testing.
For few first test cases everything seems to be working fine and Liana is impressed on the work Dean has done. She knows though that as this is the very first test run run against the Cart she expects some issues to be found in it. And so it is that the third time is the charm and the “Remove items from cart” test case does not work as expected. A confirmation dialog is missing when she removes all items for a specific product from the Shopping Cart. Liana marks the test case step and the test case as failed and adds an issue “DEMO-1 – Removing all items for a product does not show confirmation dialog” for Dean to resolve.
Liana goes on runs all the remaining test cases and finds three more issues with failing test cases. She adds the issues and assigns them to Dean.
Dean knows that the first milestone version should be in acceptance testing by the week four. He gets notifications from Testlab that Liana has found some issues in her first tests. Dean goes thru the issues and adds some comments to them when applicable. He goes on and fixes the issues, adds comments and marks the issues as resolved. He knows that the test cases for the issues should be retested before the acceptance tests so he assigns the issues back to Liana for closing.
Liana gets informed that the issues should now be fixed. She adds a new test run “Milestone 1 Sprint 1 retest” for test cases that failed in previous run and starts testing. While she is testing she closes the issues found in the previous run. She is satisfied to find that the issues are now resolved.
Liana has a talk with Dean and Pedro and gets informed that the implementation is nicely in schedule and the acceptance testing can be run as planned. Acceptance tests are run to verify that the requirements are covered and the planned functionality works as expected before installing.
Liana starts testing. The acceptance test run has all the test cases needed to pass for the first milestone version. She is glad to find that only one test case has some issues: An event log file that should be written to server’s file system was not found. She is a bit baffled as this file did exist previously as expected and adds a comment about the fact with the issue. She assigns the issue to Dean for resolving.
Dean reacts quickly and notices that the server has some file permission problems with the directory where the application is trying to write the event log file to. He quickly fixes the issue, marks the issue as resolved and adds a comment to the issue. As it was only a server specific issue Liana decides that she re-tests and closes this issue in the production after the installation.
The team meets up to discuss whether the version in testing environment is acceptable to production. Pedro presents the coverage of the Milestone 1 version and everything shows up green expect the traceability requirement regarding the event log file. As Liana explains in the comment of the issue the issue is environment specific and should not show up in the production environment. In addition to this the team goes thru the issues with issue list report and concludes that everything seems to be ok for production installation.
Victor meets up with Boris and reports that all should be green for the deployment of the first milestone version of the Shopping Cart. They agree that Wednesday the 30th of January is a good day for a small downtime in Boris’ web store. They plan to take the version to production and Liana is on standby to run the test cases scheduled after Dean reports back that installation has succeeded.
On the 30th Dean goes to work and sets up a maintenance break page for Boris’ web store. The maintenance break page is set up in a way that Liana has access to the production server but customers from the outside internet do not. Dean installs the Shopping Cart application to the servers and tells Liana to start testing.
Liana runs the test cases in “Milestone 1 installation tests” test run and is happy to find out that the Shopping Cart works as expected! Victor reports back to Boris and tells the good news. They decide to take down the maintenance break page and open up the brand new enhanced web store to the public!
For the project to go on as smoothly as possible Victor knows to start planning the next version of the Shopping Cart as soon as possible. He kicks of the design of the PayPal enabled Cart at the same time with the finishing of the previous milestone.
Victor gives a call to the User Interface Designer and makes sure that he is able to deliver the updated UI design at the end of week five at the latest so that the test case design and implementation can proceed as planned.
Victor knows that the next version is quite simple. He talks with Boris and gathers up the requirements for the PayPal payment option. Victor documents the findings and adds a couple of new user stories: “US1.5 – Choose to pay with PayPal” and “US2.3 – Get payments from PayPal“. He adds a new milestone Milestone 2 [PP] to the project and configures it to inherit all assets of the previous milestone.
Then Victor goes on and goes thru all the existing requirements to see if they need any refinement because of the new added payment option. He updates “US1.4 – Checkout and place an order” user story which creates a new version to the Testlab project for the user story. Finally Victor assigns the created and updated user stories to Boris and marks them ready for review and marks the user stories as not covered.
Boris logs in to Testlab and gets notified about requirements needing some work. He reads thru the user stories Victor has asked him to review, likes what he sees and approves the requirements as ready.
Liana talks with Dean and they go thru the changes in user stories. She finds the relevant requirements from the project easily by looking which requirements and user stories are currently marked as not covered. They agree that the changes in the application are quite small so they agree that it’s probably enough to test the implementation of Milestone 2 once and for all in acceptance testing phase. Dean is confident in his skills and promises Liana to deliver a working version in one go.
Liana knows that she must go thru the current test plan and make sure the test cases cover the new and updated user stories. She does this and ends up adding a new test case “Pay with PayPal” and updating a few: “Enter shipping address“, “List and view confirmed orders“, “Verify event log” and “Submit an order“. She assigns the test cases to Pedro for review so that she knows the test cases are given an another look before testing.
Pedro‘s e-mail inbox pings and he gets notified about the test cases to be reviewed. Luckily he has spare time in his hands so he goes on to read through the updated test cases and the requirements verified by them. He approves the test cases as ready and marks the requirements as covered.
Pedro knows about Liana’s and Dean’s plan to test the changes in one go at acceptance testing. He starts to think about the test runs needed for the testing and adds the “Milestone 2 Acceptance tests” test set with a list of test cases to verify that the Shopping Cart still works with the PayPal payment option included.
By using the test set added Pedro adds a new test run “Milestone 2 acceptance run“, assigns it to Liana due to be run between 18th and 20th of February.
Dean, Pedro and Liana meet up and agree that everything is in place for acceptance testing. On Monday the 18th Liana starts testing. She works hard for few days and finds that some issues are present in the application. Liana adds her findings to Testlab as issues and assigns them to Dean. The handling of shipping address for the order has some regression compared to the previous version and PayPal paid orders payment identifiers are missing from the Boris’ order management view.
Dean gets notified about new issues. He adds some comments to the issues, fixes them and marks the issues as resolved. Pedro gets informed about issues in acceptance. He reports this to Victor and schedules a re-run for the test cases related to the issues to be run on Wednesday 27th of February.
Third milestone is a next version of Shopping Cart with discount related features including handling of coupons.
At the same time when the previous second milestone version is in it’s finishing stage Victor thinks ahead and starts to plan user stories for the third milestone. He adds the third Milestone 3 [C] milestone to the project and, three new requirements: “US1.6 – Enter a coupon code for a discount“, “US2.5 – Validate coupons” and “US3.6 – Add new coupons“.
The first milestone with basic Shopping Cart features is happily in production and customers can now enjoy themselves more shopping the books Boris is selling. The second version with PayPal enabled payments had some issues which are fixed and a re-test test run is scheduled for it but it is not yet tested. The third milestone with coupons has been started with requirement planning.
You can and should log on to a Demo-project in your Testlab. The data included in this project represents the assets and actions executed in the above description and is left in the state of 25th of February 2019 – Only the automated testing is done after that.
Liana has noticed that much of her testing involves repeating similar patterns. Adding more automated tests to handle the routine inputs would allow her to spend more time on refining tests instead. Liana had heard of a testing framework called Robot Framework which would allow hew to create and maintain automated tests easily. Even better, Meliora Testlab reads the test results in allowing the same operations for automated tests as for manual one. The output of test results is elaborate enough for Pedro and Dean easily understand what the tests do. Dean gets super exited about this plan as it means the tests can be run right after each software build automatically, which in turn means the problems will be found right away – not just after the manual regression testing phase.
Liana goes through the manual regression test cases and picks ones that are most suitable for automation. She values the ones that are generally executed always the same way, tests that are executed most often and the ones that are most important. When Liana has created her fist automated tests that run well, she creates a ruleset Bookstore ruleset to Testlab, and adds an Add and Map – rule, that will import all test results by mapping them to Testlab test cases, creating new test cases when necessary. This allows her to import test results to Testlab right away when she is still developing the tests.
Later on Liana adjusts the ruleset by ignoring a test she knows does not currently work. She also wants to organise her tests a bit, and adds a rule that maps all tests related to ordering products to a single test case named Complete Order. She also adds another ignore-rule, ignoring those tests so they won’t get mapped twice. When Liana runs the tests using her automation tool she just drags & drops the test result file to Testlab. Now, using the newly created rules Testlab gets the results and adds the test cases to Testlab. Liana then updates the requirement verification situation for the new automated test cases so that she will be able to see what effect test execution has for the requirements.
This section contains some pointers on what to look for in the Testlab project. We recommend that you log on to the Demo project in your Testlab and follow the steps explained here to help you familiarize yourself to the concepts in Testlab.
For a single test run
For a single test run the status of the testing is seen by going to Test Runs view and selecting a test run. The pane below lists all test cases run in the test run, results for them and some additional details. You can expand each row to show a result card of an individual test case. To drill down into issues found for each test case you can click the value on the “Issues added” column.
Current status of requirements
As said, the project is in finishing stages of Milestone two which has some issues with it, milestone one is accepted and successfully installed and third milestone has been just started with requirement planning. To get an understanding of the project’s status is probably easiest achieved by looking at the Test Coverage view.
For a milestone
Go to Test Coverage view and select a milestone. By default, the coverage status is shown against all current requirements in the project’s up-to-date specification. For milestones “in the past” this is probably not what you want and you’d prefer a view in the past milestone which includes only the test cases intended to be run or have results for. You can get this by checking off the “All requirements” control. To go further please go on and select the “Milestone 2” from the coverage controls to get a glance at the status of the second milestone.
For a version
Go to Test Coverage view and select a version. A version here is a technical software version installed and is tracked as a typical software version number such as 1.1. As explained for Milestones above similarly with versions, you can control if you’d prefer to report the coverage against all requirements in your current specification or against only the requirements which relate to your version through executed test cases.
If you’d prefer to report the progress of your testing for a time period the easiest way is to generate a fitting report. Testlab provides you a set of reports out-of-box including progress reports for covered requirements, test case results and issues. Try it out and generate a progress report for project’s requirements and user stories by selecting Reports > Requirements > Testing progress for requirements from Testlab’s menu bar. You should get a report formatted as a PDF document about how the requirements and user stories of the project have been covered in testing during the time period.
The same kind of progress report can be generated for test cases of the project by selecting Reports > Test cases > Testing progress for test cases and entering similar report parameters as earlier. The report shows you how the results of test cases has been developing during a time period. Try it out with issues too by selecting Reports > Issues > Issues’ development.
Remember – there is a number of configurable report templates available out-of-box for your use. You can find descriptions of different report types in Testlab’s help manual by reading the Reporting chapter.
You can use and setup your dashboard with widgets you prefer to keep you up to date with your project. Try it out by configuring your dashboard and dragging and dropping the “Current issues” widget to some slot. The widget you dropped shows the current issues in the project – in the case of Shopping Cart project the few resolved issues still to be re-tested and closed for Milestone 2. The “Project time line” widget is also handy for a quick glance on recent events in your project.
Thank you for your attention! It should be noted that the Shopping Cart project chosen and described above is just one kind of quality related project you can manage with Testlab. The practices and methodologies used are picked to suit a small and agile project with a distributed team. If you for example think that reviewing is time consuming and unnecessary you can apply a simpler workflow for you project.