Posts tagged with: reporting


5.3.2018

How to: Migrate from TestLink to Meliora Testlab

transSome of our customers have moved from using TestLink to our tool, Meliora Testlab. We’ve been asked to make this transition easier and so we decided to document this migration path, and this  post here describes how the migration works. Basically the migration moves your important test data from TestLink to Testlab so you can continue to work in your new tool.

 

When changing the tool, do I need to change the way I work?

This is a tough subject that deserves another post completely dedicated to that matter. To put it shortly, Testlab offers a lot of features that allow working in new ways, but most of the Testlink features are also in Testlab, so if there is no need to alter your way of working, you can do most of your work in the same way as before. TestLink’s “Test specification” view’s data can be seen in Testlab in “Test case design”. Assigning test cases to be tested in the simple form in Testlab is done by picking the cases to work set and then creating a test run, which tells the what release / version is being tested. Test execution is again pretty much the same again.

In nutshell – how does this all work?

In TestLink there is an export feature that allows you to export your test case data as XML. We can transform this data in to a format that can be directly put in in to the Testlab. When custom fields have been used in Testlink, the tool needs to be instructed how the custom fields are mapped in to Testlab fields. After the definition, a click of a button transform the data in to desired format.

For your convenience, Meliora will do this transformation free of charge (*). For our On-premise customers we can also deliver the tool to allow doing the transformation yourself.

The migration has five distinct steps:

  1. Modifying the Testlab to used format. ( optional )
  2. Exporting Test cases from TestLink testsuites.
  3. Describing the data mappings ( optional )
  4. Transforming the TestLink Export XML to csv format that Testlab can read
  5. Importing the test case data from csv.

The techical part behind these steps is pretty straightforward. Once you know how you want to migrate the data, the export-import will be just a few clicks for you.

*) Meliora reserves rights to decline of the free service on cases where making the transformation would engage Meliora with exeptional workload. This could happen if you would have huge amount of projects to be migrated.

 

What do I need to do before the migration?

Well, the only required thing is becoming a Meliora customer. Any edition of Testlab will do. A very very recommended thing to do is to plan how do you want to test with Testlab in the future. As Testlab offers many features that make testing easier that are not present in Testlink, it might be wise to change the way how testing is done at the same time when the tool is being changed. For example testlab has automatic revisioning, history, built-in (optional) review etc. so you might have made customizations to TestLink for these issues. These customizations are probably not needed anymore, so you need to decide if it is better just to ditch the customizations or do you want to continue using them. If you are in a hurry, fear not! Then you can just import all custom data and ditch the unneeded later.

 

General considerations

Most important thing to ensure is that your company’s testing work is not interrupted more than it has to. When you switch the tool, it is not effective if you continue using the old tool ( in migrated projects ) as you would be getting test case updates in two tools and test results in two tools for same project. Thus it is best to decide a timeslot for the switch and ensure everything goes smoothly when the time comes.

It is best to contact Meliora as you plan the migration and prepare a timeslot in case you want to keep to switch time in absolute minimum. Just create a support ticket and Meliora will help you getting the migration done smoothly.

Migration steps

 

Modifying the Testlab

Testlab, by default, has different fields and field Values than TestLink. You can, if you want, do the migration without modifying Testlab. Then the fields are going to be mapped by default. In case you decide to do the modification, these are the things to consider:

Choosing / Modifying a workflow

workflow-editIn testlab, the way how statuses are used are controlled by workflows. Workflows allow logic behind changing statuses – what statuses can be reached from what status, what fields are mandatory in which state and who has privileges to make changes. Testlab comes with two default workflows, called “simple” and “review”. The difference for test cases here is that review has a phase for review, where simple skips this. Basically you need to decide what statuses you wish to see in Testlab. The default Status transformations are depicted in following table:

 

 

 

 

 

TestLink Status
Testlab Simple workflow
Testlab With review workflow
Draft In design In design
Ready for review Ready for review In design
Review in progress Ready for review In design
Rework In design In design
Obsolete Deprecated Deprecated
Future Ready Ready
Final Ready Ready

You can add additional data transformations for the migration in addition to simply changing statuses. For example the “Testlab way” for handling test cases that are not yet runnable, is using a “Milestone” field to define when the test case is planned to be used. More on this on chapter “Describing the data mappings”.

Modifying the Testlab project

In case you have used custom fields and wish to keep the data in the custom fields in the future as well, you need to configure your Testlab’s project to include those. You’ll find the instructions for that in Testlab’s manual – it’s very simple operation.

Keep in mind, that you also have an option to import data from custom fields to description field. This makes sense when you do not want to lose this information, but you do not need to filter data in reports using custom field data.

If you are migrating data to multiple Testlab projects you can do the common modifications once and then copy the project with modifications. This way you do not need to do the modifications for all the projects separately.

 

Exporting test cases from TestLinkTestlink export options

In Testlink, you need to export test cases as XML for each project you wish to migrate.

  • In Testlink, go to “Test specification”
  • Choose “Actions” -> Export All Test Suites
  • make sure you have at least four boxes checked
  • Save your XML export file

 

Describing the data mapping

Before the transforming the data you can define how the TestLink data is to be transformed in the migration. Our on-premise customers that wish to do the transformation themselces do not need to do this as they are going to put this data in to tool themselves. Send the following information  to Meliora:

  • What custom fields are to be migrated. For each field describe to what field you want to import the data in to.
    • Example1 ) TestLink custom field “risk” values to Testlab field “risk”
    • Example 2) TestLink custom field “legacy link” added to the end of “description” field
  • How you want to change the data values
    • Example 1) TestLink custom field “risk” value “petty” to be “Low” in Testlab
    • Example 2) TestLink status “Future” to Testlab status “In design” + Milestone to value “Future”
Transforming the XML to Testlab csv

Here you have a few options:

  1. For SaaS users you can just send the XML to Meliora, and Meliora will do the transformation and import the data to your Testlab project. Just wait for the confirmation and you can start using your Testlab with imported data!
  2. For On-premise / SaaS users that want Meliora to just do the transformation, send the XML to Meliora and Meliora will deliver you an CSV that is ready to be imported in to testlab.
  3. For On-Premise users that want to do the transformation themselves, contact Meliora for details. Meliora will deliver you the tool along with instructions to do the transformation.
Importing the data from csv

The import itself is just a few click really:

  1. From Testlab menu choose Import -> Test cases
  2. Choose the csv file
  3. Try the import first with “dry run” option on. This will show errors / warnings should there be any. Here you will see if, for example, if you do not have a custom field in this project where you try to input data with the csv file.
  4. After you are satisfied with dry run results, uncheck the dry run box to really load the data in to the Testlab.
  5. Refresh the test case tree and start using your Testlab.

Final words

This document describes the basic setup of the migration process. As with all project more complicated than Hello World!, there will be unknown factors. You might have in-house customization done to your TestLink or you might want to include data from completely another source. Fear Not! Meliora can work with migrations that does not follow the ordinary path. Just contact our great support and we will work trough the migration together!

Meliora team

 

Facebooktwitterlinkedinmail


Tags for this post: announce features product release reporting screenshots 


26.1.2018

Testlab – Helios release

A new version of Meliora Testlab – Helios – has been released. In this version, one of the goals was to optimize the speed of the user interface which makes this version of Testlab faster than ever. Please read more about the changes below.

 

Faster UI

Several enhancements have been made to optimize the UI, rendering and backend services. Some are workarounds to browser-related bugs, some are different rendering strategies and some are optimizations in situations where there are lots of assets in your project. 

We hope these changes make the use of Testlab snappier than ever. As the changes are browser-specific and may benefit some users more depending on the configuration of the workstation, we are more than happy to receive any feedback from you on your experiences.

 

Comments for test results
File attachments for steps of test cases

Previously, one could attach files to test cases. In Helios, you can also attach files to individual steps of the test case. The editor for editing steps has a new configurable column available for which you can attach any files relevant to the specific step.

The files are presented to the tester when the test case is in testing. The files are also included to reports and to e-mail messages sent for the test cases if preferred.

 

Expand attached images to listing reports

Requirements, test cases, and issues in Testlab can be attached with files. These files might be images, which is most common in case of issues where images such as screenshots are used to describe the issues encountered.

In Helios, the detailed listing reports (List of requirements, List of test cases and List of issues) have a new option to expand the attached images. This way, when the report is rendered, the images are expanded and shown in the printed report.

And there is more

In addition to the changes above:

  • SAML-authentication can be used to authenticate to web-published reports.
  • When hovering the assets in Testlab’s UI, the hovers now show also the files attached to the assets.
  • Help-browser window can now be resized and left open while using Testlab. As previously, you can also open up the help to separate browser tab.
  • API: The responses of REST API have been enhanced in a way that attached files are also included in responses. The related objects of assets also provide HAL-compatible links now for easier navigation in the response graph.
  • Concurrent editing and testing of test cases is now handled with better and more clear warning messages. 
 

Thanking you for all your feedback,

Meliora team


Helios 2

Helios 1 and Helios 2 are probes launched in 1974 and 1975 to study solar processes. They are no longer operational but remain in their orbits around the Sun.

The probes were once fastest man-made objects at 252 792 km/h (157 078 mph) – over 6 times around the earth in an hour. That is 70.22 kilometers per second. From London to New York in 79 seconds. On the other hand, it is only 586 times faster than the fastest production car Bugatti Veyron 16.4 Super Sport. For Helios 2, It would take 18028 years to reach Alpha Centauri, the nearest star system to the Sun. With Bugatti Veyron, it would take 10.5 million years. And while driving, you would have to fill the tank 459 900 000 000 times which equals 45 263 661 534 000 liters of gasoline. If we construct a cube as a gasoline tank for the fuel, we would need a cube that is approx. 3.6 kilometers per side.

(Source: Wikipedia)

Facebooktwitterlinkedinmail


Tags for this post: announce features product release reporting screenshots 


13.10.2017

Testlab – Parallax Denigrate release

We are proud to bring you a new version of Testlab – Parallax Denigrate – with usability-related features and risk analysis reporting.

 

Custom fields for projects
Targeted filtering of tree assets

Assets organized in tree structures, such as specification related assets (requirements, test cases, and test sets, can be filtered by a search field at the top of the tree control. This makes it easy to filter into the tree the assets which you are interested in.

This search field has been enhanced in a way which enables you to target the search better. For example, for test cases, filtering with

  • “priority:critical” shows all test cases prioritized as critical,
  • “name:plane tag:customer” shows all test cases with a word “plane” in the name of the test case and with a tag “customer”.

As said, this logic works with all trees in Testlab – including the pickers which are used to select assets related when editing. The UI has also been added with an easy help on the syntax as a tooltip.

 

Comments for test results
Risk analysis reporting

Often, the people responsible for the project are asked tough questions on the status of the project or product: How well is it tested? Can we go to production? Do you think the product is in a good enough for release? The manager must basically make an educated guess with the best possible information available – often from the data in the test management tool.

Testlab has been added with a Risk Analysis report which analyzes the requirements, test cases, and issues in the project and combines the information from risk view-point as a single report. The report includes

  • requirements not marked as covered, with failing or blocking tests or requirements not tested at all – grouped by requirement risk,
  • test cases not yet marked as ready, not passed or test cases which have not been tested at all – grouped by priority and
  • issues not yet resolved or closed – grouped by severity or priority.

As a combined single report, this report provides essential information for the manager to make his decision.

 

Issue-centric testing

Features have been added to support issue-centric testing better. Previously, issues in Testlab could be linked to a test case and a test run but in this new version of Testlab, it is possible to link multiple test cases (optionally via a test run as executed tests) to a single issue. The controls while editing the issue have been changed appropriately to make it easier to link an existing issue to a test case.

New functions have also been added to better utilize these links:

  • While running tests, you can now easily add the test cases linked to an issue as executable tests to your current run. To do this, click the “Add linked test cases to this run” control via the Issues tab while running tests.
  • While running a test, you can also add a link to an existing issue by clicking the “Link to current test case” control via the Issues tab.

Note: Due to this change the REST-API model has been updated. The old format is still supported for a single link but is due to be deprecated in future releases. Please see the documentation of DefectResource-endpoint in your Swagger instance (https://yourtestlab.melioratestlab.com/api).

 

Showing results in test case tree

The controls at the bottom of the test case tree are used to highlight the test cases with testing results. In previous versions of Testlab, the controls limited you to fetching the not passing results only. The controls were designed from the re-testing and regression testing point of view.

In Parallax Denigrate, these controls were changed in a way so that you have an option to simply filter in results for test cases from your preferred viewpoint (latest results in the project, results for a milestone, for a version or for a single test run). The actual change in the UI is that you also get the results for passed test cases. This makes these controls more flexible and the use of these controls is easier in different usage scenarios.

 

Better tree-pickers

The popup controls used to pick related assets are now much easier to use:

  • The pickers now feature the same searching feature introduced earlier (see “Targeted filtering of tree assets” above). This makes finding the asset from the tree much easier.
  • When picking assets, you have an option to add or remove the selected assets from the set of current values.

 

Reporting enhancements
  • The “Do not include sums of zero” option on the requirement, test case, issue and project grouping reports now filter out sums of zeroes from both axes (“Field to report” and “Group by”). Previously, the option filtered out sums of zeroes only from “Field to report” axis.
  • When old revisions (or otherwise deprecated assets) are included in reports, the assets are now highlighted with a gray color and also, the date when the asset was revised/deprecated is shown.
  • When reporting the latest results only on “Results of run tests” report, the report now also includes the test cases not yet run.
  • “Execution status of test cases” report has been added with two new options to customize the report’s behavior (“Test cases in test runs only” and “Result for latest revision”). Please see the inline tooltip helps for these two options on this report on the logic of these options.
 
Other miscellaneous changes
  • When editing a single issue, you can save the edits without closing the window by keeping Shift-key pressed.
  • When batch editing assets in the table view, top-right corner change event notifications are no longer generated to your teammates.
  • For deprecated test cases in the list of test cases in test execution view now includes the timestamp of the deprecation (when the test case was revised or when the test case was deprecated).
  • The state of “Show events” checkbox at the top right corner is now saved and remembered between sessions.
  • Copying and pasting with keyboard shortcuts now work as expected to step editor’s pre- and post-conditions.
 

Thanking you for all your feedback,

Meliora team


Bookhouse Boys

Usenet is a distributed discussion system which enables you to read and post messages, often termed as “news”, still existing but more popular back in the days. Usenet features newsgroups on various different topics and was a popular place for discussion.

In 1996, a series (hundreds) of unexplained word puzzles were posted to different newsgroups. The title for all these posts was “Markovian Parallax Denigrate” and each post contained a series of words which cryptographers and hackers have been trying to decipher – with little success. Some theories exist but the “Markovian Parallax Denigrate” has still been referred as “The Internet’s oldest and weirdest mystery”.

(Source: Wikipedia, The Daily Dot, Last (?) existing post in Google’s archive)

Facebooktwitterlinkedinmail


Tags for this post: announce features product release reporting screenshots 


13.7.2017

Testlab – Bookhouse Boy release

To celebrate the warmth of the summer months, we are proud to release a new feature version of Testlab: Bookhouse Boy.

 

Custom fields for projects
Custom fields for projects

Just like requirements, test cases, and issues, the project asset can now also be defined with custom fields. You can find the field settings for your project asset from Testlab > Manage company … view.

When you customize the fields for your project asset these field settings will take effect in all your projects. For example, you can use these fields to track some high-level project details inside Testlab such as project deadlines, resourcing related details or something else.

For reporting out these details two new project templates have been added:

  • “List of projects” which is basically a list of project details from your Testlab instance. The project filters in projects by the criteria set and lists out project details as a listing.
  • “Project grouping” which can be used to group project data by defined fields. This report includes a bar graph for the grouped data and a listing of projects matching the set criteria.

Please note: As these two reports expose details on project level, “report.all” permission does not grant permission for these two new reports. This way, these two reports might not show up with your permission set automatically. You should adjust the role your have in your projects to include “report.projectlistreport” and “report.projectgroupingreport” to have access to these reports.

 

Comments for test results
Test result comments

Each result of the test case can now be set with a distinct comment. Earlier, test case assets themselves and also steps of executed tests could be commented.

Now as you create test runs, the test cases get bound as items in these runs. As you list out test cases in your scheduled runs in the Test execution view, there is a new column in this table for an execution comment. Note that you set the comment when you wish – You don’t have to execute the test and give it a result to include a comment: This way you can also use this comment field for some information before execution of the actual tests.

The comments for executed (or scheduled) tests are also shown in test cases’ execution history view, on related reports and on test coverage view (for tests with a result set). Also for Jenkins integration, the comments sent via the API are also set to this field, if any.

 

Report page sizing and orientation

Reports can now be rendered in different page sizes and in chosen orientation. The size of the page can be set as A4 or the larger A3 and the report can be laid out in portrait or in landscape orientation. Please experiment with these settings if you have trouble fitting all your data on your reports.

 

Reporting enhancements
  • Test case related reports now support filtering in test cases from test sets and test runs. If you filter in test cases with test runs, keep in mind, that the latest execution status of the test case is then reported against these filtered in runs.
  • “Results for run tests” report now has an option to filter in latest executed result for each test case. This way, if the same test case is included multiple times, only the latest executed results are left in on the report.
  • A project can now be set with a logo which is included in each report. You can upload the logo of your choice by editing the project details.
  • Grouping reports can now be grouped also with custom field values.

 

Progress indication

The UI now includes automatic notification on server operations that might take some time to complete. Also, most operations expected to run some time such as imports of data also support showing the progress of the operation in this notification dialog.

 
Support for new file formats

Exporting data is now possible as modern Excel-files (.xlsx). Also, reports can now be exported as .docx-files for Microsoft Word.

 

Other changes
  • The changes in the tags of the asset are now stored and included in the changes history of requirements, test cases, and issues.
  • The number of defects found -column in test execution view can now be used as a filter in the table.

 

Thanking you for all your feedback,

Meliora team


Bookhouse Boys

With Bookhouse Boy, we wish to celebrate and give homage to the true visionaries behind Twin Peaks – one of the greatest television drama series created. After 25 years, a new season of this great television is currently running offering new bizarre twists in the story of interesting characters including the small town of Twin Peaks itself. With the excellent writing of Frost and visionary directing of Lynch, the series is sure to leave it’s mark to the history of television.

All Testlab releases are code-named with some weird fact, theory, historical event or something else that should give some food for your imagination. As such, a code-name from Twin Peaks is more than fitting.

(Image from Twin Peaks (the original series) / Lynch/Frost Productions)

Facebooktwitterlinkedinmail


Tags for this post: announce features jenkins product release reporting screenshots 


12.4.2017

Testlab – Lilliput Sight released

Meliora is proud to announce a release of new major Testlab version: Lilliput Sight. This release includes some new major features such as asset templating but in the same time, bundles various smaller enhancements and changes for easier use. New features are described below.

 

Asset templating

 
Asset templating

Creating assets for your projects in a consistent way is often important. This release of Testlab includes support for templates which you can apply when adding new requirements, test cases or issues.

A template is basically a simple asset with some fields set with predefined values. As you are adding a new asset, you have an option for applying templates. When you apply a template, the asset is set with the values from the applied template. You can also apply multiple templates, so designing templates to suit the needs of your testing project is very flexible.

A set of new permissions (testcase.templates, requirement.templates, defect.templates) has been added to control who has access to using the templates. These permissions have been granted to all your roles which had the permission to edit the matching asset type.

 

Robot Framework output support
Robot Framework output support

When pushing automated results to Testlab, we now have native support for Robot Framework’s custom output file. By supporting the native format, the results include detailed information of keywords from the output which are pushed to Testlab as the steps executed.

The support for Robot Framework’s output has also been added to the Jenkins CI plugin. With the plugin, it is easy to publish Robot Framework’s test results to your Testlab project in a detailed format.

 

Copying and pasting steps

The editor for test case steps now includes a clipboard. You can select steps (select multiple steps by holding your shift and ctrl/cmd keys appropriately), copy them to the clipboard and paste them as you wish. The clipboard also persists between test cases so you can also copy and paste steps from a test case to another.

 

 

Filtering with operators in grids

The text columns in Testlab’s grid now feature operator filters which allow you to filter in data from the grid in more specific manner. You have an option of choosing the operator the column is filtered with such as “starts with”, “ends with”, … , and of course the familiar default “contains”.

With a large number of data in your project, this makes it easier to filter in and find the relevant data from your project.

 

 

Mark milestones, versions and environments as inactive

When managing milestones, versions, and environments for your project, you now can set these assets as active or inactive. For example, if a version is set as inactive, it is filtered out from relevant controls in the user interface. If your project has numerous versions, environments or milestones, keeping only the relevant ones active makes the use easier as the user interface is not littered with the non-relevant ones.

For versions and environments, the active flag is set in the Project management view. For milestones, the completed flag is respected as in completed milestones are interpreted as inactive.

 

Usability related enhancements
  • Editing asset statuses in table views: You can now edit statuses of assets in table views – also in batch editing mode.
  • New custom field type – Link: A new type of custom field has been added which holds a single web link (such as http, https or mailto -link).
  • Support deletion of multiple selected assets: The context menu for requirements and test cases now includes “Delete selected” function to delete all assets chosen from the tree.
  • Delete key shortcut: The Delete key is now a keyboard shortcut for context menu’s Delete-function.
  • Execution history respects revision selection: The execution history tab for a test case previously showed a combined execution history for all revisions of the chosen test cases. This has been changed in a way that the tab respects the revision selection in a similar manner to the other tabs in the Test case design view. When you choose a revision, the list of results in the execution history tab is only for the chosen revision of the test case.
  • Custom fields hold more data: Earlier, custom fields were limited to a maximum of 255 characters. This has been extended and custom fields can now hold a maximum of 4096 characters.
  • Test cases already run in test runs can be run again: If the user holds the permission for discarding a result of an already run test case, you can now choose and execute test cases already with a result (pass or fail) directly from the list of test cases in Test execution view. Earlier, you needed to discard all the results one by one for the tests you wish to run again.
  • Enhancements for presenting diagrams: The presentation view for relation and traceability diagrams has been improved – you can now zoom the view and pan the view in a more easier manner by dragging and by double clicking.
  • Copy link to clipboard: The popup dialog with a link to open up an asset from your Testlab has been added with a “Copy to clipboard” button. Clicking this button will copy the link directly to your clipboard.

 

Reporting enhancements
  • “Filter field & group values” option added for grouping reports: Requirement, test case, and issue grouping reports have been added with an option to apply the filter terms of the report to the values which are fetched via “Field to report” and “Group by”. For example, if you filter in a requirement grouping report for importance values “High” and “Critical”, choose to group the report by “Importance” and check the “Filter field & group values” option, the report rendered will not include reported groups for any other importance values than “High” and “Critical”.

 

Enhancements for importing and exporting
  • Export verified requirements for test cases: Test case export now includes a “verifies” field which includes the identifiers of requirements the test cases are set to verify.
  • Input file validation: The input file is now validated to hold the same number of columns in each read row than the header row of the file has. When reading the file, if rows are encountered with an incorrect number of columns, an error is printed out making it easier to track down any missing separator characters or such.

 

Thanking you for all your feedback,

Meliora team


Alice

Various disorientating neurological conditions that affect perception are known to exist. One of them is broadly categorized as Alice in Wonderland Syndrome in which the people experience size distortion of perceived objects. Lilliput Sight – or Micropsia – is a condition in which objects are perceived to be smaller than they actually are in the real world.

The condition is surprisingly common: episodes of micropsia or macropsia (seeing objects larger than they really are) occur in 9% of adolescents. The author of Alice’s Adventures in Wonderland – Lewis Carroll – is speculated to have had inspiration for the book from his own experiences in Micropsia. Carroll had been a well-known migraine sufferer which is one possible cause of these visual manifestations.

(Source: WikipediaThe Atlantic, Image from Alice’s Adventures in Wonderland (1972 Josef Shaftel Productions))

 

Facebooktwitterlinkedinmail


Tags for this post: announce features integration jenkins plugin product release reporting 


23.2.2017

Unifying manual and automated testing

Automating testing has been an ongoing practice to gain benefits for your testing processes. Manual testing with pre-defined steps is still surprisingly common and especially during acceptance testing we still often put our trust in the good old tester. Unifying manual testing and automated testing in a transparent, easily managed and reported way is particularly important for organizations pursuing gains from testing automation.

 

All automated testing is not similar

The gains from testing automation are numerous: Automated testing saves time, it makes the tests easily repeatable and less error-prone, makes distributed testing possible and improves the coverage of the testing, to bring up few. It should be noted though that not all automated testing is the same. For example, modern testing harnesses and tools make it possible to automate and execute complex UI-based acceptance tests and in the same time, developers can implement low-level unit tests. From the reporting standpoint, it is essential to be able to combine the testing results from all kinds of tests to a manageable and easily approachable view with the correct level of details.

 

I don’t know what our automated tests do and what they cover

It is often the case that testers in the organization waste time on manual testing of features that are already covered with a good set of automated tests. This is because the test managers don’t always know the details of the (often very technical) automated tests. The automated tests are not trusted on and the results from these tests are hard to combine to the overall status of the testing. 

This problem is often complicated by the fact that many test management tools report the results of manual and automated tests separately. In the worst case scenario, the test manager must know how the automated tests work to be able to make a judgment on the coverage of the testing. 

 

Scoping the automated tests in your test plan

Because the nature of automated tests varies, it is important that the test management tool offers an easy way to scope and map the results of your automated tests to your test plan. If is not often preferred to report the status of each and every test case (especially in the case of low-level unit tests) because it makes it harder to get the overall picture of your testing status. It is important to pay attention to the results of these tests though so that failures in these tests get reported.

Let’s take an example on how the automated tests are mapped in Meliora Testlab.

In the example above is a simple hierarchy of functions (requirements) which are verified by test cases in the test plan:

  • UI / Login -function is verified by a manual test case “Login test case“,
  • UI / User mgmnt / Basic info and UI / User mgmnt / Credentials -function is verified by a functional manual test case “Detail test case” and
  • Backend / Order mgmt -functions are verified by automated tests mapped to a test case “Order API test case” in the test plan.

Mapping is done by simply specifying the package identifier of the automated tests to a test case. When testing, the results of tests are always recorded to test cases:

  1. The login view and user management views of the application are tested manually by the testers and the results of these tests get recorded to test cases “Login test case” and “Detail test case“.
  2. The order management is tested automatically with results from automated tests “ourapp.tests.api.order.placeOrderTest” and “ourapp.tests.api.order.deliverOrderTest“. These automated tests are mapped to test case “Order API test case” via automated test package “ourapp.tests.api.order“.

The final result for the test case in step 2 is derived from the results of all automated tests under the package “ourapp.tests.api.order“. If one or more tests in this package fail, the test case will be marked as failed. If all tests pass, the test case is also marked as passed.

As automated tests are mapped via the package hierarchy of the automated tests, it makes it easy to fine tune the detail level you wish to scope your automated tests to your test plan. In the above example, if it is deemed necessary to always report out the detailed results on order delivery related tests, the “ourapp.tests.api.order.deliverOrderTest” automated test can be mapped to a test case in the test plan.

 

Automating existing manual tests

As testing automation has clear benefits to your testing process, it is preferred for the testing process and the used tools to manage it to support easy automation of existing manual tests. From the test management tool standpoint, it is not relevant which technique is used to actually automate the test, but instead, it is important that the reporting and coverage analysis stays the same and the results of these automated tests are easily pushed to the tool.

To continue on with the example above, let’s presume that login related manual tests (“Login test case“) are automated by using Selenium:

The test designers record and create the automated UI tests for the login view to a package “ourapp.tests.ui.login“. Now, the manual test case “Login test case” can be easily mapped to these tests with the identifier “ourapp.tests.ui.login“. The test cases themselves, requirements or the structure of these do not need any changes. When the Selenium based tests are run, later on, the result of these tests determine the result for the test case “Login test case“. The reporting of the testing status stays the same, the structure of the test plan is the same, and related reports are easily approached by the people formerly familiar with them.

 

Summary

Testing automation and manual testing are most often best used in a combination. It is important that the tools used for test management support the reporting of this kind of testing in as flexible way as possible.

 

(Icons used in illustrations by Thijs & Vecteezy / Iconfinder)

Facebooktwitterlinkedinmail


Tags for this post: automation best practices example features product reporting usage 


15.11.2016

Testlab – Earthquake Light released

Meliora Testlab – Earthquake Light – has been released. In addition to set of fixes, this release comes with a new advanced reporting mode which allows you to customize the criteria of data on your reports in an intuitive manner. We’ve also integrated with Stripe to make subscription handling and credit card payments as easy as possible.

Please read on for more detailed description of the new features.

 

Inline table editing
Reporting criteria in advanced mode

Most of the report templates available have been added with an option to switch the criteria form to the new advanced mode. In this mode, the criteria for picking the data on the report can be specified as a set of rules the reporting engine will use when rendering the report.

Reports in advanced mode work in similar manner to the earlier so called simple mode. You can save them, schedule them for publishing and so on.

 

Advanced operators
Various operators for each field

A rule on an advanced reports consists of the targeted field, an operator executed to determine if the rule matches and an optional value for the operator. The mode has been implemented with a full set of operators allowing you to define a complex set of rules.

The operators available depend on the type of the field.

 

 

Boolean operators and sub-clauses

The criteria may be defined with sub-clauses. Each clause is set with an boolean operator (match all [and], match any [or], match none [not]) which declares how the list of rules in the clause should be interpreted.

This allows you to define a complex set of rules to pick the data you need on your report.

 
Stripe subscriptions for credit card billing

The hosted Meliora Testlab has been integrated with Stripe – a leading credit card payment gateway – to make subscription and credit card handling for you as easy as possible.

If any action is needed from existing customers, we will contact all customers directly for instructions.

 

Meliora team


earthquakelights

For a very long time, people have occasionally reported seeing strange bright lights in the sky before, during or after an earthquake. For much of the modern times, these reports were considered questionable and skeptics say, there is no considerable proof of the existence of such phenomenon.

The lights reported are usually blueish and greenish dancing lights in the sky comparable a bit to the aurora borealis. Theories exist on the cause of the lights such as the electricity in certain rocks of the earth gets activated and gets discharged during the teutonic stress. It has even been suggested that the existence of the lights could be used to predict upcoming earthquakes.

The latest the earthquake lights were reported was with the New Zealand earthquake in November 2016 and there are even videos circulating documenting the event.

(Source: Wikipedia, National Geographic, Youtube, Photo from UC Berkeley Online Archive)

 

Facebooktwitterlinkedinmail


Tags for this post: announce features product release reporting usage 


23.1.2016

New feature release: Oakville Blob

We are proud to announce a new major release of Meliora Testlab: Oakville Blob. A major feature included is a Rest-based API for your integration needs. In addition, this release includes numerous enhancements and fixes. Read on.

 

Integration API

 
Integration API

Meliora Testlab Integration API offers Rest/HTTP-based endpoints with data encapsulated in JSON format. The first iteration of the API offers endpoints for

  • fetching primary testing assets such as requirements, test cases and issues,
  • fetching file attachments and
  • pushing externally run testing results as test runs. 

We publish the API endpoints with Swagger, a live documentation tool, which enables you to make actual calls against the data of your Testlab installation with your browser. You can access this documentation today by selecting the “Rest API …” from the Help menu in your Testlab. You can also read more about the API from the corresponding documentation page.

Tagging users in comments

 
 
Tagging team members to comments

When commenting assets in your project such as issues or test cases, you can now tag users from your project to your comments. Just include the ID, e-mail address or name of the team member prefixed with @ character and this user automatically gets a notification about the comment. The notice scheme rules have also been updated to make it possible to send e-mail notifications for users who are tagged to the comments of assets. Also, a new out-of-box notice scheme “Commenting scheme” has been added for this.

See the attached video for an example how this feature works.

 

Execution history of a test case

Sometimes it is beneficial to go through the results of earlier times of execution for a test case. To make this as easy as possible, Test design view has been added with a new “Execution history” tab.

This new view lists all times the chosen test case has been executed with the details similar to the listing of test run’s items in Test execution view. Each result can be expanded to show the detailed results of each step.

 

Notification enhancements

When significant things happen in your Testlab projects, your team members are generated with notifications always accessible from the notifications panel at the top. Oakville Blob provides numerous enhancements to these notifications such as

  • notifications panel has been moved to the top of viewport and is now accessible from all Testlab views,
  • notifications panel can now be checked as kept open meaning, that the panel won’t automatically close on lost focus of the mouse cursor and
  • detailed documentation for all notification types has been added to the help manual.

In addition, the following new notification events have been added:

  • if an issue, a requirement or a test case which you have been assigned with is commented, you will be notified,
  • if during testing, a test case or it’s step is commented, the creator of the test run will get notified,
  • if a step of test which you have been assigned with is added with a comment during testing, you will be notified.

 

Targeted keyword search

The quick search functionality has been enhanced with a possibility to target the keyword to specific information in your project’s assets. By adding a fitting prefix: to your keyword, you can for example target the search engine to search from the name of a test case only. For example, 

  • searching with “name:something” searches “something” from the names of all assets (for example, test cases, issues, …) or,
  • searching with “requirement.name:something” searches “something” from the names of requirements.

For a complete listing of possible targeting prefixes, see Testlab’s help manual.

 

Usability enhancements

Several usability related changes have been implemented.

  • Easy access to test case step’s comments: Test execution view has been added with a new “Expand commented” button. By clicking this button the view expands all test run’s test cases which have been added with step related comments.
  • Rich text editing: The rich test editors of Testlab have been upgraded to a new major version of the editor with simpler and clearer UI.
  • Keyboard access:
    • Commenting: keyboard shortcuts have been added for saving, canceling and adding a new comment (where applicable).
    • Step editing: Step editor used to edit the steps of a test case has been added with proper keyboard shortcuts for navigating and inserting new steps.
  • Test coverage view: 
    • Test case coverage has been added with a new column: “Covered requirements”. This column sums up and provides access to all covered requirements of listed test cases.
    • Test run selector has been changed to an auto-complete field for easier use with lot of runs.

 

Reporting enhancements

Several reporting engine related changes have been implemented.

  • Hidden fields in report configuration: When configuring reports, fields configured as hidden are not anymore shown in the selectors listing project’s fields.
  • Sorting of saved reports: Saved reports are now sorted to a natural “related asset” – “title of the report” -order
  • Better color scheme: The colors rendered for some fields of assets are changed to make them more distinct from each other.
  • Test run’s details: When test runs are shown on reports, they are now always shown with the test run’s title, milestone, version and environment included.
  • Wrapped graph titles: When graphs on reports are rendered with very long titles, the titles are now wrapped on multiple lines.
  • Results of run tests report: The report now sums up the time taken to execute the tests.
  • Test case listing report: Created by field has been added.

 

Other changes

With the enhancements listed above, this feature release contains numerous smaller enhancements.

  • New custom field type – Unique ID: A new custom field type has been added which can be used to show an unique non-editable numerical identifier for the asset. 
  • Editing of test case step’s comments: The step related comments entered during the testing can now be edited from the Test execution view. This access is granted to users with “testrun.run” permission.
  • Confluence plugin: Listed assets can now be filtered with tags and, test case listing macro has an option to filter out empty test categories.
  • Test run titles made unique: Testlab now prevents you from creating a test run with a title which is already present if the milestone, version and environment for these runs are the same. Also, when test runs are presented in the UI, they are now always shown with this information (milestone, version, …) included.
  • “Assigned to” tree filter: The formerly “Assigned to me” checkbox typed filter in the trees has been changed to a select list which allows you to filter in assets assigned to other users too.
  • File attachment management during testing: Controls have been added to add and remove attachments of the test case during testing.
  • Dynamic change project -menu: The Change project -selection in Testlab-menu is now dynamic – if a new project is added for you, the project will be visible in this menu right away.
  • Permission aware events: When (the upper right corner floating) events are shown to users, the events are filtered against the set of user’s permissions. Now the users should be shown only with events they should be interested in.
  • Number of filtered test runs: The list of test runs in Test execution view now shows the number of runs filtered in.
  • UI framework: The underlying UI framework has been upgraded to a new major version with many rendering fixes for different browsers.

 

Thanking you for all your feedback,

Meliora team


Oakville

“On August 7, 1994 during a rainstorm, blobs of a translucent gelatinous substance, half the size of grains of rice each, fell at the farm home of Sunny Barclift. Shortly afterwards, Barclift’s mother, Dotty Hearn, had to go to hospital suffering from dizziness and nausea, and Barclift and a friend also suffered minor bouts of fatigue and nausea after handling the blobs. … Several attempts were made to identify the blobs, with Barclift initially asking her mother’s doctor to run tests on the substance at the hospital. Little obliged, and reported that it contained human white blood cells. Barclift also managed to persuade Mike Osweiler, of the Washington State Department of Ecology’s hazardous materials spill response unit, to examine the substance. Upon further examination by Osweiler’s staff, it was reported that the blobs contained cells with no nuclei, which Osweiler noted is something human white cells do have.”

(Source: WikipediaReddit, Map from Google Maps)

 

Facebooktwitterlinkedinmail


Tags for this post: announce features integration jenkins plugin product release reporting 


24.10.2015

Testlab Girus released

Meliora Testlab has evolved to a new major version – Girus. The release includes enhancements to reporting and multiple new integration possibilities.

Please read more about the new features and changes below. 

 

Automatic publishing of reports

Testlab provides an extensive set of reporting templates which you can pre-configure and save to your Testlab project. It used to be that to render the reports, you would go to Testlab’s “Reports” section and choose the preferred report for rendering.

Girus enhances reporting in a way where all saved projects can be scheduled for automatic publishing. Each report can be set with a

  • recurrence period (daily, weekly and monthly) for which the report is refreshed,
  • report type to render (PDF, XLS, ODF),
  • optionally, a list of e-mail addresses where to automatically send the report and
  • an option to web publish the report to a pre-determined web-address.

When configured so, the report will then be automatically rendered and sent to interested parties and optionally, made available to the relatively short URL address for accessing.

 

Slack integration

slack_rgb (1)

Slack is a modern team communication and messaging app which makes it easy to set-up channels for messaging, share files and collaborate with your team in many ways.

Meliora Testlab now supports integrating with Slack’s webhook-interfaces to post notifications about your Testlab assets to your Slack. The feature is implemented as a part of Testlab’s notice schemes. This way, you can set up rules for events which you prefer to be publishes to your targeted Slack #channel.

You can read more about the Slack integration via this link.

 

Webhooks

Webhooks are HTTP-protocol based callbacks which enable you ro react on changes in your Testlab project. Webhooks can be used as a one-way API to integrate your own system to Testlab.

Similarly to the Slack integration introduced above, the Webhooks are implemented as a channel in Testlab’s notice schemes. This way you can set up rules to pick the relevant events you wish to have the notices of.

An example: Let’s say you have an in-house ticketing system and you need to mark a ticket resolved each time an issue in Testlab is closed. With Webhooks, you can implement a simple HTTP based listener on your premises and set up a notification rule in Testlab to push an event to your listener everytime an issue is closed. With little programming, you can then mark the ticket in your own system as resolved. 

You can read more about the Webhooks via this link.

 

Maven plugin

Apache Maven is a common build automation tool used primarily for Java projects. In addition, Maven can be used to build projects written in C#, Ruby, Scala and other languages.

Meliora Testlab’s Maven Plugin is an extension to Maven which makes it possible to publish xUnit-compatible testing results to your Testlab project. If you are familiar with Testlab’s Jenkins CI plugin, the Maven one provides similar features easily accessible from your Maven project’s POM.

You can read more about the Maven plugin via this link.

 

Automatic creation of test cases for automated tests

When JUnit/xUnit-compatible testing results of automated tests are pushed to Testlab (for example from Jenkins, Maven plugin, …), the results are mapped and stored against the test cases of the project. For example, if you push a result for a test com.mycompany.junit.SomeTest, you should have a test case in your project with this identifier (or sub-prefix of this identifier) as a value in the custom field set up as the test case mapping field. The mapping logic is best explained in the Jenkins plugin documentation.

To make the pushing of results as easy as possible the plugins now have an option to automatically import the stubs of test cases from the testing results themselves. This way, if a test case cannot be found from the project for some mapping identifier, the test case is automatically created to a configured test category.

 

Jenkins plugin: TAP support

Test Anything Protocol, or TAP, is a simple interface between testing modules in a test harness. Testing results are represented by simple text files (.tap) describing the steps taken. The Jenkins plugin of Testlab has been enhanced in a way, that the results from .tap files produced by your job can be published to Testlab.

Each test step in your TAP result is mapped to a test case in Testlab via the mapping identifier, similarly to the xUnit-compatible way of working supported previously.

The support for TAP is best explained in the documentation of the Testlab’s Jenkins plugin.

 

Support for multiple JIRA projects in Webhook-based integration

When integrating JIRA with Testlab using the Webhooks-based integration strategy, the integration now supports integrating multiple JIRA projects to a single Testlab project. When multiple projects are specified and an issue is added, the user gets to choose which JIRA project the issue should be added to.

Due to this change, it is now also possible to specify which JIRA project the Testlab project is integrated with. Previously, the prefix and the key of the projects had to match between the JIRA and Testlab project.

 

Miscellaneous enhancements and changes
  • Results of run tests report added with new options to help you on reporting out your testing results:
    • Test execution date added as a supported field,
    • “Group totals by” option added which sums up and reports totals for each group of specified field,
    • reports have been added with columns and rows of sum totals and
    • test categories selection supports selection of multiple categories and test cases for filtering.
  • Updates on multiple underlying 3rd party libraries.
  • Bugs squished.

 

Sincerely yours,

Meliora team


Pithovirus

Pithovirus sibericum was discovered buried 30 m (98 ft) below the surface of late Pleistocene sediment in a 30 000-year-old sample of Siberian permafrost. It measures approximately 1.5 micrometers in length, making it the largest virus yet found. Although the virus is harmless to humans, its viability after being frozen for millennia has raised concerns that global climate change and tundra drilling operations could lead to previously undiscovered and potentially pathogenic viruses being unearthed.

A girus is a very large virus (gigantic virus). Many different giruses have been discovered since and many are so large that they even have their own viruses. The discovery of giruses has triggered some debate concerning the evolutionary origins of the giruses, going so far as to suggest that the giruses provide evidence of a fourth domain of life. Some even suggest a hypothesis that the cell nucleus of the life as we know it originally evolved from a large DNA virus.

(Source: Wikipedia, Pithovirus sibericum photo by Pavel Hrdlička, Wikipedia)

 

Facebooktwitterlinkedinmail


Tags for this post: announce features integration jenkins plugin product release reporting 


7.7.2015

Meliora Testlab – Polybius – released

Meliora Testlab team is proud to announce a new version of Testlab – Polybius. In addition to smaller enhancements many of the new features revolve around parameterization of test cases. It is now possible to create test cases as templates and when executing them, enter the parameters of the template to easily create variants of the test case.

Please read more about the new features and changes below. 

 

Test case parameters

 
Test case templating with parameters

Test cases can now be embedded with parameters. Parameters are basicly tags entered to test cases’ description, preconditions, steps or to the expected end result. In the content text, parameters should be entered as

    ${PARAMETER}

When test case is entered with parameter values later on, the actual description shown to the tester will include parameter tags replaced with entered values.

Using parameters is an easy way to create many executable variants for a test case with similar description content. For example, if you have a set of test cases intended to be run with each different web browser variant, you can describe your test case with a appropriate ${BROWSER} tag. This way you can easily plan & execute this test case for different web browsers by entering fitting values for this parameter later on.

 

Parameter typing

Testlab includes a management view for all project’s parameters. You can configure your parameters as selection lists, which restricts the values that can be selected for these parameters.

 

Planning and running tests with parameters

To create test case variants for parameterized test cases you can now enter values for the parameters in Execution planning view. The editor features a column showing all possible parameters for the test case and allows you to enter values for them. When a parameterized test case (template) is added to a test set and it is set with parameter values, the test case becomes a parameterized instance to be run.

Test execution view has similar controls where you can enter parameter values for test cases already added to a test run.

 

Issue management enhancements
  • Issue is added with a new field “Found with parameters” which always includes the parameters of the test case the issue was added for (if any).
  • The issue listing includes a similar field as a column which allows you to easily filter in issues with specific parameter value.
  • When adding a new issue while executing a test case, the failed steps and their comments so far are now automatically copied to the issue’s description field.

 

Coverage and reporting with parameterized tests

Test coverage view with coverage views via requirements and test cases have been enhanced to include parameterized test cases. Each parameterized variant of a test case is now calculated as a single verifying test case in the coverage calculation. The view also shows the parameters the test cases were run with.

The reports also now include the test case parameters in appropriate places (Issue listing, requirement coverage, results of run tests and execution status of test cases report). Issue listing, results of run tests and execution status of test cases reports also have test case parameter values as filters which allows you to filter in specific results from your test runs.

 

Jenkins plugin changes

Meliora Testlab Jenkins plugin now supports passing test case parameters from Jenkins’ environmental variables. You can list the variables which are passed as they are with the results and will be included as parameters at Testlab side.

For example, test case(s) in your Testlab project might have a parameter titled ${BROWSER}. When running automated UI tests in your Jenkins, enter a value “BROWSER” to this new setting and ensure that your Jenkins job sets an environmental variable BROWSER to some sensible value. This way running the job sends and sets the ${BROWSER} test case parameter value to all run tests to a value matching to the environmental variable.

 

Importing test runs

A CSV import of test runs has been added. Importing test runs supports importing test run details, test cases run in them and testing results of test cases and their steps.

 

Centralized authentication (CAS) improvements

Registration of new user accounts

Previously, when users authenticated via an external source via CAS came into Testlab, if they had not yet been created with Testlab user account, they could not log on. This is because authorization to Testlab’s projects are done via Testlab’s user accounts the needed roles and permissions must be granted to these users for the access.

This version is improved in a way, that when a user without Testlab account gets authenticated via CAS, he/she is shown with a welcome screen allowing her to register a personal Testlab account by entering few details (e-mail address, full name). The Testlab user account is then automatically created and an e-mail messages is sent to the company administrators notifying them about the new account.

Granting roles to new accounts (Onpremise)

On-premise version of Testlab can be configured to automatically grant a set of roles in a set of projects when a new user account is registered via CAS.

Relaxing the SSL verification of the host name for ticket validations

When setting up CAS, the client can now be configured to skip the host name check of the SSL certificate when verifying CAS tickets. This may help setting up CAS with self-signed certificates or with proxy configurations.

 

Sincerely yours,

Meliora team


eisernermann

Fancy a game ? Polybius is an arcade game which is said to have induced various psychological effects on players. The story describes players suffering from amnesia, night terrors, and a tendency to stop playing all video games. Around a month after its supposed release in 1981, Polybius is said to have disappeared without a trace. [Wikipedia]

In the span of a week, three children really did fall ill upon playing video games at arcades in the Portland area. Michael Lopez got a migraine, the first he’d ever had, from playing Tempest. Brian Mauro, a 12-year-old trying to set the world record for playing Asteroids for the longest time, fell ill after a 28-hour stint. And only a week later 18-year-old competitive gamer Jeff Dailey died due to a heart attack after chasing the world record in Berserk. One year later 19-year-old Peter Burkowski followed suit for the same reason playing the same game. [Eurogamer]

(Photo by DocAtRS [CC BY-SA 3.0], via Wikimedia Commons)

 

 

Facebooktwitterlinkedinmail


Tags for this post: announce features jenkins product release reporting video 


 
 
Best-of-class cross-browser hosted SaaS quality and test management, testing and issue management tools to improve your quality. Site information.