Test Configuration


Configuration Setting Overview

The configuration is done using a JSON object. Please note the text must use double quotes (") around all strings. The following parameters can be set:

  • xtcType: this should always be "XTC", unless a custom project template was created by Matrix Requirements staff
  • cloneSources: this should always be ["TC", "UC"], unless a custom project template was created by Matrix Requirements staff
  • presetFields: if set the "create test forms" dialog will ask for default values of these to set them in created forms. The available options depend on the project setup. The default are version, tester and executionDate
  • render: [] an array defining how test tables are rendered in each category (see below)
  • defaultTestResultResult: this is the initial value a an executed test case should have when created. Must be a 'code' from the values in 'automatic' or 'manual' section below.
  • automatic: if the overall test result should be generated from the individual test step results, this can be configured here. It is an array of items defining rules. The first rule which applies determines the status of the test result.
    • human: a human readable text explaining the test result
    • code: a unique identifier for the status of the test result
    • render: whether test result means 'ok' (all good), 'error' (test failed), 'warning' (e.g. not completed): this is used in report renderings
    • rule: the type of rule to determine from the test steps the overall status. Can be 'one', 'all' or ''. The first rule matching determines the status
    • param: parameter for the rule, i.e. a code of a test step
  • manual: These allow the user to set the overall test result manually.
    • human: a human readable text used in reports explaining the test result
    • code: a unique identifier for the status of the test result
    • render: whether test result means 'ok' (all good), 'error' (test failed), 'warning' (e.g. not completed): this is used in report renderings
    • command: the string in the drop down menu
  • perStep: Possible test results per test step.
    • human: a human readable text used in reports explaining the test result
    • code: a unique identifier for the status of the test result
    • render: whether test result means 'ok' (all good), 'error' (test failed), 'warning' (e.g. not completed): this is used in report renderings
    • command: the string in the drop down menu
    • key: the keyboard shortcut
    • image: so far only internally used to show icons for common states (passed, fail and docu)

Changing a test configuration:

Most of the changes can be done in the administration client's main UI

In the following video, see how to change the overall test results options:

Changing Test Table Columns

To change the columns of test tables, you need to click on the the advanced button the admin page (1).

When the script editor opens you can click on format (compact) (2) to more easily edit the columns (3).

The render object has one table definition for each category which contains tests, e.g. UC, TC and XTC.

Each definition is an object which defines the columns, e.g. "UC": {"columns": []} would define a table without columns. To add columns you need to specify for each column

  • a name (shown in user interface)
  • a field (an identifier under which the values are stored)
  • an editor (which specifies how data is entered, or none for readonly cells). See Table Configuration for a list of editors.

Note: When generating test forms from tests (e.g. an XTC from a TC) values are copied from the TC if the field identifiers are identical in both tables. So in the example above the column Action (with id action) from the TC is copied into the column Action Result (because it has the same id action). 



Example Configurations

Example Configuration
{
    "xtcType": "XTC",
    "cloneSources": ["TC", "UC"],
    "presetFields":[ 
	  {"field":"name", "value":"Version"},
      {"field":"name", "value":"Tester" } ],
    "render": {
      "UC": {
       "columns":[
         {"name": "Action UC", "field": "action", "editor": "text"  },
         {"name": "Expected Result UC", "field": "expected", "editor": "text"  }]
      },
      "TC":{
        "columns":[
          {"name": "Action 1 TC", "field": "action","editor": "text"  },
          {"name": "Action 2 TC", "field": "action2", "editor": "text" },
          {"name": "Expected Result", "field": "expected", "editor": "text"  }]
      },
      "XTC":{ 
        "columns":[
          {"name": "Action", "field": "action","editor": "none" },
          {"name": "Action Alt","field": "action2","editor": "text" },
          {"name": "Expected","field": "expected","editor": "none" },
          {"name": "Passed/Failed","field": "result","editor": "result" },
          {"name": "Comment","field": "comment","editor": "text"}]
      }
    },
    "defaultTestResultResult":"an",
    "automatic": [
        {"human": "passed", "code": "ap", "render": "ok", "rule": "all", "param": "p"},
        {"human": "failed", "code": "af", "render": "error", "rule": "one", "param": "f"},
        {"human": "not started", "code": "an", "render": "warning", "rule": "all", "param": ""},
        {"human": "documentation", "code": "ad", "render": "warning", "rule": "one", "param": "d"},
        {"human": "in progress", "code": "ai", "render": "warning", "rule": "", "param": ""}
    ],
    "manual": [
        {"human": "passed", "command": "passed", "render": "ok", "code": "p"},
        {"human": "failed", "command": "failed", "render": "error", "code": "f"},
        {"human": "", "command": "not executed", "render": "warning", "code": "r"},
        {"human": "to be decided", "command": "to be decided", "render": "warning", "code": "tbd"},
        {"human": "documentation", "command": "documentation", "render": "warning", "code": "d"},
        {"human": "in progress", "command": "in progress", "render": "warning", "code": "i"}
    ],
    "perStep": [
        {"human": "passed", "command": "passed", "render": "ok", "code": "p", "key": "p", "image": "success.png"},
        {"human": "failed", "command": "failed", "render": "error", "code": "f", "key": "f", "image": "fail.png"},
        {"human": "documentation", "command": "documentation", "render": "warning", "code": "d", "key": "d", "image": "docu.png"},
        {"human": "", "command": "not executed", "render": "warning", "code": "", "key": "r", "image": ""}
    ]
}


This example

  • adds a column to TC where users can select and design item
  • another column to XTC where user can select a FILE item

example with refs as columns
// 1) in the traceability of XTC add downrule, e.g. to a NEW category FILE (which you need to create....)
{
  "message": "files", "name": "can have file", "rule":"can_have", "any_of": [ "FILE" ] 
} 
// modify the xtc_config
{
  "xtcType": "XTC",
  "cloneSources": ["TC", "UC"],
  "presetFields":[  {    "field":"meaning",     "value":"Version"   },   {      "field":"name",     "value":"Tester"     } ],
  "render": {
    "UC": {
      "columns":[    
         {"name": "Action",          "field": "action",          "editor": "text"          },
         {"name": "Expected",          "field": "expected",          "editor": "text"          }]    },
    "TC":{
      "columns":[
        {
          "name": "Links",
          "field": "links",
          "editor": "design"  
        },
        {
          "name": "Action",
          "field": "action",
          "editor": "text" 
        },
        {
          "name": "Expected",
          "field": "expected",
          "editor": "text"  
        }]
    },
    "XTC":{
      "columns":[
        {
          "name": " Links",
          "field": "dref",
          "editor": "none" 
        },
        {
          "name": "Action",
          "field": "action",
          "editor": "none" 
        },
        {
          "name": "Expected",
          "field": "expected",
          "editor": "none" 
        },
        {
          "name": "Passed/Failed",
          "field": "result",
          "editor": "result" 
        },
        {
          "name": "Test Results",
          "field": "files",
          "editor": "downrules"  
        },
        {
          "name": "Comment",
          "field": "comment",
          "editor": "text"
        }]
    }
  },
  "defaultTestResultResult":"an|warning|not started",
  "automatic": [
    {
      "human": "passed",
      "code": "ap",
      "render": "ok",
      "rule": "all",
      "param": "p"
    },
    {
      "human": "failed",
      "code": "af",
      "render": "error",
      "rule": "one",
      "param": "f"
    },
    {
      "human": "not started",
      "code": "an",
      "render": "warning",
      "rule": "all",
      "param": ""
    },
    {
      "human": "documentation",
      "code": "ad",
      "render": "warning",
      "rule": "one",
      "param": "d"
    },
    {
      "human": "in progress",
      "code": "ai",
      "render": "warning",
      "rule": "",
      "param": ""
    }
  ],
  "manual": [
    {
      "human": "passed",
      "command": "passed",
      "render": "ok",
      "code": "p"
    },
    {
      "human": "passed with deviation",
      "command": "passed with deviation",
      "render": "ok",
      "code": "pf"
    },
    {
      "human": "failed",
      "command": "failed",
      "render": "error",
      "code": "f"
    },
    {
      "human": "",
      "command": "not executed",
      "render": "warning",
      "code": "r"
    },
    {
      "human": "to be decided",
      "command": "to be decided",
      "render": "warning",
      "code": "tbd"
    },
    {
      "human": "documentation",
      "command": "documentation",
      "render": "warning",
      "code": "d"
    },
    {
      "human": "in progress",
      "command": "in progress",
      "render": "warning",
      "code": "i"
    }
  ],
  "perStep": [
    {
      "human": "passed",
      "command": "passed",
      "render": "ok",
      "code": "p",
      "key": "p",
      "image": "success.png"
    },
    {
      "human": "failed",
      "command": "failed",
      "render": "error",
      "code": "f",
      "key": "f",
      "image": "fail.png"
    },
    {
      "human": "documentation",
      "command": "documentation",
      "render": "warning",
      "code": "d",
      "key": "d",
      "image": "docu.png"
    },
    {
      "human": "",
      "command": "not executed",
      "render": "warning",
      "code": "",
      "key": "r",
      "image": ""
    }
  ]
}