The Linux Foundation

 
LsbinstallTest

From The Linux Foundation

Lsbinstall Test Development

Please insert notes on lsbinstall test development here

Test Assertions

http://cvs.gforge.freestandards.org/cgi-bin/cvsweb.cgi/tests/lsb-runtime-test/wip/lsbinstall/tset/?cvsroot=lsb


Some initial comments on the lsbinstall assertions... June 2 2005

We're missing a categorization of the assertions according to IEEE 1003.3. This will help us to understand whether a test is for a mandatory or optional feature, and also whether its practical to test it.

We should also have some form of reference including the categorization to enable bug reporting against individual assertions, since its primarily a manual page we could just have the section number within the specification and then sequence numbers for the assertions You would then also embed the same reference numbers within the test code so that references can be made between testcases and individual assertions.

for example

Reference 3.4-1 (A) (A is a mandatory case, shall...) Reference 3.4-2 (C) (C is an optional case, may, if the ... then)

As well as the Reference and the assertion text please include the results that can be returned by the assertion, for example

Reference 3.4-1 (A) The /bin directory exists and is searchable Result: PASS/FAIL

Reference 3.4-37 (C) If the implementation supports a C Shell or equivalent (tcsh):

The path name of the C Shell shall be /bin/csh.

Result: PASS/FAIL/UNSUPPORTED Notes: UNSUPPORTED will be returned if no csh or tcsh is present, this will be denoted by a boolean configuration variable to indicate whether a C shell is provided


If you decide to omit some tests categorizing them as (B) or (D) as per IEEE 1003.3 then give some reasons for the omission, for example:

Reasons for test omissions:

1. This would be too difficult to test, i.e. would require a system reboot.

2. The feature is optional, little value is seen in testing its presence. Its presence could be made a documentation assertion if felt necessary

Looking specifically at one of your assertions:

@ Test general.1

>># Test general.1 "If the -c or --check option is specified, lsbinstall shall exit with a zero status if an object of the specified type and name is already installed, or non-zero otherwise. Otherwise, lsbinstall shall exit with a zero status if the object with the specified type and name was successfully installed (or removed if the -r or --remove option was specified), and non-zero if the installation (or removal) failed."

Problem:

This needs to be split into several assertions, since there are multiple test cases here as indicated in your strategy. See criteria for test development enclosed later--- its best to have an assertion for each requirement for traceability, rather than having one assertion testing multiple requirements.

Action: For example:

Reference # description.1-1 (A)

When the -c option is specified, a valid object type is specified by -t type , and the operand respresents an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of zero, and shall print a diagnostic message to STDOUT

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails.

Reference # description.1-2 (A)

When the -c option is specified, a valid object type is specified by -t type , and the operand does not represent an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of non-zero.

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails. It appears to be unspecified whether a diagnostic message is returned.

Reference # description.1-3 (A)

When the -c option is specified, an invalid object type is specified by -t type , and the operand represents an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of non-zero.

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails. It appears to be unspecified whether a diagnostic message is returned.

Reference # description.1-4 (A)

When the -c option is specified, an invalid object type is specified by -t type , and the operand does not represent an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of non-zero.

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails. It appears to be unspecified whether a diagnostic message is returned.


Reference # description.1-5 (A)

When the --check option is specified, a valid object type is specified by -t type , and the operand respresents an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of zero, and shall print a diagnostic message to STDOUT

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails.

Reference # description.1-6 (A)

When the --check option is specified, a valid object type is specified by -t type , and the operand does not represent an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of non-zero.

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails. It appears to be unspecified whether a diagnostic message is returned.

Reference # description.1-7 (A)

When the --check option is specified, an invalid object type is specified by -t type , and the operand represents an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of non-zero.

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails. It appears to be unspecified whether a diagnostic message is returned.

Reference # description.1-8 (A)

When the --check option is specified, an invalid object type is specified by -t type , and the operand does not represent an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of non-zero.

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails. It appears to be unspecified whether a diagnostic message is returned.


Reference # description.1-9 (A)

When the --check option is specified, a valid object type is specified by --type=type , and the operand respresents an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of zero, and shall print a diagnostic message to STDOUT

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails.

Reference # description.1-10 (A)

When the --check option is specified, a valid object type is specified by --type=type , and the operand does not represent an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of non-zero.

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails. It appears to be unspecified whether a diagnostic message is returned.

Reference # description.1-11 (A)

When the --check option is specified, an invalid object type is specified by --type=type , and the operand represents an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of non-zero.

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails. It appears to be unspecified whether a diagnostic message is returned.

Reference # description.1-12 (A)

When the --check option is specified, an invalid object type is specified by --type=type , and the operand does not represent an object name that has been successfully installed, then the lsbinstall utility shall have an exist status of non-zero.

Result: PASS/FAIL/UNRESOLVED

Notes: Its possible that UNRESOLVED might be returned if the test setup to instance an object type of the appropriate name fails. It appears to be unspecified whether a diagnostic message is returned.

You probably need the combinations of -c and --type=type, and also to add assertions for the end piece of your original


For background guidance on assertion writing and general test acceptance criteria that we have used previously. See

http://www.opengroup.org/openbrand/testing/testprocs/process.html and http://www.opengroup.org/testing/lsb-fhs/process.html#criteria

These are also at the TestDevelopmentGuidelines page.


[Article] [Discussion] [View source] [History]