The Linux Foundation

 
CairoTesting40

From The Linux Foundation

Contents

Distribution Testing Status for Cairo

New test planning for libcairo aren't in the schedule until 2009. For the short term, cairo ships with a number of unit tests which can be adapted and patched for TET journal reporting. Although not of the typical LSB assertion -> test style tests, they can at least check for the presence of a number of libcairo interfaces.

Build Time Issues

There is a bug in cairo-1.0.2 that prevents building, I'm using the patch included in freedesktop.org bugzilla to fix this issue.

Patching for TET reporting proved to be fairly easy. The build of the test reporting is in cairo-test.c, although pthread-show-text.c, and xlib-surface.c use a slightly different model than the rest of the tests. user-data is just a bunch of asserts, did not try to tet'ify it. (tetj patch)

We were lacking some png functions to build with lsbcc. These have now been added, with png_jmpbuf to go in soon.

Also a couple of issues with freetype code in the cairo source. Use of deprecated ft_glyph_format_bitmap vs FT_GLYPH_FORMAT_BITMAP, and for some reason fterrors.h needs to be include explicitly to get Ft_Error_OK defined. This is handled with another patch.

Build Time Test Issues

Some tests fail at build time (make check builds and runs the tests against the just built libcairo). To avoid running the xlib tests, I just unset DISPLAY (we're not really interesting in running all the tests at build time).

Even so, some tests still fail on SLES10 (our build host):

FAILS on ia32 at build time (disabled):

text-antialias-none

FAILS on ppc64 at build time:

text-antialias-subpixel-image

FAILS on ia64, s390 at build time:

clip-operator-image
operator-clear-image
operator-source-image
show-text-current-point-image
text-antialias-gray-image
text-antialias-subpixel-image
text-pattern-image
unbounded-operator-image

Since we're not really interesting in testing the just-built libcairo, the thing to do here is seperate the build and check functions at build time.

Distribution Testing

Running the resulting tests (packaged as part of desktop test) also show a number of failures, the failures increasing with the respective cairo version installed on the distribution in question. We enable the xlib tests at runtime, which required making sure the render extension is enabled for lsb-xvfb (disabled for the other desktop tests). Fortunately the failures from the lesser versions of cairo are subsets of those from the greater versions, so disabling the superset could allow for clean runs on a variety of systems.

Failed test Images (all)

SLES10 (ia32, cairo 1.0.2)

clip-operator-xlib 1 FAIL
unbounded-operator-xlib 1 FAIL

Test was run: 20080808 14:16:21
Test Suite Version: unset
Test Suite Architecture: unset
Total Tests Passed: 112
Total Tests Failed (including waived): 2
Total Tests Failed (excluding waived): 2

Info from test log:

Error: 30897 pixels differ from reference image ./clip-operator-ref.png
Error: 11890 pixels differ from reference image ./unbounded-operator-ref.png

Failed test Images

RHEL5 (ia32, cairo-1.2.4)

clip-operator-image 0 FAIL
  Error: 112333 pixels differ from reference image ./clip-operator-ref.png

clip-operator-xlib 1 FAIL
  Error: 127843 pixels differ from reference image ./clip-operator-ref.png

dash-caps-joins-image 0 FAIL
  Error: 796 pixels differ from reference image ./dash-caps-joins-ref.png

dash-caps-joins-xlib 1 FAIL
  Error: 796 pixels differ from reference image ./dash-caps-joins-ref.png

ft-font-create-for-ft-face-image 0 FAIL
  Error: Function under test failed

ft-font-create-for-ft-face-xlib 1 FAIL
  Error: Function under test failed

gradient-alpha-image 0 FAIL
  Error: 100 pixels differ from reference image ./gradient-alpha-ref.png

gradient-alpha-xlib 1 FAIL
  Error: 100 pixels differ from reference image ./gradient-alpha-ref.png

linear-gradient-image 0 FAIL
  Error: 38287 pixels differ from reference image ./linear-gradient-ref.png

linear-gradient-xlib 1 FAIL
  Error: 38287 pixels differ from reference image ./linear-gradient-ref.png

mask-image 0 FAIL
  Error: 50476 pixels differ from reference image ./mask-ref.png

mask-xlib 1 FAIL
  Error: 50476 pixels differ from reference image ./mask-ref.png

operator-clear-image 0 FAIL
  Error: 10358 pixels differ from reference image ./operator-clear-ref.png

operator-clear-xlib 1 FAIL
  Error: 10358 pixels differ from reference image ./operator-clear-ref.png

operator-source-image 0 FAIL
  Error: 27117 pixels differ from reference image ./operator-source-ref.png

operator-source-xlib 1 FAIL
  Error: 27117 pixels differ from reference image ./operator-source-ref.png

text-antialias-gray-image 0 FAIL
  Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png

text-antialias-gray-xlib 1 FAIL
  Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png

text-antialias-subpixel-image 0 FAIL
  Error: 295 pixels differ from reference image ./text-antialias-subpixel-ref.png

text-antialias-subpixel-xlib 1 FAIL
  Error: 295 pixels differ from reference image ./text-antialias-subpixel-ref.png

text-pattern-image 0 FAIL
  Error: 1017 pixels differ from reference image ./text-pattern-ref.png

text-pattern-xlib 1 FAIL
  Error: 1017 pixels differ from reference image ./text-pattern-ref.png

trap-clip-image 0 FAIL
  Error: 22269 pixels differ from reference image ./trap-clip-ref.png

trap-clip-xlib 1 FAIL
  Error: 22269 pixels differ from reference image ./trap-clip-ref.png

unbounded-operator-image 0 FAIL
  Error: 19441 pixels differ from reference image ./unbounded-operator-ref.png

unbounded-operator-xlib 1 FAIL
  Error: 28203 pixels differ from reference image ./unbounded-operator-ref.png

Test was run: 20080808 14:32:20
Test Suite Version: unset
Test Suite Architecture: unset
Total Tests Passed: 88
Total Tests Failed (including waived): 26
Total Tests Failed (excluding waived): 26

Failed test Images

Debian4 (ia32, cairo-1.2.4)

same 26 as RHEL5

OpenSuSE-10.3 (cairo-1.4.10)

clip-operator-image 0 FAIL
  Error: 58949 pixels differ from reference image ./clip-operator-ref.png

clip-operator-xlib 1 FAIL
  Error: 80770 pixels differ from reference image ./clip-operator-ref.png

dash-caps-joins-image 0 FAIL
  Error: 796 pixels differ from reference image ./dash-caps-joins-ref.png

dash-caps-joins-xlib 1 FAIL
  Error: 796 pixels differ from reference image ./dash-caps-joins-ref.png

ft-font-create-for-ft-face-image 0 FAIL
  Error: Function under test failed

ft-font-create-for-ft-face-xlib 1 FAIL
  Error: Function under test failed

gradient-alpha-image 0 FAIL
  Error: 100 pixels differ from reference image ./gradient-alpha-ref.png

gradient-alpha-xlib 1 FAIL
  Error: 100 pixels differ from reference image ./gradient-alpha-ref.png

linear-gradient-image 0 FAIL
  Error: 55738 pixels differ from reference image ./linear-gradient-ref.png

linear-gradient-xlib 1 FAIL
  Error: 55738 pixels differ from reference image ./linear-gradient-ref.png

mask-image 0 FAIL
  Error: 90916 pixels differ from reference image ./mask-ref.png

mask-xlib 1 FAIL
  Error: 90916 pixels differ from reference image ./mask-ref.png

operator-clear-image 0 FAIL
  Error: 5894 pixels differ from reference image ./operator-clear-ref.png

operator-clear-xlib 1 FAIL
  Error: 5894 pixels differ from reference image ./operator-clear-ref.png

operator-source-image 0 FAIL
  Error: 25366 pixels differ from reference image ./operator-source-ref.png

operator-source-xlib 1 FAIL
  Error: 25366 pixels differ from reference image ./operator-source-ref.png

paint-with-alpha-image 0 FAIL
  Error: 64 pixels differ from reference image ./paint-with-alpha-ref.png

paint-with-alpha-xlib 1 FAIL
  Error: 64 pixels differ from reference image ./paint-with-alpha-ref.png

set-source-image 0 FAIL
  Error: 10 pixels differ from reference image ./set-source-ref.png

set-source-xlib 1 FAIL
  Error: 10 pixels differ from reference image ./set-source-ref.png

text-antialias-gray-image 0 FAIL
  Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png

text-antialias-gray-xlib 1 FAIL
  Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png

text-antialias-subpixel-image 0 FAIL
  Error: 361 pixels differ from reference image ./text-antialias-subpixel-ref.png

text-antialias-subpixel-xlib 1 FAIL
  Error: 361 pixels differ from reference image ./text-antialias-subpixel-ref.png

text-pattern-image 0 FAIL
  Error: 1024 pixels differ from reference image ./text-pattern-ref.png

text-pattern-xlib 1 FAIL
  Error: 1024 pixels differ from reference image ./text-pattern-ref.png

trap-clip-image 0 FAIL
  Error: 44592 pixels differ from reference image ./trap-clip-ref.png

trap-clip-xlib 1 FAIL
  Error: 44592 pixels differ from reference image ./trap-clip-ref.png

unbounded-operator-image 0 FAIL
  Error: 11137 pixels differ from reference image ./unbounded-operator-ref.png

unbounded-operator-xlib 1 FAIL
  Error: 21978 pixels differ from reference image ./unbounded-operator-ref.png

Test was run: 20080814 16:39:54
Test Suite Version: unset
Test Suite Architecture: unset
Total Tests Passed: 84
Total Tests Failed (including waived): 30
Total Tests Failed (excluding waived): 30

Failed test Images

FC8 (ia32, cairo-1.4.14)

same 30 FAILS as OpenSuSE, plus "Journal indicates expected 114 tests, found 112"

FC9 (ia32, cairo-1.6.4)

caps-joins-image 0 FAIL
  Error: 285 pixels differ from reference image ./caps-joins-ref.png

caps-joins-xlib 1 FAIL
  Error: 285 pixels differ from reference image ./caps-joins-ref.png

caps-sub-paths-image 0 FAIL
  Error: 12 pixels differ from reference image ./caps-sub-paths-ref.png

caps-sub-paths-xlib 1 FAIL
  Error: 12 pixels differ from reference image ./caps-sub-paths-ref.png

clip-operator-image 0 FAIL
  Error: 59315 pixels differ from reference image ./clip-operator-ref.png

clip-operator-xlib 1 FAIL
  Error: 81047 pixels differ from reference image ./clip-operator-ref.png

clip-twice-image 0 FAIL
  Error: 96 pixels differ from reference image ./clip-twice-ref.png

clip-twice-xlib 1 FAIL
  Error: 96 pixels differ from reference image ./clip-twice-ref.png

dash-caps-joins-image 0 FAIL
  Error: 1059 pixels differ from reference image ./dash-caps-joins-ref.png

dash-caps-joins-xlib 1 FAIL
  Error: 1059 pixels differ from reference image ./dash-caps-joins-ref.png

fill-and-stroke-image 0 FAIL
  Error: 38 pixels differ from reference image ./fill-and-stroke-ref.png

fill-and-stroke-xlib 1 FAIL
  Error: 38 pixels differ from reference image ./fill-and-stroke-ref.png

ft-font-create-for-ft-face-image 0 FAIL
  Error: Function under test failed

ft-font-create-for-ft-face-xlib 1 FAIL
  Error: Function under test failed

gradient-alpha-image 0 FAIL
  Error: 100 pixels differ from reference image ./gradient-alpha-ref.png

gradient-alpha-xlib 1 FAIL
  Error: 100 pixels differ from reference image ./gradient-alpha-ref.png

leaky-polygon-image 0 FAIL
  Error: 10 pixels differ from reference image ./leaky-polygon-ref.png

leaky-polygon-xlib 1 FAIL
  Error: 10 pixels differ from reference image ./leaky-polygon-ref.png

linear-gradient-image 0 FAIL
  Error: 56719 pixels differ from reference image ./linear-gradient-ref.png

linear-gradient-xlib 1 FAIL
  Error: 56719 pixels differ from reference image ./linear-gradient-ref.png

mask-image 0 FAIL
  Error: 91308 pixels differ from reference image ./mask-ref.png

mask-xlib 1 FAIL
  Error: 91308 pixels differ from reference image ./mask-ref.png

operator-clear-image 0 FAIL
  Error: 6020 pixels differ from reference image ./operator-clear-ref.png

operator-clear-xlib 1 FAIL
  Error: 6020 pixels differ from reference image ./operator-clear-ref.png

operator-source-image 0 FAIL
  Error: 25741 pixels differ from reference image ./operator-source-ref.png

operator-source-xlib 1 FAIL
  Error: 25741 pixels differ from reference image ./operator-source-ref.png

paint-with-alpha-image 0 FAIL
  Error: 64 pixels differ from reference image ./paint-with-alpha-ref.png

paint-with-alpha-xlib 1 FAIL
  Error: 64 pixels differ from reference image ./paint-with-alpha-ref.png

rectangle-rounding-error-image 0 FAIL
  Error: 53 pixels differ from reference image ./rectangle-rounding-error-ref.png

rel-path-image 0 FAIL
  Error: 3 pixels differ from reference image ./rel-path-ref.png

rel-path-xlib 1 FAIL
  Error: 3 pixels differ from reference image ./rel-path-ref.png

set-source-image 0 FAIL
  Error: 10 pixels differ from reference image ./set-source-ref.png

set-source-xlib 1 FAIL
  Error: 10 pixels differ from reference image ./set-source-ref.png

text-antialias-gray-image 0 FAIL
  Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png

text-antialias-gray-xlib 1 FAIL
  Error: 291 pixels differ from reference image ./text-antialias-gray-ref.png

text-antialias-subpixel-image 0 FAIL
  Error: 295 pixels differ from reference image ./text-antialias-subpixel-ref.png

text-antialias-subpixel-xlib 1 FAIL
  Error: 295 pixels differ from reference image ./text-antialias-subpixel-ref.png

text-pattern-image 0 FAIL
  Error: 1024 pixels differ from reference image ./text-pattern-ref.png

text-pattern-xlib 1 FAIL
  Error: 1024 pixels differ from reference image ./text-pattern-ref.png

transforms-image 0 FAIL
  Error: 2 pixels differ from reference image ./transforms-ref.png

transforms-xlib 1 FAIL
  Error: 2 pixels differ from reference image ./transforms-ref.png

trap-clip-image 0 FAIL
  Error: 45107 pixels differ from reference image ./trap-clip-ref.png

trap-clip-xlib 1 FAIL
  Error: 45107 pixels differ from reference image ./trap-clip-ref.png

unantialiased-shapes-image 0 FAIL
  Error: 1427 pixels differ from reference image ./unantialiased-shapes-ref.png

unantialiased-shapes-xlib 1 FAIL
  Error: 19 pixels differ from reference image ./unantialiased-shapes-ref.png

unbounded-operator-image 0 FAIL
  Error: 11376 pixels differ from reference image ./unbounded-operator-ref.png

unbounded-operator-xlib 1 FAIL
  Error: 22129 pixels differ from reference image ./unbounded-operator-ref.png

Test was run: 20080808 10:40:49
Test Suite Version: unset
Test Suite Architecture: unset
Total Tests Passed: 65
Total Tests Failed (including waived): 47
Total Tests Failed (excluding waived): 47

Journal indicates expected 114 tests, found 112

The missing tests look to be:

path-data-image 0 PASS
path-data-xlib 1 PASS

Which when run on these platforms exits with status 1 and no log or journal. The test states it is "Tests calls to path_data functions: cairo_copy_path_data, cairo_copy_path_data_flat, and cairo_append_path_data". Looking at the code, it too, varies from the other test code in that it's coded to be able to exit with status 1 before and logging or journal reporting.

Failed test Images

MDV2008.1 (ia32, cairo-1.6.4)

same as FC9

Ubuntu-8.04 (ia32, cairo-1.6.0)

same as FC9

Test Updates

ft-font-create-for-ft-face

Patched in a partial uplift to this test using the testing method used in cairo-1.6.4 - now passes on SLES10, RHEL5, Mdv2008.1, FC9, OpenSuse10.3 (08/19/08)

path-data

Patched the test to not do a "return 1" until the intermediate setup code has all run, capturing the state as "setup_stat". If this variable is "1", then pass UNREPORTED (with the CAIRO error code) for both the image and xlib tests.

Results (FC9):

path-data-image 0 UNREPORTED
  Error: CAIRO_STATUS_NULL_POINTER (cairo_append_path(cr, &path) \
         path.data = NULL, path.num_data = 0, path.status = CAIRO_STATUS_SUCCESS)

path-data-xlib 1 UNREPORTED
  Error: CAIRO_STATUS_NULL_POINTER (cairo_append_path(cr, &path) \
         path.data = NULL, path.num_data = 0, path.status = CAIRO_STATUS_SUCCESS)

Note: This test is not present in 1.4.14 or 1.6.4. Candidate to drop?

image differences

Thinking about the tests in an LSB context, perhaps we're not that interested in subtle differences in the reference and generated image, which can be driven by the system's font or X server differences. What we're primarily interested in is whether the interface being tested exists, and that it returns as expected. With this in mind, I've updated (8/22/08) the tetj patch to return TETJ_WARNING for these cases.

This (and the other changes above) reduces the FAIL count considerably:

  • SLES10 - 0 FAIL, 2 WARNING, 1 UNTESTED
  • RHEL5 - 0 FAIL, 24 WARNING, 1 UNTESTED
  • Debian4 - 0 FAIL, 24 WARNING, 1 UNTESTED
  • OpenSuSE-10.3 - 0 FAIL, 28 WARNING, 1 UNTESTED
  • FC8 - 0 FAIL, 28 WARNING, 1 UNTESTED, 2 UNREPORTED (UNREPORTEDs are path-data-{image,xlib}
  • FC9 - 0 FAIL, 45 WARNING, 1 UNTESTED, 2 UNREPORTED (UNREPORTEDs are path-data-{image,xlib}
  • Mandriva-2008.1 - 0 FAIL, 45 WARNING, 1 UNTESTED, 2 UNREPORTED (UNREPORTEDs are path-data-{image,xlib}
  • Ubuntu-8.04 - 0 FAIL, 45 WARNING, 1 UNTESTED, 2 UNREPORTED (UNREPORTEDs are path-data-{image,xlib}

[Article] [Discussion] [View source] [History]