The Linux Foundation

 
LibToLSB

From The Linux Foundation

Contents

How to Add a Library to LSB

Overview

To add a library to the LSB, a few things are required:

  1. The library must have a stable ABI and be considered "best practice" or in common use across the major Linux distributions.
  2. The library interfaces must be added to the LSB database
  3. The interfaces, description, arguments, results must be added the LSB specification
  4. Tests need to be defined to test the behavior of the interfaces and added to the LSB test suites.

To illustrate the process, we'll walk through adding a common library, libbz2, to the LSB.

Build a version of the library with debugging enabled

wget http://www.bzip.org/1.0.4/bzip2-1.0.4.tar.gz
tar -xzf bzip2-1.0.4.tar.gz
cd bzip2-1.0.4
CC="gcc -g" make -f Makefile-libbz2_so 

(-g enables debugging information in the compiled output. libbz2 doesn't use a "configure" script but provides the above mentioned Makefile to compile a shared library.)

file libbz2.so.1.0.4
libbz2.so.1.0.4: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), not stripped

Display symbols from the dynamic section:

readelf -DWs libbz2.so.1.0.4

Looking at this you'll see entries like:

50  39: 0000ed00     7    FUNC GLOBAL DEFAULT  10 BZ2_bzflush

Those entries with an Ndx entry of 10, are the interfaces that we're interested in in defining, specifying in the LSB, and creating tests for. So we could filter this down to just a list of the interesting interfaces:

readelf -DWs libbz2.so.1.0.4 | grep "GLOBAL DEFAULT  10" | awk '{print $9}' > interfaces.list

You should carefully compare this list with the header files and remove any interfaces that are exported, but not really part of the supportable and public API of the library (Rather than edit the file, we'll create another list of excluded interfaces that we'll use for libtodb). [1]

So, for the case of libbz2, our original list of 33 interfaces would get trimmed some more:

for i in `cat interfaces.list`;do grep -q $i bzlib.h;if [ "$?" -eq 1 ]; then echo $i;fi;done
BZ2_blockSort
BZ2_compressBlock
BZ2_indexIntoF
BZ2_bsInitWrite
BZ2_hbMakeCodeLengths
BZ2_bz__AssertH__fail
BZ2_hbAssignCodes
BZ2_decompress
BZ2_hbCreateDecodeTables

These interfaces will become part of our exclude list in the next step.

Add the library to the LSB database

It's best to get a copy of the LSB database you can setup locally. The transactions to examine and add a library are too numerous to effectively be able to do it over the internet, and you'll want to work out the issues on a local copy before official submission anyway: Spec Database

Install MySQL - instructions for this will vary according to your Linux distribution.

All LSB tools and tests can be checked out from bzr. The repositories can be browsed at: http://bzr.freestandards.org/lsb/

You will need to install bzr to check out source: Bazaar-NG

Check out the specdb:

bzr branch http://bzr.freestandards.org/lsb/devel/specdb/ specdb

Read 00README:

SPECDB
======
The files in this directory reflect the current state of the LSB
database. There are some basic rules that must be followed when 
using these files:

make restore
        This will create a new database copy using the environment
        variables LSBUSER, LSBDBPASSWD, LSBDBHOST and LSBDB.
        *** NEVER use "make restore" on the central database
        (dbservices.freestandards.org) unless that database has been hoplessly
        lost!  "make restore" is ONLY intended for users requiring to
        initialize local copies of their own database.

You'll note the mention of LSB* environment variables. These should be set appropriately to access your local database copy, in ~/.bashrc or similar:

export LSBUSER=stew
export LSBDBPASSWD=foobar
export LSBDBHOST=norris
export LSBDB=lsb

So, once these are set, "make restore" should populate your local LSB database.

Now that we have our library and the database, we can use the tool "libtodb" to examine the library and it's headers, and eventually add it to the database: LibtoDB. This is an interative process, and depending on the complexity of the library in question, and whether you want to exclude certain interfaces from the LSB (due to the fact they may not have a stable ABI or may be deprecated), it can take several iterations of using libtodb to get a satisfactory import into the database that will allow one to build the stub headers and specification.

Check out libtodb and friends:

bzr branch http://bzr.freestandards.org/lsb/devel/scripts/ scripts

At this point you have a decision to make if your library is 'complex' and has multiple header files associated with it. If it is a simple library with a single header covering all of the exported symbols you are pretty much ready to go. If it is a complex library like OpenSSL with many headers exposing different portions of the ABI you have a choice to make. Either you need to fix things such that the entire libraries interface can be imported from a single header or you need to go through a process of importing exports relative to each individual header file. Choosing to put everything into a single header should only be done with the advise and consent of the upstream package maintainer since doing so may break some applications that depend on the existing upstream header layout and side effects of header inclusion (symbols defined, symbols not defined). For complex libraries I have worked out a sort of 'pipeline' to assist in importing these libraries. It requires a bit more time to setup, but at that point is automated and the results will be more true the upstream package than the LSB header modification approach. addopenssl.sh in the scripts directory is an example of this pipeline in action. [1]

If you look at one of the libraries that are already in the LSB, you'll see the captured data used during the import:

[scripts]$ find libtodb_data/jpeg/
libtodb_data/jpeg/
libtodb_data/jpeg/jpeglib.h.def
libtodb_data/jpeg/jpeglib_ex.list
libtodb_data/jpeg/jpeglib_opaque.list
libtodb_data/jpeg/libs
libtodb_data/jpeg/libs/libjpeg.so.62.0.0

First we want to make a constants file from the library's header file, in this case bzlib.h:

./mkconstfile -h bzlib.h > bzlib.h.def

The above tool tends to pull in macros imported in the header file that aren't actually defined in the header file. What you really want is just those macros that are part of the library in question. Using Tracy's "pipeline" method you can narrow this down to a more succinct list:

cpp -dD bzlib.h | ./trim_macros.pl -m bzlib.h | grep -v "#define HEADER" \
    | ./mkconstfile -h - | grep -v "^<macro>" | sort -u > bzlib.h.def

Even with this, the file will still need to be edited, until we end up with something like this:

<>#define BZ_FINISH 2
<>#define BZ_FINISH_OK 3
<>#define BZ_FLUSH 1
<>#define BZ_FLUSH_OK 2
<>#define BZ_MAX_UNUSED 5000
<>#define BZ_OK 0
<>#define BZ_RUN 0
<>#define BZ_RUN_OK 1
<>#define BZ_STREAM_END 4
<unknown>#define BZ_EXPORT 
<unknown>#define _BZLIB_H 

Then create the excluded interfaces list:

for i in `cat interfaces.list`;do grep -q $i bzlib.h;if [ "$?" -eq 1 ]; \
    then echo $i;fi;done > bzlib_ex.list

Likewise, opaque interfaces would be listed in bzlib_opaque.list. The opaque list is important for cropping the exposed API into a portable ABI. If the API exposes internal structures that are not guaranteed to be stable across many releases it is best to cause the ABI to treat these as opaque types. The opaque file lists types that should be treated as opaque. These will have to be manually adjusted, even in the complex pipeline. [1]

For libbzip2, an opaque listing will not be necessary.

Before proceeding with the import with libtodb, two more choices need to be made. Those are the module the new library will be part of, and the standard that will be referred do.

For the purposes of our example, we'll create a new LSB module: LSB_Toolkit_Compression. To be able to use this, we'll need to add an entry to the "Module" table in the LSB database, using mysql, picking the next unused module number. Alternatively, the library could have gone into an existing module. In this case, the libtodb -M 9 argument below would be adjusted to refer to the appropriate module number. The DB Navigator can be used to browse the modules.

INSERT INTO Module VALUES (9, 'LSB_Toolkit_Compression', 'LSB Toolkit Compression Module', '3.2', NULL, '3.2');

The standards question comes down to whether there is good API-level specifications available for the library. If so, the LSB should reference that specification, and only a small addition needs to be made to the LSB specification. If not, the specification text has to be written and added to the LSB itself, in which case LSB itself (specification ID 10) becomes the reference specification - that's the case for the example we're using, the argument below will be -s 10. If a new standard is to be referenced, it must be inserted into the database Standard table before the import, and the -s argument adjusted. Again, the DB Navigator can be used to browse the existing standards.

As a first pass, run libtodb without the "-i" argument which inserts the interfaces into the database.

./libtodb -q -H bzlib.h -L "Bzip Compression Library" -M 9 -s 10 -S libbz2.so.1 \
    -A IA32 -c bzlib.h.def -x bzlib_ex.list libbz2.so.1.0.4 > out.log

So we're doing a test run of adding libbz2 to the compression module (-M 9), standard is LSB (-s 10), SONAME is libbz2.so.1 (-S) host architecture is IA32 (-A), and we've specified our constants file (-c) and excludes file (-x). So if we look at out.log, we find 26 items in the proposed list of interfaces to add to LSB:

$ grep "DWARF" out.log
DWARF: BZ2_bzflush: Function
DWARF: BZ2_bzDecompressInit: Function
DWARF: BZ2_bzlibVersion: Function
DWARF: BZ2_bzBuffToBuffDecompress: Function
DWARF: BZ2_rNums: Data
DWARF: BZ2_bzread: Function
DWARF: BZ2_bzReadOpen: Function
DWARF: BZ2_bzReadGetUnused: Function
DWARF: BZ2_bzCompress: Function
DWARF: BZ2_bzCompressInit: Function
DWARF: BZ2_bzDecompress: Function
DWARF: BZ2_bzerror: Function
DWARF: BZ2_bzopen: Function
DWARF: BZ2_bzReadClose: Function
DWARF: BZ2_bzCompressEnd: Function
DWARF: BZ2_bzDecompressEnd: Function
DWARF: BZ2_bzWrite: Function
DWARF: BZ2_bzBuffToBuffCompress: Function
DWARF: BZ2_bzWriteClose64: Function
DWARF: BZ2_bzWriteClose: Function
DWARF: BZ2_bzdopen: Function
DWARF: BZ2_bzWriteOpen: Function
DWARF: BZ2_bzclose: Function
DWARF: BZ2_bzRead: Function
DWARF: BZ2_crc32Table: Data
DWARF: BZ2_bzwrite: Function

BZ2_rNums and BZ2_crc32Table are data structures, and can be added to our exclude list also.

If the list looks reasonable, then you can proceed with actually adding the library to the database, by including the "-i" argument:

./libtodb -i -q -H bzlib.h -L "Bzip Compression Library" -M 9 -s 10 -S libbz2.so.1 \
    -A IA32 -c bzlib.h.def -x bzlib_ex.list libbz2.so.1.0.4 > out.log
Module used is LSB_Toolkit_Compression
Standard used for interface(s) is LSB
Architecture has id 1(All)
Host Architecture has id 2(IA32)
Library has id 263
LibGroup has id 493
Header has id 488
Header Group has id 929
24 interfaces added to database.

Generate the LSB headers and Stub libraries

Headers

Now that you've imported the header into the database, you should be able to regenerate the LSB headers.

$ bzr branch http://bzr.freestandards.org/lsb/devel/build_env
$ cd build_env/headers
$ make headers-All

You should get a generated bzlib.h header as part of this build, generated from the database.

Since we added our own module, we'll need to edit "mkfilelists" to be aware of it, so that the new header appears in the correct header/library package. There are two packages that include generated header files and libraries, lsb-build-base (core modules headers/libraries) and lsb-build-desktop (desktop modules headers/libraries). Starting from the headers directory in the previous step:

$ cd ..
$ bzr diff mkfilelists
=== modified file 'mkfilelists'
--- mkfilelists 2007-04-12 12:37:31 +0000
+++ mkfilelists 2007-04-30 16:03:47 +0000
@@ -16,7 +16,8 @@

 lsbversion = sys.argv[1]

-core_modules = ["LSB_Core", "LSB_Graphics", "LSB_Cpp"]
+core_modules = ["LSB_Core", "LSB_Graphics", "LSB_Cpp",
+               "LSB_Toolkit_Compression"]
 desktop_modules = ["LSB_Toolkit_Gtk", "LSB_Toolkit_Qt",
                   "LSB_Graphics_Ext", "LSB_XML", "LSB_Toolkit_Qt3"]

So we've just added the LSB_Toolkit_Compression to the core_modules group. We can now generate the lists:

$ ./mkfilelists
$ ls -1 */*filelist
headers/core_filelist
headers/desktop_filelist
stub_libs/core_filelist
stub_libs/desktop_filelist

Stub libraries

Now we can build the stub libraries.

$ cd stub_libs
$ make distclean
$ make dbfiles
$ make

You should find libbz2 (appropriate for the build arch):

$ find . -name 'libbz2*'
./IA32/libbz2.c
./IA32/libbz2.Version
./IA32/libbz2.o
./IA32/libbz2.so
./IA64/libbz2.c
./IA64/libbz2.Version
./PPC32/libbz2.c
./PPC32/libbz2.Version
./PPC64/libbz2.c
./PPC64/libbz2.Version
./S390/libbz2.c
./S390/libbz2.Version
./S390X/libbz2.c
./S390X/libbz2.Version
./x86-64/libbz2.c
./x86-64/libbz2.Version

Test the new headers / stub libraries

So, we now have a stub library and an LSB header. It should be possible at this point to copy these to /opt/lsb and use them with lsbcc.

headers$ sudo cp bzlib.h /opt/lsb/include/
cd ../stub_libs/IA32
sudo cp libbz2.so /opt/lsb/lib

We need to alter lsbcc to accept -lbz2 as an LSB library, rather than an external library to link statically. lsbcc is part of the same build_env tree we checked out earlier:

cd lsbdev-cc
gendiff . .bz2
--- ./lsbcc.c.bz2       2007-03-13 16:56:28.000000000 -0400
+++ ./lsbcc.c   2007-03-13 16:56:52.000000000 -0400
@@ -270,6 +270,7 @@
        "rt",
        "util",
        "z",
+       "bz2",
        "GL",           /* graphics module */
        "ICE",
        "SM",

lsbcc should compile ok with this change:

make

Now we can compile a test app using our LSB header and the patched lsbcc (bztest.c):

#include <stdlib.h>
#include "bzlib.h"

#define OFILE "outfile.bz2"
#define BUFSTR "01234567890123456789012345678901234567890123456789"
#define WRITE_ERR "BZ2_bzWrite set bzerror to"

int main(void)
{
        BZFILE* b;
        FILE* f;
        char buf[] = BUFSTR;
        int bzerror;
        
        f = fopen(OFILE, "w");
        
        if (!f) {
                printf("Cannot open %s for writing\n", OFILE);
                exit(1);
        }
        
        b = BZ2_bzWriteOpen(&bzerror, f, 9, 0, 0);
        if (bzerror != BZ_OK) {
                BZ2_bzWriteClose(&bzerror, b, 0, NULL, NULL);
                printf("Cannot open %s for bzWrite\n", OFILE);
                exit(1);
        }
        
        BZ2_bzWrite(&bzerror, b, buf, strlen(buf));
        if (bzerror == BZ_IO_ERROR) {
                BZ2_bzWriteClose(&bzerror, b, 0, NULL, NULL);
                printf("%s %d\n", WRITE_ERR, bzerror);
                exit(1);
        }

        BZ2_bzWriteClose(&bzerror, b, 0, NULL, NULL);
        if (bzerror == BZ_IO_ERROR) {
                printf("Cannot close %s\n", OFILE);
                exit(1);
        }
}

Compile the test app with our patched lsbcc and our new header/stub lib:

 
./lsbcc bztest.c -lbz2 -o bztest
ldd bztest
linux-gate.so.1 =>  (0xffffe000)
libbz2.so.1 => /lib/libbz2.so.1 (0xf7ef1000)
libpthread.so.0 => /lib/libpthread.so.0 (0xf7edd000)
libm.so.6 => /lib/libm.so.6 (0xf7eb8000)
libc.so.6 => /lib/libc.so.6 (0xf7d98000)
/lib/ld-lsb.so.3 (0xf7f0f000)

An unpatched lsbcc would have statically linked libbz2.a:

lsbcc -lbz2 bztest.c -o bztest
ldd bztest
linux-gate.so.1 =>  (0xffffe000)
libpthread.so.0 => /lib/libpthread.so.0 (0xf7f20000)
libm.so.6 => /lib/libm.so.6 (0xf7efb000)
libc.so.6 => /lib/libc.so.6 (0xf7ddb000)
/lib/ld-lsb.so.3 (0xf7f41000)

Run the app (simply creates a bzip file outfile.bz2):

./bztest 
ls -l outfile.bz2 
-rw-r--r-- 1 stew stew 51 Mar 13 19:16 outfile.bz2
bzcat outfile.bz2 
01234567890123456789012345678901234567890123456789

It's possible that you might encounter issues here and have to go back to the libtodb step and re-import your library into the database, after making corrections. You can restore your local LSB database using (in the specdb directory):

make restore

You would then have a pristine database ready to re-add your module and library.

Regenerate checkers

add content here on regenerating in misc-test

Add the library to the specification

The spec content is in the lsbspec branch of bzr:

bzr branch http://bzr.freestandards.org/lsb/devel/lsbspec

In this tree there is a document named docbook_tutorial.txt. This describes an overview of how the specification is built and edited.

Create Toolkit_Compression module tree

Since we've opted to create a new module, we'll need the directory structure to support it. Looking at the existing structure, it looks like we can perhaps model our new lib from Graphics-Ext/generic/libjpeg, with the thought that Toolkit_Compression may someday have other compression libs (like gzip, zip, lzma, etc.). If we were integrating into an existing module, the steps below would be modified appropriately.

cd lsbspec
install -d Toolkit_Compression/intro
install -d Toolkit_Compression/generic/libbz2

We need a toplevel makefile (in Toolkit_Compression):

SUBDIRS= generic intro

all:
        for dir in $(SUBDIRS);do (cd $$dir && make all);done

autobuild:
        for dir in $(SUBDIRS);do (cd $$dir && make gensrc all);done

gensrc:
        for dir in $(SUBDIRS);do (cd $$dir && make gensrc);done

source:
        for dir in $(SUBDIRS);do (cd $$dir && make all);done

clean:
        for dir in $(SUBDIRS);do (cd $$dir && make clean);done
        rm -f $(HTML)

spotless:
        for dir in $(SUBDIRS);do (cd $$dir && make spotless);done
        rm -f $(HTML)

And one in intro:

.SUFFIXES: .sgml .cpp .m4

FILES=intro.sgml
TABLES=standards.sgml

.cpp.sgml:
        cpp -I../../../code/headers $*.cpp | grep -v '^# ' >$@

.m4.sgml:
        m4 -Uindex -Uformat $*.m4 >$@

all: $(FILES) $(TABLES)

gensrc:
        ../../mkstandardsgmltable -a All -s "'ISOC99','libbz2'" >standards.sgml
        ../../mklibsgmltable -a All -m LSB_Toolkit_Compression >libraries.sgml


clean:
        rm -f $(FILES)

spotless: clean
        rm -f $(TABLES)

intro.sgml: standards.sgml libraries.sgml

And one in generic:

SUBDIRS= libbz2                                          

all:
        for dir in $(SUBDIRS);do (cd $$dir && make all);done

gensrc:
        for dir in $(SUBDIRS);do (cd $$dir && make gensrc);done

source:
        for dir in $(SUBDIRS);do (cd $$dir && make source);done

clean:
        for dir in $(SUBDIRS);do (cd $$dir && make clean);done
        rm -f $(HTML)

spotless:
        for dir in $(SUBDIRS);do (cd $$dir && make spotless);done
        rm -f $(HTML)

And one in generic/libbz2 (only listing our 2 sample functions for the moment):

.SUFFIXES: .sgml .cpp .m4

MANPAGES= BZ2_bzRead.sgml\
BZ2_bzWrite.sgml

FILES=libbz2.sgml
TABLES= bz2.sgml

.cpp.sgml:
        cpp -I../../../code/headers $*.cpp | grep -v '^# ' >$@

.m4.sgml:
        m4 -Uindex -Uformat $*.m4 >$@

all: $(FILES) $(TABLES)

gensrc:
        ../../../mklibspec -a All -c 3 -l libbz2 >bz2.sgml

clean:
        rm -f $(FILES)

spotless: clean
        rm -f $(TABLES)

libbz2.sgml: bz2.sgml
libbz2.sgml: $(MANPAGES)

And generic/libbz2/libbz2.m4:

<PART ID="toclibbz2">
<TITLE>BZ2 library</TITLE>

<CHAPTER id=libbz2>
<TITLE>Libraries</TITLE>

include(bz2.sgml)

</CHAPTER>

</PART>

We also need an into/intro.m4 document and intro/terms.sgml. I won't paste those here. I basically copied them from a neighboring directory and edited intro.m4 appropriately.

Write our libbz2 interface documents

Then we need to write the "foo.sgml" document for each of our imported interfaces (in generic/libbz2), let's pick a couple for examples, BZ2_bzWrite and and BZ2_bzRead:

BZ2_bzWrite.sgml:

<refentry id="libbz2.BZ2.bzWrite.1">
<refmeta>
<refentrytitle>BZ2_bzWrite</refentrytitle>
<refmiscinfo>libbz2</refmiscinfo>
</refmeta>

<refnamediv>
<refname>BZ2_bzWrite</refname>
<refpurpose>Absorbs len bytes from the buffer buf, eventually to be compressed a
nd written to the file</refpurpose>
<indexterm id="ix.libbz2.BZ2.bzWrite.1">       
<primary>BZ2_bzWrite</primary>
</indexterm>
</refnamediv>
<refsynopsisdiv>
<funcsynopsis>
<funcsynopsisinfo>
#include <bzlib.h>
</funcsynopsisinfo><funcprototype>
<funcdef>void
<function>BZ2_bzWrite</function>
</funcdef>
<paramdef>int
<parameter>*bzerror</parameter>
</paramdef>
<paramdef>BZFILE
<parameter>*b</parameter>
</paramdef>
<paramdef>void
<parameter>*buf</parameter>
</paramdef>
<paramdef>int
<parameter>len</parameter>
</paramdef>
</funcprototype>
</funcsynopsis>
</refsynopsisdiv>
<refsect1>
<title>Description</title>
<para>
The function <function>BZ2_bzWrite</function> shall absorb len bytes from
the buffer buf, eventually to be compressed and written to the file.
</para>
</refsect1>
<refsect1>
<title>Errors</title>
<para>
Possible assignments to bzerror:
<VARNAME>BZ_PARAM_ERROR</VARNAME>
  if b is NULL or buf is NULL or len < 0
<VARNAME>BZ_SEQUENCE_ERROR</VARNAME>
  if b was opened with BZ2_bzReadOpen
<VARNAME>BZ_IO_ERROR</VARNAME>
  if there is an error writing the compressed file.
<VARNAME>BZ_OK</VARNAME>
  otherwise
</para>
</refsect1>
</refentry>

BZ2_bzRead.sgml:

<refentry id="libbz2.BZ2.bzRead.1">
<refmeta>
<refentrytitle>BZ2_bzRead</refentrytitle>
<refmiscinfo>libbz2</refmiscinfo>
</refmeta>

<refnamediv>
<refname>BZ2_bzRead</refname>
<refpurpose>Reads up to len (uncompressed) bytes from the compressed file b into
 the buffer buf. If the read was successful, bzerror is set to BZ_OK and the num
ber of bytes read is returned. If the logical end-of-stream was detected, bzerro
r will be set to BZ_STREAM_END, and the number of bytes read is returned. All ot
her bzerror values denote an error.</refpurpose>
<indexterm id="ix.libbz2.BZ2.bzRead.1">       
<primary>BZ2_bzRead</primary>
</indexterm>
</refnamediv>
<refsynopsisdiv>
<funcsynopsis>
<funcsynopsisinfo>
#include <bzlib.h>
</funcsynopsisinfo><funcprototype>
<funcdef>void
<function>BZ2_bzRead</function>
</funcdef>
<paramdef>int
<parameter>*bzerror</parameter>
</paramdef>
<paramdef>BZFILE
<parameter>*b</parameter>
</paramdef>
<paramdef>void
<parameter>*buf</parameter>
</paramdef>
<paramdef>int
<parameter>len</parameter>
</paramdef>
</funcprototype>
</funcsynopsis>
</refsynopsisdiv>
<refsect1>
<title>Description</title>
<para>
The function <function>BZ2_bzRead</function> shall read up to len (uncompressed)
 bytes from the compressed file b into the buffer buf. If the read was successfu
l, bzerror is set to BZ_OK and the number of bytes read is returned. If the logi
cal end-of-stream was detected, bzerror will be set to BZ_STREAM_END, and the nu
mber of bytes read is returned. All other bzerror values denote an error.
</para>
</refsect1>
<refsect1>
<title>Errors</title>
<para>
Possible assignments to bzerror:
<VARNAME>BZ_PARAM_ERROR</VARNAME>
  if b is NULL or buf is NULL or len < 0
<VARNAME>BZ_SEQUENCE_ERROR</VARNAME>
  if b was opened with BZ2_bzWriteOpen
<VARNAME>BZ_IO_ERROR</VARNAME>
  if there is an error reading from the compressed file.
<VARNAME>BZ_UNEXPECTED_EOF</VARNAME>
  if the compressed file ended before 
  the logical end-of-stream was detected
<VARNAME>BZ_DATA_ERROR</VARNAME>
  if a data integrity error was detected in the compressed stream
<VARNAME>BZ_DATA_ERROR_MAGIC</VARNAME>
  if the stream does not begin with the requisite header bytes 
  (ie, is not a bzip2 data file).  This is really 
  a special case of BZ_DATA_ERROR.
<VARNAME>BZ_MEM_ERROR</VARNAME>
  if insufficient memory was available
<VARNAME>BZ_STREAM_END</VARNAME>
  if the logical end of stream was detected.
<VARNAME>BZ_OK</VARNAME>
  otherwise
</para>
</refsect1>
</refentry>

Generate content from the database

The remaining documents get generated from the database:

[Toolkit_Compression]$ make gensrc
for dir in generic intro;do (cd $dir && make gensrc);done
make[1]: Entering directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/generic'
for dir in libbz2 ;do (cd $dir && make gensrc);done
make[2]: Entering directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/generic/libbz2'
../../../mklibspec -a All -c 3 -l libbz2 >bz2.sgml
Skipped 22 missing interfaces
make[2]: Leaving directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/generic/libbz2'
make[1]: Leaving directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/generic'
make[1]: Entering directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/intro'
../../mkstandardsgmltable -a All -s "'ISOC99','libbz2'" >standards.sgml
../../mklibsgmltable -a All -m LSB_Toolkit_Compression >libraries.sgml
make[1]: Leaving directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/intro'

You'll note the mention of 22 missing interfaces. These are the BZ2* interfaces we haven't yet written documents for. Looking at our directory now, we see:

Toolkit_Compression]$ find .
.
./intro
./intro/makefile
./intro/standards.sgml
./intro/intro.m4
./intro/libraries.sgml
./intro/terms.sgml
./generic
./generic/libbz2
./generic/libbz2/makefile
./generic/libbz2/BZ2_bzWrite.sgml
./generic/libbz2/bz2.sgml
./generic/libbz2/BZ2_bzRead.sgml
./generic/libbz2/libbz2.m4
./generic/makefile
./makefile

Create the "book" structure

I directory up, you'll find a directory named "book". Since this a new module, we'll need a similar structure here to generate the remaining documents:

[book]$ find Toolkit_Compression Toolkit_Compression Toolkit_Compression/makefile Toolkit_Compression/buildversion Toolkit_Compression/contents Toolkit_Compression/Toolkit_Compression.sgml.sed

buildversion:

3.1

contents:

&toolkit-compression-generic-bz2;

makefile: ARCH=generic DOC=Toolkit_Compression

include ../Makefile.common </pre>

Toolkit_Compression.sgml.sed:

<!DOCTYPE BOOK PUBLIC "-//OASIS//DTD DocBook V4.1//EN" [

<!ENTITY % entities SYSTEM "../../entities">
<!ENTITY contents SYSTEM "contents">

<!ENTITY specversion "VERSION">

%entities;

<!ENTITY copyrightyear "2005">
<!ENTITY copyrightholder "Free Standards Group">
<!ENTITY license "&fdl;">
<!ENTITY doccopyright SYSTEM "../../matters/fsgcopyright.sgml">

]>

<BOOK>
<BOOKINFO>
<TITLE>LSB Toolkit Compression Specification</TITLE>
<COPYRIGHT>
<YEAR>2005</YEAR>
<YEAR>2006</YEAR>
<HOLDER>Free Standards Group</HOLDER>
</COPYRIGHT>
&legal;
</BOOKINFO>

&toolkit-compression-intro;
&contents;

</BOOK>

To include this in the spec document build:

[lsbspec]$ gendiff . .libbz2
--- ./book/makefile.libbz2      2007-03-16 12:53:31.000000000 -0400
+++ ./book/makefile     2007-03-16 12:54:03.000000000 -0400
@@ -17,7 +17,7 @@
        LSB-CXX-PPC32 LSB-CXX-PPC64 LSB-CXX-S390 LSB-CXX-S390X \
        Packaging-generic Packaging-AMD64 Packaging-IA32 Packaging-IA64 \
        Packaging-PPC32 Packaging-PPC64 Packaging-S390 Packaging-S390X \
-       Graphics-Ext XML $(QT4_SUBDIRS)
+       Graphics-Ext XML Toolkit_Compression $(QT4_SUBDIRS)
        # $(GTK_SUBDIRS) $(QT3_SUBDIRS) 
 
 all:
--- ./makefile.libbz2   2007-03-15 17:48:52.000000000 -0400
+++ ./makefile  2007-03-15 17:49:18.000000000 -0400
@@ -1,5 +1,5 @@
 
-DOCDIRS=ELF Graphics LSB Packaging Graphics-Ext Toolkit_Gtk Toolkit_Qt3 Toolkit_Qt XML Desktop
+DOCDIRS=ELF Graphics LSB Packaging Graphics-Ext Toolkit_Gtk Toolkit_Qt3 Toolkit_Qt XML Desktop Toolkit_Compression
 BOOKDIRS=book booksets
 SUBDIRS= $(DOCDIRS) $(BOOKDIRS)
 
--- ./entities.libbz2   2007-03-16 13:04:21.000000000 -0400
+++ ./entities  2007-03-16 13:08:03.000000000 -0400
@@ -184,6 +184,11 @@
 <!ENTITY toolkit-gtk-s390-gtk SYSTEM "Toolkit_Gtk/S390/GTK/GTK.sgml">
 <!ENTITY toolkit-gtk-s390x-gtk SYSTEM "Toolkit_Gtk/S390X/GTK/GTK.sgml">
 
+<!-- Toolkit-Compression Chapters -->
+<!-- Generic -->
+<!ENTITY toolkit-compression-intro SYSTEM "Toolkit_Compression/intro/intro.sgml">
+<!ENTITY toolkit-compression-generic-bz2 SYSTEM "Toolkit_Compression/generic/libbz2/bz2.sgml">
+
 <!-- Graphics-Ext Chapters -->
 <!-- Generic -->
 <!ENTITY graphics-ext-intro SYSTEM "Graphics-Ext/intro/intro.sgml">

Writing tests

Ideally, tests should be designed to test each one of the newly added interfaces. A complete test should check not only "normal" function behavior, but also verify the interface correctly handles corner and error cases, returning expected error codes.

The LSB uses a test framework known as "TET" - Test Environment Toolkit. TET includes APIs for C as well as modules for ksh, posix_sh, perl, and python.

Creating tests to verify ABI compliance can be approached in a number of ways, depending on what the library in question might provide. If the vendor already provides a test suite, it may be possible to adapt or wrap these tests in a fashion that will provide "TET" output in the form of a journal file, than can then be processed by the TET tools and submitted for certification.

LSB provides the packages lsb-tet3-lite as well as lsb-tet3-lite-devel which can be used to develop tests.

To run TET code, the environment variable TET_ROOT must be set. For these examples, setting it to `pwd` should be sufficient:

export TET_ROOT=`pwd`

Perl 4 example (from the tet tarball)

#!/usr/bin/perl
@iclist=(ic1);
@ic1=("tp1","tp2");

$tet'startup="startup";
$tet'cleanup="cleanup";

sub startup {
        &tet'infoline("This is the startup");
        &tet'delete("tp2", "deleted in startup");
}

sub cleanup {
        &tet'infoline("This is the cleanup");
}

sub tp1{

        &tet'infoline("This is the tp1 test case"); 
        &tet'result("PASS");
}

sub tp2{

        &tet'infoline("This is the tp2 test case"); 
        &tet'result("PASS");
}

require "$ENV{\"TET_ROOT\"}/lib/perl/tcm.pl";

Run the test:

./tc1

Results are in tet_xres:

15|0 3.7 1|TCM Start
520|0 0 14116 1 1|This is the startup
400|0 1 2 15:37:39|IC Start
200|0 1 15:37:39|TP Start
520|0 1 14116 1 1|This is the tp1 test case
220|0 1 0 15:37:39|PASS
200|0 2 15:37:39|TP Start
520|0 2 14116 1 1|deleted in startup
220|0 2 6 15:37:39|UNINITIATED
410|0 1 2 15:37:39|IC End
520|0 0 14116 1 1|This is the cleanup

Python example (from the tet tarball)

#!/opt/lsb/appbat/bin/python
from pytet import *
import sys

def startup():
        print "tc1: Calling startup"
        tet_delete(3, "Marking test 3 as uninitiated")

def cleanup():
        print "tc1: Calling cleanup"

def test1():
        try:
                var = tet_getvar("PYTET_TC1_VAR");
        except:
                tet_infoline("Failed to get a value for PYTET_TC1_VAR")
                tet_result(TET_UNRESOLVED)
                return

        tet_infoline("PYTET_TC1_VAR is set to " + var);
        tet_result(TET_PASS)

def test2():
        tet_infoline("platform = " + sys.platform)
        tet_result(TET_PASS)

def test3():
        tet_result(TET_NOTINUSE)

testlist = { 1:test1, 2:test2, 3:test3 }
pytet_init(testlist, startup, cleanup)

Run the test:

./tc1
tc1: Calling startup
tc1: Calling cleanup

Results are in tet_xres:

15|0 3.7-lite 3|TCM Start
400|0 1 1 15:30:21|IC Start
200|0 1 15:30:21|TP Start
520|0 1 00013957 1 1|Failed to get a value for PYTET_TC1_VAR
220|0 1 2 15:30:21|UNRESOLVED
410|0 1 1 15:30:21|IC End
400|0 2 1 15:30:21|IC Start
200|0 2 15:30:21|TP Start
520|0 2 00013957 1 1|platform = linux2
220|0 2 0 15:30:21|PASS
410|0 2 1 15:30:21|IC End
400|0 3 1 15:30:21|IC Start
200|0 3 15:30:21|TP Start
520|0 3 00013957 1 1|Marking test 3 as uninitiated
220|0 3 6 15:30:21|UNINITIATED
410|0 3 1 15:30:21|IC End

C example (from the tet tarball)

#include <stdlib.h>
#include <tet_api.h>

/* 
 *  The tet_startup / tet_cleanup functions will be explained
 *  in detail later in the course, as will the tet_testlist
 *  structure 
 */

void (*tet_startup)() = NULL, (*tet_cleanup)() = NULL;
void tp1();

struct tet_testlist tet_testlist[] = { {tp1,1}, {NULL,0} };

void tp1()
{
        (void) tet_printf("test case: %s, TP number: %d ",
                tet_pname, tet_thistest);
        tet_result(TET_PASS);

/*      tet_result(TET_FAIL);*/
}

Compile the test:

cc -I/opt/lsb-tet3-lite/inc/tet3 -o tc1 tc1.c /opt/lsb-tet3-lite/lib/tet3/tcm.o \
    /opt/lsb-tet3-lite/lib/tet3/libapi.a

Run the test:

./tc1

Results are in tet_xres:

15|0 3.7-lite 1|TCM Start
400|0 1 1 15:48:52|IC Start
200|0 1 15:48:52|TP Start
520|0 1 00014320 1 1|test case: ./tc1, TP number: 1 
220|0 1 0 15:48:52|PASS
410|0 1 1 15:48:52|IC End

Many more examples are include with the tet tarball source.

Whatever API you choose, the final test setup should be put together such that it can readily be integrated with the existing test suites. You can review the current test suites in bzr under the directories "something-test"

libbz2 test example

Now if we go back to our libbz2 example, we'll make tests for the BZ2_bzRead, BZ2_bzWrite interfaces, using the TET C API:

You can build and run the tests like this:

tar xzf bzip2-test-0.1lsb.tar.gz
cd bzip2-test-0.1lsb
make test

Prerequisites to build are lsb-tet3-lite-devel and whatever your distribution package for libbzip2-devel may be called.

Looking at the bzRead.c and bzWrite.c test code, you'll see we've setup a test for each of the possible return code scenarios for each function, testing not only normal operation but that the error codes returned when the function fails agree with the specification. (bzRead test2,10 are incomplete (UNTESTED)).

Each test follows the sample API code above, with a START/RESULT/END, as well as some explanation text of the assertion being tested. Although the text isn't required, it can be helpful for the end user in debugging test failures:

200|0 1 16:11:31|TP Start
520|0 1 00016060 1 1|Test 1
520|0 1 00016060 1 2|When BZ2_bzRead is called with appropriate arguments, the function 
shall return the number of bytes read and set bzerror to BZ_OK
220|0 1 0 16:11:31|PASS
410|0 1 1 16:11:31|IC End

It's important to try and catch every possible exit path from your test so that meaningful results are captured in the journal file. If the test exits with no result code, it makes it very difficult for the user to debug why the test counts and/or journal analysis may be inconsistent.

That's about it. To complete libbzip2, we'd continue to edit spec files for the remaining functions and write test cases for each. The new library could then either be integrated into an existing LSB module or become a module of it's own and added to the next LSB release.

libbz2 test example

Now if we go back to our libbz2 example, we'll make tests for the BZ2_bzRead, BZ2_bzWrite interfaces, using the TET C API:

You can build and run the tests like this:

tar xzf bzip2-test-0.1lsb.tar.gz
cd bzip2-test-0.1lsb
make test

Prerequisites to build are lsb-tet3-lite-devel and whatever your distribution package for libbzip2-devel may be called.

Looking at the bzRead.c and bzWrite.c test code, you'll see we've setup a test for each of the possible return code scenarios for each function, testing not only normal operation but that the error codes returned when the function fails agree with the specification. (bzRead test2,10 are incomplete (UNTESTED)).

Each test follows the sample API code above, with a START/RESULT/END, as well as some explanation text of the assertion being tested. Although the text isn't required, it can be helpful for the end user in debugging test failures:

200|0 1 16:11:31|TP Start
520|0 1 00016060 1 1|Test 1
520|0 1 00016060 1 2|When BZ2_bzRead is called with appropriate arguments, the function 
shall return the number of bytes read and set bzerror to BZ_OK
220|0 1 0 16:11:31|PASS
410|0 1 1 16:11:31|IC End

It's important to try and catch every possible exit path from your test so that meaningful results are captured in the journal file. If the test exits with no result code, it makes it very difficult for the user to debug why the test counts and/or journal analysis may be inconsistent.

That's about it. To complete libbzip2, we'd continue to edit spec files for the remaining functions and write test cases for each. The new library could then either be integrated into an existing LSB module or become a module of it's own and added to the next LSB release.

Credits

[1] Thanks to Tracy Camp for assistance with this document and for the detailed explanations of the issues involved with more complex libraries like OpenSSL (some of which are pasted verbatim into this document). Also thanks to Mats Wichmann for pointing me in the right direction(s) and helping me get up to speed working on the development side of LSB, vs. being on the customer side, certifying a distribution.


[Article] [Discussion] [View source] [History]