The Linux Foundation

 
ImportGotchas

From The Linux Foundation

Specific Issues/Workarounds encountered using headertodb2 to import libtiff

Just a rundown of issues I hit trying to use the headrtodb2 tool to inport libtiff for LSB 5.0. Although not apparent from the workarounds, this was very time consuming in learning my way around the database schema and how the headers get generated from the database data. With the following post-sql I get generated headers that appear to be similar to the upstream header (diff doesn't do much good, things are too re-arranged, comments dropped, etc.).

Contents

Setup:

Used tiff-4.0.2.tar.gz from upstream, configured and build locally (Mageia's headers are altered for multiarch support and they tend to give the tools additional fits).

Checkout libtodb2/headertodb2 from bzr:

bzr branch http://bzr.linuxfoundation.org/lsb/devel/libtodb2/
cd libtodb2/headertodb2

Prepare/load the data:

Setup specdb, set env variables, do make restore (documented elsewhere)

Because I was doing this a number of times and needed to "make restore" on specdb several times, I made a small script for the steps (also set_appeared.pl still has 4.0, changed it to 5.0):

# split the headers into 2 passes, as I can't seem to force ordering as I want
./headertodb2.pl --lib libtiff -a -I ./tiff-4.0.2/libtiff tiff-4.0.2/libtiff/tiffvers.h tiff-4.0.2/libtiff/tiffconf.h
./prepare_sql.pl -l libtiff >add_libtiff.sql
mysql $LSBDB -h $LSBDBHOST -u $LSBUSER --password=$LSBDBPASSWD < add_libtiff.sql
../set_appearedin.pl -l libtiff -a -f -v 5.0 > appeared_libtiff.sql

./headertodb2.pl --lib libtiff -a -I ./tiff-4.0.2/libtiff tiff-4.0.2/libtiff/tiff.h tiff-4.0.2/libtiff/tiffio.h
./prepare_sql.pl -l libtiff >add_libtiff.sql
mysql $LSBDB -h $LSBDBHOST -u $LSBUSER --password=$LSBDBPASSWD < add_libtiff.sql
../set_appearedin.pl -l libtiff -a -f -v 5.0 > appeared_libtiff.sql
mysql $LSBDB -h $LSBDBHOST -u $LSBUSER --password=$LSBDBPASSWD < appeared_libtiff.sql 
mysql $LSBDB -h $LSBDBHOST -u $LSBUSER --password=$LSBDBPASSWD < finish_libtiff.sql

finish_libtiff.sql:

SET @Lid=(SELECT Lid FROM Library WHERE Lname='libtiff');
SET @Mid=(SELECT SMid FROM SubModule WHERE SMname='LSB_Graphics');
INSERT INTO SModLib VALUES( @Mid, @Lid, '5.0', NULL );
INSERT INTO ArchLib VALUES( @Lid, 1, 'libtiff.so.5', '5.0', NULL );
/* FIXME */
/* INSERT INTO ArchLib VALUES( @Lid, 1, 'libtiffxx.so.5', '5.0', NULL ); */

Problems/Workarounds

The above procedure works as far as importing, and seems to work OK to generate stub libs and libchk. There are problems though in header generation.

Mysql returns "out of memory" error and/or anonymous structs are missing in generated headers

More than once I ran into trying to generate headers from the above load and just had things stall until mysql gave up with "out of memory". The headers I was able to get out seemed to be missing any anonymous structs.

bzr branch http://bzr.linuxfoundation.org/lsb/devel/build_env
cd build_env/headers
make tiffvers.h tiffconf.h tiff.h tiffio.h

The issue seems to originate in headertodb2. The generated flat files (__res/res_type_member) from this step have entries like: "._N" with N being a number that corresponds to the anon-struct-N entries. It looks like these should really have some other value, and the "._N" values end up in the sql and ultimately in the database, and subsequently the data is insufficient for mkheader to generate the anonymous structs, and in some cases exhausts memory trying to gather the data together.

Workaround

The "._N" entries seem to be safe to remove from TypeMember

For an anonymous struct, there are 2 entries in Type, one with Ttype "Struct" and one should have Ttype "Typedef". The first has Tname of the type anon-struct-N, while the 2nd should have the actual struct name.

The entries in TypeMember for the struct members need TMappearedin set to '5.0' The entries in ArchType need ATappeared in set also and also need ATbasetype set to point to the anon-struct entry from Type.

fix_libtiff.sql:

/* clean up cruft from the "._N" entries - anonymous structs
   before this is run the header gen aborts with out of memory */
delete from TypeMember where TMname like '._%';
create temporary table Tdefs (Tdef INT);
insert into Tdefs (select Tid+1 as Tdef from Type where Tlibrary='libtiff' and Ttype='Struct' and Tname like 'anon%');
update Type set Ttype='Typedef' where Tid in (select Tdef from Tdefs);
drop table Tdefs;
update TypeMember set TMAppearedin='5.0' where TMmemberof in (select Tid from Type where Tlibrary='libtiff' and Ttype='Struct' and Tname like 'anon%');
create temporary table Tanon (Tid INT, Tidplus INT);
insert into Tanon (select Tid, Tid+1 as Tidplus from Type where Tlibrary='libtiff' and Ttype='Struct' and Tname like 'anon%');
update ArchType set ATappearedin='5.0', ATbasetype=ATtid-1 where ATtid in (select Tidplus from Tanon);
drop table Tanon;

macros defined in tiffconf.h, to be used in tiff.h, get replaced by actual values

I don't think this one hurts anything, but the generated headers do look different than the upstream. Haven't come up with a fix/workaround for this yet.

Example (upstream vs generated, define is in the generated tiffconf.h):

typedef TIFF_UINT64_T uint64;

vs

unsigned long long int uint64;

uint{18,6,32}, uint16_vap used in tiff.h not defined

While uint64 does get defined, uint{8,13,32} and uint16_vap seem to get dropped. uint16_vap isn't used elsewhere in the headers, but the comments seem to indicate it's intended to be used.

Workaround: add the missing defines, enable uint16_vap:

SET @HGid=(SELECT HGid FROM Header JOIN HeaderGroup ON HGheader=Hid WHERE Hname='tiff.h' LIMIT 1);
INSERT INTO Type (Tid,Theadgroup,Ttype,Tname,Tlibrary) VALUES (0,@HGid,'Typedef','uint8','libtiff');
SET @Tid=(SELECT last_insert_id());
INSERT INTO ArchType (ATaid,ATtid,ATappearedin) VALUES(1,@Tid,'5.0');
SET @ATtid=(SELECT Tid FROM Type WHERE Tname='unsigned char' AND Ttype='Intrinsic');
UPDATE ArchType SET ATbasetype=@ATtid WHERE ATtid=@Tid;

SET @HGid=(SELECT HGid FROM Header JOIN HeaderGroup ON HGheader=Hid WHERE Hname='tiff.h' LIMIT 1);
INSERT INTO Type (Tid,Theadgroup,Ttype,Tname,Tlibrary) VALUES (0,@HGid,'Typedef','uint16','libtiff');
SET @Tid=(SELECT last_insert_id());
INSERT INTO ArchType (ATaid,ATtid,ATappearedin) VALUES(1,@Tid,'5.0');
SET @ATtid=(SELECT Tid FROM Type WHERE Tname='unsigned short' AND Ttype='Intrinsic');
UPDATE ArchType SET ATbasetype=@ATtid WHERE ATtid=@Tid;

SET @HGid=(SELECT HGid FROM Header JOIN HeaderGroup ON HGheader=Hid WHERE Hname='tiff.h' LIMIT 1);
INSERT INTO Type (Tid,Theadgroup,Ttype,Tname,Tlibrary) VALUES (0,@HGid,'Typedef','uint32','libtiff');
SET @Tid=(SELECT last_insert_id());
INSERT INTO ArchType (ATaid,ATtid,ATappearedin) VALUES(1,@Tid,'5.0');
SET @ATtid=(SELECT Tid FROM Type WHERE Tname='unsigned int' AND Ttype='Intrinsic');
UPDATE ArchType SET ATbasetype=@ATtid WHERE ATtid=@Tid;

SET @Tid=(SELECT Tid from Type where Tname='uint16_vap' LIMIT 1);
UPDATE ArchType SET ATappearedin='5.0' where ATtid=@Tid;

tweak the ordering in tiff.h with the new adds

With the new typdefs added later, they fall after where they are used.

Workaround:

To avoid any issues, re-order part of tiff.h:

/* tweak the tiff.h ordering */
SET @Hid=(SELECT Hid from Header where Hname='tiff.h' LIMIT 1);
INSERT INTO HeaderGroup (HGName,HGHeader,HGorder) VALUES('Secondary Section for tiff.h',@Hid,1);
SET @Hgid=(SELECT last_insert_id());
update Type set Theadgroup=@Hgid where Tname='TIFFHeaderCommon';
update Type set Theadgroup=@Hgid where Tname='TIFFHeaderClassic';
update Type set Theadgroup=@Hgid where Tname='TIFFHeaderBig';
update Type set Theadgroup=@Hgid where Tname='TIFFDataType';

tiff.h should include tiffconf.h

This include got dropped in the generated headers.

Workaround:

The Constant table has a mechanism to force a header include. You add an entry with the Ctype "header_depend" and the HeaderGroup HGid for the header in Constant, as well as an entry in ArchConst with ACcid set to this new Constant record and ACvalue set to the Hid from the Header table.

/* tiff.h should include tiffconf.h */
SET @Hid=(SELECT Hid from Header where Hname='tiff.h' LIMIT 1);
SET @Hgid=(SELECT HGid from HeaderGroup where HGheader=@Hid and HGorder=0 LIMIT 1);
INSERT INTO Constant (Cname,Ctype,Cheadgroup) VALUES('tiff.h depends on tiffconf.h','header_depend',@Hgid);
SET @Accid=(SELECT last_insert_id());
SET @Hid=(SELECT Hid from Header where Hname='tiffconf.h' LIMIT 1);
INSERT INTO ArchConst (ACaid,ACcid,ACvalue,ACappearedin) VALUES(1,@Accid,@Hid,'5.0');

_TIFFRGBAImage struct should not be anonymous

This could well be fallout from my post-processing, but since mysql gave up before generating headers, it's hard to tell.

Workaround:

To fix this, remove the second Typedef entry in Type, change Tname for the anon entry to "_TIFFRGBAImage". Then delete any entries in ArchType with ATbasetype of the deleted Type entry (there seem to be 2 sets plus some odd random things). The entries in ArchType we want are actually assigned to the "TIFFRGBAImage" Type entry, so we need to reset these to the underbar version we just modified.

/* this struct should not be anonymous (_TIFFRGBAImage) */
SET @Tid=(SELECT Tid from Type where Tname='_TIFFRGBAImage');
DELETE FROM Type WHERE Tid=@Tid;
UPDATE Type SET Tname='_TIFFRGBAImage' WHERE Tid=@Tid-1;
DELETE from ArchType where ATtid=@Tid;
SET @Tid2=(SELECT Tid from Type where Tname='TIFFRGBAImage');
UPDATE ArchType set ATbasetype=@Tid-1 where ATtid=@Tid2;
DELETE from TypeMember where TMmemberof=@Tid;

typdefs of the style 'typdef xxx (*Pointer) (arg1, arg2, arg3...)' are generated as 'typdef xxx (*Pointer) (void)'

Workaround:

Appears to just be a lack of the entries in TypeMember not having TMappearedin set. I just blanket set everything that was Ttype 'FuncPtr' in Type and was using that basetype in TypeMember. Seemed to do the right thing.

/* all of these have (void) in generated header and should have something else
/* tileContigRoutine, tileSeparateRoutine
TIFFInitMethod TIFFErrorHandler TIFFErrorHandlerExt TIFFReadWriteProc
TIFFSeekProc TIFFCloseProc TIFFSizeProc TIFFMapFileProc TIFFUnmapFileProc
TIFFExtendProc TIFFVSetMethod TIFFVGetMethod TIFFPrintMethod */
UPDATE TypeMember SET TMappearedin='5.0' WHERE TMmemberof IN (select Tid from Type where Ttype='FuncPtr' and Tlibrary='libtiff');

TIFFFieldInfo is obsolete - disable it, same for TIFFMergeFieldInfo

I could have avoided this just by not importing in the first place, easy to disable in ArchType, ArchInt.

/* TIFFFieldInfo is obsolete - disable it, same for TIFFMergeFieldInfo */
SET @Tid=(SELECT Tid from Type where Tname='TIFFFieldInfo');
UPDATE ArchType SET ATappearedin=NULL WHERE ATtid=@Tid;
SET @Iid=(SELECT Iid from Interface where Iname='TIFFMergeFieldInfo');
UPDATE ArchInt SET AIappearedin=NULL WHERE AIint=@Iid;

tiffio.h should include tiffvers.h

Similar issue to tiff.h needing tiffconf.h

/* tiffio.h should include tiffvers.h */
SET @Hid=(SELECT Hid from Header where Hname='tiffio.h' LIMIT 1);
SET @Hgid=(SELECT HGid from HeaderGroup where HGheader=@Hid and HGorder=0 LIMIT 1);
INSERT INTO Constant (Cname,Ctype,Cheadgroup) VALUES('tiffio.h depends on tiffvers.h','header_depend',@Hgid);
SET @Accid=(SELECT last_insert_id());
SET @Hid=(SELECT Hid from Header where Hname='tiffvers.h' LIMIT 1);
INSERT INTO ArchConst (ACaid,ACcid,ACvalue,ACappearedin) VALUES(1,@Accid,@Hid,'5.0');

mkheader insists on including nspr4 headers

mkheader tries to derive any required headers and adds #include lines for them in the generated header. tiff*.h use uint(16,32} which are only in LSB as part of nspr4 so the following includes get inserted, which we don't really want (I *think* it's uint16/32 when I enable trace in mkheader and try and follow the sql trail that pulls them in):

#include <nspr4/obsolete/protypes.h>
#include <nspr4/prtypes.h>

An experiment seems to support that this is what is happening:

mysql> select * from Type where Tname='uint32';
+-------+--------+---------+------------+--------------+----------+--------+-----------+------------+-----------+-------------+----------+--------+
| Tid   | Tname  | Ttype   | Theadgroup | Tdescription | Tsrconly | Tconly | Tindirect | Tunmangled | Tmemberof | Tinstanceof | Tlibrary | Tclass |
+-------+--------+---------+------------+--------------+----------+--------+-----------+------------+-----------+-------------+----------+--------+
| 35593 | uint32 | Typedef |       1452 |              | No       | No     | No        | NULL       |         0 |           0 | libssl3  |      0 |
| 40899 | uint32 | Typedef |       1597 |              | No       | No     | No        | NULL       |         0 |           0 | libtiff  |      0 |
+-------+--------+---------+------------+--------------+----------+--------+-----------+------------+-----------+-------------+----------+--------+
2 rows in set (0.01 sec)

mysql> select * from Type where Tname='uint16';
+-------+--------+---------+------------+--------------+----------+--------+-----------+------------+-----------+-------------+----------+--------+
| Tid   | Tname  | Ttype   | Theadgroup | Tdescription | Tsrconly | Tconly | Tindirect | Tunmangled | Tmemberof | Tinstanceof | Tlibrary | Tclass |
+-------+--------+---------+------------+--------------+----------+--------+-----------+------------+-----------+-------------+----------+--------+
| 33950 | uint16 | Typedef |       1452 |              | No       | No     | No        | NULL       |         0 |           0 | libnss3  |      0 |
| 40898 | uint16 | Typedef |       1597 |              | No       | No     | No        | NULL       |         0 |           0 | libtiff  |      0 |
+-------+--------+---------+------------+--------------+----------+--------+-----------+------------+-----------+-------------+----------+--------+
2 rows in set (0.00 sec)

mysql> select * from ArchType where ATbasetype IN (35593,33950);
+-------+-------+--------+--------------+---------------+------------+-------------+
| ATaid | ATtid | ATsize | ATappearedin | ATwithdrawnin | ATbasetype | ATattribute |
+-------+-------+--------+--------------+---------------+------------+-------------+
|     1 | 35369 | 0      |              | NULL          |      33950 | NULL        |
|     1 | 35452 | 0      |              | NULL          |      33950 | NULL        |
|     1 | 40838 | 0      |              | NULL          |      33950 | NULL        |
|     1 | 40833 | 0      |              | NULL          |      35593 | NULL        |
+-------+-------+--------+--------------+---------------+------------+-------------+
4 rows in set (0.00 sec)

mysql> delete from ArchType where ATbasetype IN (35593,33950);
Query OK, 4 rows affected (0.11 sec)

mysql> delete from Type where Tid IN (35593,33950);
Query OK, 2 rows affected (0.10 sec)

After the above deletions from Type, ArchType for the nspr entries, only tiffconf.h gets included in tiff.h.

Workaround:

Denis provided the fix for this:

SET @Tid=(SELECT Tid FROM Type WHERE Tname='uint32' AND Tlibrary='libtiff');
UPDATE Parameter JOIN Interface ON Iid=Pint SET Ptype=@Tid WHERE Ptype=35593 
AND Ilibrary='libtiff';
UPDATE Interface JOIN ArchInt ON AIint=Iid SET AIreturn=@Tid WHERE
AIreturn=35593 AND Ilibrary='libtiff';

SET @Tid=(SELECT Tid FROM Type WHERE Tname='uint16' AND Tlibrary='libtiff');
UPDATE Parameter JOIN Interface ON Iid=Pint SET Ptype=@Tid WHERE Ptype=33950
AND Ilibrary='libtiff';
UPDATE Interface JOIN ArchInt ON AIint=Iid SET AIreturn=@Tid WHERE
AIreturn=33950 AND Ilibrary='libtiff';

SET @Tid=(SELECT Tid FROM Type WHERE Tname='uint8 *' AND Tlibrary='libtiff');
UPDATE Parameter JOIN Interface ON Iid=Pint SET Ptype=@Tid WHERE Ptype=35501
AND Ilibrary='libtiff';
UPDATE Interface JOIN ArchInt ON AIint=Iid SET AIreturn=@Tid WHERE
AIreturn=35501 AND Ilibrary='libtiff';

[Article] [Discussion] [View source] [History]