From The Linux Foundation
Revision as of 23:12, 4 February 2008 by Oedipus (Talk | contribs)

(diff) ←Older revision | view current revision (diff) | Newer revision→ (diff)
Jump to: navigation, search

Minutes of the Open A11y Expert Handlers SIG Call 2007/11/26

Note: The following minutes are quite intensively linked to issues and entities where there is cross-over between the Expert Handlers work at Open A11y and work in other fora. It is suggested that one first obtain an overview of the meeting's transactions by first listening to the OGG-formatted audio archive of the 26 November 2007 Expert Handlers call as a means of orienting oneself to the detailed discussions minuted below. For more information about accessing the audio archive of this meeting, please consult Janina's announcement concerning the audio archive of this meeting.


  • Neil Soiffer (NS/chair)
  • Vladmir Bulatov (VB)
  • Alfred S. Gilman (AG/chair, WAI PF)
  • Gregory J. Rosmaita (GJR/scribe)
  • Janina Sajka (JS)
    • regrets: none logged

Agenda Review

Approval of Last Meeting's Minutes

Background Materials for the 2007/11/26 Expert Handlers Conference Call

Minutes of Expert Handlers Conference Call 26 November 2007

Topic 1: Liaising with the W3C's Protocols & Formats Working Group

Topic 1A: Background Review

NS: PDF tag "formula" (usually a graphic) -- can tag each part of pieces of formula (corresponding MathML presentation syntax); heading, paragraph, blockquote

JS: any kind of expert markup -- most XML-based; when handler has something to handle, going to be XML

NS: universally the case -- most PDF not tagged, but when tagged have mapping to XHTML; each tag should be marked as "here is what it maps to in XHTML" (default) other more generic measures; not just for a11y but for fidelity of export as well

JS: XML-to-AT correspondence

NS: if embed chemical markup into PDF, should do so with an XML-based markup language that is standardized and publicly documented

[scribe's note: Al Gilman (AG) joins]

Topic 1B: Introductions

NS: chair of Expert Handlers SIG; interested in math a11y -- member of MathML activity since its inception; sat on DAISY group and PDF universal access group

VB: work with Jim Gardner -- accessible graphics through SVG and integration of SVG into DAISY

JS: chair, Open A11y working group;

Topic 1C: Genesis of Expert Handlers activity in Open A11y

JS: context: Neil came to FSG when IAccessible2 was submitted last September (2006); could help DesignScience and others presenting expert markup for PWDs -- speech output, speech input, alternate input, tactile output; IAccessible2 group decided that it was a general issue to be covered in ATK/AT-SPI; Open A11y formed Expert Handlers sub-team; been concentrating so far on use cases and requirements gathering -- finishing first round of use cases and architectural framework -- considering where opportunities lay, almost anything can conceive of is XML-based, even if embedded in a Compound Document (even PDF); platform agnostic expert handler is the goal, XML based handler best candidate

NS: one thing that IAccessible2 SIG felt was they had done their job to solve problem -- access to document their goal; recognized that AT vendors just receives a lot of data, not interested in interpreting; hole having a11y to something but no means of making information useful to AT vendor that doesn't want to code to specific expert markup languages

NS: complicated in sense that tables are as far as most AT vendors want to go; math is just too much of an expert domain for mainstream AT vendors to tackle; in our conversations with AT vendors, tell us math support would be great -- when are you going to make it available to us; aim: generic framework that can call to say i have something, it is math, hands off to math handler which does intermediary work

Topic 1D: Working in Harmony/Brainstorming

GJR: generic handler defined in XML Events 2 does not include "purpose" element; nexus between this issue and CDF and Web API work at W3C; want to ascertain how Expert Handlers SIG can compliment and extend work of PF Working Group in liaising with other fora; division of labor with common goal to accelerate implementation of expert handlers and the reinstatement of a "purpose" element into the generic handler framework

AG: inside W3C my understanding that the group that is working on "purpose" element function for generic handlers is the Ubiquitous Web domain -- need separate statements of what does (click does it, follow hyperlink does)

GJR: 4 necessary components of a generic handler purpose element captured for me by "IISI" acronym: identity, integrity, state and intent - identity (i am X), integrity (i am X and here is proof that i am X), state (i am verifiably X and am in Y state), and intention (i am verifiably X in Y state so user has a choice of the following actions) --

AG: in case of MouseOver in GUI no documented label as to what a widget does; therefore, label for INPUT provides user-processable information -- if you do this, what happens as well as "where am i, what's there and what can i do?" -- need explicit means of expressing this; don't expect XHTML2 or XML Events to get that distinction -- that each object has to declare what it does; "purposing belongs in a later processing layer, not in the formatting layer" is reply from XHTML2 group; repurposing at a higher processing level as well -- same functionality in different skins (triggering bindings, labeling bindings); purpose equals intent -- what does it do? not a lot of traction in W3C for purpose declaration; assumption is that can be achieved by some kind of labelling

NS: how does compare with role attribute?

AG: role's usage is object class in target space -- tree has expand and collapse, navigate forward and back, navigate up and down; using role to expose fact that here is object that is capable of interaction -- ultimate outcome not captured in role, but in rendering -- how am i going to interact with it -- palette of gestures available to user; argument has been users can learn to deal with objects because used to desktop model that widgets based upon; indicate to user how to deal with object if identified for them, rather than spelling everything out

NS: purpose element - gives more detail than role element? more precision?

AG: yes, instance property -- what is labelledby (in ARIA markup) -- XML Events in XML data structure -- user understandable explanation -- request for labelling and "what does it do in data structure" (gestures available, AT reprogrammed gestures); that's why in Ubiquitous Web Applications (formerly the Device Independence group) is a key ally

AG: take dropping of "purpose" to UWA working group -- ask them to please help us -- do you (the UWA) need "purpose" to declare events in user-presentable form? explain requirement -- it is the intent, that is being expressed, not the binding -- query UWA if have an intent element; handler is a specific function -- can be generic, but point was to have documentation on what it does that can be presented to user to provide assistance/context -- tells user not only what hot-keys are available, but what each does

NS: doesn't seem that relevant for needs of expert handlers -- we don't interact with user, trying to aid AT software as intermediary between document and what can be presented to user

AG: not sure that's where you want to go;

AG: what the screen readers are going to want is likely a fairly superficial level of interaction; screen reader doesn't have to understand the order of integration, but user who understands math does; domain specific application software (math for math and astrophysics, genomic expressions and other uses); AT wants label of method -- ARIA work showed that domain knowledge is user understandable -- this explanation goes with this feature of document source; relationship between explanation and ????

NS: some action the user takes to get the explanation of math?

AG: assuming that expert application would transform MathML element through interface

NS: document itself shouldn't have anything in it that refers to Expert Handlers that is invoked to interpret it

AG: yes and no; correlation between map structures such as tree and methods expand/collapse; math is structures

NS: can write math as tree, can be used in tree-like manner

AG: equation math needs are simple -- can tackle as tree sturcuture (like DAISY book works like tree); when get into diagrams and graph structure, user has to understand what expert handler is doing for them; not clear that linearizing from tree structure is sufficient

VB: superimpose organization at some level that makes semantic sense;

JS: can define that and iterate tree-type structure to level can create framework

AG: don't want to go to trawl level, but may want to include concept of "loose" -- break arcs in graph to get into tree, for example; consumer example is the transit map -- how do i get from here to there -- one part of solution is trip planner that gives one a linear answer, can be cross-checked against graph -- put in information for user -- iterating graph navigation by neighbors hasn't been explored; plausible case throw into tree, but can't garuntee what comes out is usefull

NS: if doesn't fit tree, don't present as tree; graphs rich structures that are difficult to present if have lots of options; means of comprehending graph

AG: in my opinion, the point is that the Access API can deal with is likely to deal with dual layer of "chicken wire" -- interactive widgets and ARIA work -- critical properties -- last graph before pixels -- migration towards mixed presentation and less structure; have to take graph can get and annotate it with semantics; SRs can only go so deep in terms of interpretation -- will do tree as well as table, not clear that even if change occurs domain knowledge going to be in expert handlers

NS: we found that screen-reader developers say they don't want to know about this specialized markup natively, you tell us what to do and how to control interaction; same issues for SVG -- want something like D-Bus to provide handling

NS: if transit map coded in SVG, how to speak? at meta-level, that's what expert handlers are for --

JS: AT can support all because of standardized input to AT

NS: right; domain agnostic AT

JS: application navigating XML, reaches specific domain knowledge markup, identifies it and grabs expert handler, how does it wire together -- same parser?

AG: word that comes to mind is "ontology" -- transit map represented in SVG as a host language/file format relating to expert handler using 3 tiers of specialization: diagram level (symbolic graph), transit-map specific ontology (group segments, train routes by number or destination) would follow single-threaded arc graph to produce an object class -- could be declared in an ontology language

For Reference in Regards Ontology:

NS: not going to describe transit at ontology level; application hits SVG or MathML tag in document, AT software calls handler that understands the specialized markup language; expert software has to interpret DTD -- don't break up SVG by purpose (train route, geometric shapes) -- is that what you are talking about with ontology? -- coarser grain of data format and who is going to handle

AG: SVG player that UA going to call to get pixels on screen

VB: UA doing the job

AG: state of art, handled by plug-in; even if UA implements plug-in, is a delegated job for specialized formats -- that's the compound document peice of it; there is another call for expert handler to AT extracted from basic DOM that core application builds (including plug-ins) -- filling gap between markup and comprehensibility -- that is where role enters into it

JS: FireFox may have different handler for MathML than OpenOffice -- parallel track for AT is going to know begin MathML end MathML -- expose all tagging to AT, runs parallel interpreter to provide specific output modality

AG: like tab order -- brute force approach is too large to be realistic; slicing or linearizing to filter domain specific data structure appropriate to presentation and input capabilities of user

AG: this is precisely the area -- publish ontology, someone in AT developer goes extra mile to interpret ontology; that is a way of interpreting domain knowledge; if UA tries to help AT and layers between object and a11y API -- tool that goes out, grabs ontology for specific markup, and interprets it for user -- need to get new SQL-type query processing for domain knowledge from model knowledge for specific domain which is not processed except as graphics; expert handlers can fetch ontology to interpret role, class, microformats, etc. -- additional ontology that relates to how basic graphic format being used

NS: understand concern about graphics formats not being understood well by generic SVG tool -- not sure applies to math, music or chemistry -- graphs are very broad; if say we want to add more info to SVG to obtain a more specific tool, still a generic issue to be solved: how is AT to discover application, what are API calls app can make to obtain info, navigational functions and conversion functions are missing layer; solving problem 3 steps beyond where we can go now -- haven't yet figured out first step

JS: dichotomy between specialized markup languages -- semantic meaning in elements that can be interpreted by AT divorced from renderer; 2 levels of thinking -- figure out different ways of navigate and interact with specialized markup -- bring closer to tree-level

AG: the AT doesn't know novel navigation methods added to spec; can be exposed via menus from expert application -- label for name user can understand -- AT can't do it, but expert handler should

NS: case of SVG or Math, it is all data in the end; specific application that displays it either presents a menu or doesn't present a menu; expert handler not what is handling renderer, unless have API that tells AT software "here's what we want to expose" in API

AG: there is "menurole" (akin to ARIA menu role) in Accessibility API -- telling authors that they have to document

NS: if author decides to put in menu over SVG to enable specific action, but independent of description of what the math is, independent of describing SVG or formula -- independent of data standard for that domain; not there for math not there for SVG;

NS: specific case of math: in DOM encounters math tag, may be as simple as saying "one-half" -- don't need menus, but something has to render "one-half" to user from markup -- expert has to have interface to get info out

AG: speech and braille software does that

NS: no one takes strings out of specialized MLs -- data structure to string to speak is problem -- if have big data structure something has to be able to parse tags and present info in proper order

NS: for math, some ways of making accessible might be to say "one-half" "one-over-two" "start fraction 1 over 2 end fraction" -- speech rule dependent -- encoded as tree in XML, software still has to take tree that IA2 or MSAA gave the AT software has to be farmed out to expert handler to obtain what to announce, what to write to braille display

NS: implementing to standard

AG: hidden from AT?

NS: all there for AT -- that's why why the IAccessible2 group said "we don't need to do anything" -- too complicated for them to worry about -- want canned solution for them; have proprietary solution, but would like an open solution that applies to more than just math so doesn't need to be re-invented evrery time if standard changes, OS version changes interaction model;

NS: for genetics have chunk of data described in genome -- just a bunch of strings -- something needs to know how to deal with it (scale properly, speak properly)

AG: standard exposition -- tree from equation, passes into literate speech

NS: standard should nto say this is how you read something, but here is data, give me back string to speak

AG: in java, text content of object could be data value or indication of method; want ambiguity so that can give text content from expert handler at higher level of knowledge of domain, or just transliterates it, in which case you get garbage; ambiguity needs to be built in -- AT can process product of expert handler process to create literate reading of equation

JS: then the "degrade gracefully" case becomes -- "ding! don't have handler for this type of markup"

AG: annotate MathML or SVG with metadata what domain of knowledge this markup language represents, present to user with metadata -- that's the point of the ongoing work at the Fluid project (

NS: adding metadata to specific data

AG: Expert Handler competency in terms of understanding domain specific knowledge -- here are the user preferences, follow them to present domain specific information; standard identifiers for vocabularies -- what needs to be represented and how -- amenable to specialization; generic and more fine-grained

NS: can state handle certain data formats

AG: data may be identified more specifically using handler

NS: i handle this data format, if tagged correctly very good interpretatation, can generate user specified output (braille encoding); how to get meta-data

JS: point of specifying this is what Handlers do

NS: set of capabilities advertised by each handler; data required optionally specified;

JS: present list of those on initial encounter -- install something registered with AT so in real time, can grab cached specified handler -- one for braille and one for speech runing at same time

AG: 85% correlation; web services parallel -- things that could be understood

JS: could do over web, but will take longer; need caching mechanism

AG: not going to get metadata unless someone defines it; Fluid Project trying to get data with purpose-driven handlers

NS: there's no income tax math in the fluid project -- no formulae

AG: formulas in the pay-per-service, such as Turbo Tax, online -- not delivered to user/client

NS: question for AG: is there a W3C activity that we should be involved in -- how to take XML (tagged with metadata at document or instance level, optimally) and produce

AG: Compound Document Formats and Web API groups

GJR: process of joining Web API WG; informally liaising with Doug Schepers (W3C staff contact for WebAPI, SVG, and CDF) and Kenny Johar on CDF and SVG accessibility and integration into DAISY issues

NS: how should we put data together?

AG: can't focus myopically on API group without some knowledge of handshake -- how are they cooperating and communicating; have to be able to navigate in and out of documnent in different namespace of embedded object; has to be programmatic communication in terms of navigation throughout document;

NS: MathML working group starting an effort to get MathML in CDF specification -- will be addressing accessibility issues with them

VB: good to have discussion -- 3 steps ahead of what we want to accomplish

GJR: can we discuss relationship to ARIA?

AG: to a man with a hammer, everything looks like a nail; what i know about a11y API learned from working on ARIA markup; a11y API to expert handler interface -- go from DOM to pipe data to AT after processed by expert handler;

NS: expert processor should modify DOM?

AG: it may, but reads DOM and presents processed data to AT

NS: way that design science currently works with AT, but other groups wanted it the other way around

JS: there's another step missing

AG: expert refers back to source -- start at topmost level -- preferences that user understands,

JS: should be able to specify what is in our spec and what kind of data gets processed to hand to API

AG: platform specific application, but underlying ideas are portable

JS: helps a lot

NS: thanks for joining call --

AG: keep PF posted on what is going on with expert handlers

GJR: will keep bilateral lines of communication open; thank you, AG, for an extra half-hour of your time

JS: should have asked at beginning, but does AG have any objections to Open A11y making an audio archive of meeting?

AG: yes, you may retroactively record audio and share with Open A11y WG; may want to share more widely within W3C;

JS: still have the wai-pf archive on -- can use that to provide opportunity for broadening brainstorming exercise begun here