Implementing an MSAA Server

Practical Tips for Developers, and How Mozilla Does It

Contents

This document is for people working to support MSAA in an application in order to make it accessible with 3rd party assistive technologies, as well as for hackers wishing to be involved in Mozilla's MSAA support specifically.
You may also wish to read Gecko Info for Windows Accessibility Vendors, a primer for vendors of 3rd party accessibility software, on how MSAA clients can utilize Gecko's MSAA support.

1. Intro: What is MSAA

2. Deciding Which MSAA Features to Support
Methods
Events
States
Roles
Object Identifiers

3. MSAA's Quirks and Workarounds
MSAA can be crash prone
Hacky caret tracking not working
Event window confusion
Confusion with system-generated events
No unique child ID for object in window
Not all MSAA features utilized by 3rd party vendors
Missing functionality in MSAA
Dueling text equivalents
Issues with Links
Performance Problems
Differing client implementations
Undocumented Window Class Usage
Vendor quirks

4. Example: How Gecko and Mozilla Implement MSAA
Creation of IAccessible Objects
The Accessible Tree vs. the DOM Tree
The Various Implementations of IAccessible
Generating MSAA Events

5. Feedback

1. Intro: What is MSAA?

2. Deciding Which MSAA Features to Support

MSAA Methods - Cheat Sheet for Developers

MSAA Events Cheat Sheet

EVENT_SYSTEM_SOUND
EVENT_SYSTEM_ALERT
EVENT_SYSTEM_FOREGROUND
EVENT_SYSTEM_MENUSTART
EVENT_SYSTEM_MENUEND
EVENT_SYSTEM_MENUPOPUPSTART [important]
EVENT_SYSTEM_MENUPOPUPEND [important]
EVENT_SYSTEM_CAPTURESTART
EVENT_SYSTEM_CAPTUREEND
EVENT_SYSTEM_MOVESIZESTART
EVENT_SYSTEM_MOVESIZEEND
EVENT_SYSTEM_CONTEXTHELPSTART
EVENT_SYSTEM_CONTEXTHELPEND
EVENT_SYSTEM_DRAGDROPSTART
EVENT_SYSTEM_DRAGDROPEND
EVENT_SYSTEM_DIALOGSTART
EVENT_SYSTEM_DIALOGEND
EVENT_SYSTEM_SCROLLINGSTART
EVENT_SYSTEM_SCROLLINGEND [possibly important, talk to AT vendor]
EVENT_SYSTEM_SWITCHSTART
EVENT_SYSTEM_SWITCHEND
EVENT_SYSTEM_MINIMIZESTART
EVENT_SYSTEM_MINIMIZEEND
EVENT_OBJECT_CREATE [don't implement, watching system generated versions of this event causes assistive technology crashes]
EVENT_OBJECT_DESTROY [don't implement, watching system generated versions of this event causes assistive technology crashes]
EVENT_OBJECT_SHOW
EVENT_OBJECT_HIDE
EVENT_OBJECT_REORDER [important for mutating docs in future, but not yet]
EVENT_OBJECT_FOCUS [important]
EVENT_OBJECT_SELECTION
EVENT_OBJECT_SELECTIONADD
EVENT_OBJECT_SELECTIONREMOVE
EVENT_OBJECT_SELECTIONWITHIN
EVENT_OBJECT_STATECHANGE [important for checkboxes and radio buttons]
EVENT_OBJECT_LOCATIONCHANGE
EVENT_OBJECT_NAMECHANGE
EVENT_OBJECT_DESCRIPTIONCHANGE
EVENT_OBJECT_VALUECHANGE
EVENT_OBJECT_PARENTCHANGE
EVENT_OBJECT_HELPCHANGE
EVENT_OBJECT_DEFACTIONCHANGE
EVENT_OBJECT_ACCELERATORCHANGE

MSAA States Cheat Sheet

STATE_UNAVAILABLE [important]
STATE_SELECTED [important]
STATE_FOCUSED [important]
STATE_PRESSED           
STATE_CHECKED [important]
STATE_MIXED
STATE_READONLY [important]
STATE_HOTTRACKED        
STATE_DEFAULT [important]
STATE_EXPANDED [important]
STATE_COLLAPSED [important]
STATE_BUSY [important]
STATE_FLOATING          
STATE_MARQUEED 
STATE_ANIMATED        
STATE_INVISIBLE      

STATE_OFFSCREEN [important]
STATE_SIZEABLE       
STATE_MOVEABLE       
STATE_SELFVOICING    
STATE_FOCUSABLE [important]
STATE_SELECTABLE [important]
STATE_LINKED [important]
STATE_TRAVERSED [important]
STATE_MULTISELECTABLE [important]
STATE_EXTSELECTABLE  
STATE_ALERT_LOW      
STATE_ALERT_MEDIUM   
STATE_ALERT_HIGH     
STATE_PROTECTED [important]
STATE_HASPOPUP

MSAA Roles Cheat Sheet

ROLE_TITLEBAR [inserted by system]
ROLE_MENUBAR [important if you don't use native menus]
ROLE_SCROLLBAR
ROLE_GRIP
ROLE_SOUND
ROLE_CURSOR
ROLE_CARET
ROLE_ALERT
ROLE_WINDOW [inserted by system]
ROLE_CLIENT [important]
ROLE_MENUPOPUP [important]
ROLE_MENUITEM [important]
ROLE_TOOLTIP
ROLE_APPLICATION
ROLE_DOCUMENT
ROLE_PANE [important]
ROLE_CHART
ROLE_DIALOG
ROLE_BORDER
ROLE_GROUPING
ROLE_SEPARATOR [important]
ROLE_TOOLBAR
ROLE_STATUSBAR [important]
ROLE_TABLE [important]
ROLE_COLUMNHEADER
ROLE_ROWHEADER
ROLE_COLUMN
ROLE_ROW
ROLE_CELL [important]
ROLE_LINK [important]
ROLE_HELPBALLOON
ROLE_CHARACTER
ROLE_LIST [important]
ROLE_LISTITEM [important]
ROLE_OUTLINE [important]
ROLE_OUTLINEITEM [important]
ROLE_PAGETAB [important]
ROLE_PROPERTYPAGE [important]
ROLE_INDICATOR
ROLE_GRAPHIC [important]
ROLE_STATICTEXT [important]
ROLE_TEXT [important]
ROLE_PUSHBUTTON [important]
ROLE_CHECKBUTTON [important]
ROLE_RADIOBUTTON [important]
ROLE_COMBOBOX [important]
ROLE_DROPLIST [important]
ROLE_PROGRESSBAR [important]
ROLE_DIAL
ROLE_HOTKEYFIELD
ROLE_SLIDER
ROLE_SPINBUTTON
ROLE_DIAGRAM
ROLE_ANIMATION
ROLE_EQUATION
ROLE_BUTTONDROPDOWN
ROLE_BUTTONMENU
ROLE_BUTTONDROPDOWNGRID
ROLE_WHITESPACE
ROLE_PAGETABLIST [important]
ROLE_CLOCK
ROLE_SPLITBUTTON
ROLE_IPADDRESS
ROLE_NOTHING

MSAA Object Identifiers Cheat Sheet

For information on what each object identifier does, see the MSDN Object Identifiers Constants page.

Check with our assistive technology partners to find out what object identifiers you need to support. There's a very good chance they won't ask for more than the object itentifiers marked [important]:
OBJID_ALERT
OBJID_CARET
OBJID_CLIENT [important]
OBJID_CURSOR
OBJID_HSCROLL
OBJID_NATIVEOM [important? might be useful for supporting custom interfaces, need to research]
OBJID_MENU
OBJID_QUERYCLASSNAMEIDX
OBJID_SIZEGRIP
OBJID_SOUND
OBJID_SYSMENU
OBJID_TITLEBAR
OBJID_VSCROLL
OBJID_WINDOW

3. Dealing with the Quirks of MSAA

MSAA has a well deseved reputation for quirkiness. It is not "plug and play", and will take a lot of testing/refinement before your solution works with any product. Here are some of its quirks and some solutions/workarounds:

MSAA can be crash prone

Problem: Many of MSAA's crash occur because more than one process is refcounting the same objects, and because pointers are being shared between processes. When your application closes, different signals are typically broadcast. For example, the application window closes and the window is blurred. It is impossible to know if and when the 3rd party assistive technology will use one of these signals to release the objects of yours that is is refcounting. This can lead to crashes where it releases something and the wrong time, when some of your dll's are unloaded but not others, and a destructor is called in an unloaded DLL.

Solution: Create a "shutdown" method for each internal accessible object, to remove any references to other internal objects before any of your dll's are unloaded. In order to do this effectively, you will have to keep track of every accessible object that you create. The shutdown method for an accessibility object should be called whenever the document or UI object it refers to goes away. The easiest way to do that is to keep a pointer to an accessible in each internal UI object. If that pointer is non-null, then there is an accessible object for it. Whenever the UI object is destroyed, shutdown its accessible object as well. In Gecko/Mozilla we are not allowed to keep this extra pointer for each accessible object, so when accessibility is turned on we use a hash table to cache these objects. Such a cache must be kept in perfect sync with the tree of UI and document objects, which is difficult. Therefore, unless 4 bytes extra on each object is criticial in your application, just keep the extra pointer around instead of using a hash table.

Also, don't implement EVENT_OBJECT_CREATE or EVENT_OBJECT_DESTROY. Vendors have found that watching these events causes crashes.

Hacky caret tracking causes problems

Problem: Assistive technologies do not use the MSAA caret. They follow screen draws, looking for a vertical blinking line. Unfortunately, some products can get confused by the vertical lines on other objects, such as list boxes, even though those lines are not blinking. The assistive technology may not see your caret at all.

Solution: Make sure there is a configuration file for each assistive technology specific to your application. Read the manual or help, and find the keystroke or commands for training the caret, and save this information in the configuration file. Don't support the MSAA caret, none of the vendors use it.

Event window handle is incorrect

Problem: The screen reader or other assistive technology does not track the focus or other events correctly.

Solution: This may be because you are reporting that the events in a different window from the current system focused. The assistive technology may be asking GetGUIThreadInfo for its hwndFocus, and throwing away MSAA events that are not in the currently focused window. Even if you are visibly showing window focus on the correct window, you must also tell the operating system to focus this window before any other accessibility events get fired in it.

Confusion with system-generated events

Problem: When you test with Accessible Event Watcher in the MSAA SDK, you will see many events that your application did not generate. Microsoft Windows generates some events for you. This is extremely annoying. The assistive technology has no way to tell whether the event came from your application or from Windows. For example, when you set window focus, Windows will generate an EVENT_OBJECT_FOCUS event an a ROLE_WINDOW object it also generated for you. If you happen to set window focus after you fired your own EVENT_OBJECT_FOCUS event on an object in the widnow, your event will be ignored, because assistive technology software tends to pay attention only to the last focus event.

Solution: When an object is about to get focused in a different window, make sure you focus a window before you fire your own focus events for objects inside it. Test using Accessible Event Watcher in the MSAA SDK, and use the settings panel to watch subsets of accessibility events. Count on the assistive technology to make sense out the jumble of extra system-generated events, it's not your problem.

No unique child ID for event target in window

Problem: MSAA expects events to be reported using NotifyWinEvent(eventNum, hWnd, worldID, childID), and there may not be an obvious way to get a window handle and a 32 bit childID for an object. You may be using windowless controls, or have an entire document with lots of objects in a given window. The child ID must be unique, because this is what the assistive technology will use to retrieve the accessible via AccessibleObjectFromEvent() which ends up using get_accChild on the accessible for the given window.

Solution: In Gecko/Mozilla, we did not want to store an extra 32 bit unique ID value on every object. Instead, we hand back a 32 bit value derived from the UI object's pointer, which is unique. We ensure that the value we hand back is always negative. When the get_accChild call comes back, we check our hash table cache for that window to see if there's an accessible object still associated with that unique value. This means the client must use AccessibleObjectFromEvent immediately, because there is a danger that the object will go away, and another different object will be created with the same pointer value.That case seems extremely remote, because information from events is generally retrieved right after the event.

If you're not using a hash table to keep track of unique ID's, store the child ID's and objects for the last 50 or so events in a circular array. In practice, this is enough to keep AccessibleObjectFromEvent() happy.

Not all MSAA features utilized by 3rd party vendors

Problem: The assistive technology does not use 50% of what's available in MSAA, e.g. MSAA caret, a lot of events, roles, states and methods. It's difficult to know which things to support.

Solution: Use this document to see what is generally considered important by assistive technology manufacturers. Contact the the top vendors early and often as you plan and implement your architecture, to see what's important to them. Implement only what's needed -- supporting everything would take too long for zero results.

Missing functionality in MSAA

Problem and solutions: Assistive technology vendors need some things which MSAA does not provide, such as:
Dueling text equivalents

Problem: There are three kinds of text equivalents, and it is difficult to know when to use each. Different applications behave differently in this regard. For example, Acrobat uses accessible value for text labels where as most programs use accessible name. There are different roles for text objects, ROLE_STATICTEXT and ROLE_TEXT (editable text), which seems to be used for non-editable text in many places.

Solution: Be as consistent with Internet Explorer as possible. Use accessible name for most text equivalents, and accessible value for URL's. Don't use accessible description unless you really do have a long description for the object you need to expose -- most assistive technology makes little use of it. Use ROLE_STATICTEXT for labels specific to dialog and UI controls, and always use ROLE_TEXT for document text even if the text is not editable (in that case use ROLE_TEXT with STATE_READONLY).

Issues with Links

Problem: The assistive technology has inflexible heuristics when it comes to reading links. First, it won't read the object unless the states are correctly set. Second, it can mishandle the object if it cannot parse the whitespace according to its own rules.

Solution: Make sure the ROLE_LINK object and its child ROLE_TEXT objects all have STATE_LINKED set. For multi-line links with a line break in the middle, make sure there is no whitespace at the beginning or end of any of the accessible names, and make sure there is a \r\n where the line breaks occur in the accessible name for the ROLE_LINK. For an example of how to do this properly, see Internet Explorer or Gecko. Again, if it's not done exactly this way, some links will not be read.

MSAA Implementation is Not Performant

Problem: The assistive technology may interact slowly with your application.

Solution: Try not to calculate the same things more than once or create the same objects more than once. For example, create and cache an object's children when you look for them in get_accChildCount(), so that you can just hand them back when asked for using get_accChild() or accNavigate(). Support IEnumVARIANT so that the MSAA client can ask for a number of children in one call. In custom interfaces, create methods that hand back a lot of data in one call, rather than requiring a large number of calls. Fewer calls are much better better because COM Marshaling is slow.

Differing client implementations

Problem: Every assistive technology uses MSAA differently.

Solution: We don't know of any outright conflicts in the differing uses of MSAA (yet). However, be on guard. If a vendors asks you to do something different from the spec, it's better to check with the other vendors before moving forward. Check to see what applications from Microsoft do in a similar situation.

Undocumented Window Class Usage

Problem: most assistive technologies won't use your MSAA implementation out of the box. They must list your window classes somewhere in their implementation, and then turn on MSAA support when a window of that class receives focus. The window class is also used to determine a host of hard-coded behaviors, such as whether or not a screen reader will load the entire MSAA tree into a special buffer for the user to navigate with screen reader commands. This is only supposed to occur for document navigation, not for UI/dialogs. where your application's keyboard commands will be solely used to navigate.

Solution: Contact each vendor and let them know what window classes you will be using MSAA for. If possible, use a different window class name for documents/content than you use for UI/dialogs. Or, do what Mozilla does  - expose a control ID (GWL_ID) of 1 for content, and 0 for UI. Consistent window class names are important for the assistive technology vendors, so that they can determine what code to run for a given window. Don't change window class names after you have shipped a version.

Vendor quirks

Problem: Because assistive technology tends to utilize MSAA as an additional solution resting on top of old hacks, rather than a completely clean and separate way to deal with an application, and because of the quirky nature of MSAA and of the inflexible heuristics that screen readers use, we do not have a "plug and play solution". In addition, assistive technology vendors are tiny companies, often scrambling to keep up with changes in the applications they already support, or new products other than yours which they need to support. It's very difficult to get vendors to spend time testing an MSAA implementation, send feedback or help find out why things aren't working. Time and version commitments often fall through. There is always a belated new version due around corner, after which you will be promised to be the next priority.

Solution: Try to reach out in a friendly manner to the assistive technology company. Be as easy to work with as you possibly can -- this includes being extremely responsive to their bug reports with new test builds, as well as being very communicative about what you have changed and when. Do as much work as you possibly can without their help. See if your organization can offer something they can't get for themselves. Be patient, and set your expectations to a reasonable level. Realize that it's about both pride and revenue for these companies, and that they need to sell a lot of copies of their software to make up the work they put in to support your app. Remember that no matter how small they are, you need them more than they need you, unless your application's accessibility is being demanded by end-users.

4. Example: How Gecko and Mozilla Implement MSAA

The Accessible module is where the Mozilla MSAA implementation lives. Feel free to peruse the source code in the accessible module whenever you want to see how something can be implemented.

The accessible module is also where support for Sun's ATK accessibility API for Linux and UNIX is implemented. For documentation specific to the Mozilla ATK effort, supported by Sun Microsystems, see the Mozilla accessibility on Unix page.

Creation of IAccessible Objects

The Accessible Tree vs. the DOM Tree

MSAA tree vs. DOM tree - what's the relationship?

A Variety of Implementations for IAccessible

There are two main kinds of classes in Mozilla's accessibility class hierarchy, platform-specifc and cross-platform. All of the platform-specific classes have the word "Wrap" appended to them. The Wrap classes contain implementations and interfaces specific to MSAA or ATK. These platform-specific classes inherit from cross-platform classes, where the most of the implementation is done. For example, nsAccessibleWrap inherits from nsAccessible. Every accessible object in the MSAA tree has an implementation dertived from nsAccessible, which exposes accessibility information through nsIAccessible, in a generic cross-platform manner.

This default implementation for nsIAccessible knows how to use nsAccessibleTreeWalker to walk Mozilla's content DOM and frame tree, exposing only the objects that are needed for accessibility. The nsAccessibleTreeWalker class knows what it needs to expose by asking each DOM node's primary frame (a Gecko formatting object) for an nsIAccessible, using the nsIFrame::GetAccessible() method. If nsAccessibleTreeWalker gets an nsIAccessible back, then the DOM node considered to be an accessible object. The nsIAccessible that is returned is either a new one, or reused from the accessibility cache, and the correct type of accessibility object to correctly expose that DOM node through the cross-platform nsIAccessible and MSAA-specific IAccessible interfaces.

Every accessibility object created must be cached, and must inherit from nsAccessibleWrap so that it supports a base implementation of nsIAccessible and IAccessible. Apart from that, it is free to override IAccessible or nsIAccessible methods. In this way each class is tailored to the specific abilities and properties of the HTML or XUL/UI objects it applies to, and can support both MSAA, ATK and hopefully any future accessibility API's we need to support. For example nsHTMLButtonAccessible overrides nsIAccessible::GetAccRole to expose ROLE_BUTTON for IAccessible::get_accRole which uses that.

Generating MSAA Events

5. Feedback

How can this document be improved? Was it useful? Questions? Contact aaronl@netscape.com