This post reminds devs that unless the threading model being used in a UIA app is carefully considered, there's a very real risk that your customers will encounter severe delays when using the app.
Apology up-front: When I uploaded this post to the blog site, the contained image did not get uploaded with the alt text that I'd set on it. The image is followed by a textual description of the image.
Introduction
I got a reminder recently of how important it is for app devs who are using the Windows UIA Client API or Windows UIA Provider API, to be aware of the threading model being used by their app. I was asked by some devs who'd built their own UIA provider, about unexpected delays being experienced when a screen reader was interacting with their app. As it happened, I didn't know the cause of the problem, but they figured it out. It related to the threading model being used by the app. So this was a helpful learning experience for me, and a great reminder to always keep the threading model in mind.
The following are a few thoughts on threading considerations when using the Windows UIA API.
The UIA Client app
Many assistive technology (AT) apps use the Windows UIA Client API, including the Narrator screen reader, Windows Magnifier, and the Windows On-Screen Keyboard.
Any UIA Client app can experience unexpected delays beneath calls into UIA, if the threading model is not considered. Some very important points are discussed at Understanding Threading Issues. For example, making sure CoInitializeEx() is called with COINIT_MULTITHREADED if required.
To give an example of where an inappropriate threading model can be problematic, I once worked on a single-threaded UIA client app which, called CoInitialize(NULL) on startup. It then called AddFocusChangedEventHandler(), (remember, that's being called on the app's UI thread,) to register for notifications as keyboard focus moved around the system. Importantly, in that call to AddFocusChangedEventHandler(), it passed in a request to have the UIA BoundingRectangle property of the element gaining keyboard focus cached with the event. This meant that when handling the event, the app could call get_CachedBoundingRectangle() and get the bounding rect without an additional cross-proc call being made back into the UIA provider. Having got the bounding rect of the element with keyboard focus, the app could react however it pleased, and all seemed well.
Well, yes, all did seem well until the call to get_CachedBoundingRectangle() was replaced with a call to get_CurrentBoundingRectangle(). For some reason, it seemed necessary in some scenario to call back into a provider to get the bounding rect of the element with keyboard focus after the event had been received. (Perhaps the provider hadn't updated the element's UIA BoundingRectangle property until after it raised the event saying the element had got keyboard focus, and so whatever bounding rect was cached with the event, was stale.) But once the call was made in the UIA client to go back to the provider, that was happening for any element getting keyboard focus, including an element in the UI of the UIA client app itself. With this change, long delays intermittently starting being hit beneath the call to get_CurrentBoundingRectangle().
So really, if the app is expected to be able to handle a UIA event from any UI, (including its own UI,) and expected to be able to call back into providers while inside event handlers, it was on thin ice going with a single-threaded approach. If the app wants to be reliable, it'd spin up a background thread, initialized with CoInitializeEx(NULL, COINIT_MULTITHREADED). Inside that thread, it'd CoCreateInstance() a CLSID_CUIAutomation8, and call AddFocusChangedEventHandler() and RemoveFocusChangedEventHandler() on that thread. Whatever UIA calls it needs to make in the UIA event handler would be made on that background thread. And if some related action ends up needing to be taken on the UI thread, a Win32 app might post a custom message to that thread, or a WinForms app might call BeginInvoke(), to make that happen. And all would really be well.
Back in the Windows 7 days, I created a set of UIA client samples at Developer Code Samples. My goal was to demonstrate use of some of the most common actions a UIA client app might take. I really worked to feel confident that there was no chance that the threading model made the sample less robust. (In fact a good chunk of the sample code relates to threading, and not really to the UIA client calls.) I think it's possible that since Windows 7, the threading constraints around UIA have been relaxed a little, and so perhaps not all the threading action in my samples is still necessary, but I don't know the exact details if that is the case. I believe that at least some of the action I took must still be accounted for. For example, as Understanding Threading Issues states, calls to RemoveAutomationEventHandler() must be made on the same thread as the matching AddAutomationEventHandler() was called on, and they must be called in a non-UI Multithreaded Apartment (MTA) thread. So if you start calling either of those in a UIA event handler, are you sure that you're accounting for those constraints? For me, I'd never call either AddAutomationEventHandler() or RemoveAutomationEventHandler() in a UIA event handler. (Sometimes it can be tempting to want to do that, in order to add property changed handlers to an element, whilst inside the event handler that's letting you know the element's got keyboard focus.)
The UIA Provider app
I've less experience with implementing the UIA Provider API than I have with the UIA Client API. This is because, like most other devs, I let the UI framework that I'm using implement the UIA Provider API for me. For example, whether I'm building UI using Win32, WinForms, WPF, UWP XAML, or web UI hosted in Edge, I use standard controls and widgets, and let the UI framework take what I've built and generate the appropriate UIA representation for me. (Of course, I still take UI framework-specific steps which will enhance that default UIA representation if my customers need me to.) But there are still some very important scenarios where devs choose to implement the UIA Provider API themselves.
So I was contacted recently by the devs experiencing delays when a screen reader interacted with their custom UIA provider. One point in the description of the issue seemed particularly interesting; the delays went away when they changed their implementation of IRawElementProviderSimple::ProviderOptions() to not return ProviderOptions_UseComThreading. However, without that flag, the app started experiencing crashes. According to ProviderOptions, the ProviderOptions_UseComThreading option affects what thread the provider's methods are called on, and that's affected by the type of apartment in which the threads are running. For threads running in a Single-Threaded Apartment (STA), COM will synchronize the calls into the provider, to protect against multiple methods in the provider running concurrently when the provider's not designed to support that. But this can mean that UIA will be blocked from calling into your provider while the provider's busy doing work elsewhere on the same thread. Perhaps this was related to the unexpected delays, and it would be necessary to examine exactly what the thread of interest was doing while UIA was waiting to call the provider.
As it happens, the devs came back to me really quick, saying that they'd replaced the call that they were making to CoInitialize(NULL) with a call to CoInitializeEx(NULL, COINIT_MULTITHREADED), and the delays had gone away. In their case the delays that they'd encountered related to the time it took the UIA events that they raised to propagate through UIA to the UIA client app, rather than delays in their methods being called by UIA. But the threading model's going to affect a ton of what goes on inside UIA.
So this really was a learning experience for me. I'd seen the use of CoInitialize() in a UIA client app lead to delays, but I now know that that's one of the first things to be considered in a UIA provider too. With the change to initialize the provider such that it's running in a thread in an MTA, and by having its IRawElementProviderSimple::ProviderOptions() return ProviderOptions_UseComThreading, COM will not block the UIA calls into the provider's methods.
Important: The above approach does mean that the provider had better be thread-safe. If COM is going to happily allow concurrent UIA calls into the provider's various methods, then the provider needs to be designed to handle that. In this scenario, it's the provider's responsibility to implement whatever thread synchronization is required to account for concurrent access.
Summary
The apartments in which the threads are running in a UIA client or UIA provider app are rarely at the forefront of a dev's mind when building the app. Rather, thought is typically more focused on all the ways in which you can use UIA to help your customers. But making sure the appropriate threading model is used in your app is critically important to delivering a robust experience.
A long time ago I wondered how I could encourage devs to make sure that their threading model was included in the list of things they considered when designing their apps. I'd seen severe delays being investigated just before products were about to ship, and it was a serious blow for a dev to learn that the only way that they could make the app reliable was to change the threading model. No-one should be doing that just before shipping. So I created the cartoon below in the hope that it might grab devs' attention and prevent at least some devs from discovering the problem just before shipping. Like so much of what I've done, it seemed like a good idea at the time.
It's almost midnight… DO YOU KNOW WHERE YOUR THREADS ARE?
Guy
The above cartoon contains eleven frames, and is titled "Proxy & Stub Present... Live at the Inproc". At the top of the cartoon, above the contained frames, the character called "Proxy" is standing behind a microphone, saying "A funny thing happened on the way to the apartment", and seems to be wearing historical attire. The character called "Stub" looks on, saying "Nice threads!".
Attached to one of the frames at the top of the cartoon are the lollipop symbols used in technical diagrams to represent interfaces supported by a component. One of the symbols is labelled "IGBarker".
The images in the first ten frames are identical, with Proxy and Stub not looking at the reader, and having the following conversation.
Proxy: We're going to talk about a couple of COM threading models that you can use when developing client applications or server objects.
Stub: Threading Model? What's that?
Proxy: You can think of a threading model as a way for you to control how your client and server objects will work with each other when you have more than one thread running.
Stub: That may be fascinating, but why do I care?
Proxy: Because it may be easy to create an application with lots of threads, but if you don't understand how your objects work across those thread boundaries, you're jeopardizing stability and performance.
Stub: So if I don't design it right up front, I'll regret it later?
Proxy: Right! The first thing to know is that a thread runs in an "apartment". How an object in one thread works with objects in other threads depends which apartments those other threads are in.
Stub: Are there different types of apartments?
Proxy: Yes, you can create your thread in a "Single Threaded Apartment" (STA), and you can have any number of STAs. Or you can create your thread in a "Multithreaded Apartment" (MTA).
Stub: ...in which case it's not an empty MTA eh?
Proxy: Well said. There's at most one MTA in a process, and any object created in the MTA can be accessed by all threads running in that MTA at the same time.
Stub: So those objects better provide synchronization to make them thread-safe?
Proxy: Right! If the object's not thread-safe, then it must run in an STA where only one thread can run. This means that COM will synchronize all calls to the object from all other threads automatically.
Frame notes: Musical notes appear in the background.
Stub: STA, All my troubles seem so far away...
Proxy: Not quite. It takes time for COM to do all this work, and you still have extra work if you're going to pass interface pointers around from one STA to another.
Stub: So I'm better off with MTA?
Proxy: Performance will probably be better with MTA, because you know exactly how and where your objects need to be synchronized.
Stub: HASTA LA VISTA STA! I'm going MTA!
Proxy: The point is you should understand what threads your objects use and which apartments they live in. This will improve stability and performance...
Stub: ...and you'll know what to expect when external components work with yours.
Frame notes: The background of the frame darkens, and both Proxy and Stub look at the reader. In unison they say, "It's almost midnight... DO YOU KNOW WHERE YOUR THREADS ARE?"