Microsoft Windows UI Automation Blog

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 4: Postscript

Guy Barker MSFT — Fri, 12 Apr 2013 03:19:00 GMT

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 4: Postscript
Guy Barker

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 1: Introduction
Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 2: The XAML app
Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 3: The WinJS app

I recently had some fun exploring how I can customize the accessibility of a very simple Windows 8 app. I first built this app with XAML and then with WinJS, and it does the bare minimum required for me to demonstrate how to do the following four specific things.

Assign localized accessible names to elements.
Associate localized strings with values along a slider.
Make a screen reader aware that some text has just appeared and should to be spoken.
Present images appropriate to the customer’s current high contrast theme.

Having done that, I thought it might be interesting to upload the apps to the Windows Store. By doing that, people can download them to their Windows 8 device and get a feel for the user experience provided as a result of the actions taken in parts 2 and 3 of this blog. And importantly, I could get feedback from people to let me know where I’ve missed something with the accessibility of the app’s current UI, or where the app’s UI could be expanded to demonstrate some other topic which can be problematic with shipping apps.

So off I went, submitting the apps to the Store, checking the Accessibility box along the way...

This app has been tested to meet accessibility guidelines, and should be shown to people who are specifically looking for apps that meet those guidelines. Learn more

Not long after that, I got told that one of the apps had failed the content compliance, and so I canceled the submission of the other app and looked into the problem.

The first issue was that I’d broken the rule around not enhancing screenshots unless I’ve made it really obvious that I’ve done that. As the rule says, “Graphically enhanced screenshots are not allowed, unless they are clearly marked as having been enhanced.” I thought it’d be fun to stick a big screen reader speech bubble over the screenshot to show what the screen reader’s saying. Turns out that’s not such a good idea, but it’s something that’d be easy for me to fix.

The second issue was the more interesting one, and related to the requirement “Your app must offer customers unique, creative value or utility”. In my case, the apps are only a resource relating to a developer blog, and not useful at all as end-user apps. So it’s fair enough that I got told that the value of the apps wasn’t clear.

But it was fun submitting them anyway, and if I do get end up expanding them based on feedback – who knows, one day perhaps they will be useful as apps in their own right.

Given that apps aren’t available at the Store, I thought I’d make a short video of some of the accessibility-related functionality that the apps support. And as I did this, I also mentioned a few other aspects of the apps’ accessibility which I’d not mentioned in the other parts of this blog.

So here’s the video. Please excuse the poor quality of the filming and the dialogue. But really, when I do this sort of thing at the crack of dawn, I’m less concerned about the quality of the video, and more concerned about whether the kettle’s boiled.

http://www.youtube.com/watch?v=xzJf3_Kx-JY&feature=share&list=UUy69QyrZ8wApDLD8l5Zp_IQ

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 3: The WinJS app

Guy Barker MSFT — Tue, 02 Apr 2013 23:57:00 GMT

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 3: The WinJS app
Guy Barker

Say I’m fairly familiar with HTML, css and javascript, so it’s quick for me to build my new app in WinJS. I’ve found that I can use the app fine when only using the keyboard, and by pointing the Inspect SDK tool at the UI I can see all sorts of accessibility-related properties exposed by the app. I’ve also found that there are four places in my app’s accessibility story where there are gaps which I need to fill in. The blog describes what I need to do in order to deliver an app that’s fully accessible to all the customers I want to reach.

Accessible names

As I point the Inspect SDK tool to my UI, I can see an accessible name for the button and for the various text elements, but not for the slider or the image. This means as a customer uses the Narrator screen reader to examine the slider, they won’t hear a useful name describing what the slider relates to. (This will be case regardless of whether they’re using the physical keyboard or touch to interact with my app.) If the meaning of my slider is not made clear, then this significantly impacts the usefulness of my app.

The image below shows no accessible name being exposed by the slider UI.

So I need at add a helpful accessible name to the slider. I could do this by simply adding an “aria-label” property to the slider tag that defines my slider. An aria-label is a way to specify an accessible name for some element, and is one of a set of properties you can add to UI, as listed at http://msdn.microsoft.com/en-us/library/windows/apps/hh767301.aspx.

However, I must assign a localizable aria-label to the UI in order for the slider to have a useful name in all the geographical regions in which the app ships. So given that I’d not yet taken the steps to add localizable strings to my app, I’ll do that now.

Add a new “Resources File (.resjson)” item to my project.
Create a “strings\en-us” folder in my project and move the new resjson file into there.
Add an WinJS.Application.onloaded() function in default.js which gets called when the page is loaded.
Near the top of the onloaded() function, add a call to WinJS.Resources.ProcessAll().

Now that localizable strings are available in my app, I’ll insert the following in my slider’s markup.

data-win-res="{attributes: {'aria-label' : 'appSliderName'}}"

And I’ll then add the localizable string to my new string resource file.

"appSliderName" : "Risk level"

So having done that, when I run the app and point the Inspect tool at the UI, I can now see the localizable friendly name.

This means that when a customer uses a screen reader to access the slider, they’ll hear “Risk level” and immediately know what the slider relates to.

Now, a customer using a screen reader can examine UI regardless of whether the UI can get keyboard focus. For example, they may use touch to learn what UI is beneath their finger. So it’s important that all UI that’s not purely decorative has an accessible name. As such, I want the image that fills up most of the UI to have a useful, localized name.

This is where things get interesting. When I originally added the image to the app, I decided to make the image a background-image property on a div. When I did this this, I was interested in some of the css styling that’s possible with background images. Having built the app, I expect I didn’t need to have used a background image and I could have used a regular Img tag. But I’m going to stick with a background image here anyway, as it introduces a few interesting topics on accessibility.

By the way, when building your UI in HTML, don’t forget to add explicit closing tags to elements where you can, rather than doing “/>” to close it. Without the explicit closing tag on some elements, your layout won’t following the grid rules you try to set up in css. I forgot this when building the app, and so got some unexpected layout for a while.

So anyway, I added the image resource to my app, added the div in HTML and the related styling in css.

    #appImage {
        background-image:url(/images/Herbi.png);
        background-size: contain;
        background-position: center;
        background-repeat: no-repeat;
…
    }

And sure enough, when I ran the app, the image appeared. However, when I point the Inspect tool at the image, it couldn’t see the image at all. The image was nowhere to be found in the UIA tree of elements. This means a screen reader won’t be able to find it either. The image below shows the slider, the button and the text controls in the UIA element hierarchy, but not the image.

So this is a critical problem from an accessibility perspective, and I need to take action to get the image added to the UIA tree. The problem is triggered by the fact that I used a div to create the UI. By default a div is not an accessible element, and isn’t exposed to a screen reader user. Other elements, such as a Button or Input are added to the UIA tree. A description of how HTML maps by default to UIA can be found at http://msdn.microsoft.com/en-us/library/windows/apps/hh452708.aspx.

I can make a div element appear in the UIA tree by giving it some accessible role. It’s most appropriate in this case to give it a role of “img”, because it’s displaying an image. Once I’ve done that, I can point Inspect to the image, and I now see the element appear the UIA tree.

However, I can now see that the element has no accessible name, so I need to fix that next. This is easy to do in the same way I added an accessible name to the slider earlier. I’ll add a localizable string to my string resources, and update my HTML to have the elements aria-label property set to that accessible name.

data-win-res="{attributes: {'aria-label' : 'appImageName'}}">

"appImageName" : "Herbi",

Note that I don’t make the name something like “Picture of Herbi”, because including “picture” or “image” is redundant. When I gave the element a role of “img”, its control type was exposed as an Image control type, so screen readers can already inform my customers that the element is an image. The Name property should be constrained to making it clear exactly what it’s an image of.

One additional note on the onLoaded() function that I added earlier; this is a good moment to stop and think about any other action that might be useful when the app starts. For example, where’s the most useful place for keyboard focus to be? For my customers who only use the keyboard, if it’s likely that the button is going to be the element that they’ll want to interact with first, then I’ll set focus to the button in the following way.

appAskButton.focus();

Descriptive Slider values

By default, my slider supports the UIA RangeValue pattern. This makes sense, as is it has a minimum, a maximum and a current value, and all this is reported by the Inspect SDK tool.

When the Narrator screen reader examines slider UI that supports the RangeValue pattern, but which doesn’t support the Selection pattern or Value pattern, it reports the current value as a percentage of the full range. For my app, I really need friendly text to be associated with each value, to describe what each value means along this “Risk level” slider. I first wondered whether I could achieve this by adding support for the Selection pattern to the slider, by giving it some specific role, but my attempts there didn’t work given that I could find a way to create the element that was the associated selected item. But it turns out that in WinJS, there’s a far simpler way of achieving what I need.

The first thing I should do is add a changeValue event handler for my slider. I’ll add this in the onLoaded() function that I added earlier.

document.getElementById("appSlider").addEventListener('change', changeValue, false);

Then in this event handler I can set the aria-valuetext property of the slider based on its current value.

    var risklevelIds = new Array(
        "riskLevelLowId",
        "riskLevelMediumId",
        "riskLevelHighId",
        "riskLevelTopId"
    );

    // Set the localized slider value text based on the supplied value of the slider.
    function changeSliderValue() {
        var riskLevel = appSlider.value;
        var riskLevelText = WinJS.Resources.getString(risklevelIds[riskLevel]).value;
        appSlider.setAttribute("aria-valuetext", riskLevelText);

}

The aria-valuetext is described up at http://msdn.microsoft.com/en-us/library/windows/apps/Hh968009.aspx.

However, doing that alone did not get the value text spoken by Narrator as I’d hoped. I then noticed that the description of the related aria-valuenow at http://msdn.microsoft.com/en-us/library/windows/apps/hh465728.aspx, says that that property “applies to elements with an ARIA role of progressbar, slider, or spinbutton.” So I added the following to my slider HTML.

role="slider"

When I then pointed the Inspect SDK tool at the UI, I could see that UIA Value pattern support had been added to the slider, and the Value property was set to my friendly text. And sure enough when I use my app with Narrator, as I changed the slider value, Narrator would speak the helpful text associated with each slider value.

By the way, I also explicitly forced the aria-valuetext setting action on startup to, (before the slider value had been changed by my customer), I order to make sure the first time Narrator saw the slider, it had its friendly text set.

Announcing visual updates

And next we move to the part of the UI which is its raison d'être – the advice. When my customer invokes the app button, I present the result on the screen. I have an “h2” tag for the element which presents the text, and I set the innerText property on that when I want to show the text. If I point the Inspect SDK tool to the text after I’ve done that, I can see the result exposed in the UIA tree, and so it’s accessible to a screen reader.

The problem is that when I presented the text visually, nothing happened to let a screen reader know that the text was appearing. If I don’t give a screen reader a heads-up that text has appeared, it can’t know to speak the text, and so my customer misses out on the most important information that the app presents. So I must make sure that a screen reader is notified of the text appearing. I can do this my marking the element that shows the text as being a “Live Region”, as described at http://msdn.microsoft.com/en-us/library/windows/apps/hh465711.aspx.

By making the element a Live Region, I can have text-related changes lead to a UIA LiveRegionChanged event being raised. In response to that event, a screen reader can go back to the source element and speak whatever details about the element that it wants to. So I’ll add the following attribute to the “h2” tag.

aria-live="assertive"

And by declaring it as being assertive, I’m telling screen readers that I want changes spoken as soon as the change in text happens, even if that means the screen reader interrupting itself if it was already in the middle of speaking. If I’d prefer a screen reader to finish what it was currently speaking before announcing the change in my text, I would make the Live Region “polite”.

By the way, some roles for elements automatically lead to the element being declared to be a Live Region. As described at http://msdn.microsoft.com/en-us/library/windows/apps/hh452704.aspx, if you give an element the role of “log”, “alert” or “status”, it will become a Live Region. One of these three roles leads to the element being an “assertive” Live Region and the other two are “polite”. Which would you guess is the assertive one?

So I could have given the element one of the roles, but instead I left it as an “h2” with “aria-live” explicitly applied. This results in Inspect showing a Text control which is an assertive live region, and whose accessible Name property is the text shown visually for the element. (Interestingly, I also see that the Text control contains another Text control which has the same accessible Name property. I don't know why this hierarchy exists, but having two elements here does not affect what I’ll do when it comes to building my app, so I'll not worry about it.)

Having made my Text control a Live Region, I thought I'd run the AccEvent SDK tool to verify that the LiveRegionChanged events are getting raised as expected. I then noticed something really interesting, in that I'm getting two events raised with each click.

That seemed strange, given that I thought I only set the text once. But then I noticed that before I set the text, I explicitly set the Text control's text to be "", (an empty string,) before I set the actual results. I did this to force the text to change. If the text I display happened to be the same text that I previously displayed, then the displayed text wouldn't change, and I don't think I'd get the LiveRegionChanged event raised. So I set the text on the element twice in rapid succession. I'll bet what's happening is that when AccEvent sees the first event, it goes back to my app to get the new text shown in the UI. But rather than finding the empty string, it find the results text which was set on the UI since the first event was raised. Shortly after that AccEvent will react to the second event and get the same text, (which explains the results shown in AccEvent above). None of this will affect my customer's experience, as the effects of the second event will immediately stomp on the effects of the first, because the element is an assertive Live Region. So I'll leave all this as it is.

As I poke around my app with Inspect, I do notice one other thing that I’d like to tidy up. Before I invoke the button, the advice text element appears in the tree, but with no accessible Name property. This isn’t too surprising really, as I’ve yet to set any text on the element. So perhaps I should keep the element out of the tree whenever there’s no text set on it. I can do that through the use of the aria-hidden property, (http://msdn.microsoft.com/en-us/library/windows/apps/hh465705.aspx). I’ll add the following to my HTML.

aria-hidden="true"

Then when I set the text on the element later, I’ll change the state of the aria-hidden property.

appResult.setAttribute("aria-hidden", "false");

Now when I point Inspect to my app, the element doesn’t appear in the UIA tree until it’s presenting some text.

Presenting appropriate high contrast images

When my customers have specifically chosen a high contrast color theme to be used when my app presents its visuals, I want all visuals to be appropriately displayed. WinJS will do a great job at automatically changing the visuals for the text, button and slider in my app, but if I present a whacking great image in colours that contradict my customers’ wishes, then that’s hardly a good user experience. As it happens, WinJS makes it a piece of cake for me to have appropriate images shown.

My first step is to generate versions of my image which are black-on-white and white-on-black. The black-on-white version is to be shown when the “High Contrast White” theme is active, and the white-on-black image is to be shown when one of the light-on-black themes are active, (these being “High Contrast Black”, “High Contrast #1” or “High Contrast #2”). Having made my images, (doing my best with the Paint program), I add my images as assets in my app.

But here’s the key thing; I name the assets such that WinJS will know they’re appropriate for use when specific high contrast themes are active. Given that my original image was called Herbi.png, I give my high contrast image assets the following names.

Herbi.contrast-black.png
Herbi.contrast-white.png

My css for the original image-related element the following:

background-image:url(/images/Herbi.png);

WinJS will automatically show the appropriate one of Herbi.png, Herbi.contrast-black.png or Herbi.contrast-white.png depending on whether a high contrast theme is active. Pretty handy!

There is one very important step I do need to take. When I was building the app, I’d deliberately decided to present the image as a background image, given that at the time I thought I might be leveraging some of the scaling and cropping functionality that’s possible through css for background images. But WinJS considers background images to be potentially problematic for people who use high contrast themes. If an image is in “the background”, that suggests that there may well be something in “the foreground”. The point of high contrast themes is to show a high contrast between the foreground and the background. If text lies on top of some image, then this could really interfere with the goals of using a high contrast theme. So WinJS reasonably decides that by default, background images should not be shown when a high contrast theme is active.

In my cases however, I’ve chosen to present an image as a background image when it’s not really a background image at all. And I would like some image to be shown regardless of whether a high contrast theme is active. So I can override the default WinJS action of hiding background images, by setting the following css.

    @media screen and (-ms-high-contrast)
    {
      #appImage {
        -ms-high-contrast-adjust: none;
      }
    }

This means that an image will be shown, and WinJS will select the appropriate one based on which high contrast theme is active.

I should point out that the appropriate image will be selected based on which high contrast theme, (if any,) was active when my app starts. This will normally be sufficient for my customers. But if my app showed some small UI where I felt that a customer might need to switch to use a high contrast theme while performing a particular task, and switch out of high contrast once that task is complete, then I might choose to have my images react to changes in the state of high contrast while the app is running. I can achieve this will the css below.

    @media screen and (-ms-high-contrast)
    {
      #appImage {
        -ms-high-contrast-adjust: none;
        background: url(/images/herbi.contrast-black.png);
        background-size: contain;
        background-position: center;
        background-repeat: no-repeat;
      }
    }

    @media screen and (-ms-high-contrast:black-on-white)
    {
      #appImage {
        background: url(/images/herbi.contrast-white.png);
        background-size: contain;
        background-position: center;
        background-repeat: no-repeat;
      }
    }

In fact, having written that css here, I might as well stick it in my app, and support changing the image as the state of high contrast changes while my app’s running, as shown below.

Summary

So by default, my app was fully keyboard accessible. This means that a customer who has mobility impairments, (including those with severe impairments and who use a foot switch or head switch), would be able to use an on-screen keyboard to leverage the features in my app. Also by default, my app exposed a lot of accessibility-related information and visual behaviors which could be leveraged by my customers who have visual impairments. By taking the additional steps described above, I’ve completed the accessibility story for those customers, such that whether they interact with my app using a high contrast theme or through touch with a screen reader, all of my customers can access the full functionality of my app.

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 1: Introduction

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 2: The XAML app

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 4: Postscript

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 2: The XAML app

Guy Barker MSFT — Tue, 02 Apr 2013 22:12:00 GMT

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 2: The XAML app
Guy Barker

So I’ve built my cool new Windows 8 in XAML. I’ve checked out its accessibility, and found there’s a lot I’ve got by default. I’ve also decided there are four specific things I need to do in order to deliver the fully accessible app my customers expect. This blog describes the steps I took around enhancing my XAML app’s accessibility.

Accessible names

If I point the Inspect SDK tool to my slider, it tells me that the slider has no accessible Name property, (as shown in the image below). That’s because there's no label for the control that XAML can use to set the accessible Name from. So I need to give the slider an accessible Name, such that when a screen reader user tabs to the control, they’re told what the meaning of the slider is.

I could set the Name in the slider’s XAML, by setting the AutomationProperties.Name property. This is really easy to do, and would seem to fix the issue. However, I mustn’t hard code English strings in the XAML, as what good is English to a screen reader user who doesn’t speak English? I want this app to be fully accessible in every market I ship.

So instead, I’m going to give the slider a localizable accessible Name property. First I have to add the resource file for localizable strings. (I’d have to do this anyway, to localize the strings shown visually in the app.) There are lots of details up at MSDN around localizable resources, and it’s pretty much a case of doing the following:

Right click on your project, do Add->New Item, Resources file (.resw)
Move the file into a Strings\en-us folder, (for US English strings).
Add strings to the file, matching ids of elements to the strings associated with them.

So in my case, I’m going to give the slider an id, like this:

x:Uid="sliderRiskLevelId"

I’m then going to update the resources file to match the AutomationProperties.Name for that element with a localizable string:

sliderRiskLevelId.[using:Windows.UI.Xaml.Automation]AutomationProperties.Name Risk level

When I next point Inspect to the slider, I see the required Name property.

I can then take the same steps for the image element in the app, so that when a screen reader examines the image, a relevant localized name can be presented to my customers.

Note, as someone once said, “What’s in a name?” It can be tempting to think that if your XAML element has an “x:Name” set, then that will become the accessible Name property, as exposed through UIA. But that’s not the case. The “x:Name” property is exposed as the AutomationId property. The AutomationId is intended to be a unique identifier for that element relative to whatever else is shown in your app. The AutomationId is not localized, so a screen reader or your automation tests can use it to access the element regardless of the language of the app.

Also, it's worth a comment here on how useful Inspect is when it comes to helping you become aware of things in your UI element hierarchy which could be a problem for your customers. When I ran Inspect, I noticed that the UIA tree of elements for my app included a text-related element with no name. This element was associated with the TextBlock which presents the advice that gets shown after the Ask button is invoked, but initially no text is shown in the TextBlock. As such, it's pretty confusing for it to be exposed in the UIA tree. To fix this, I initialized the TextBlock to not be shown, with the following XAML:

Visibility="Collapsed"

That removes the element from the UIA tree. When I later set the text on the TextBlock, I made it visible and so it then appeared in the UIA tree.

adviceTextBlock.Visibility = Windows.UI.Xaml.Visibility.Visible;

Descriptive Slider values

While the AutomationProperties class can make it really easy to easily control some aspects of the app’s accessibility, sometimes you need the deeper customization available through a custom AutomationPeer. AutomationPeers are what make XAML elements accessible, and by introducing your own custom AutomationPeer, you might do a little work to customize a single accessible property, or take more involved steps to add specific behavioral support to a control. This MSDN article on custom automation peers, http://msdn.microsoft.com/en-us/library/windows/apps/xaml/jj219211.aspx gives loads of useful details on what they can be used for. (Some of what follows here will only make sense if you’ve read that article.)

After I first built my XAML app containing the slider, I ran Narrator and used the keyboard to change the slider position. Narrator notified me of new the slider values, but that took the form of speaking the percentage values “0”, “33”, “67” and “100”. The purpose of my slider is to set a risk level, and numeric values don’t express much meaning, other than perhaps assuming that a higher number is riskier than a lower number. What’s more, why don’t I hear the absolute values “0”, “1”, “2” and “3”? To answer this, we should point the Inspect tool at the slider again, and look closely at what UIA patterns are supported by the slider. (UIA patterns relate to the behaviors of elements.) Of interest to me here is that Inspect tells me that the RangeValue pattern is supported, but the Value pattern and Selection pattern are not, (as shown in the image below).

When Narrator examines an element, it might look for specific pattern support based on the element’s UIA control type. (Inspect shows me that the slider has a control type id of UIA_SliderControlTypeId.) In the case of a slider, Narrator will first check for support of the Selection pattern, then the Value pattern and finally the RangeValue pattern. If only the RangeValue pattern is supported, it will access the current value and speak this as a percentage of the full slider range. This is the behavior I get by default in my app. If I wanted to override this to have absolute values spoken by Narrator, I would need to add Value pattern support to the slider. And if I want to have Narrator speak some descriptive text associated with each slider value, I need to add Selection pattern support. So I’ll add Selection pattern support next. (I would add Value pattern support in a similar way, if I’d needed that.)

So here I go, adding Selection pattern support to my slider.

The first thing I’ll do is create my own class, through which my custom AutomationPeer can be accessed.

    public class CustomSlider : Slider
    {
        protected override AutomationPeer OnCreateAutomationPeer()
        {
            return new CustomSliderAutomationPeer(this);
        }
    }

I’ll then reference my custom class in the XAML, by replacing the use of the “Slider” element with “local:CustomSlider”. This means that when Narrator asks for accessible data associated with the element, that data will be provided by a CustomSliderAutomationPeer class that I now need to create.

Given that most of the accessible data that I want exposed by the element will be the same as that exposed by a regular slider, I want my custom AutomationPeer to be derived from the AutomationPeer class that provides the accessible data for a regular slider.

    public class CustomSliderAutomationPeer : SliderAutomationPeer
    {
        public CustomSliderAutomationPeer(CustomSlider owner)
            : base(owner)
        {
        }
    }

Next I’ll override the GetPatternCore() method, to say that the element supports the Selection pattern in addition to the RangeValue pattern that it supports by default.

        protected override object GetPatternCore(PatternInterface patternInterface)
        {
            if (patternInterface == PatternInterface.RangeValue)
            {
                return this;
            }
            else if (patternInterface == PatternInterface.Selection)
            {
                return this;
            }

return null;
}

Now, having declared that my custom AutomationPeer class supports the Selection pattern, I’d better add that support. To add support, I’ll need to implement all the members of the ISelectionProvider interface, (as described at http://msdn.microsoft.com/en-us/library/windows/apps/windows.ui.xaml.automation.provider.iselectionprovider.aspx). The IsSelectionRequired and CanSelectMultiple properties are easy, but the GetSelection() method’s going to take a bit more thought.

    public class CustomSliderAutomationPeer : SliderAutomationPeer, ISelectionProvider
    {
        public bool IsSelectionRequired
        {
            get
            {
                return true;
            }
        }

        public bool CanSelectMultiple
        {
            get
            {
                return false;
            }
        }

        public IRawElementProviderSimple[] GetSelection()
        {
…
        }
    }

The GetSelection() method returns an array of IRawElementProviderSimple objects, but in my case there’s only going to be one element in the array. This element represents the selection, and will only be used to provide an accessible name to be spoken by Narrator. The object must implement the ISelectionItemProvider interface if it’s to be the item returned by the slider as the element that is the current selection, so I first wondered if I needed to define some new class for the selection item. But then I took a look through http://msdn.microsoft.com/en-us/library/windows/apps/dd319586(v=vs.85).aspx to see what standard XAML controls support the SelectionItem pattern. The ListBoxItem seemed pretty tempting, so I decided to leverage that class as my selection item.

I created an instance of a ListBoxItem when my AutomationPeer is created, and cached the ListBoxItem for later use. I also cached my single-element array which I’d return from the calls to GetSelection(). In fact I now have all the objects I need to return a selection.

private ListBoxItem m_item;
private IRawElementProviderSimple[] m_collection;

        public CustomSliderAutomationPeer(CustomSlider owner)
            : base(owner)
        {
            m_item = new ListBoxItem();

AutomationPeer itemPeer = FromElement(m_item);

            m_collection = new IRawElementProviderSimple[1];
            m_collection[0] = ProviderFromPeer(itemPeer);
        }

Finally, I fill in the blanks in the GetSelection() method, by setting an appropriate AutomationProperties.Name property on the ListBoxItem I created, and returning that in the array representing the selection.

    public IRawElementProviderSimple[] GetSelection()
    {
        string[] sliderItemTextIds = {
            "riskLevelLowId",
            "riskLevelMediumId",
            "riskLevelHighId",
            "riskLevelTopId"
        };

        int idxSlider = (int)base.Value;
        var loader = new Windows.ApplicationModel.Resources.ResourceLoader();
        string sliderText = loader.GetString(sliderItemTextIds[idxSlider]);

m_item.SetValue(Windows.UI.Xaml.Automation.AutomationProperties.NameProperty, sliderText);

return m_collection;
}

The bulk of that function simply gets the localized text that I want to be spoken for the slider value.

By making the changes above, I’ve added Selection pattern support to my slider. When Narrator examines the slider and finds the Selection pattern supported, it retrieves an IRawElementProviderSimple associated with the selection. In my case that’s provided by the ListBoxItem whose accessible Name property I’ve set to be a string associated with the current slider value. Narrator speaks that Name as I change the slider value.

So this all works great for me. In theory another screen reader might examine other properties of the object that I return as the selection, and those properties might be rather unexpected from the user’s perspective. For example, why would a slider’s selection have a control type of ListBoxItem? If that ever turned out to be an issue, I’d need to look into changing that.

By the way, for completeness I thought it might be worth setting a ThumbToolTipValueConverter on my slider too. This would allow me to have the tooltips shown which match the text spoken by Narrator. So I created the necessary class, and referenced an instance of it on my slider.

sliderRiskLevel.ThumbToolTipValueConverter = new TooltipStringGenerator();

    public class TooltipStringGenerator : IValueConverter
    {
        public TooltipStringGenerator() {}

        public object Convert(object value, Type type, object parameter, string language)
        {
            string[] sliderItemTextIds = {
                "riskLevelLowId",
                "riskLevelMediumId",
                "riskLevelHighId",
                "riskLevelTopId"
            };

            double dv = (double)value;
            int idxSlider = (int)dv;
            var loader = new Windows.ApplicationModel.Resources.ResourceLoader();
            return loader.GetString(sliderItemTextIds[idxSlider]);
        }

        public object ConvertBack(object value, Type type, object parameter, string language)
        {
            throw new NotSupportedException();
        }
    }

It’s worth pointing out here that setting the tooltip strings alone isn’t sufficient to meet the meets of the screen reader experience. When the tooltip pops up with the friendly text, Narrator might speak that text, depending on what else is going on at the time. But as the slider value changes, Narrator won’t speak the updated tooltip. So the custom AutomationPeer is necessary to provide the experience I want my customers to have.

The image below shows the Narrator cursor on the slider, and the visual tooltip showing the same text as Narrator is speaking for the slider value.

Announcing visual updates

When I invoke the app button, the answer I’m looking for appears visually in the UI. But if Narrator doesn’t announce that text when it appears, then the app’s as good as useless. I need a way to tell a screen reader to pay attention to the text that appeared, even though it’s appearing at some point distant from where keyboard focus is. (In general, screen readers need to be careful not to announce all visual changes on the screen, because many of those changes would only be a distraction to the user.)

So I’ll explicitly declare the TextBlock that shows the answer to be what’s known as a “live region”, with a handy one-line change.

AutomationProperties.LiveSetting="Assertive"

(Here we see the same useful AutomationProperties class that we saw earlier when setting the accessible Name.)

There are two aspects of having some UI be a live region. The first is that when the contents of the element change, an event should be raised so that UIA clients such as screen readers are aware that’s something’s changed. This is done by raising a UIA LiveRegionChanged event. If I’d built some custom control I’d need to raise the event itself. But conveniently, if I've declared the TextBlock to be a live region, then XAML raises the event for me whenever the text in the element changes.

The second part of using live regions is declaring what type of live region the element is. When Narrator receives the event, it will examine the AutomationProperties.LiveSetting property of the element that raised the event. If the LiveSetting is assertive, Narrator will let the user know about the current state of the element immediately, (including the new text set on it). So this announcement will interrupt Narrator if it happened to be saying something else at the time of the event. If I’d given the element a LiveSetting of “polite”, then Narrator would finish what it was in the process of saying when the event was raised, before announcing the new state of the element to my customer.

To get confirmation that all was as I expected when the contents of the TextBlock in my app changed, I pointed the AccEvent SDK tool to the TextBlock. As the image shows below, I could see the event getting raised, and that the LiveSetting property on the element that raised the event was “Assertive”.

I can’t stress the importance of the AutomationProperties.LiveSetting enough. If your customers interact with some controls in your app, and some critical information is displayed elsewhere in the app as a result of the interaction, a screen reader will very likely need to announce that critical information at that time. Often the simple step of declaring a TextBlock to have an assertive LiveSetting is all you need to do.

Presenting appropriate high contrast images

When I first built my app, I created an Image element and slapped in an existing image that I had lying around. That was easy. But then I turned on one of the high contrast themes and ran my app, and realized I’m not quite done yet. I don’t want to display large chunks of a white background when my customer wants light text shown on a black blackground. In many cases, presenting colours that are inappropriate to the current theme will make things difficult or impossible to make out, and in some cases it can be physically painful.

I could choose to show no images at all when a high contrast theme is active. But in the case of my images I think it would generally be preferable to present high contrast versions of them rather than not showing any image at all. So I set to work in Paint, creating black-on-white and white-on-black versions. The results aren’t too inspiring, but they illustrate the point I’m making here.

There were three steps to leveraging my new high contrast masterpieces.

1. Add the new png files as asserts to my app project.

2. Change the source property in my Image element’s declaration to use an image loaded as a resource.

Source="{StaticResource imageHerbi}"

3. Update the existing ResourceDictionary in the project’s StandardStyles.xaml, such that an image appropriate to the active high contrast theme will get loaded.

    <ResourceDictionary.ThemeDictionaries>
        <ResourceDictionary x:Key="Default">
…
            <x:String x:Key="imageHerbi">Assets/Herbi.png</x:String>
        </ResourceDictionary>

        <ResourceDictionary x:Key="HighContrast">
…
            <x:String x:Key="imageHerbi">Assets/Herbi.contrast-black.png</x:String>
        </ResourceDictionary>

        <ResourceDictionary x:Key="HighContrastWhite">
           <x:String x:Key="imageHerbi">Assets/Herbi.contrast-white.png</x:String>
        </ResourceDictionary>
    </ResourceDictionary.ThemeDictionaries>

To test my high contrast change, I need to set the high contrast theme before running my app. I tend to pick a high contrast theme by right-clicking on the desktop, choosing Personalize, and then selecting the theme I’m interested in. I could instead just turn on high contrast from the Ease of Access settings, but that will only use the most recent high contrast theme used, (or the default). I always check my UI in High Contrast White and at least one of the light-on-dark themes.

By the way, in Visual Studio the XAML Design view reported the following exception when I added the resource string:

Exception: An object of the type "System.String" cannot be applied to a property that expects the type "Windows.UI.Xaml.Media.ImageSource".

I ignored that exception given that everything seemed to work fine when I ran the app.

The images below show the app running in two high contrast themes.

Summary

Ok, so having done the above, my app now exposes localized friendly names for my slider and image, and also provides meaningful strings for the slider values. It announces the results as they appear in the UI, and also presents colours which are appropriate for high contrast themes. It’s now good to be declared “Accessible” when I upload it to the Windows Store!

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 1: Introduction

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 3: The WinJS app

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 4: Postscript

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 1: Introduction

Guy Barker MSFT — Tue, 02 Apr 2013 21:42:00 GMT

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 1: Introduction
Guy Barker

Windows 8 apps built using XAML or WinJS, automatically support a certain level of accessibility by default. These two UI frameworks provide controls which are programmatically accessible, can be controlled through the keyboard and can present visuals appropriate for the current high contrast theme. In some cases however, the nature of the app’s UI means that the UI frameworks alone can’t automatically provide the fully accessible experience that your customers need. As such, there are steps that you as the app developer can take to elevate the default accessibility to the level required.

This blog explores some of the more common actions you might do to build that fully accessible experience. To illustrate the points, I’ve built a new app called “Ask Herbi”. (The app helps me with decision-making when I’m not in a decision-making mood.) I use a slider to set the level of risk that I find acceptable, and then invoke a button to have a response generated to help me with my course of action. I built two pretty much identical versions of the app using XAML and WinJS, both of which were based on the “blank” app templates available in VS Express 2012 for Windows 8.

The apps fully support keyboard accessibility by default. I can tab to, and control, the slider and button, and that’s really important from an accessibility perspective. If I point the Inspect SDK tool to the controls, I can see all sorts of accessibility-related properties being exposed by the apps through the UI Automation (UIA) API. And if I run the apps while a high contrast theme is active, the controls show colours appropriate to the high contrast theme. So the apps are well on the way to being accessible.

In the case of my particular apps, there are some gaps in their accessibility which I now need to address. These gaps are listed below.

Accessible names

The slider has no accessible name, and so a screen reader user doesn’t know what the slider refers to. For some controls, (such as a button), the UI frameworks can consider the text displayed with the control to be the control’s accessible name. But if the control has no associated text, (such as a slider or list), then the app developer needs to explicitly give the control an accessible name. I also need to do a similar thing with the image displayed by the app. Even though keyboard focus doesn’t move to the image, a screen reader user can explore the full UI in the app, and examine all the accessible properties of the image.

Descriptive Slider values

As I change the value of the slider, its value exposed through the UIA API is based on the numeric range. (In my case, that’s 0 to 3.) When the Narrator screen reader speaks the current value, it speaks the value as a percentage of the slider’s full range. For my customer’s I don’t want a numeric value spoken at all. Rather I’d like a helpful text string spoken as the value changes, explaining the meaning of the slider at its new value.

Announcing visual updates

When I tab to the button, Narrator will tell me what has focus, and so I know I’m ready to ask the app to generate the answer I’m after. So I invoke the button and the result appears visually on the screen. However, Narrator says nothing when the result appears. Knowing this result is the whole point of using the app, so if it’s not accessible, my app’s serving no purpose. Technically, I could use the screen reader to navigate through the UI that’s not keyboard focusable, but there’s no way I want to do that for such a critical piece of data. Instead, I simply want to invoke the button, and then hear the result spoken.

Presenting appropriate high contrast images

When a high contrast theme is active, I want to present an image that’s appropriate to the theme. Due to the way I present the image in my WinJS app, no image appears at all when a high contrast theme is active. In my XAML app, I see my regular image, which is really not helpful to my customers, as shown below.

Instead, I want an image presented that's appropriate to the active high contrast theme, such as that shown below.

So while the UI frameworks have given me a ton of accessibility-related functionality by default, I’ll now take the steps necessary to address the issues just listed. And as it happens, the UI frameworks make most of this pretty easy for me to do.

The following two parts of this blog detail the changes I made to my XAML app and WinJS app in order to make them fully accessible. For your app, there might be some different steps around accessibility that you’ll be interested in, but the steps discussed in this blog are some of the more common ones.

Having built these exploratory apps, I’ll probably upload them to the Windows Store next. I’ll check the box declaring them as accessible during the upload process. If I’ve missed something around accessibility while building the apps, hopefully I’ll get feedback from customers describing the issue, and I can update the apps accordingly. And if you’d like to have this blog updated to demonstrate something about accessibility that it’s not describing today, let me know. It’d be interesting for me to explore other aspects of accessibility with Windows 8 apps.

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 2: The XAML app

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 3: The WinJS app

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 4: Postscript

What can I do to make my Windows 8 WinJS.UI.ListView app as accessible as possible?

Guy Barker MSFT — Wed, 20 Feb 2013 15:02:00 GMT

I recently spent five years working in the Windows Accessibility team, working on most of the accessibility-related features in Windows. This included the UI Automation API, Narrator, Magnifier and the On-Screen Keyboard. A few months ago I started working in a Windows Apps team, and so was very interested in studying accessibility from the App developer’s perspective. In order to learn more about this, I examined a new app that I created from the HTML/Javascript Grid App Template. My goal was to understand exactly how WinJS.UI.ListView UI is accessible by default, and what additional steps a developer can do to make the UI even more accessible. At the end of this blog, I’ve listed a summary of my findings.

Programmatic Access

Many assistive technology products will want to access and interact with your UI. For example, a screen
reader will speak the details of your UI to users who have visual impairments, and magnifiers will follow keyboard focus around the screen. Your UI is made accessible through the UI Automation (UIA) API available in Windows. The WinJS framework builds up a hierarchical tree of elements representing the UI on the screen, and that tree is accessed by UIA client apps.

So having created my new app from the HTML Grid App template, I pointed the Inspect SDK tool to it, and examined the results reported by Inspect. These results are shown in Figure 1.

Figure 1: Inspect reporting the UIA tree for an app based on the Grid App template.

A number of things caught my attention when I did this, and I could see that there was already a lot of support for accessibility built into the WinJS.UI.ListView control. Great! But I was particularly interested in whether there was additional action I could take myself to further enhance the accessibility of the UI in some way.

The follow topics were some of the things I considered.

1. Should I try to name the unnamed “pane” element high up in the tree?

In general, the user really doesn’t want to encounter elements that don’t have accessible names. For example, a button without a name means that the user won’t know what’s going to happen if they click it. Similarly an app with multiple lists which don’t have names can make the experience very challenging for a screen reader user. (Sometimes container elements that are only used for layout purposes appear in the tree, and while it’s preferable for them to have names, it’s not such a problem when they don’t given that the user will usually not interact with them.)

It turns out that that the unnamed pane shown by Inspect is not a part of the UI defined in the HTML that describes the UI in the pages that got generated when I created my app. So this is not something I can easily control myself. As such I’ll leave the unnamed pane as it is, and when the user encounters it, they’ll ignore it and move on to the more interesting elements.

By the way, there are a bunch of other pane elements shown high up in the UIA tree, with names of “WinJS_NewAppFromGridAppTemplate_1”. These names all came from the name I supplied for the new app when I created it form the template. So your pane elements there will have whatever name you gave.

2. Should I set the accessible names of other elements that are defined in my HTML files?

The answer here is a resounding Yes! This is one of the most important things you should do as part of delivering the most accessible app possible. The template used to create the app has already populated many elements with names, such as “List of groups” for the list shown on the first page when I run the app. But you’ll want to change this to best describe your list. So for example, say I was presenting a list of houses for sale. The accessible name on the list is set through the use of the aria-label attribute in the groupedItems.html file, and so by replacing the default text of “List of groups” with “Houses for sale”, my customers who user screen readers can know what the list refers to.

However, to do that would only be useful to my customers who speak English. Given that I want a worldwide customer base, I want to present a localized accessible name for the list. I wouldn’t present English text visually to my Spanish speaking customers, so why would I present English text audibly? So, when setting the aria-label attribute here, I want to use a localized string. I can do this by replacing the default aria-label attribute with the following in the HTML, (where “stringGreeting” is the id of some localized string).

data-win-res="{attributes: {'aria-label' : 'stringGreeting'}}"

Of course, this means you need to update your app to include the localized string. I just did this with my new app with the steps below. (You’ll want to do this regardless of accessibility, to make it easy to localize you app for worldwide distribution.)

Add a new “Resources File (.resjson)” item to my project.
Create a “strings\en-us” folder in my project and move the new resjson file into there.
Near the top of the “ready” function in my page’s js file, add a call to WinJS.Resources.ProcessAll().
Edit the new file to contain the localized string I want to show. For example:
"stringGreeting" : "Welcome to Houses For Sale!"

Similarly I’ll want to replace all the other fixed strings that are set through aria-label in the VS solution, with localized strings. I can then point the Inspect SDK tool at my UI and verify that no elements defined on my pages have missing accessible names.

So with the above few steps, I have an app which is ready for worldwide distribution regardless of whether my customers will consume the text visually or through a screen reader.

But hold on – what’s that unnamed group element shown in the Inspect results? The results show an unnamed group element as the first and last child elements of another group with the name “Scrolling Container”. It turns out that all these group elements are created by WinJS when I use the ListView and their names are not under my control. As such, they’re not something I should spend time on changing.

3. The order of the list items and group headers in the UIA tree does not match their order shown on the screen.

Figure 2 shows the UIA tree for the last set of descendant elements beneath the list. It shows a number of list items, followed by elements representing the group headers shown in the list. So this raises the question, does this mean that when a user moves through the list via a screen reader which relies on the UIA tree, the user will be taken to all the items, and then to all the group headers? This seems less helpful than moving to a group header, then to all the items in the group (in order), then onto the next group header.

Figure 2: Descendants of the list element in the UIA tree.

This more preferable navigation order is supported, thanks to the WinJS.UI.ListView’s use of the UIA FlowsTo and FlowsFrom properties. These properties allow an element to specify which element logically precedes it and which follows it in the UI, when those elements aren’t adjacent in the UIA tree. So there’s nothing I need to do here to enable a useful navigation path through my items and group headers.

By the way, you may wonder why the Inspect SDK tool reports an ARIA role of “option” for the list items. (The ARIA specification includes a role of “listitem” after all.) This is because the ARIA “listitem” role doesn’t support a selected state, whereas the WinJS.UI.ListView makes it very easy to enable single- or multi-selection of items. The ARIA “option” role does support selection, so that’s the role the list gives to its items. The end result is that when UIA exposes data on this element, the element has an ARIA role and localized control type sting of “option”, and a UIA Control Type of UIA_ListItemControlType.

4. Do the elements in my UI have unique identifiers?

Sometimes assistive technology apps such as screen readers want a unique identifier to search for a particular element in your UI. Automated test frameworks certainly want to do this. So how do you give your element such an identifier? In many cases, if you set an id on your div, that id will be exposed through UIA as the UIA AutomationId property for the element. For example, if I add some id to the list in my new app, say id=”listHouses”, then I can see the AutomationId set by using the Inspect tool, as shown in Figure3.

Figure 3: Inspect reporting the AutomationId for the list in my new app.

Note that setting the id of divs contained within WinJS.UI.ListView items does not result in the UIA AutomationId being set. So if you want to set some id on the divs contained within the list items for automation test purposes, you’d want to set some custom attribute on the div, and have your automated test framework examine the DOM.

So that wraps up what struck me as I looked at the UIA tree associated with my new WinJS.UI.ListView app. A great UIA element hierarchy, (along with very helpful properties being set on those elements), is critical to allowing a screen reader to provide a fully functional and efficient experience to your customers. While there are a number of screen readers available for Windows, I then wanted to explore the experience provided by one in particular; the in-box screen reader – Narrator!

Narrator

As I’ve just said, having your app present a great representation of its UI through UIA is essential if a screen reader is to describe the UI to your customers. The Narrator screen reader only uses UIA to access your app’s UI, and so if some data’s not exposed through UIA, Narrator can’t present it. But verifying a great UIA representation alone doesn’t necessarily mean your customers are going to get the best screen reader experience they can. So in addition to verifying the UIA representation, it’s really helpful to your customers to consider the resulting screen reader experience.

For example, say your customer wants to navigate through the items in your list to find a particular item. As they move between items, they’ll not want to find that they have to listen to a lot of text that’s the same for a bunch of nearby items, before they hear the text that differentiates the new item from the previous. That’ll be a really slow and frustrating experience. Instead, they want to hear something immediately that lets them know whether they’ve found an item of interest. By default WinJS will build up an accessible name for a list item based on the text elements contained within the item. So take a look at that accessible name with the Inspect tool and consider whether the user’s going to hear the most valuable text almost immediately or not. And to really get an appreciation for the user experience, turn on Narrator, turn off your monitor, and arrow around your list. Remember that this is not only about building an app that’s technically accessible, but providing a fully functional and efficient experience.

Another important consideration is around images shown in the list items. In my new app, the list items include an “img” tag, and also set the alternate text for the images to some bound strings. So it’s great that the template reminds app devs to consider alternate text here. (As with the aria-label mentioned earlier, the alternate text that finally gets supplied to the user needs to be localized.) But say for some reason, you wanted to set the background-image attribute of some div to present an image, rather than using the img tag. (Sometimes it can be more straightforward to get a particular cropping and scaling that you want for an image though the styling of the background-image.) So you add your div, and set its background-image attribute accordingly through its css.

background-image: url(house.jpg);

If you then point the Inspect SDK tool to your new div, no element shows up in the UIA tree for the new div. As such Narrator is unaware of the image, and cannot make your customer aware of it. While it’s important not to distract the screen reader user with information which is not generally useful to them, I prefer to make the presence of the useful image discoverable to the user if they choose to explore the UI in detail. As such I want to make sure some UIA element associated with the image-related div exists in the UIA tree. This is straightforward by adding the following to the javascript associated with the UI:

myBackgroundImageDiv.setAttribute("role", "img");
myBackgroundImageDiv.setAttribute("aria-label", WinJS.Resources.getString("stringImageDescription").value);

Once I’ve added a localized image description, Inspect will then show that a UIA element exists for the image-related div, and it has the useful alternate text.

Controlling the list item

Programmatic accessibility isn’t only about exposing details of what you’re showing on the screen. It’s also essential that your customers can use a screen reader to control that UI too. For example, your customer needs to be able to invoke or select an item. This is enabled by default, thanks to the WinJS.UI.ListView supporting specific UIA patterns. (A UIA pattern is used to describe the behavior of UI.) Because the list item supports the Invoke pattern, the Narrator user can invoke it. And because the list item supports the SelectionItem pattern the Narrator user can select and deselect it.

Navigation

The Narrator user can use touch gestures or keyboard shortcuts to move to the next or previous UI element, so thanks to WinJS.UI.ListView’s use of UIA’s FlowsTo and FlowsFrom properties, the Narrator user can navigate between list items. By default the navigation will not include the individual elements within a list item, but the user can navigate through these contained elements if they choose to. They would do this with a keyboard by using the CapsLock+RightArrow shortcut. (Note that if the item contains an image div, then this could be included in the navigation by changing Narrator’s cursor movement mode setting to “Advanced” first.)

There’s some interesting background on Narrator’s touch gestures at Using your feedback to make Narrator work better with touch.

Using your app with the keyboard

Many of your customers will want to access the full functionality of your app through use of the keyboard alone. There are a whole host of reasons for wanting to do that, ranging from some users simply finding it most efficient to only use the keyboard, to users who have severe mobility impairments and interact with their device through a head switch or foot switch. (Your customers who use a switch device will interact with your app with an on-screen keyboard rather than a physical keyboard.)

The great point here is that the list items in the WinJS.UI.ListView supports keyboard access by default. Your customers can use the tab key to move focus into the list, and then use the arrow keys to move between items once the list has focus. Use of the Page Up/Down and Home/End keys allow for faster navigation through the list. If you want an item in the list to have focus when the app starts, be sure to set focus to the list when you create it. When a list item first gets added to the list, focus will move to that list item.

If you’ve modified the list from the HTML Grid App template such that it supports selection, a press of the Spacebar will select or deselect the focused list item. A new app created from the template also contains an invoke event handler which gets called when a list item has focus and you press the Enter key.

All of the above keyboard action can be taken either at a physical keyboard or an on-screen keyboard. Figure 4 shows the Windows OSK docked beneath my new app. The OSK is scanning the keys to allow the user to navigate through the list of items by only using a switch device.

Figure 4: Interacting with the app by using the OSK in scan mode.

Group headers

An important point that I discovered while working with my new app with the keyboard is that WinJS.UI.ListView doesn’t have built-in support for interacting with group headers via the keyboard. As I tabbed around in my app I found that keyboard focus didn’t go to the group headers, even though I could invoke the header with a mouse click. The reason why focus didn’t go to the headers is because the template used for the list has a “tab-index” property of -1. (This meant the group headers should not be included in the tab order.) So as a test I removed that, and sure enough keyboard focus navigation included the group headers. But having done that, if I moved focus to a group header and pressed the Enter key, the UI reacted as though focus was really on the most recently focused list item. So to avoid confusion, I added the “tab-index” of -1 back in.

But it’s essential that your customers who only use a keyboard can access all the functionality in your app. If a group header can be invoked with the mouse, you must allow it to be invoked through the keyboard. This requires the page’s javascript to take action to enable this, and it turns out my newly created app already has an example of just how this can be done. If I set focus to a particular list item, and then press Ctrl+Alt+g, this invokes the group header associated with the selected list item. I can see how this is done by looking at the keydown event handler that automatically got created in my app’s groupedItems.js file.

So the end result is that while the WinJS.UI.ListView doesn’t have built-in support for invoking the group headers with the keyboard, with the additional javascript that automatically gets created in your app, your customers can invoke the headers just fine. And that’s good news – because it’s critical that all your functionality can be leveraged using only the keyboard.

High Contrast

Your sighted customers need to be able to read all the text that you show, and that means there needs to be a useful contrast between the text and its background. (My keyboard at home shows the Function Key text in dark blue on a black background, and it’s really irritating spending time scrutinizing which key’s which.) So make sure that when using the default Windows theme, your text stands out clearly against the background.

Now some of you customers will have selected a high contrast theme to make it practical for them to read text on the screen, and so your text needs to respect those high contrast colours in order to be useable. So I selected a high contrast colour theme, and took a look at the results in my new app. (In order to experiment with different themes, I select these by right clicking on the classic desktop, selecting Personalize, and then picking one of the High Contrast themes. I always at least try out High Contrast Black and High Contrast White.) You don’t have to restart a WinJS app after changing the theme, as the app automatically gets updated to reflect the new system colours. Having changed the theme, I found that the items in my list got updated just fine. The background of the list items was the newly active window background colour and the text was the newly activate window text colour.

Given that everything worked just great, it’s tempting to leave high contrast now, and move onto the next topic. But this is where you can stop to consider if there's anything you can do beyond checking that the minimum expectations around high contrast have been met, and elevate the experience to something more useful in practice. For example, when I run my new app with the High Contrast Black theme active I find that the items have no borders showing, which means if my items only contained text, it would be a challenge for some of my customers with low vision to quickly determine where one item ends and the next begins. So once the default text shown in Figure 5 has been replaced with all your actual text, it might be less efficient to use your app than your customers would like.

Figure 5: High contrast list items with no clear delineator between items.

So you might consider changing the styling of the items when a high contrast theme is active, to make each item stand out more. For example, all it takes is to add the following styling to the new app’s css for the .item-overlay, in order to get the borders shown in Figure 6. And another great thing here is that the colour of the border will automatically be set to the colour appropriate for the currently active high contrast theme.

border-style: solid;
border-width: 4px;

Figure 6: High contrast list items with a clear border around each item.

Images

While the following isn’t specific to images presented in lists, given that many lists do show images, I thought it was worth mentioning it here.

Another important question you’ll want to ask yourself relates to background images. As you turn your app from the one created fresh from a template into the one that exactly meets the need of your business, you may have chosen to present some images through styling as background images, rather than with “img” tags. By default WinJS will hide background images set through styles when a high contrast theme is active. This default WinJS action is helpful, because if background images continued to appear behind text, that would interfere with the goal of making text clearer by maximizing the contrast between text and its background.

But there may be times when you feel that your image might be of use to some of your customers when a high contrast theme is active, and you still want it to appear. All you need to do then is to set the following css to override the default behavior, and to have the background image appear.

-ms-high-contrast-adjust: none;

If having done that, the image appears behind text when a high contrasts theme is active, you’ll want to move or size it as necessary to make sure it can’t appear behind text.

Also, your app may ship with some fixed images which aren’t easy for some of your customers who use high contrast themes to make out. So for any fixed image you ship, you’ll want to ship a white-on-black version and black-on-white version which can be presented when a particular high contrast theme is active. The following css shows how you can do this.

@media screen and (-ms-high-contrast)
{
.mypage #houseImage {
-ms-high-contrast-adjust: none;
background: url(/images/houseimage.contrast-black.png);
}
}

@media screen and (-ms-high-contrast:black-on-white)
{
.mypage #houseImage {
background: url(/images/houseimage. contrast-white.png);
}
}

In fact, given that above I’ve used the WinJS image file naming convention for high contrast, I don’t even need to specify the files in the css there. Rather I can leave it to WinJS to select the appropriate file based on the active high contrast theme, and I only retain the css to say that I don’t want background images to be hidden by default.

@media screen and (-ms-high-contrast)
{
.mypage #houseImage {
-ms-high-contrast-adjust: none;
}
}

Having said that, when I tried this, I found that the image displayed was appropriate only for the theme that was active when I started up my app. So if I changed the active theme I had to restart my app. If the file name was specifically included in the css, (as shown earlier), then the image presented would change in my running app as the active theme changed.

Conveying information through colour alone

Many of your customers will be colour blind, and so it’s really important that information isn’t conveyed only through use of colour. The classic example of this is a shape which indicates that all is well by being green, and all is not well by being exactly the same shape, but red. That’s not going to help your customers who find it a challenge to differentiate red and green.

So I took a look at my new app, and found that no information was conveyed through use of colour alone. As such, I didn’t need to add additional text, (either in the form of text that appears next to images or shapes, or as tooltips.)

Conveying information through audio alone

If your app presents some information through audio and no matching visuals change, then your deaf customers may miss out on something very important, such as an alarm. (Depending on how you output the audio, the system may help to get visuals shown.) My new app outputs no audio at all, so this isn’t a concern for me here.

Make everything bigger

Now this Windows Ease of Access setting is another great feature for many customers. On devices with a resolution that supports the setting, everything in your app can be made bigger on the screen. While this helps a lot of people with low vision, (and many people who don’t consider themselves to have a disability – but just find it more efficient to work with things that are bigger,) you don’t need to do anything specific to an accessibility setting for your app to support this. Rather, you just need to support all the resolutions that Windows Store apps should support. When your customers turn on the “Make everything on the screen bigger” setting, your app will just be presented at a particular resolution.

Given that new apps created from the HTML Grid Template support the required resolutions, I didn’t need to change anything in my app related to this setting.

Note that an HTML app does need to be restarted after the user has changed the state of the “Make everything on the screen bigger” setting. (If the app is not restarted, then the UI will not be sized correctly after the setting has been changed.)

Respecting system settings

There are a few additional system settings which you can leverage, and doing so can help you to provide a great user experience to some customers, where otherwise the app might be unusable. For example, respecting the duration for which notifications are to appear on the screen. Some of your customers need a certain amount of time in order to digest the contents of notifications that appear, and so if the notifications only appear for a few seconds, they might as well have not appeared at all.

My new app presents no notifications, and so I won’t check this system setting. In fact I won’t be changing the app to leverage any other systems settings either.

Summary

So having considered all the above, what did I actually change in my new app to make the list as accessible as possible? There were only a few things:

Set a useful localized accessible name on the list. I did this by using the “aria-label” attribute. The Inspect SDK tool is invaluable for checking the UIA properties in your UI, and you can use it to make sure all of the UI that you add to your app exposes a localized accessible name.
I gave some divs unique ids in order to expose useful UIA AutomationId properties which can be accessed by automated tests.
Show a border around my list items when a high contrast theme is active, to make sure the items really stand out.
Include high contrast versions of my shipping images, such that when the High Contrast Black and High Contrast White themes are active, the images are as clear as possible to all my customers.

Other things I considered while doing carrying out these investigations included:

Can the Narrator user quickly determine whether a list item’s of interest when they navigate to it?
Should a list item get focus when the app starts?
If I use the background-image style on a div, what’s the best experience for the user when they’re using a high contrast theme? (Certainly I don’t want an image to appear behind text when a high contrast theme is active.)

There are lots of interesting topics when considering how you can make your app as accessible as possible, and so reach the most customers. The above investigation relates to a list shown in a new app based on the HTML Grid App template, but many of the things mentioned relate to UI in general.

So try out your cool new Windows 8 apps using only the keyboard, in high contrast and with Narrator. It’s fascinating to consider how you can enhance your app to make it really efficient for all your customers to leverage its full functionality, regardless of how they interact with it!

One story of building a useful Assistive Technology tool from a UI Automation sample

Guy Barker MSFT — Sat, 08 Sep 2012 17:35:00 GMT

Background

At the 2011 Annual International Technology and Persons with Disabilities Conference (aka “the CSUN conference”), I gave a presentation on the Windows UI Automation (UIA) API. This API allows apps to find out what things are shown visually on the screen and to programmatically interact with those things. As such, it can be very helpful to Assistive Technology (AT) tools such as screen readers.

I was new to UIA at the time, and so needed to learn some things on how it’s used. The most effective way for me to learn about an API is to build an app which uses it, and to step through the code in a debugger watching what happens. I felt people at the conference would be more interested in the client-side aspects of the API rather than the provider-side. (The “client” is the app gathering up the data about what’s shown in other apps, and the “provider” is the app that contains the UI of interest.) So I built a sample app which gathers details on all links shown in a browser and which can programmatically invoke those links. A twist to this sample which is unconnected to UIA is that I also added use of the Window Magnification API, and I’d not built an app which used both these API’s before. The UIA API and Magnification API are not related, and it was interesting for the sample to clearly separate the data gathering steps from the mechanism by which the results are presented to the user. Other apps might choose different ways to present the results depending on the needs of their users. For example, using speech, (as a screen reader does), or presenting the results using large text elsewhere on the screen, (if it helps with the user’s comprehension of what’s on the screen.)

The picture below shows the sample app listing all the links shown in the browser, and highlighting one of the links.

After presenting this sample app, (which I’d built in C++), I was asked if I had a C# version. So I set to work on an equivalent C# version of the sample. It’s worth noting that all my samples use the native-code version of the UIA API that’s included in Windows, not the managed version that’s included in the .NET Framework. Since then I’ve completed a set of sample apps which use the client-side UIA API, and together the samples demonstrate all the commonly used parts of the API. My goal has always been to build samples which might be the starting point for real-world AT tools. Interacting with hyperlinks in a browser is an extremely common action. Another sample uses UIA to track keyboard focus and magnifies the UI with focus. And a third sample uses UIA’s “Text Pattern” to programmatically interact with e-mails. While none of the samples are feature-complete as AT apps, I had hoped that they would be a good base for people who want to build AT apps.

Actually, one of the samples is pretty much complete – the Key Speaker. This uses UIA to find the text shown on the Prediction Keys in the On-Screen Keyboard. This satisfied a specific need that someone described to me and it was an important reminder that sometimes an AT tool doesn’t have to do much to be useful; it just has to do what’s needed. I’ve had a few thousand downloads of my samples, and so I’m pleased that they are proving to be a useful reference.

Deciding to build an AT app

I was recently in contact with an AT consultant who has a client with no gross motor skills, but does have some fine motor skills. This got me thinking about users who can’t use a mouse but can move their fingers to press keys on the number pad on a keyboard. How might the user easily browse the web? (I’m thinking about browsing again, give that it’s such a common thing that people do.) So I then broke the potential actions down into three stages.

1. Pressing a key to “highlight” things that can be clicked in the browser.

2. Pressing keys to specify which thing is to be clicked.

3. Pressing a key to do the click.

So I thought I’d build an AT tool which does this. Given that my original sample already had classes for calling UIA and then highlighting results, it was natural that I’d use this sample as starting point for my new AT app. (In Windows 7, the UIA API has certain threading requirements which means that depending on what client apps do, they sometimes need to do a certain amount of thread-related work. That’s all standard thread stuff, but I don’t want to write that code from scratch again if I again avoid it.)

I then grabbed my sample and got to work. I decided that I’d build an app which I knew would be useful but which would have some constraints. I’d look into addressing those constraints based on future user feedback. One constraint that would not change would be that the browser would have to be accessible through the UIA API. (Pointing the Inspect SDK tool to UI is a great way to get a feel for how accessible an app is.)

Getting things which can be clicked

I concentrated on the UIA aspects of the app next, as that’s core to what I’m trying to achieve. The original sample has a file called LinkProcessor.cpp, and that’s that where all the UIA calls in the sample are made. So this is where all the UIA calls would be made in my new AT app. In the sample, all the calls are made on a background thread, and we can see a call to CoInitializeEx() being made with COINIT_MULTITHREADED. This is necessary if the app might interact with its own UI through UIA. In the case of my new app, this isn’t actually necessary, but there was no reason for me to change the threading code. Having the calls done on the background thread would work perfectly fine for my app, and if in the future I change the app such that it does interact with its own UI, it’ll continue to work fine.

So I left all the threading-related code in, (including the mechanism for communication between threads), and then pulled out lots of action taken by the sample which I’m not interested in. This left me with one function of real interest; the one that gets me a list of elements on the screen. The sample built up a list of hyperlinks, and I now want a list of things that can be clicked. I decided to use UIA caching in my app in order to get the highest performance possible. (The original sample demonstrated both using caching and not using caching.) So this had now all boiled down to a single call to FindAllBuildCache().

FindAllBuildCache() takes a condition which describes what elements are to be retrieved. I needed to figure out a condition which would get me all the elements that I’m interested in, and only them. By using the Inspect SDK tool I could examine the properties of hyperlinks shown in the browser, and I decided on the following conditions for the element I would retrieve:

1. They would appear in UIA’s Control view of the element tree. (This is a subset of the Raw view, and the Raw view contains lots of the elements that the user usually doesn’t care about.)

2. The IsOffscreen property is false. If the IsOffscreen property if true, then the link isn’t visible on the screen.

3. The IsEnabled property is true. If the IsEnabled property is false, the link can’t be clicked.

4. The IsInvokePatternAvailable property is true.

This last condition is the most interesting one. In thinking about the condition, I decided that I’m not just interested in links. Rather, I want anything that can be clicked. This means my new AT app can be used to click buttons like the Back button.

I then combined the conditions with a number of calls to CreateAndCondition() and passed that into the call to FindAllBuildCache(). I also passed in a cache request for the bounding rectangle and InvokePattern. This meant that by calling FindAllBuildCache(), a single cross-process call would be made to gather the bounding rectangles of potentially hundreds of elements which can be clicked. (Cross-process calls are slow and so I want to make as few of them as possible.)

Having made this call, I did indeed get many elements returned, including links and buttons which could be clicked. However, I didn’t quite get all the things I expected. I found that the tabs shown in the browser weren’t included. So again I used Inspect and pointed it to the tabs. By doing this I could see that that tabs do not support the InvokePattern, and this explained why they were missing from the results returned from FindAllBuildCache(). Given that the tabs are so important to my app, I decided to change the condition I’d built to make sure the tabs were included in the result. Inspect showed me that the tabs support the LegacyIAccessible pattern. (IAccessible is related to the legacy MSAA API. That API has been replaced by the more powerful and performant UIA API.) Since the tabs do support that pattern and not the Invoke pattern, I decided to include that pattern in my condition. An element that supports the Invoke pattern will later be clicked by calling the Invoke() method on that interface, but if that pattern is not supported and LegacyIAccessible() is supported, then LegacyIAccessible’s DoDefaultAction() method would be used.

The final code looked something like this:

hr = _pUIAutomation->CreateCacheRequest(&pCacheRequest);
if (SUCCEEDED(hr))
{
    hr = pCacheRequest->AddProperty(UIA_BoundingRectanglePropertyId);
    if (SUCCEEDED(hr))
    {
        hr = pCacheRequest->AddPattern(UIA_InvokePatternId);
        if (SUCCEEDED(hr))
        {
            hr = pCacheRequest->AddPattern(UIA_LegacyIAccessiblePatternId);
        }
    }
}

if (SUCCEEDED(hr))
{
    hr = _pUIAutomation->get_ControlViewCondition(&pConditionControlView);
    if (SUCCEEDED(hr))
    {
        VARIANT varPropOffScreen;
        varPropOffScreen.vt = VT_BOOL;
        varPropOffScreen.boolVal = VARIANT_FALSE;

        hr = _pUIAutomation->CreatePropertyCondition(
                 UIA_IsOffscreenPropertyId,
                 varPropOffScreen,
                 &pConditionOffscreen);
        if (SUCCEEDED(hr))
        {
            VARIANT varPropIsEnabled;
            varPropIsEnabled.vt = VT_BOOL;
            varPropIsEnabled.boolVal = VARIANT_TRUE;

            hr = _pUIAutomation->CreatePropertyCondition(
                     UIA_IsEnabledPropertyId,
                     varPropIsEnabled,
                     &pConditionIsEnabled);
            if (SUCCEEDED(hr))
            {
                VARIANT varPropHasInvokePattern;
                varPropHasInvokePattern.vt = VT_BOOL;
                varPropHasInvokePattern.boolVal = VARIANT_TRUE;

                hr = _pUIAutomation->CreatePropertyCondition(
                         UIA_IsInvokePatternAvailablePropertyId,
                         varPropHasInvokePattern,
                         &pConditionHasInvokePattern);
                {
                    VARIANT varPropHasLegacyIAccessiblePattern;
                    varPropHasLegacyIAccessiblePattern.vt = VT_BOOL;
                    varPropHasLegacyIAccessiblePattern.boolVal = VARIANT_TRUE;

                    hr = _pUIAutomation->CreatePropertyCondition(
                             UIA_IsLegacyIAccessiblePatternAvailablePropertyId,
                             varPropHasLegacyIAccessiblePattern,
                             &pConditionHasLegacyIAccessiblePattern);
                }
            }
        }

        if (SUCCEEDED(hr))
        {
            hr = _pUIAutomation->CreateOrCondition(
                     pConditionHasInvokePattern,
                     pConditionHasLegacyIAccessiblePattern,
                     &pConditionPatterns);
            if (SUCCEEDED(hr))
            {
                hr = _pUIAutomation->CreateAndCondition(pConditionControlView,
                         pConditionOffscreen,
                         &pConditionCombined1);
                if (SUCCEEDED(hr))
                {
                    hr = _pUIAutomation->CreateAndCondition(
                             pConditionCombined1,
                             pConditionIsEnabled,
                             &pConditionCombined2);
                    if (SUCCEEDED(hr))
                    {
                        hr = _pUIAutomation->CreateAndCondition(
                                 pConditionCombined2,
                                 pConditionPatterns,
                                 &pCondition);
                    }
                }
            }
        }
    }
}

if (SUCCEEDED(hr))
{
    hr = pElementWindow->FindAllBuildCache(TreeScope_Descendants,
                                           pCondition,
                                           pCacheRequest,
                                           &pElementArray);
}

The above code snippet doesn’t include the calls to release all the objects once they’re no longer needed.

Hold on - where am I getting the clickable things from?

The above call to FindAllBuildCache() is being made from an IUIAutomationElement called pElementWindow, but what is that? My goal here is to get clickable things in a browser, but I thought it would also be interesting to get the clickable things in whatever the foreground window is. So the code for getting pElementWindow is:

HWND hWndForeground = GetForegroundWindow();
if (hWndForeground != NULL)
{
    IUIAutomationElement *pElementWindow = NULL;
     HRESULT hr = _pUIAutomation->ElementFromHandle((UIA_HWND)hWndForeground,
                                                    &pElementWindow);
     if (SUCCEEDED(hr) && (pElementWindow!= NULL))
     {
        // Now get the clickable elements…

In practice my AT app wasn’t able to click things in a number of apps, and I’ll look into fixing that at some point. Given that my main goal related to browser usage, and that worked fairly well, the above approach met my needs.

One approach that I’d certainly not want to take would be to get the root element of the UIA tree and then call one of the Find* methods from that with TreeScope_Descendants in the hope of getting every clickable things on the screen. MSDN points out that calling Find* must not be done from the root element with TreeScope_Descendants.

Another approach I’d not want to take is to call ElementFromPoint() to hit-test for every clickable thing on the screen. To feel confident that I’d not missed small elements, I’d need to make many thousands of calls to ElementFromPoint(). Given that each call would be cross-process, the performance would be grim.

But I did think that if at some point my AT app would be useful for working in some apps outside of the browser, then in addition to helping the user work in the foreground app, it should also help in starting apps and switching between apps. In that case, the app would need to interact with the Taskbar. So in addition to working with the foreground window, the app also finds the Taskbar with the code below and then finds the clickable things that are shown on it.

IUIAutomationElement *pElementRoot = NULL;
HRESULT hr = _pUIAutomation->GetRootElement(&pElementRoot);
if (SUCCEEDED(hr))
{
    VARIANT varPropClassName;
    varPropClassName.vt = VT_BSTR;
    varPropClassName.bstrVal = SysAllocString(L"Shell_TrayWnd");

    IUIAutomationCondition *pConditionClassName = NULL;
    hr = _pUIAutomation->CreatePropertyCondition(UIA_ClassNamePropertyId,
                                                 varPropClassName,
                                                 &pConditionClassName);
    if (SUCCEEDED(hr))
    {
        IUIAutomationElement *pElementTaskBar;
        hr = pElementRoot->FindFirst(TreeScope_Children,
                                     pConditionClassName,
                                     &pElementTaskBar);
        if (SUCCEEDED(hr) && (pElementTaskBar != NULL))
        {
            // Now get the clickable elements…

The above code finds the element with the accessible Name property of “Shell_TrayWnd” which is a direct child of the root element. It’s ok to call one of the Find* functions from the root element when TreeScope_Children is used. I used the Inspect SDK tool to look at the properties of the Taskbar in order to figure out a way to programmatically access it.

It turned out that in practice the gathering the Taskbar buttons was only partially useful in my app as it stands today. The user can start apps by clicking a Taskbar button with my app and sometimes switch between apps, but there were situations where it didn’t work. I’ll look into this further if I get related feedback about it.

So with the above code, my app had gathered a long list of elements which can be programmatically clicked in the browser. It keeps this list around, as the app will need to retrieve an element from the list later when the user wants to click it. The next step was to show these elements to the user.

Highlighting the results

The original UIA sample has a file called Highlight.cpp which is responsible for all the work to present results to the user. The sample did this by magnifying the element, but for my app I want to present something at the location of every clickable element. This will allow the user to specify that they want to click a particular element.

So I decided to create a fullscreen window that would be completely transparent except for identifiers that would lie over the clickable elements. I pulled out all the magnification related code from the sample, and modified the existing code which creates a window such that it created the big window I needed.

In the call to CreateWindowEx(), I passed the flags:

WS_EX_LAYERED | WS_EX_TRANSPARENT | WS_EX_NOACTIVATE | WS_EX_TOPMOST

I then called SetLayeredWindowAttributes() with LWA_COLORKEY and a colour that I thought would never be used as a system colour in practice. (I used system colours for the display of the element identifiers.)

By doing the above, I had a fullscreen window that was transparent, and which would not react at all to mouse clicks in it. (When I say “fullscreen” here, a constraint I have in the app at the moment is that I’m not supporting multi-monitor. I’ll fix that if I get feedback about it.)

Having created the window which will present the results to the user, I added a public method to the existing CHighlight object from the sample. This method allowed the object that gathered all the bounding rectangles of the things that can be clicked, to inform the CHighlight object where each identifier is to be displayed on the screen. So CHighlight ends up with an array of points, and that’s all it needs. (CHighlight also now calls GetTextExtentPoint32() to take into account the size of the identifier to be displayed, and as such, center the identifier over the element.)

Finally, whenever the window processes WM_PAINT, it draws the identifiers on the screen. It simply iterates through the array, building up strings as the identifiers based on the index in the point array. So the identifier for the first element in the array is “ 1 ”, the next is “ 2 ”, and so on.

The picture below shows clickable elements in the browser.

For many people the highlighting shown will be usable, but I’m conscious of the fact that the person with the mobility impairments which prompted me to build this app, also has visual impairments. So another aspect of the app which I may be enhancing is to try to make the identifiers easier to see. I specifically used theme-based colours for the text colour and background colour, (through calls to GetSysColor() with COLOR_WINDOWTEXT and COLOR_WINDOW), but the identifiers are still very small and often overlap.

The picture below shows the identifiers when the High Contrast #1 theme has been selected in the Personalization Control Panel.

Clicking something

The last step in all this is providing some way for the user to click the thing that’s highlighted. Different users might prefer different ways of doing this, (for example, typing a number and then automatically having the element clicked after say a couple of seconds). But I went with the following approach:

1. Press the ` key near the top left of the English keyboard to turn the identifier display on or off.

2. Type the identifying number of the thing you want to click.

3. Press the Escape key to click it.

An important question here is, why choose the ` and Escape keys? I did this deliberately to make it as fast as possible for someone using the Windows 7 On-Screen Keyboard (OSK) in its Scan mode. The OSK’s Scan mode allows someone to control the keyboard with a hardware switch. So if someone has severe mobility impairments and uses either a foot switch or head switch to interact with the computer, my new app is optimized to make it quick for those users to type the keys they need to. The picture below shows the OSK in Scan mode:

I realize the keys I’ve chosen will be far from ideal for some people. For example, a user might prefer to be able to use the number pad on a physical keyboard, such that with only small movements of their fingers, they can click things in the browser. If I get feedback asking for the app to be enhanced to allow the user to configure which keys are used, I’ll do that.

The original sample had no code for detecting key presses in the way my app needed to, so I added a low-level keyboard hook through a call to SetWindowsHookEx(). The hook detects VK_OEM_3 to turn the identifier display on and off in response to a press of the ` key. (I’ve only used my app with an English keyboard, and so might have to update the hook for non-English keyboards.) In response to key presses of ‘0’-‘9’ while the identifiers are on, the app builds up a string to cache the full number typed. Then in response to a press of the VK_ESCAPE key while identifiers are on, the app performs the clicking of the element. Whenever it does take action in response to the key press, the app blocks the system from also processing the key press. If it did not do this, then the number that the user’s typing to specify an identifier would also appear in whatever control currently has keyboard focus. (And I realize that doing the above, I’m assuming that the user never wants to type a ` character into any app.)

When a click is made, the app uses the number typed to figure out which element is of interest. It still has the long list of elements that it originally gathered through UIA, and so it can retrieve the element of interest. That is, if the user typed “1”, then the app will use the first element found earlier.

The code below performs what the user considers to be the click on the element.

IUIAutomationInvokePattern *pPattern = NULL;
IUIAutomationLegacyIAccessiblePattern *pPatternLegacyIAccessible = NULL;

hr = pElement->GetCachedPatternAs(UIA_InvokePatternId, IID_PPV_ARGS(&pPattern));
if (SUCCEEDED(hr))
{
    if (pPattern != NULL)
    {
        hr = pPattern->Invoke();

        pPattern->Release();
        pPattern = NULL;
    }
    else
    {
        hr = pElement->GetCachedPatternAs(UIA_LegacyIAccessiblePatternId,
                                          IID_PPV_ARGS(&pPatternLegacyIAccessible));
        if (SUCCEEDED(hr) && (pPatternLegacyIAccessible != NULL))
        {
            hr = pPatternLegacyIAccessible->DoDefaultAction();

            pPatternLegacyIAccessible->Release();
            pPatternLegacyIAccessible = NULL;
        }
    }
}

The calls to GetCachedPatternAs() above do not result in cross-process calls, because I explicitly asked for information about these patterns to be cached in the call to FindAllBuildCache(). The calls to Invoke() or DoDefaultAction() will result in cross-process calls, as this is where the app tells the browser to have the element perform some action. If the original element no longer exists by the time these calls are made, then the calls will return UIA_E_ELEMENTNOTAVAILABLE.

And that’s it

Having built the working app, I did pull out some more parts of the sample which I didn’t need. This included all the code related to UIA event handling. Looking at the final code, the amount of code which I’d had to add myself in order to build my app was fairly small, so I was pleased about that. The componentization of the original sample into parts for (i) the main app UI, (ii) the code which calls UIA, and (iii) the code which highlights the results worked well for me.

So in a few hours over a weekend, I’d built a functioning app which can help someone with very limited mobility to browse the web. I’ve made the app freely available at my own AT site, and documented the current constraints of the app there. I’ll enhance it as I get feedback from people who want to use it.

If you know of someone who could benefit from an app which helps them interact with things that are shown on the screen, perhaps one of my UI Automation samples could help you build it for them.

Guy

Using Dynamic Annotation with Child IDs

Michael Bernstein — Wed, 25 Apr 2012 04:50:18 GMT

The Dynamic Annotation API in the Windows Automation API is a convenient way to make simple accessibility changes to the accessible properties of Win32 Common Controls without writing a lot of code. There is good reference documentation available on MSDN, and some samples, too. That said, I’ve received some questions about it and wanted to talk specifically about how it works with Child IDs.

A quick refresher

The basic usage model for Dynamic Annotation is simple. Suppose I have a static control in my application with an image on it instead of a text label, and I need to set the accessible name to be “Picture of a thermometer” in order to allow a screen reader to read the control correctly. No problem: my app uses CoCreateInstance() to create an AccPropServices object, and calls IAccPropServices::SetHwndPropStr() to annotate the name property (PROPID_ACC_NAME):

HWND hwndIcon = GetDlgItem(hwnd, IDC_ICON1); 
CHECKHR(pAccPropSvc->SetHwndPropStr(hwndIcon, static_cast<DWORD>(OBJID_CLIENT), CHILDID_SELF, 
    PROPID_ACC_NAME, L"Picture of a thermometer"));

In this code snippet, CHECKHR is just a macro that I’m using to log any errors that occur. A real application would use an error handling strategy consistent with what it was using for the rest of the codebase.

I make sure to call IAccPropServices::ClearHwndProps() afterwards to clear off the property:

HWND hwndIcon = GetDlgItem(hwnd, IDC_ICON1); 
MSAAPROPID propids[1] = { PROPID_ACC_NAME }; 
CHECKHR(hr = pAccPropSvc->ClearHwndProps(hwndIcon, static_cast<DWORD>(OBJID_CLIENT), CHILDID_SELF, 
    propids, 1));

The samples I mentioned above cover this well.

One new extension, added in Windows 7, is that I can now annotate with UIA properties. The property IDs for UIA properties are all in UIAutomationCoreApi.h (in the Platform SDK). So, if I want to set new UIA properties like AutomationId or ItemStatus, I’ve got a good way to do that:

CHECKHR(pAccPropSvc->SetHwndPropStr(hwndIcon, static_cast<DWORD>(OBJID_CLIENT), CHILDID_SELF, 
    AutomationId_Property_GUID, L"ThermometerAutomationId"));

How do I see if this is working? I can use the Inspect tool, available in the Windows SDK, to inspect the properties on my annotated object and verify that the new properties are showing up.

Annotating child items

Suppose my application uses a standard menu (CreateMenu()) and I want to add some special accessibility properties to menu items. This is a little different than our button example: we don’t want to annotate the whole menu, but rather one of its children. Well, I can do that, too – I just use IAccPropServices::SetHmenuPropStr() and IAccPropServices::ClearHmenuProps(). They both take a child ID argument, which specifics which menu item to annotate. The Inspect tool can show you the child ID of the menu item. If you are just annotating one or two menu items, this is a good approach. This technique works on toolbars as well. In this code snippet, the child ID is 1.

// Annotate the menu item with a string... 
CHECKHR(pAccPropSvc->SetHmenuPropStr(g_hMenuBar, 1, PROPID_ACC_NAME, L"NewText"));

However, It often happens that you want to annotate the properties of all of the children of a container in a certain way. For example, suppose I have an owner-drawn menu and I want to set the accessible names for all of the items in the menu, even though the menu items don’t officially have any textual content. Calling SetHmenuPropStr() for every child ID would be tedious and inflexible – if the menu structure changes, I have to remove all the annotations and reset them. A better solution would be to use a callback function that can be called with a child ID to request the accessible name (or another property) for that child ID. The MSDN sample for Dynamic Annotation demonstrates this in a formal way; the core idea is to create an object that implements IAccPropServer and use dynamic annotation to mark the HWND or HMENU with the IAccPropServer, rather than with a string.

The annotation looks very much as it did before:

    AutoRelease<MenuAccServer *> pMenuAccSrv = new MenuAccServer(pAccPropSvc);
    MSAAPROPID propid = PROPID_ACC_NAME;
    CHECKHR(pAccPropSvc->SetHmenuPropServer(g_hPopup, CHILDID_SELF, &propid, 1, pMenuAccSrv, ANNO_CONTAINER));

The essential function in MenuAccServer looks like this:

    BOOL GetPropValue(
        const BYTE *    pIDString,
        DWORD           dwIDStringLen,
        MSAAPROPID      idProp,
        VARIANT *       pvarValue)
    {
        HMENU hmenu = NULL;;
        DWORD idChild = 0;
        BOOL fSuccess = TRUE;

        // Break down the identity string into components and check the idChild
        if(S_OK != _pAccPropSvc->DecomposeHmenuIdentityString(pIDString, dwIDStringLen,
                    &hmenu, &idChild) ||
            idChild == CHILDID_SELF)
        {
            fSuccess = FALSE;
        }

        // Respond to requests for PROPID_ACC_NAME with the override
        // accessible name
        if (fSuccess)
        {
            if (idProp == PROPID_ACC_NAME)
            {
                if ((idChild - 1) < ARRAYSIZE(g_ColorInfo))
                {
                    pvarValue->vt = VT_BSTR;
                    pvarValue->bstrVal = SysAllocString(g_ColorInfo[ idChild - 1 ]._pName);
                }
                else
                {
                    fSuccess = FALSE;
                }
            }
            else
            {
                fSuccess = FALSE;
            }
        }

        return fSuccess;
    }

And with that, I can mark up all of the children of the HMENU differently depending on their child IDs. Very convenient.

Don’t forget to clear the annotation when the HWND or HMENU is destroyed – leaving stale annotations around is a good way to cause memory leaks or crashes during shutdown.

       propid = PROPID_ACC_NAME;
       CHECKHR(pAccPropSvc->ClearHmenuProps(g_hPopup, CHILDID_SELF, &propid, 1));

So, if you have an accessibility problem with a common control with children, now you know a useful technique to add or override accessibility properties on the control.

Windows 7 UI Automation Client API C# sample (e-mail reader) Version 1.1

Guy Barker MSFT — Sun, 18 Mar 2012 00:09:39 GMT

I recently built a very simple C# app which uses the Windows 7 native-code UI Automation (UIA) API to gather text from a Windows Live Mail window, as part of having that text spoken. Sometimes having the text spoken can make text in a mail being read easier to understand, or to detect issues with mail being composed. The app used the UIA TextPattern to gather the text, but did so with a single call to collect all the text in the mail. The app might be useful as it is in some situations, but I've just updated it to leverage some more of the TextPattern and TextRange features.

As it happens, with UIA's TextPattern and TextRange, you can still do a lot more to manipulate and traverse text that my updated app does, but the updated sample introduces the use of methods such as:

GetBoundingRectangles() - Get all the bounding rectangles for the text in a range.
GetSelection() - Get only the text that's selected in the mail.
Move() - Move an entire range forward or background by some unit, (for example, one paragraph).
MoveEndpointByRange() - Move one end of a range to be equal to the start or end of another range (or even the same range.)
MoveEndpointByUnit() - Move one end of a range by some amount, (for example, by one paragraph).
ScrollIntoView() - Tell the provider app to scroll its view, such that text is actually visible on the screen.

The full set of functionality for TextPattern and TextRange is available at http://msdn.microsoft.com/en-us/library/windows/desktop/ee696214(v=vs.85).aspx and http://msdn.microsoft.com/en-us/library/windows/desktop/ee696221(v=vs.85).aspx.

The comments in the sample app show why and how the above calls are made. I should say that I've not done a great deal of testing with this app, but I'm sure the UIA related code is working great, which is what the sample's really all about. It's worth pointing out that when working with TextPattern and TextRange, your client app can only use what the provider app supports. If the provider has chosen to only partially implement TextPattern, then there's nothing you or UIA can do to add the missing functionality. Rather, you make the best of what the provider does support.

This is what you can do with the app once you’ve opened a Windows Live Mail window to read or compose a mail…

To read the whole mail, just hit the Play button.
Whenever text is being spoken, pressing the Stop button will restore the app to its initial state.
Whenever text is being spoken, pressing the Play button again will stop the speech in such a way that pressing Play again will start speaking the same text again.
To start reading from the current selection, (or where the caret is), check the "Start reading the mail from the selection" radio button, and press the Play button.
To have some visual highlight of the paragraph being read, check the "Highlight the text being read paragraph by paragraph" checkbox. For this sample app, I reused a magnification class I created for another sample, but you could choose to highlight the text in some other way. (The sample app assumes the Windows Live Mail window is not obscured by anything, and the magnified text fits on the screen. I've not looked into a black flash that can sometimes appear in the magnification lens when the magnification is refreshed.)
If you’ve checked the "Highlight the text being read paragraph by paragraph" checkbox, then when the text is being read, you can jump to the next or previous paragraph by clicking the Next and Previous button respectively.

So while I've only leveraged a subset of what the TextPattern and TextRange can do, the updated app still can do some very useful things.

The updated sample's out at http://code.msdn.microsoft.com/Windows-7-UI-Automation-9ce18fd5.

Finally - if you happen to know of someone who might find the sample app a useful tool in practice for working with e-mail, but it needs a few tweaks before it'll really work well for them, let me know. I'd be happy to try to help you use UIA to extend the functionality further in order to turn the app into a working tool for someone.

Thanks,

Guy

Windows 7 UI Automation Client API C# sample (e-mail reader) Version 1.0

Guy Barker MSFT — Tue, 06 Mar 2012 02:36:32 GMT

In preparation for a demo of the native-code Windows UI Automation (UIA) API last year, I built a sample app which introduces most of the commonly used features of UIA. This included the following topics; properties, events, patterns, cache requests and conditions. The original C++ sample is up at http://code.msdn.microsoft.com/Windows-7-UI-Automation-9131f729, and a version I ported to C# is at http://code.msdn.microsoft.com/Windows-7-UI-Automation-0625f55e.

Probably the most interesting functionality that’s not mentioned in that sample relates to the Text Pattern. If a UIA provider supports the Text Pattern, then a client can perform a number of operations to move through the text shown in the provider and to access that text. For example, the client can move through the text, examining the text character-by-character, word-by-word, line-by-line or paragraph-by-paragraph. Details of the Text Pattern are up at http://msdn.microsoft.com/en-us/library/windows/desktop/ee815688(v=vs.85).aspx.

So I thought it would be interesting to build a new sample that uses the Text Pattern. As I considered this, I wondered whether I could build a very simple app which hardly does anything, but could still be a useful assistive technology (AT) tool in practice. While UIA is great for enabling UI automated tests, the API is also designed to support people building AT, and so I’m always very excited to build tools which relate to a real-world AT scenarios. (For example, building the tool to speak the text shown on the OSK prediction keys, http://code.msdn.microsoft.com/Windows-7-UI-Automation-433a961f.)

With this in mind, I decided that a future sample would demo some of the more powerful aspects of the Text Pattern, but my next sample would do the absolute minimum required to have the Text Pattern support a useful tool. What I settled on was an app to read e-mail text. Some people find it easier to comprehend an e-mail message if the text is spoken to them as they read it. And sometimes when important e-mail is being composed, if it’s spoken prior to being sent, that can help to detect issues with the text which aren’t caught by running a spell checker alone. So the new sample app should read both received e-mail and e-mail being composed.

I decided to build the app in C# and my first step was to copy one of my existing C# samples, http://code.msdn.microsoft.com/Windows-7-UI-Automation-6390614a. I then removed a few existing things which weren’t relevant to the new app, (for example the UIA event handler, and use of the Magnification API.) One handy aspect about my new app was that I didn’t have to account for the UIA threading rules. That is, if an app uses UIA to access its own UI, or it has UIA event handlers, then certain things must be done on background threads, (as described in my other samples). But since the new app doesn’t do any of that, it keeps things really simple.

I then updated the .cs file for the main app WinForm UI. There’s little work done here other than to use a SpeechSynthesizer to speak the text using the current default voice on the computer, and also to change the visuals for a button to start or stop the speaking of the text.

The more interesting .cs file is the one that actually uses UIA. It initializes and uninitializes an IUIAutomation object, (as any UIA client app does), and then has a single function to access the e-mail text. That function looks like this:

// Try to find a Windows Live Mail window for composing and reading e-mails.
// Using the Spy tool, the class of the main window can be found. This test
// app assumes there's only one Windows Live Mail window of interest.
IntPtr hwnd = Win32.FindWindow("ATH_Note", null);
if (hwnd != IntPtr.Zero)
{
    // We found a window, so get the UIA element associated with the window.
    IUIAutomationElement elementMailAppWindow = _automation.ElementFromHandle(
                                                    hwnd);

    // Find an element somewhere beneath the main window element in the UIA
    // tree which represents the main area where the e-mail content is shown.
    // Using the Inspect SDK tool, we can see that the main e-mail content
    // is contained within an element whose accessible name is "NoteWindow".
    // So create a condition such that the FindFirst() call below will only
    // return an element if its name is "NoteWindow".
    IUIAutomationCondition conditionNote = _automation.CreatePropertyCondition(
                                                _propertyIdName, "NoteWindow");

    IUIAutomationElement elementNoteWindow = elementMailAppWindow.FindFirst(
                                                TreeScope.TreeScope_Descendants,
                                                conditionNote);

    // As it happens, the actual element that supports the Text Pattern is
    // somewhere beneath the "NoteWindow" element in the UIA tree. Using
    // Inspect we can see that there is an element that supports the
    // Text Pattern. Once we have that element, we can avoid a cross-process
    // call to access the Text Pattern object by using cache request.
    IUIAutomationCacheRequest cacheRequest = _automation.CreateCacheRequest();
    cacheRequest.AddPattern(_patternIdTextPattern);

    // Now find the element that supports the Text Pattern. This test app assumes
    // there’s only one element that can be returned which supports the Text Pattern.
    bool fTextPatternSupported = true;
    IUIAutomationCondition conditionTextPattern = _automation.CreatePropertyCondition(
                                                    _propertyIdIsTextPatternAvailable,
                                                    fTextPatternSupported);

    IUIAutomationElement elementMailContent = elementMailAppWindow.FindFirstBuildCache(
                                                    TreeScope.TreeScope_Descendants,
                                                    conditionTextPattern,
                                                    cacheRequest);

    // Because the Text Pattern object is cached, we don't have to make a cross-process
    // call here to get object. Later calls which actually use methods and properties
    // on the Text Pattern object will incur cross-proc calls.
    IUIAutomationTextPattern textPattern = (IUIAutomationTextPattern)
                                                elementMailContent.GetCachedPattern(
                                                    _patternIdTextPattern);

    // This test app is only interested in getting all the e-mail text, so we get that through
    // the DocumentRange property. A more fully featured app might be interested in getting a
    // a collection of Text Ranges from the e-mail. Each range might relate to an individual
    // word or paragraph. Given that a provider which supports the Text Pattern allows a
    // client to find the bounding rectangles of these ranges, the client could choose to use
    // its own method of highlighting the text as the text is being spoken.
    IUIAutomationTextRange rangeDocument = textPattern.DocumentRange;

// Pass -1 here because we're not interested in limiting the amount of text here.
strMailContent = rangeDocument.GetText(-1);
}

This app specifically only works with Windows Live Mail, but with a small adjustment the code above could target other window-based e-mail apps.

As Text Pattern samples go, this isn’t very interesting as all it does is access the text for the entire document as a single string, (through the DocumentRange property.) But I did find this an exciting experiment. In around an hour and a half I built a UIA app which contains very little code, but which could still be a useful AT tool for some people. My next step is to figure out which other parts of the Text Pattern and Text Range interfaces would help most in practice for an e-mail related AT like this, and to update the sample accordingly. For example, the AT tool might get call IUIAutomationTextRange::Move() to access each line in turn, and call GetBoundingRectangles() and render its own highlighting for the line being spoken. The app could also call IUIAutomationTextPattern::GetVisibleRanges() to make sure it only gathers text that’s shown on the screen when working with e-mail clients that handle hidden text.

The e-mail reading sample app is out at http://code.msdn.microsoft.com/Windows-7-UI-Automation-28b2b537.

UI Automation Featured on Win8 Blog

Michael Bernstein — Thu, 16 Feb 2012 16:33:00 GMT

You may have wondered why this blog has gone quiet. The UIA team has been very busy with Windows 8. We're excited to link to a new blog post on the Building Windows 8 blog that goes into depth on Windows 8 Accessibility as a whole:

Enabling Accessibility

That post can give you a sense of what we've been up to. While you're at it, you may also want to check out this fine talk by blog contributor Guy Barker from the //BUILD/ conference, about how to build accessible Metro style applications for Windows 8:

Reaching more customers with accessible Metro style apps in HTML5

In the meantime, if you have pressing questions, be sure to visit our MSDN forum for Accessibility and ask us there.

- Michael

Microsoft Windows UI Automation Blog

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 4: Postscript

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 4: PostscriptGuy Barker

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 3: The WinJS app

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 3: The WinJS appGuy Barker

Accessible names

Descriptive Slider values

Announcing visual updates

Presenting appropriate high contrast images

Summary

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 2: The XAML app

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 2: The XAML appGuy Barker

Accessible names

Descriptive Slider values

Announcing visual updates

Presenting appropriate high contrast images

Summary

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 1: Introduction

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 1: IntroductionGuy Barker

Accessible names

Descriptive Slider values

Announcing visual updates

Presenting appropriate high contrast images

What can I do to make my Windows 8 WinJS.UI.ListView app as accessible as possible?

Programmatic Access

Narrator

Using your app with the keyboard

High Contrast

Conveying information through colour alone

Conveying information through audio alone

Make everything bigger

Respecting system settings

Summary

One story of building a useful Assistive Technology tool from a UI Automation sample

Using Dynamic Annotation with Child IDs

A quick refresher

Annotating child items

Windows 7 UI Automation Client API C# sample (e-mail reader) Version 1.1

Windows 7 UI Automation Client API C# sample (e-mail reader) Version 1.0

UI Automation Featured on Win8 Blog

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 4: Postscript
Guy Barker

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 3: The WinJS app
Guy Barker

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 2: The XAML app
Guy Barker

Customizing the accessibility of my XAML and WinJS Windows 8 apps – Part 1: Introduction
Guy Barker