This post describes considerations when porting an accessible solitaire game to the Xbox, and focuses on the experience when using the Narrator screen reader or a switch device.
Apology up-front: When I uploaded this post to the blog site, the images did not get uploaded with the alt text that I'd set on them. So any images are followed by a title.
Introduction
Earlier this year, at The Sa11ytaire Experiment: Part 1 – Setting the Scene I described how a colleague and I built a Microsoft Store app which explored a variety of input and output mechanisms that could be used to play a solitaire game. I felt this was a fascinating topic, involving input mechanisms such as speech, eye control, switch device, keyboard access keys, and also considering the most effective Narrator experience we could build. While the app was far from being complete, (for example, it didn't provide a usable experience on small screens or when different themes were active,) it had reached a point where we could start tuning the app based on feedback.
Since we built the app, Microsoft announced the Xbox Adaptive Controller, and so it was inevitable that I wasn't going to be able to relax until I'd played Sa11ytaire with an Adaptive Controller. This post describes the steps I took to get Sa11ytaire running on an Xbox. I've never built an Xbox game before, and so I mention a few places where there were hiccups along the way, (mostly caused by me not paying much attention to all the helpful resources available at docs.microsoft.com).
Please do send feedback on the Sa11ytaire Xbox experience. I know there are many things that could be improved with the app, (for example the visuals are still rudimentary,) but I'm hoping the current switch device and Narrator experiences provide the foundation for a usable experience.
The Sa11taire Xbox game is available at Sa11ytaire at the Microsoft Store.
Figure 1: The Sallytaire game being played on the Xbox, with a switch device plugged into an Adaptive Controller. A ten of diamonds is being moved onto a jack of clubs.
A few other thoughts
Before we get started, a few additional comments.
Is the game really usable?
There are a fair number of known bugs and limitations in the game, but I do believe it's worth sharing it out as it is, in order to help us focus on the top priority changes first. The most significant constraint with the switch device experience, is that on Xbox, the switch scan speed cannot be adjusted through switch input. So the speed needs to be set using something other than the switch device.
What about Magnifier on the Xbox?
As far as I know, Magnifier works with Sa11ytaire just as it does with any other app. I couldn't find a way to have Magnifier follow keyboard focus in the app as the Magnifier feature can on the desktop, so the view needs to be manually updated to bring the cards of interest into view.
What about colors used in the app?
Color usage is a critical part of building an app. Providing sufficient contrast, and respecting customers' choice of light-on-dark or dark-on-light, is such a fundamental concept. That said, I didn't focus on that for this porting of the game to Xbox, simply due to time constraints. The visuals today provide a means to learn about the switch device and Narrator experiences, and we can prioritize updates to the app based on your feedback.
What about my Windows Phone?
Since we'd built a UWP XAML app, it should work on any Windows 10 device right? I was porting the Windows desktop app to the Xbox, and who knows, maybe the Sa11ytaire experiment will move to Hololens at some point. But what about my Windows Phone? When I'm sitting on the 255 bus, why wouldn't I want to play the game and consider the next steps for improving it? Well, it looks like I'm out of luck. I tried deploying the app to my Microsoft Lumia 950, and Visual Studio told me the phone's OS was too old. Perhaps I could address that my changing the Sa11ytaire app to build for the older OS, but there's no way I'm doing that. One of the key points of Sa11ytaire app is to explore how to get the most out of the great new accessibility-related features available in whatever the latest version of Windows 10 is. And as far as I can tell, there is no later version of the OS available for my phone. So I think it might be time for me and my Windows Phone to part ways. That's too bad really, I'll miss it.
That last point raises the question: Should the app be turned into a Xamarin app, enabling it to run on multiple mobile platforms? That is a tempting idea, but I'm not sure if a Xamarin app running on Windows 10 can leverage all the latest accessibility features of Windows. The Sa11ytaire app makes some very deliberate use of the UIA Notification event through the UWP XAML platform, as part of trying to deliver the most efficient Narrator experience possible. If I can't do that with a Xamarin app today, then I'll continue with the regular UWP XAML app.
Porting the desktop app to the Xbox
Typically, I was going to do the least amount of preparation that I could while porting the game to Xbox. I tell myself that that's because I just don't have time to read all the material out there which explains what I should be doing. But I have been called a slacker in the past, so perhaps that's the real problem. Either way, I wondered if I could simply publish the existing desktop game as-is for the Xbox, and maybe it would "just work".
So the first step was publishing for Xbox. At the Services section for the Sa11ytaire app at the Dev Center, I selected "Xbox Live" and viewed all the things I could do there. The only thing I actually did there was invoke the Settings button, and then say that the app supports both Desktop and Xbox One. Back in the main Properties for the app, the app was already declared to be a "Card + board" game, and since Xbox seems to require that the app is a game, that was all fine. I set the "Game settings" to be the most basic they could be. That is, single player for PC and Xbox. (I left the Broadcasting setting checked as that seemed like that might be handy if in the future someone wants to provide feedback through a broadcast.)
Having done that, an error string appeared on the submission page, saying "Document: accesspolicies.xml Error: AccessPoliciesRequired The access policies document is not present in the config set. This document is required for all publish operations". I searched high and low for information about that, and in the end got help from someone in Xbox support. It turned out that all I had to do was press the "Test" button on the Xbox Live Services page at the Dev Center, and the error went away. I suspect others might not hit this error as I did, as I think my dev account might have been in an unusual state. (The account was created way back, perhaps when the process for setting up Xbox accounts was still evolving.) But if you ever do hit that mysterious error, try invoking the "Test" button. (You would have done that already of course, if you were actually testing your app on an Xbox before publishing it…)
At that point I published the app, and to my delight soon after found I could download the app on my Xbox. It technically worked, but there were a couple of things I'd need to do to make it usable. The first related to preventing some UI clipping, and so needed to support a smaller screen resolution that the app currently supported. Addressing that was routine UWP XAML UI work, including making sure I reduced minimum widths for some elements. And while the results still led to some text clipping, it would do for now. The second point related to input via the Xbox controller. By default the controller was moving the pointer on the screen, and instead I needed it to move focus between the cards in the app. In order to address that, I added the following to the app, after which, the controller's D-pad moved me left/right/up/down just great.
RequiresPointerMode = ApplicationRequiresPointerMode.WhenRequested;
(For more details, visit ApplicationRequiresPointerMode.)
While making the above changes, I did re-publish the app a couple of times. But then after one attempt, I was told the app submission was rejected due to:
"Create at least one active user and allow the user to sign the user in to Xbox. Display the user's Xbox gamertag as the primary display and profile name. Please add this behavior and resubmit your game."
It was becoming clear I couldn't keep publishing the app and hoping for the best. I needed to develop and test using my own Xbox, and publish when I was reasonably confident that the app would actually work. So I took a look at a variety of resources, including:
Setting up my Xbox in dev mode and connecting to it through the Dev Portal from a browser on my laptop was surprisingly straightforward once I'd got used to configuring the Xbox's Remote Access Settings, as described at Introduction to Xbox One tools. (I found that page after first encountering the "This site is not secure" thing described there, and I have to say, I did find that a little disconcerting at the time.)
Having effectively turned my Xbox into a dev machine, the next step was to debug Sa11ytaire running on it. I learned about deploying the app from Visual Studio to the Xbox, as described at Set up your UWP on Xbox development environment, and successfully entered the VS pin when requested. Following that, I got stuck. All my attempts to actually deploy the app ended with me being told "Failed to launch remote debugger with the following error: 'Command failed: 0x800705b4'". I then spent 90 minutes scouring the web for tips on how to deal with this, but was not successful. (I must admit that by the end of that, I felt my original attempts to publish the app without testing it, weren't perhaps that outrageous after all.) Since that day when I hit the 0x800705b4 problem, it was suggested to me that given that this is a ERROR_TIMEOUT, I should consider moving to a wired, rather than wireless connection. And in fact after reviewing Create your first app again, it does say "A wired network connection is recommended". I've yet to try that, but it's next on my list of things to do…
With my attempt to debug the app on hold, I instead installed the app from my laptop through the Dev Portal, and ran it on the Xbox. Not being able to debug made diagnosing problems rather inefficient, but I could still make a ton of progress anyway. The first interesting thing I learned was that the app failed to start. This was because I'd not set up the sandboxing as required. But once I'd set up the Xbox's sandbox to match that of the Sa11ytaire app I was installing, the app started fine.
The next step was to add the Xbox Live sign-in code for app start-up, and so copied in the C# code sample at Authentication for UWP projects. Initially, this seemed to work great. I ran the app, worked through various sign-in-related UI on app start-up, and displayed the player tag in the UI. At this point, I thought I was good to go. I then found that after playing the game for a while, it would crash at apparently random moments. As I recall, I was encountering a NullReferenceException beneath Microsoft.Xbox.Services.dll, with none of my own code shown in the callstack. I have no clue what was going on there, but after some experimenting, I found that if I removed the creation of the unused XboxLiveContext object, the app stopped crashing. As such, for now, I'll not be creating an XboxLiveContext object.
And so I'd reached the point where I had an Xbox app that could be published to the Microsoft Store, and could charge ahead on considering the switch control and Narrator experiences.
Switch device control
The switch device control of the Sa11ytaire app on the desktop, worked through Space key input. I tested this with a switch device and an adaptor, such that in response to a switch device press, the app would receive the Space key input. (Limiting the type of input to the Space key on the desktop was only due to time constraints.) In order to test this out on the Xbox, I hooked up the Y button to toggle the state of switch control, and pressed the Space key on my controller's chatpad. As a result, the switch control scan highlight cycled through the UI, and I could play the game with the controller.
But supporting only a chatpad's Space key input is not sufficient for the app on Xbox, so I hooked up the left and right bumpers to control the app when switch control is active. I did this by adding a Window.Current.CoreWindow.KeyDown handler, and responding to presses of VirtualKey.GamepadLeftShoulder/GamepadRightShoulder. (The original Space key action was triggered in response to the page's overridden OnPreviewKeyDown() being called.) All in all, that seemed pretty straightforward.
One additional modification that was required related to interaction with standard buttons in a ContentDialog. (For example, the "Are you sure you want to restart the game?" dialog.) When switch device control is enabled, when one of those dialogs appears, the app simply moves keyboard focus between the buttons, and whenever the Space key is pressed, then the focused button gets invoked. So the fact that this worked on the desktop was a side-effect of having the app react to a Space key press when switch control is active. In order to make this work on Xbox, I added the following code, to be run when a bumper is pressed and a ContentDialog is up. (Sorry about the dodgy indentation of the code. The blog site won't seem to let me fix that.)
var buttonWithFocus = FocusManager.GetFocusedElement() as Button;
if (buttonWithFocus != null)
{
ButtonAutomationPeer peer = FrameworkElementAutomationPeer.FromElement(buttonWithFocus) as ButtonAutomationPeer;
if (peer != null)
{
peer.Invoke();
}
}
I don't actually remember building a UWP XAML app that invokes one of its own buttons through UIA, but it seemed to work ok. (Note that I've not yet had a chance to update the app such that its appbar UI is usable with an Xbox controller.)
Now I could get to the bit that I was really excited about: Playing the game with the Xbox Adaptive Controller. Playing the game using a bumper or any other specific button on the controller might be fine for some players, but I want to make the game playable for as many players as I can. I've pre-ordered an Adaptive Controller, but it won't arrive for a while yet. So a colleague kindly lent me a device, and I tried it out. The Adaptive Controller supports customization such that its big buttons can be configured to effectively become other buttons, and I did consider doing that. But instead, I just paired the Adaptive Controller with my Xbox, plugged a switch device into the back of it, and hey-presto, I could play Sa11ytaire on my Xbox with a switch device. That was just fantastic!
I can't wait for my own Adaptive Controller to arrive now, so I can get familiar with all that it can do.
Figure 2: The Sa11ytaire app being played with the right bumper of a Xbox controller. A ten of diamonds and nine of clubs are being moved on to a jack of spades.
Narrator
As far as Narrator goes, I pretty much knew what the experience was going to be like with Sa11ytaire on the Xbox. After all, the same Narrator, UIA and UWP XAML is running on both the Windows desktop and Xbox. We'd built Sa11ytaire on the desktop to have Narrator make specific announcements which we felt would be valuable to the player, and those announcements would also be made on the Xbox. That said, we did adjust the experience, simply due to further consideration while we played the game.
For example, I added an announcement to confirm that the game had been restarted by the player. When I originally added that I felt that the announcement was too important to be truncated by another announcement, and so when I raised the related notification event, I supplied ImportantAll. That turned out to be the wrong thing to do. Say a player restarts the game many times in quick succession. With my change, an entire "Game restarted" announcement will be made for each restart of the game. The game and the player may well be ready for further play long before the full set of announcements have completed. So that's a tedious experience. As it happens, Tim caught this error and fixed it by replacing ImportantAll with ImportantMostRecent. (Thanks Tim!)
Another change Tim made relates to the announcement on whether a move is available between dealt card piles. Previously if no move is available there would be no announcement, and so the player might wonder if they really issued the command to learn about available moves or not. So now, if no moves are available, Narrator announces "No move is available".
By the way, I also removed some of the UIA Help properties from a few elements. When the game was originally built, I think I went over the top with the Help properties. I was effectively stuffing the instructions on how to play the game into the Help properties, and I don't think that's appropriate. It led to far too much information being announced simply when trying to play the game.
I'd say the most interesting aspect of using Narrator when porting Sa11ytaire to the Xbox, was how the controller should be used as a form of input at the game. With the D-pad working great, and the A button interacting as required with the focused card by default, the game was usable. I could move to one card, press A to select it, move to another card, and press A to move the first card over to it.
But this is where the classic question arises: Is this the most efficient experience that can be delivered?
The desktop app supports F6 to move between the three main groups in the app. So I updated the app such that a press of the left/right bumpers, (when switch device control is not enabled,) would move focus to the first control in the previous/next group respectively. What's more, given that the access keys on the desktop can provide for some players an extremely efficient means of playing the game, I said that once a bumper is pressed, the keys on the chatpad keyboard would behave in the same way as the access keys on the desktop. For example, press N to turn over a new card, or press U then C to move the upturned card to the Clubs pile, or press 4 then 6 to move a card from the fourth dealt card pile to the sixth dealt card pile.
On the desktop, function keys are used for a variety of actions, and so it was fun to consider how a controller might be used to access that same functionality. In the end, I implemented access to the function key functionality using a mix of buttons on the controller.
By the end of all this, this is how the Sa11ytaire app reacts to input at the Xbox controller:
- D-Pad: Moves keyboard focus left/right/up/down through the app.
-
A Button:
- Invoke the Next Card button.
- Check the Upturned Card button or Target Card Pile buttons.
- Select an item in the Dealt Card Pile lists.
- LeftThumbstick Button: Replicates A button action, in order for the game to be playable with controls only on the left side of the controller.
-
Left/Right Bumpers:
- If switch device control is not enabled, moves keyboard focus to the first card in the three main areas in the app. (Equivalent to Shift+F6 or F6 on the desktop.)
- If switch device control is enabled, then triggers the switch device action in the app. (Equivalent to Space key.)
- RightThumbstick Left/Up/Right: Have Narrator announce the state of the remaining cards, target card piles, or dealt card piles respectively. (Equivalent to F2/3/4.)
- RightThumbstick Down: Have Narrator announce the hint on whether a card can be moved between the dealt card pile lists. (Equivalent to F7.)
- RightThumbstick Button: Restart game. (Equivalent to F5.)
- Y Button: Toggle the state of switch device control. (Equivalent to F10.)
- X Button: Deselect and uncheck everything. (Equivalent to Escape.)
Figure 3: The Sa11ytaire app running with Narrator on an Xbox. The four of clubs is being moved onto the five of diamonds.
Summary
I'm really excited to have been able to port the Sa11ytaire experiment to the Xbox. While it was a fun learning experience for me to have published an Xbox game to the Microsoft Store for the first time, what I'm so thrilled about is to now have the opportunity to learn from game players as to what the app really needs to do if it's to provide an efficient experience in practice for players using Narrator or a switch device. And if some of the feedback we get is "You're nowhere near that", then that's exactly what we need to know. And who knows, maybe for some players, we're already pretty close today.
So let us know, and together we can all be a part of delivering that great experience on Xbox.
Guy