How AOS permissions to the TempDB are handled in Dynamics AX 2012

With Dynamiscs AX 2012, the concept of TempDB temporary tables was introduced to leverage the power of SQL Server. The TempDB database can store temporary tables as opposed to the traditional InMemory tables. But this technique might cause issues if the access to the TempDB database for the domain account running the AOS service is not correctly setup, specifically in scenarios involving SQL AlwaysOn Availability Groups. When the failover occurs from the Primary to the Secondary Replicas, the users might see the following error message on the Dynamics AX client:

Grant TempDB permissions to the AOS account

First, to understand why this error is being thrown, it is important to explain how the permissions to the TempDB database are handled. Looking at the permissions of TempDB, we can see that the AOS needs the following permissions:

But where do these permissions come from? And another issue with the use of the TempDB is that every time the SQL Server instance is restarted, the TempDB is recreated so all the permissions are gone. Now let’s see how we can ensure these permissions end up in the TempDB database again.

Master database to the rescue

The answer to that question can be found in the Master database. When you install an AOS instance, it will use the Master database to automatically host the Stored Procedure for the AOS account as this is being persisted after a restart of the SQL Server instance. The execution of this Stored Procedure will grant the AOS rights to the TempDB database. If you have multiple AOS instances in your environment, you will find a Stored Procedure for each one of the AOS instances.

Peeking into the Stored Procedure, we see that it grants the AOS access rights to TempDB.

USE [master]
GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO

CREATE procedure [dbo].[CREATETEMPDBPERMISSIONS_WIN-40AOAH4EN9P_01] as begin exec (‘USE tempdb; declare @dbaccesscount int; exec sp_grantlogin ”<yourDomain><yourAOSServiceAccount>”; select @dbaccesscount = COUNT(*) from master..syslogins where name = ”<yourDomain><yourAOSServiceAccount>”; if (@dbaccesscount <> 0) exec sp_grantdbaccess ”<yourDomain><yourAOSServiceAccount>”; ALTER USER [<yourDomain><yourAOSServiceAccount>] WITH DEFAULT_SCHEMA=dbo; exec sp_addrolemember ”db_ddladmin”, ”<yourDomain><yourAOSServiceAccount>”; exec sp_addrolemember ”db_datareader”, ”<yourDomain><yourAOSServiceAccount>”; exec sp_addrolemember ”db_datawriter”, ”<yourDomain><yourAOSServiceAccount>”;’) end
GO

And by now, the next question pops up: When is this Stored Procedure called and by who? It is often thought that the AOS calls the Stored Procedure when it tries to access the TempDB, but this is incorrect! There is one crucial detail missing and it depends on how you get to the source of the Stored Procedure. If you right click the procedure and select Script -> Create To -> new query window, then you will get an extra line of source code.

EXEC sp_procoption N'[dbo].[ CREATETEMPDBPERMISSIONS_<yourAOSServerHostName>]’, ‘startup’, ‘1’
GO

This line of code is responsible to alter the properties stored in the Sys.Objects system table for the object referring to the Stored Procedure and marks it to be executed on startup. Therefore, restarting the SQL Server instance will immediately trigger the execution of the Stored Procedure and this will handle the AOS access rights to the TempDB.

One last thing that needs to be done is making sure that the AOS account also has the necessary rights to the Stored Procedure. In the properties of the Stored Procedure, go to the Permissions tab. Now add the AOS account and grant the permission to execute.

Possible error scenarios

Now that we have explained how the permissions are handled, we can describe some other issues you may face:

• The AOS account does not match the account in the Stored Procedure
  Cause: The AOS account has been switched to a user different from the one used at installation time.
  Resolution: Alter the Stored Procedure in the Master database to grant permissions to the currently used account and not the one registered in the Stored Procedure at installation time.
• You are using an Availability Group and one of the nodes does not have the Stored Procedure in place
  Cause: This is common when the Availability Group is already in place when the AOS is installed. The setup will do all the required steps on the active node during installation, but it will not automatically perform these actions on the secondary inactive node.
  Resolution: Recreate the same Stored Procedure on the other node. When there is a failover on the Availability Group, the right procedures are in place and will be triggered
• Scan for startup stored procedures have been disabled on the SQL Instance
  Cause: If the scan for startup procs option is set to 0, the automatic execution of Stored Procedures is not executed.
  Resolution: Re-enable the option by using the information available on the following links: https://technet.microsoft.com/en-us/library/ms179460.aspx https://technet.microsoft.com/en-us/library/ms191129.aspx

The next steps

Though this resolves the access issues the AOS might have to the TempDB database, there are also some considerations regarding the actual temporary tables residing in the TempDB during failover. The AOS keeps track of which temporary tables are stored in the TempDB. This is done because the tables are being truncated after usage instead of dropping them to enable reuse of the tables. When a failover occurs, these temporary tables are not present in the TempDB database of the failover node.

To resolve this, KB3141722 (R3) and KB4017756 (R2) were introduced. With this fix in place, a background thread within the AOS service will compare the internal pool of temporary tables with the actual contents of the TempDB. The following article was published by the support team and explains in detail how the fix works. https://blogs.msdn.microsoft.com/axsupport/2016/04/06/improved-high-availability-support-for-sql-clusters/

Greets,