Discover input/output paths of U-SQL jobs using Azure PowerShell

Do you use Azure Data Lake Analytics? Have you ever wanted to programmatically fetch the input and output paths of one of your U-SQL jobs? Now you can!

This is especially useful for developers who want to quickly debug an input file, fetch the results of an analysis, or check which files a particular job touched. With this new script, we’re helping those users by providing a simple Azure PowerShell script that reduces several steps into a one-line command.

How it works

Each time you submit a U-SQL job, a job folder is created in your Azure Data Lake Store account. This folder contains useful debugging information about the job, including a file called the U-SQL algebra file. This is an XML file containing information about your job graph, the list of input and output files, and other key U-SQL job metadata.

We’ve just published a sample script that reads the U-SQL algebra file for a specified job and returns the input or output files. Give it a try!

Running the sample

You can run this on a Windows machine with Azure PowerShell installed or via the Azure Cloud Shell.

1. First, find the Job ID of the job for which you’d like to get the input or output paths. You can easily find this in the Azure Portal by opening your ADLA account, navigating to the desired job, and copying the Job ID.
   
   Alternatively, you can get the job ID of a job that you just submitted via Azure PowerShell.

     PS C:> $job = Submit-AdlJob -Account contosoadla -ScriptPath myScript.usql
     PS C:> $job.JobId
   
     Guid
     ----
     c3426e86-85f5-4521-b376-e4b3e8d32d8c

2. Copy the contents of the sample script to a new local file called Get-AdlJobIoPath.ps1
3. Open a new PowerShell window. Navigate to the folder containing the sample script.
4. Log in to Azure.

     PS C:> Login-AzureRmAccount

5. Run the following command in PowerShell, replacing the bolded parts with your own values. This example gets the input file and table paths of the U-SQL job with Job ID c342....

     PS C:> .Get-AdlJobIoPath.ps1 -Account contosoadla -Id c3426e86-85f5-4521-b376-e4b3e8d32d8c -Direction Input
   
     Type  Path
     ----  ----
     Table master.dbo.TransactionsTbl
     File  adl://contosoadls.azuredatalakestore.net/data/SalesData.csv

Try the sample script today!

The source code for the sample is available in GitHub.

Soon, U-SQL job input and output information will be more readily accessible through our REST API, SDKs, and other interfaces.

Let us know your feedback in the comments. Are you looking for any other samples, features, or improvements? Let us know and vote for them on our UserVoice.