Make practical use of NSG Flow Logs for troubleshooting

So, you’ve found the new Azure Network Watcher features.
Cool, now I can start getting some real information about what’s going on in my Azure Network.

The NSG flow logs section of the Network Watcher blade, in the Azure Portal, lets you specify a Storage Account for each NSG (Network Security Group) to output detailed information on all the traffic hitting the NSG.
Note: The Storage Account must be in the same region as the NSG

Awesome right!
Then you try and use that information and find there are a heap of sub-folders created for each Resource Group, NSG all with separate files for each hour of logging.
Also, they are all in JSON format and really hard to find anything specific.

I don’t know about you, but I found it really hard to troubleshoot a particular network communication issue with a specific VM and/or Subnet.

So, I’ve written the following Powershell script that pulls all these files from the Storage Account down to a local folder and then builds an array of these ‘Events’.
The properties built out into the array are –
ResourceGroup, NSG, Ref, SrcIP, DstIP, SrcPort, DstPort, Protocol ‘T(TCP)/U(UDP)’, Direction ‘O(Outbound)/I(Inbound)’, Type ‘A(Accept)/D(Denied)’, Time, RuleType, and RuleName.

This is then presented in a GridView that can be sorted and filtered to find what you are looking for.
I’ve also added an option at the end to dump it out to a CSV if you need to send the output to someone else.

#Enter the Storage Account Name where the NSG Flow Logs have been configured to be stored
$StorageAccountName = "jrtnetworkwatcher"

$LocalFolder = "$($env:TEMP)AzureNSGFlowLogs"
#If you want to override where the local copy of the NSG Flow Log files go to, un-comment the following line and update
#$LocalFolder = "C:TempAzureNSGFlowLogs"

$TimeZoneStr = (Get-TimeZone).Id
#If you want to override the Timezone to something other than the locally detected one, un-comment the following line and update
#$TimeZoneStr = "AUS Eastern Standard Time"

$StorageResource = Find-AzureRmResource -ResourceNameContains $StorageAccountName -ResourceType Microsoft.Storage/storageAccounts
$ResourceGroupName = $StorageResource.ResourceGroupName

$StoKeys = Get-AzureRmStorageAccountKey -ResourceGroupName $ResourceGroupName -Name $StorageAccountName
$StorageAccountKey = $StoKeys[0].Value
$SubscriptionID = (Get-AzureRmContext).Subscription.Id

$StorageContext = New-AzureStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey

#Function to create the sub folders as required
function MakeFolders (
        [object]$Path,
        [string]$root
    )
{
    if (!(Test-Path -Path $root)) {
        $rootsplit = $root.Split("")
        $tPath = ''
        foreach ($tPath in $rootsplit) {
            $BuildPath = "$($BuildPath)$($tPath)"
            $BuildPath
            if (!(Test-Path -Path $BuildPath)) {
                mkdir $BuildPath
            }
        }
    }
    $Build = "$($root)"
    foreach ($fld in $path) {
        $Build = "$($Build)$($fld)"
        #$Build
        mkdir "$($build)" -ErrorAction SilentlyContinue
    }
}
#End function MakeFolders

#Function to get the Timezone information
function GetTZ (
        [string]$TZ_string
    )
{

    $r = [regex] "[([^[]*)]"
    $match = $r.match($($TZ_string))

    # If there is a successful match for a Timezone ID
    if ($match.Success) {
        $TZId = $match.Groups[1].Value
        # Try and get a valid TimeZone entry for the matched TimeZone Id
        try {

            $TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById($TZId)
        }
        # Otherwise assume UTC
        catch {

            $TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById("UTC")
        }
    } else {
        try {
            $TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById($TZ_string)
        } catch {
            $TZ = [System.TimeZoneInfo]::FindSystemTimeZoneById("UTC")
        }
    }
    return $TZ
}
#end function GetTZ

#Set the $TZ variable up with the required TimeZone information
$TZ = GetTZ -TZ_string $TimeZoneStr

#Get all the blobs in the insights-logs-networksecuritygroupflowevent container from the specified Storage Account
$blobs = Get-AzureStorageBlob -Container "insights-logs-networksecuritygroupflowevent" -Context $StorageContext

#Build an array of the available selection criteria
$AvailSelection = @()
foreach ($blobpath in $blobs.name) {
    $PathSplit = $blobpath.Split("/")
    $datestr = Get-Date -Year $PathSplit[9].Substring(2) -Month $PathSplit[10].Substring(2) -Day $PathSplit[11].Substring(2) -Hour $PathSplit[12].Substring(2) -Minute $PathSplit[13].Substring(2) -Second 0 -Millisecond 0 -Format "yyyy-MM-ddTHH:mm:ssZ"#2017-07-06T02:53:02.3410000Z
    $blobdate = (get-date $datestr).ToUniversalTime()
    #Write-Output "RG: $($PathSplit[4])`tNSG:$($PathSplit[8])`tDate:$($blobdate)"
    $SelectData = New-Object psobject -Property @{
        ResourceGroup = $PathSplit[4]
        NSG = $PathSplit[8]
        Date = $blobdate
    }
    $AvailSelection += $SelectData
}

#Prompt user to select the required ResourceGroup(s), NSG(s), and hourly sectioned files
$SelectedResourceGroups = ($AvailSelection | Select-Object -Property "ResourceGroup" -Unique | Out-GridView -Title "Select required Resource Group(s)" -PassThru).ResourceGroup
$selectedNSGs = ($AvailSelection | Where-Object {$_.ResourceGroup -in $SelectedResourceGroups} | Select-Object -Property "NSG" -Unique | Out-GridView -Title "Select required NSG(s)" -PassThru).NSG
$SelectedDates = ($AvailSelection | Where-Object {$_.ResourceGroup -in $SelectedResourceGroups -and $_.NSG -in $selectedNSGs} | Select-Object -Property @{n="System DateTime";e={$_.Date}},@{n="Local DateTime";e={[System.TimeZoneInfo]::ConvertTimeFromUtc((Get-Date $_.Date),$TZ)}} -Unique | Sort-Object Date -Descending| Out-GridView -Title "Select required times (1 hour blocks)" -PassThru).'System DateTime'

#Loop though blobs and download any that meet the specified selection and are newer than those already downloaded
foreach ($blob in $blobs) {
    $PathSplit = $blob.Name.Split("/")
    $blobdate = get-date -Year $PathSplit[9].Substring(2) -Month $PathSplit[10].Substring(2) -Day $PathSplit[11].Substring(2) -Hour $PathSplit[12].Substring(2) -Minute $PathSplit[13].Substring(2) -Second 0 -Millisecond 0
    if ($PathSplit[4] -in $SelectedResourceGroups -and $PathSplit[8] -in $selectedNSGs -and $blobdate -in $SelectedDates) {
        $fld = $blob.name.Replace("/","")
        $flds = $fld.Split("") | Where-Object {$_ -ne "PT1H.json"}
        $lcl = "$($LocalFolder)$($fld)"
        MakeFolders -Path $flds -root $LocalFolder
        if (Test-Path -Path $lcl) {
            $lclfile = Get-ItemProperty -Path $lcl
            $lcldate = (get-date $lclfile.LastWriteTimeUtc)
        } else {
            $lcldate = get-date "1 Jan 1970"
        }
        $blobdate = $blob.LastModified

        if ($blobdate -gt $lcldate) {
            Write-Output "Copied`t$($blob.Name)"
            Get-AzureStorageBlobContent -Container "insights-logs-networksecuritygroupflowevent" -Context $StorageContext -Blob $blob.Name -Destination $lcl -Force
        } else {
            Write-Output "Leave`t$($blob.Name)"
        }
    }
}

#Get a list of all the files in the specified local directory
$Files = dir -Path "$($LocalFolder)resourceId=SUBSCRIPTIONS$($SubscriptionID)RESOURCEGROUPS" -Filter "PT1H.json" -Recurse

#Loop through local files and build $Events array up with files that meet the selected criteria.
$Events=@()
foreach ($file in $Files) {
    $PathSplit = ($file.DirectoryName.Replace("$($LocalFolder)",'')).split("")
    $blobdate = get-date -Year $PathSplit[9].Substring(2) -Month $PathSplit[10].Substring(2) -Day $PathSplit[11].Substring(2) -Hour $PathSplit[12].Substring(2) -Minute $PathSplit[13].Substring(2) -Second 0 -Millisecond 0
    $blobResourceGroup = $PathSplit[4]
    $blobNSG = $PathSplit[8]
    if ($blobResourceGroup -in $SelectedResourceGroups -and $blobNSG -in $selectedNSGs -and $blobdate -in $SelectedDates) {
        $TestFile = $file.FullName
        $json = Get-Content -Raw -Path $TestFile | ConvertFrom-Json

        foreach ($entry in $json.records) {
            $time = (get-date $entry.time).ToUniversalTime()
            $time = [System.TimeZoneInfo]::ConvertTimeFromUtc($time, $TZ)
            foreach ($flow in $entry.properties.flows) {
                $rules = $flow.rule.split("_")
                $RuleType = $rules[0]
                $RuleName = $rules[1]
                foreach ($f in $flow.flows) {
                    $Header = "Ref","SrcIP","DstIP","SrcPort","DstPort","Protocol","Direction","Type"
                    $o = $f.flowTuples | ConvertFrom-Csv -Delimiter "," -Header $Header
                    $o = $o | select @{n='ResourceGroup';e={$blobResourceGroup}},@{n='NSG';e={$blobNSG}},Ref,SrcIP,DstIP,SrcPort,DstPort,Protocol,Direction,Type,@{n='Time';e={$time}},@{n="RuleType";e={$RuleType}},@{n="RuleName";e={$RuleName}}
                    $Events += $o
                }
            }
        }
    }
}
#Open $Events in GridView for user to see and filter as required
$Events | Sort-Object Time -Descending | Out-GridView

#Prompt to export to CSV.
$CSVExport = read-host "Do you want to export to excel? (Y/N)"
#If CSV required, export and open CSV file
if ($CSVExport.ToUpper() -eq "Y") {
    $FileName = "$($LocalFolder)NSGFlowLogs-$(Get-Date -Format "yyyyMMddHHmm").csv"
    $Events | Sort-Object Time -Descending | export-csv -Path $FileName -NoTypeInformation
    Start-Process "Explorer" -ArgumentList $FileName
}