Separation of Microsoft Unit Tests to Reduce the Load of CI Server

In case you are wondering so many ‘failed builds’ (due to unit tests failure) on Continuous Integration Server today…

The reason is that we use msbuild testall.proj on CI server and lots of memory are eaten (by C#.NET managed code). The following is the MSBuild script that scans the directory for test projects (DLL) and use mstest.exe to launch all the tests (via parameters /testcontainer)

1
2
3
4
5
6
7
8
9
10
11

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="RunMSTest">
<ItemGroup>
    <TestAssemblies Include="**\bin\*Test*.dll"/>
</ItemGroup>
<Target Name="RunMSTest">
    <Exec Condition=" '@(TestAssemblies)' != ''"
          Command="Mstest.exe /nologo /resultsfile:TestResults.trx @
(TestAssemblies ->'/testcontainer:"%(RecursiveDir)%(Filename)%(Extension)"', ' ')"
          />
</Target>
</Project>

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="RunMSTest">
<ItemGroup>
    <TestAssemblies Include="**\bin\*Test*.dll"/>
</ItemGroup>
<Target Name="RunMSTest">
    <Exec Condition=" '@(TestAssemblies)' != ''"
          Command="Mstest.exe /nologo /resultsfile:TestResults.trx @
(TestAssemblies ->'/testcontainer:"%(RecursiveDir)%(Filename)%(Extension)"', ' ')"
          />
</Target>
</Project>

Today is an unlucky day that the number of unit tests reaches a threshold and everything seems to start breaking…

according to https://social.msdn.microsoft.com/Forums/en-US/947bd836-9a0c-4e97-b2a6-828dc93d29e5/mstest-vstesthost-memory-problem?forum=vststest

Visual Studio runs tests using vstesthost.exe, both through the IDE and the commandline. Vstesthost.exe will be killed only after the whole run finishes.”

and here is a snapshot that shows you how crazy this process (mstest) eats memory..

qtagent32.exe eats memory

That is why when Test project XXX gets its turn (alphabetically) it throws out out-of-resources exception (basically the same thing as out-of-memory).

The fix is to abandon the testall.proj instead, we use the following approach.

1. all test DLL are stored line by line in “tests.txt“, so if you are adding a new Test project, you would need to update this list as well.

Project\Test1\Test1.dll
Project\Test2\Test2.dll
...

2. the total number of passed, failed tests now exclude the ignored-tests, so you will notice a slightly decline in total number of unit tests.

3. each Test project is isolated and executed via separate mstest so the memory issue is resolved. Now with the separation of Test Project, the memory usage is kept low and stable. That explains why before, on the CI server, the unit tests sometimes failed for no reason. Imaging, that two jobs are running unit tests, and the two processes are using nearly 2GB each, which makes the CI server heavily loaded.

Anyway, I am glad that it is resolved and let’s wait a few days to see how it goes.

New Powershell Build Script Snippet

The following Powershell snippet reflects the changes:

    $cnt = 0
    $cntpass = 0
    $cntfailed = 0

    $listoftests = Join-Path $WorkingCopyDir "\tests.txt"

    if (Test-Path $listoftests) {
        $reader = [System.IO.File]::OpenText($listoftests)
        try {
            for(;;) {
                $line = $reader.ReadLine()
                if ($line -eq $null) { 
                    break 
                }
                $line = $line.Trim()
                $fullpath =  Join-Path $WorkingCopyDir "$line"
                Write-Host $fullpath
                if (Test-Path $fullpath) {
                    $projname = [System.IO.Path]::GetFileNameWithoutExtension($line)
                    $trx = Join-Path  $WorkingCopyDir "TestResults_$projname.trx"
                    Write-Host "Test File: $trx"
                    Write-Host Mstest.exe /nologo /resultsfile:`"$trx`" /testcontainer:`"$fullpath`" 
                    &Mstest.exe /nologo /resultsfile:"$trx" /testcontainer:"$fullpath"  # Launch separately each Unit Test Project
                    if (Test-Path $trx) { # Count the number of tests
                        $content = Get-Content $trx -Raw
                        $cnt = $cnt + ([regex]::Matches($content, "TestMethod\ " )).Count
                        $cntpass = $cntpass + ([regex]::Matches($content, "outcome=`"[Pp]assed`"" )).Count
                        Write-Host "*** " $cnt.ToString() " Unit Tests ***"
                        Write-Host "*** " $cntpass.ToString() " Passed Unit Tests ***"
                        Write-Host "*** " ($cnt - $cntpass).ToString() " Failed Unit Tests ***"
                    } else {
                        Write-Host "$trx error"
                    }
                }
            }
        }
        finally {
            $reader.Close()
        }
        Write-Host "Total Tests = " $cnt.ToString()
        Write-Host "Passed Tests = " $cntpass.ToString()
        Write-Host "Failed Tests = " $cntfailed.ToString()
        if ($cntfailed -gt 0) {
            throw "$cntfailed unit tests failed"
        }
    } else {
        Write-Host "$listoftests not found!"
    }  

–EOF (The Ultimate Computing & Technology Blog) —