2

A test class with the following test is discovered as expected:

[Theory]
[AutoData]
public void MyDiscoveredTest() { }

However, the following test is missing:

[Theory]
[AutoNSubstituteData]
public void MyMissingTest() { }

Interestingly, if I put MyDiscoveredTest after MyMissingTest, then MyDiscoveredTest is also now missing. I have tried both the xUnit visual studio runner and xUnit console runner with the same results.

My AutoNSubstituteData attribute is defined here:

internal class AutoNSubstituteDataAttribute : AutoDataAttribute
{
    internal AutoNSubstituteDataAttribute()
        : base(new Fixture().Customize(new AutoNSubstituteCustomization()))
    {
    }
}

A related question: since the AutoNSubstituteDataAttribute above seems like a fairly common attribute, I'm wondering why it's not included with AutoFixture.AutoNSubstitute. Similarly useful would be an InlineAutoNSubstituteDataAttribute. Should I submit a pull request for these?

Nuget package versions used:
AutoFixture 3.30.8
AutoFixture.Xunit2 3.30.8
AutoFixture.AutoNSubstitute 3.30.8
xunit 2.0.0
xunit.runner.visualstudio 2.0.0
xunit.runner.console 2.0.0
NSubstitute 1.8.2.0

I am using Visual Studio 2013 Update 4 and targeting the .NET 4.5.1 Framework

Update: As recommended I tried TestDriven.NET-3.9.2897 Beta 2. The missing test now runs, however it still seems there is some bug. New example:

[Theory]
[AutoData]
public void MyWorkingTest(string s)
{
    Assert.NotNull(s); // Pass
}

[Theory]
[AutoNSubstituteData]
public void MyBrokenTest(string s)
{
    Assert.NotNull(s); // Fail
}

[Theory]
[AutoData]
public void MyWorkingTestThatIsNowBroken(string s)
{
    Assert.NotNull(s); // Fail even though identical to MyWorkingTest above!
}

Both MyBrokenTest and MyWorkingTestThatIsNowBroken fail at Assert.NotNull, while MyWorkingTest passes even though it is identical to MyWorkingTestThatIsNowBroken. So not only does the AutoNSubstituteData attribute not work properly, but it is causing the downstream test to misbehave!

Update2: Changing the definition of AutoNSubstituteDataAttribute to public instead of internal fixes everything. xunit runner now discovers and passes all the tests as does TestDriven.Net. Any idea about this behavior? Is it expected?

Eric Roller
  • 429
  • 5
  • 19
  • This looks like a test-runner bug. Could you try running the test(s) using [TestDriven.Net](http://www.testdriven.net/)? – Nikos Baxevanis Jul 07 '15 at 22:20
  • While `Inline/AutoNSubstituteData` *looks* common, none of the auto-mocking Glue Libraries reference AutoFixture.Xunit, so that Pull Request wouldn't be possible. – Even if the auto-mocking Glue Libraries reference AutoFixture.Xunit, that attribute wouldn't make a lot of sense as, in most scenarios, you'd end up composing more than one AutoFixture Customizations in an `AutoDataAttribute` derived type. – Nikos Baxevanis Jul 07 '15 at 22:25
  • @NikosBaxevanis Thanks for the explanation. The implementation of `Inline/AutoNSubstituteData` is pretty simple so not a big deal to implement myself. By the way, I updated the question after trying TestDriven.Net; it seems something is still not working properly. Could it be a problem with xUnit? – Eric Roller Jul 08 '15 at 05:13
  • All the above tests are passing on my machine with the following NuGet packages: `AutoFixture.Xunit2`, `AutoFixture.AutoNSubstitute`, and `xunit`. Could you try on an empty Class Library project? – Nikos Baxevanis Jul 08 '15 at 06:42
  • Interesting, I'm seeing the same behavior on a clean solution and class library project. I'm going to also try a clean install of VS2013PremiumUpdate4 on a different system. Which version of Visual Studio are you using? – Eric Roller Jul 08 '15 at 12:56
  • Currently I use Version 12.0.31101.00 Update 4. It shouldn't be relevant though... – Nikos Baxevanis Jul 08 '15 at 13:38
  • Same problems with clean install of VS2013. I tried both xunit VS runner and TestDriven.Net. TestDriven.Net fails the last two tests. xunit VS runner only discovers the first test. Which runner are you using? Do you have the same implementation for `AutoNSubstituteDataAttribute` as above? – Eric Roller Jul 08 '15 at 13:52
  • I found a fix and updated the question. Not sure why public vs internal matters here. – Eric Roller Jul 08 '15 at 14:23
  • Test runners may not look for internal properties - why should they? I haven't looked into any particular test runners' implementation code to investigate, but I wouldn't *expect* a test runner to pick up any internal types. – Mark Seemann Jul 08 '15 at 17:28
  • 2
    Fair enough. But if the attribute is being ignored I would expect an error like this: `System.InvalidOperationException : No data found for` .. Also, doesn't explain why the downstream tests are affected by the offending `AutoNSubstituteData` attribute from a totally different test. Seems like the runners should be a bit more robust in this case. Thanks for the responses! – Eric Roller Jul 08 '15 at 17:49
  • @MarkSeemann and Eric: Interesting and pretty inconsistent given most runners manage to be consistent on other things http://stackoverflow.com/questions/16214684/why-is-the-xunit-runner-not-finding-my-tests – Ruben Bartelink Jul 09 '15 at 10:03

1 Answers1

4

Both xUnit visual studio runner and TestDriven.Net runner are causing these weird issues because the AutoNSubstituteDataAttribute class and constructor are internal. Changing these to public resolves all the issues. If the attribute is being ignored I would expect an error like this: System.InvalidOperationException : No data found for ...

This doesn't explain why the downstream tests are affected by the offending AutoNSubstituteData attribute from a totally different test. It seems like the unit test runners should be more robust in this case.

For completeness here is the working implementation of AutoNSubstituteDataAttribute:

public class AutoNSubstituteDataAttribute : AutoDataAttribute
{
    public AutoNSubstituteDataAttribute()
        : base(new Fixture().Customize(new AutoNSubstituteCustomization()))
    {
    }
}
Eric Roller
  • 429
  • 5
  • 19