In practice, there are no really compelling points against it. I'm not a fan, but that's a personal preference, there's no empirical data to suggest that the With construct is bad.
In .NET, it compiles to exactly the same code as fully-qualifying the object name, so there is no performance penalty for this sugar. I ascertained this by compiling, then disassembling, the following VB .NET 2.0 class:
Imports System.Text
Public Class Class1
    Public Sub Foo()
        Dim sb As New StringBuilder
        With sb
            .Append("foo")
            .Append("bar")
            .Append("zap")
        End With
        Dim sb2 As New StringBuilder
        sb2.Append("foo")
        sb2.Append("bar")
        sb2.Append("zap")
    End Sub
End Class
The disassembly is as follows -- note that the calls to sb2's Append method look identical to the With statement calls for sb:
.method public instance void  Foo() cil managed
{
  // Code size       91 (0x5b)
  .maxstack  2
  .locals init ([0] class [mscorlib]System.Text.StringBuilder sb,
           [1] class [mscorlib]System.Text.StringBuilder sb2,
           [2] class [mscorlib]System.Text.StringBuilder VB$t_ref$L0)
  IL_0000:  nop
  IL_0001:  newobj     instance void [mscorlib]System.Text.StringBuilder::.ctor()
  IL_0006:  stloc.0
  IL_0007:  ldloc.0
  IL_0008:  stloc.2
  IL_0009:  ldloc.2
  IL_000a:  ldstr      "foo"
  IL_000f:  callvirt   instance class [mscorlib]System.Text.StringBuilder [mscorlib]System.Text.StringBuilder::Append(string)
  IL_0014:  pop
  IL_0015:  ldloc.2
  IL_0016:  ldstr      "bar"
  IL_001b:  callvirt   instance class [mscorlib]System.Text.StringBuilder [mscorlib]System.Text.StringBuilder::Append(string)
  IL_0020:  pop
  IL_0021:  ldloc.2
  IL_0022:  ldstr      "zap"
  IL_0027:  callvirt   instance class [mscorlib]System.Text.StringBuilder [mscorlib]System.Text.StringBuilder::Append(string)
  IL_002c:  pop
  IL_002d:  ldnull
  IL_002e:  stloc.2
  IL_002f:  newobj     instance void [mscorlib]System.Text.StringBuilder::.ctor()
  IL_0034:  stloc.1
  IL_0035:  ldloc.1
  IL_0036:  ldstr      "foo"
  IL_003b:  callvirt   instance class [mscorlib]System.Text.StringBuilder [mscorlib]System.Text.StringBuilder::Append(string)
  IL_0040:  pop
  IL_0041:  ldloc.1
  IL_0042:  ldstr      "bar"
  IL_0047:  callvirt   instance class [mscorlib]System.Text.StringBuilder [mscorlib]System.Text.StringBuilder::Append(string)
  IL_004c:  pop
  IL_004d:  ldloc.1
  IL_004e:  ldstr      "zap"
  IL_0053:  callvirt   instance class [mscorlib]System.Text.StringBuilder [mscorlib]System.Text.StringBuilder::Append(string)
  IL_0058:  pop
  IL_0059:  nop
  IL_005a:  ret
} // end of method Class1::Foo
So if you like it, and find it more readable, go for it; there's no compelling reason not to. 
(By the way, Tom, I'm interested in knowing what happened with the debugger -- I can't recall ever seeing any unusual behavior in the debugger based on a With statement, so I'm curious to know what behavior you did see.)