The sort algorithm (QuickSort) relies on a predictable IComparer implementation.  After a few dozen layers of indirection in the BCL you end up at this method:
public void Sort(T[] keys, int index, int length, IComparer<T> comparer)
{
    try
    {
        ...
        ArraySortHelper<T>.QuickSort(keys, index, index + (length - 1), comparer);
    }
    catch (IndexOutOfRangeException)
    {
        ...
        throw new ArgumentException(Environment.GetResourceString("Arg_BogusIComparer", values));
    }
}
Going a bit further into the QuickSort implementation, you see code like this:
    while (comparer.Compare(keys[a], y) < 0)
    {
        a++;
    }
    while (comparer.Compare(y, keys[b]) < 0)
    {
        b--;
    }
Basically if the IComparer misbehaves the Quicksort call with throw an IndexOutOfRangeException, which is wrapped in n ArgumentException.
Here is another example of bad IComparer's
class Comparer: IComparer<int>
{
    public int Compare(int x, int y)
    {
        return -1;
    }
}
So I guess, the short answer is, anytime your IComparer implementation does not consistently compare values as defined in the documentation:
Compares two objects and returns a
  value indicating whether one is less
  than, equal to or greater than the
  other.