56

It has generally been the case the Java source code has been forward compatible. Until Java 8, as far as I know, both compiled classes and source have been forward compatible with later JDK/JVM releases. [Update: this is not correct, see comments re 'enum', etc, below.] However, with the addition of default methods in Java 8 this appears to no longer be the case.

For example, a library I have been using has an implementation of java.util.List which includes a List<V> sort(). This method returns a copy of the contents of the list sorted. This library, deployed as a jar file dependency, worked fine in a project being built using JDK 1.8.

However, later I had occasion to recompile the library itself using JDK 1.8 and I found the library no longer compiles: the List-implementing class with its own sort() method now conflicts with the Java 8 java.util.List.sort() default method. The Java 8 sort() default method sorts the list in place (returns void); my library's sort() method - since it returns a new sorted list - has an incompatible signature.

So my basic question is:

  • Doesn't JDK 1.8 introduce a forward incompatibility for Java source code due to default methods?

Also:

  • Is this the first such forward incompatible change?
  • Was this considered or discussed when default methods where designed and implemented? Is it documented anywhere?
  • Was the (admittedly small) inconvenience discounted versus the benefits?

The following is an example of some code that compiles and runs under 1.7 and runs under 1.8 - but does not compile under 1.8:

import java.util.*;

public final class Sort8 {

    public static void main(String[] args) {
        SortableList<String> l = new SortableList<String>(Arrays.asList(args));
        System.out.println("unsorted: "+l);
        SortableList<String> s = l.sort(Collections.reverseOrder());
        System.out.println("sorted  : "+s);
    }

    public static class SortableList<V> extends ArrayList<V> {

        public SortableList() { super(); }
        public SortableList(Collection<? extends V> col) { super(col); }

        public SortableList<V> sort(Comparator<? super V> cmp) {
            SortableList<V> l = new SortableList<V>();
            l.addAll(this);
            Collections.sort(l, cmp);
            return l;
        }

    }

}

The following shows this code being compiled (or failing to) and being run.

> c:\tools\jdk1.7.0_10\bin\javac Sort8.java

> c:\tools\jdk1.7.0_10\bin\java Sort8 this is a test
unsorted: [this, is, a, test]
sorted  : [this, test, is, a]

> c:\tools\jdk1.8.0_05\bin\java Sort8 this is a test
unsorted: [this, is, a, test]
sorted  : [this, test, is, a]

> del Sort8*.class

> c:\tools\jdk1.8.0_05\bin\javac Sort8.java
Sort8.java:46: error: sort(Comparator<? super V>) in SortableList cannot implement sort(Comparator<? super E>) in List
                public SortableList<V> sort(Comparator<? super V> cmp) {
                                       ^
  return type SortableList<V> is not compatible with void
  where V,E are type-variables:
    V extends Object declared in class SortableList
    E extends Object declared in interface List
1 error
Paul
  • 3,009
  • 16
  • 33
  • 3
    `Doesn't JDK 1.8 introduce a forward incompatibility for Java source code due to default methods?` Yes, it does. :) In fact it does more than that, it introduces a continuous opportunity to break forward source compatibility. And I agree the trade-off is worth it but I'd love to know what the arguments were. – biziclop Jul 02 '15 at 15:11
  • 6
    This could have happened before with modified superclasses. Now default methods make it *less likely* that changes to an interface will break subclasses. – Andy Thomas Jul 02 '15 at 15:12
  • 4
    The problem doesn't really have to do with the addition of default methods. The problem would still have occurred even if it wasn't a default method, but just a new method on the interface without implementation. – Mark Rotteveel Jul 02 '15 at 15:13
  • 1
    @MarkRotteveel Before default methods interfaces couldn't and didn't evolve because it would've broken binary compatibility, and that was seen as the ultimate wrong. Of course technically they could've gone ahead and added new methods to interfaces but they didn't. – biziclop Jul 02 '15 at 15:15
  • 1
    @biziclop It works the same: if a new method was added to the interface, a version compiled against the previous Java version will work fine when used under the new Java version, but recompile would also fail. From that perspective nothing has changed with the introduction of default methods. The problem here is simply one of incompatible method signatures. – Mark Rotteveel Jul 02 '15 at 15:17
  • 2
    @Andy Good point. I think I missed that fact because I have tended to design and code with interfaces rather than subclassing, for the most part. I suppose it means the class name-space 'pollution' you could get from evolving base classes you can now also get from implementing interfaces too. – Paul Jul 02 '15 at 15:23
  • 2
    The difference is that in Java*, a new method called "sort" was added to the java.util.List interface. So if you wrote a class (using Java 7 or earlier) that implemented the List interface, and your class had a method called "sort", then there is now a conflict unless your sort method had the same signature as the new one added in Java8. This is not really a backward-compatibility issue. Any time a new method is added to an interface there will be a risk that user-written classes that implement that interface might have a method with the same name, and may break. – FredK Jul 02 '15 at 15:26
  • 2
    @MarkRotteveel Yes, of course, if you did provide a method with the same name and signature, it would work fine with or without default methods. The scenario I was talking about was when a new method was added to the interface and the implementer didn't provide an implementation for that method. Without default method, it would result in a `NoSuchMethodError` when someone tries to call the new method, default methods avoid that. And since default methods make it less of a risk to add new methods to interfaces, somewhat ironically it means more cases of this kind of signature clash. – biziclop Jul 02 '15 at 15:45
  • 16
    The tone of the question, and many of the comments here, are pretty overblown. All the risks introduced by default methods were _already present_ in introducing new methods into nonfinal concrete or abstract classes. Blaming default methods is mostly just shooting the messenger; its the same form of compatibility risk that always existed. (And yes -- the Expert Group deliberated on the compatibility consequences of this extensively.) – Brian Goetz Jul 02 '15 at 16:57
  • @Brian Understood - though I hadn't twigged that when I wrote the question. That said, I mostly don't create non-final or abstract classes if at all possible: they are nearly all final. For classes, at least, I type 'final' by habit. I do use interfaces a lot though. – Paul Jul 02 '15 at 17:04
  • 1
    This is a golden example of why programmers should prefer composition over inheritance. Extending a concrete class like **ArrayList** is especially problematic. – Alain O'Dea Jul 02 '15 at 17:47
  • 1
    @Alain I completely agree. The code I gave is a mocked up example. The real code where this problem surfaced was a `final` implementation of `List` that wrapped any other `List` implementation, proxying calls to it, and decorating it with about a 100 extra methods - one being `sort()`. (More of less - there was some extra complexity to allow the `RandomAccess` of the wrapped `List` to true of the wrapper too.) – Paul Jul 02 '15 at 17:57
  • 1
    @BrianGoetz I don't think anyone was envisioning massive risks, particularly not in the JDK. So far whenever I've come across real-life code that wouldn't compile because of this, it was always some not-very-well-thought-through contraption in a dire need of a rewrite anyway. But because collections are so fundamental to everything, we see a lot more of these issues than before, people are understandably worried a bit. Though I guess we won't have API changes on this scale with every single release, so this is a one-off spike. – biziclop Jul 02 '15 at 18:24

5 Answers5

59

Doesn't JDK 1.8 introduce a forward incompatibility for Java source code due to default methods?

Any new method in a superclass or interface can break compatibility. Default methods make it less likely that a change in an interface will break compatibility. In the sense that default methods open the door to adding methods to interfaces, you could say that default methods may contribute to some broken compatibility.

Is this the first such forward incompatible change?

Almost certainly not, since we've been subclassing classes from the standard library since Java 1.0.

Was this considered or discussed when default methods were designed and implemented? Is it documented anywhere?

Yes, it was considered. See Brian Goetz's August 2010 paper "Interface evolution via “public defender” methods":

  1. Source compatibility

It is possible that this scheme could introduce source incompatibilities to the extent that library interfaces are modified to insert new methods that are incompatible with methods in existing classes. (For example, if a class has a float-valued xyz() method and implements Collection, and we add an int-valued xyz() method to Collection, the existing class will no longer compile.)

Was the (admittedly small) inconvenience discounted versus the benefits?

Before, changing an interface would definitely break compatibility. Now, it might. Going from 'definitely' to 'might' can be seen either positively or negatively. On the one hand, it makes it feasible to add methods to interfaces. On the other hand, it opens the door to the kind of incompatibility you saw, not just with classes, but with interfaces too.

The benefits are larger than the inconveniences, though, as cited at the top of Goetz's paper:

  1. Problem statement

Once published, it is impossible to add methods to an interface without breaking existing implementations. The longer the time since a library has been published, the more likely it is that this restriction will cause grief for its maintainers.

The addition of closures to the Java language in JDK 7 place additional stress on the aging Collection interfaces; one of the most significant benefits of closures is that it enables the development of more powerful libraries. It would be disappointing to add a language feature that enables better libraries while at the same time not extending the core libraries to take advantage of that feature.

Andy Thomas
  • 84,978
  • 11
  • 107
  • 151
  • "Default methods make it less likely that a change in an interface will break compatibility" Whilst I take your point (previously a change to an interface signature would require all implementers to change) doesn't this depend on what happens in future? If interfaces all over the Java-verse start sprouting new default convenience methods, it seems at least possible that this will increase the chance of conflicts. (In the past adding to an interface was a big deal. Now it *seems* like it isn't. But if it has hundreds of implementers? Some of which may themselves be in libraries.) – Paul Jul 02 '15 at 15:40
  • 3
    @Paul - I agree, that is a danger. However, there is a middle course. Before, the prudent approach was to never change a public interface. (E.g., Bloch in *Effective Java*: "Once an interface is released and widely implemented, it is almost impossible to change.") Now, we could shout "it's the red hour!" and start throwing default methods in public interfaces willy-nilly. But I think the prudent approach is to recognize that we've gone from "will-break" to "might-break", and add default methods carefully and infrequently, when the benefits outweigh the chance of breaking some implementations. – Andy Thomas Jul 02 '15 at 15:49
  • 2
    Ironically, they had to resist the temptation of adding a lot of useful default methods to Collection APIs, for fear of breaking subclasses. The better question is why they thought it is absolutely necessary to add `sort` to `List`. – ZhongYu Jul 02 '15 at 16:52
  • 1
    @bayou.io: If it is possible to perform a complex action using nothing but "primitive" interface methods, but some implementations of the interface may, with the aid of their knowledge of their own internals, be able to perform the action *much more efficiently* than would be possible using primitive methods alone, then having a method to perform such action, and having a default implementation which performs the action using only interface primitives, is IMHO the best way to do things (though it would've been better if the such methods existed from the start, rather than being added later). – supercat Jul 02 '15 at 19:12
  • 1
    @bayou.io: Comparing and conditionally swapping two items of an array is apt to be much faster than loading two items from a `List`, comparing them, and if necessary storing two items back to the `List`. It is not uncommon for programs to spend a significant amount of their time sorting, and having a `List` sort itself may often offer a 50% or better speedup compared to implementing a sort that has to read and write items in a `List`. – supercat Jul 02 '15 at 19:16
  • @supercat - sure, but we have lived without `List.sort` for many years... I think it is quite odd, because it's a mutator; most commonly only the producer of the List does mutations; and the producer is not dealing with a black box. To convince me that `List.sort` is necessary, I'll have to see some real world use cases, and that they are quite common. – ZhongYu Jul 02 '15 at 20:00
  • 1
    @bayou.io: While I agree that in most cases code that wants to sort a list will know its exact type (typically `ArrayList`) but there can sometimes be considerable performance advantage to using other forms of backing storage. For example, if one has an immutable list type, it may be useful for it to include an `asMutable` method that returns a `List` which holds a reference to the immutable list along with some arrays which keep track of what has been changed. Code shouldn't need to care whether it receives such a list or an `ArrayList`, but I don't know what common supertype... – supercat Jul 02 '15 at 20:17
  • 1
    ...such a list could usefully share with `ArrayList`, other than the interface `List`. – supercat Jul 02 '15 at 20:17
  • @bayio.io "it is quite odd, because it's a mutator" Yes. That was the slightly annoying part: the method in my library that the `sort()` default method collided with was not a mutator method - it returned a new, sorted copy of the list. – Paul Aug 24 '15 at 08:39
  • 1
    @bayou.io: adding `sort` to `List` allows concrete `List` implementations to override it with an optimized implementation. That’s exactly what `ArrayList`, one of the most popular implementations, does. Even, if you keep using `Collections.sort(list)`, you’ll get the benefit, as it’s now delegating to `List.sort`. Which is even more dangerous as it can break existing, compiled code if that code happens to have a `sort` method in a `List` implementation that incidentally has the right signature but delegates to `Collections.sort`… – Holger Jan 13 '16 at 15:10
10

Doesn't JDK 1.8 introduce a forward incompatibility for Java source code due to default methods?

Yes as you've seen your self.

Is this the first such forward incompatible change?

No. Java 5 enumkeyword was also breaking because before that you could have variables named that which would no longer compile in Java 5 +

Was this considered or discussed when default methods where designed and implemented? Is it documented anywhere?

Yes Orcale Java 8 source incompatibility description

Was the (admittedly small) inconvenience discounted versus the benefits?

Yes

dkatzel
  • 31,188
  • 3
  • 63
  • 67
  • @Codebender True. I have tended not to code by subclassing when at all possible - I make most of my own classes final - which may explain why I've noticed it more now it applies to interfaces too. – Paul Jul 02 '15 at 15:25
  • @Paul It always applied to interfaces as well. – Mark Rotteveel Jul 02 '15 at 15:30
  • 1
    While that is true, Usually the JDK authors have been careful not to add methods to interfaces to avoid this problem. See `Graphics` vs `Graphics2D` as workarounds to add methods to sub-interfaces so legacy code isn't affected – dkatzel Jul 02 '15 at 15:31
  • @Codebencher. Yes, but I think people think long and hard about adding a method (pre-default methods) to an interface. I'm not aware of any library I've used that did this outside of a major new release/refactoring of the library. (Where as classes - abstract, base class, or otherwise - seem to sprout new methods at will.) – Paul Jul 02 '15 at 15:34
  • @dkatzel a well-known counter-example is JDBC – the interfaces there got new methods several times. – Paŭlo Ebermann Jul 02 '15 at 18:38
3

We can draw a parallel with abstract class. An abstract class is intended to be subclassed so that the abstract methods can be implemented. The abstract class itself contains concrete methods that invoke the abstract methods. The abstract class is free to evolve by adding more concrete methods; and this practice may break subclasses.

Therefore the exact problem you described existed even before Java8. The problem is much more manifested on Collection APIs because there are a lot of subclasses out in the wild.

While the leading motivation of default method was to add some useful methods to existing Collection APIs without breaking subclasses, they had to exercise great self-control of doing it too much, for fear of breaking subclasses. A default method is added only if it's absolutely necessary. The real question here is, why List.sort is considered absolutely necessary. I think that is debatable.

Regardless of why default method was introduced in the 1st place, it is now a great tool for API designers, and we ought to treat it the same as concrete methods in abstract classes - they need to be designed carefully up front; and new ones must be introduced with great caution.

ZhongYu
  • 19,446
  • 5
  • 33
  • 61
  • 1
    There's bound to be a "shiny new toy" phase, when everyone peppers their APIs with default methods (and it's oh so tempting to use them as traits, even though they're inadequate), then cautiousness takes over. Hopefully. :) – biziclop Jul 02 '15 at 18:30
2

Ironically default methods in interfaces were introduced to allow existing libraries using those interfaces not to break, while introducing massive new functionality in the interfaces. (backward compatibility.)

Conflicts like that sort method might arise. Something to pay for the extra functionality. In your case also something to investigate (should new functionality be used instead?).

Java forward compatibility breaks are little, more in its typing system, which was constantly enlarged. First with generic types and now with inferred types from functional interfaces. From version to version and from compiler to compiler there were slight differences.

Joop Eggen
  • 107,315
  • 7
  • 83
  • 138
  • "forward compatibility breaks are little, more in its typing system" Surely those are backward, rather than forward, compatibility problems? – Paul Jul 02 '15 at 15:46
  • @Paul yes, there were some compiler troubles with java 8, but that more concerned growing pains of the miscellaneous compilers. – Joop Eggen Jul 02 '15 at 17:44
0

Reading this issue, I was thinking of its solution.
Default methods have solved the backward compatibility problems but forward compatibility issues will exist.
I think instead of extending existing classes, in such cases, we can have our application specific interfaces to add some desired behaviour to our class. We can implement this application specific interface and use it.

pfulara
  • 41
  • 6