Example 1
You declare generic types inside the angle brackets, not construct an instance of an object.
For this:
error: illegal start of type 
            List<List<Integer> >a =new ArrayList<new List<Integer>() >();
                                                 ^
This error shows that you cannot use the new keyword in a generic type.
Example 3
Your third example shows a similar error:
error: illegal start of type
            List<List<Integer> >a =new ArrayList<new ArrayList<Integer>() >();
                                                 ^
The generic type represents a class, not an instance.
Example 2
For the second example, the error message is telling you that the generic types must be the same.  You cannot apply inheritance like this for generic types:
error: incompatible types: ArrayList<ArrayList<Integer>> cannot be converted to List<List<Integer>>
            List<List<Integer> >a =new ArrayList<ArrayList<Integer> >();
                                   ^
Using generic wildcards
To declare a generic list of List subtypes, you could use the wildcard expression ? extends List as the generic type:
List<? extends List<Integer>> a = new ArrayList<ArrayList<Integer>>();
For more on generics oracle has a good overview and the Java Tutorial has a nice guide.