The best, although still weird, solution using regex that I could come up with so far is below.
If I can figure out a solution that doesn't require to reverse the string, I'll update this.
RegexTest.java
import java.lang.StringBuilder;
public class RegexTest {
    public static String removeDupeLetters(String input) {
        // Because look-behinds have to be fixed width, we have to use look-aheads instead.
        // As a result of that we'll have to reverse the string and start from the end. (And then reverse the result again)
        return reverseString(reverseString(input).replaceAll("([A-Za-z])(?=.*\\1)", ""));
    }
    // helper function for reversing a String
    public static String reverseString(String input) {
        return new StringBuilder(input).reverse().toString();
    }
    public static void main(String[] args) {
        final String[] inputs = {"AaabBbc", "Banana", "Semester", "AaaaaaaaAaaaaa"};
        for (String input : inputs) {
            System.out.println("Input:  " + input);
            System.out.println("Output: " + removeDupeLetters(input));
            System.out.println();
        }
    }
}
javac RegexTest.java && java RegexTest
Input:  AaabBbc
Output: AabBc
Input:  Banana
Output: Ban
Input:  Semester
Output: Semstr
Input:  AaaaaaaaAaaaaa
Output: Aa
Note: As pointed out in the comments, it probably makes more sense to use a solution that does not involve regex at all, see link... was fun anyway though :D