I got RSyntaxTextArea working in JavaFX using a SwingNode, but can't seem to get code completion to work. So to remove any JavaFX related problems I implemented a Java/Swing only version with no JavaFX whatsoever. Here's that code:
package com.mystuff.swingtest;
import java.awt.*;
import javax.swing.*;
import org.fife.ui.autocomplete.*;
import org.fife.ui.rsyntaxtextarea.RSyntaxTextArea;
import org.fife.ui.rsyntaxtextarea.SyntaxConstants;
import org.fife.ui.rtextarea.RTextScrollPane;
class AutoCompleteDemo extends JFrame {
    public AutoCompleteDemo() {
        JPanel contentPane = new JPanel(new BorderLayout());
        RSyntaxTextArea textArea = new RSyntaxTextArea(20, 60);
        textArea.setSyntaxEditingStyle(SyntaxConstants.SYNTAX_STYLE_JAVA);
        textArea.setCodeFoldingEnabled(true);
        contentPane.add(new RTextScrollPane(textArea));
        
        CompletionProvider provider = createCompletionProvider();
        AutoCompletion ac = new AutoCompletion(provider);
        ac.install(textArea);
        setContentPane(contentPane);
        setTitle("AutoComplete Demo");
        setDefaultCloseOperation(EXIT_ON_CLOSE);
        pack();
        setLocationRelativeTo(null);
    }
    /**
     * Create a simple provider that adds some Java-related completions.
     */
    private CompletionProvider createCompletionProvider() {
        DefaultCompletionProvider provider = new DefaultCompletionProvider();
        // Add completions for all Java keywords. A BasicCompletion is just
        // a straightforward word completion.
        provider.addCompletion(new BasicCompletion(provider, "abstract"));
        provider.addCompletion(new BasicCompletion(provider, "assert"));
        provider.addCompletion(new BasicCompletion(provider, "break"));
        provider.addCompletion(new BasicCompletion(provider, "case"));
        // ... etc ...
        provider.addCompletion(new BasicCompletion(provider, "transient"));
        provider.addCompletion(new BasicCompletion(provider, "try"));
        provider.addCompletion(new BasicCompletion(provider, "void"));
        provider.addCompletion(new BasicCompletion(provider, "volatile"));
        provider.addCompletion(new BasicCompletion(provider, "while"));
        // Add a couple of "shorthand" completions. These completions don't
        // require the input text to be the same thing as the replacement text.
        provider.addCompletion(new ShorthandCompletion(provider, "sysout",
                "System.out.println(", "System.out.println("));
        provider.addCompletion(new ShorthandCompletion(provider, "syserr",
                "System.err.println(", "System.err.println("));
        return provider;
    }
    public static void main(String[] args) {
        // Instantiate GUI on the EDT.
        SwingUtilities.invokeLater(() -> {
            try {
                String laf = UIManager.getSystemLookAndFeelClassName();
                UIManager.setLookAndFeel(laf);
            } catch (Exception e) { /* Never happens */ }
            new AutoCompleteDemo().setVisible(true);
        });
    }
}
But when I type sysout and hit crtl+SPACE nothing happens. No popup window or anything. I checked the keystroke combination from the AutoComplete object and it says "ctrl pressed SPACE" so I know the key combination is correct.
I'm using a Mac. Tried COMMAND, ALT instead of Control, but still nothing. Any ideas?
UPDATE 1
If I press SPACE and hold down ctrl in that order, the pop up flashes on and off and when I release ctrl the pop up stays on if I catch it correctly 
UPDATE 2
This led me to try changing the Trigger Key. I've not worked out the ctrl-whatever combination yet, but I tried
ac.setTriggerKey(KeyStroke.getKeyStroke('\t'));
and pressing TAB in the editor now brings up the pop-up window without the flashing. So I'll probably be able to fix the original problem if I can just find the right key combination for KeyStroke.