From the documentation, both the class and method used for the callback must be public:
Calling back to Java from JavaScript
The JSObject.setMember method is useful to enable upcalls from
  JavaScript into Java code, as illustrated by the following example.
  The Java code establishes a new JavaScript object named app. This
  object has one public member, the method exit.
public class JavaApplication {
    public void exit() {
        Platform.exit();
    }
}
...
JavaApplication javaApp = new JavaApplication();
JSObject window = (JSObject) webEngine.executeScript("window");
window.setMember("app", javaApp);
...
The Java class and method must both be declared public.
(My emphasis.)
Your Solution class is not public, so this won't work.
In addition, when a new document is loaded, the window will lose its attributes. Since loading happens asynchronously, you need to ensure the member is set on the window after the document loads. You can do this via a listener on the documentProperty():
    webEngine.documentProperty().addListener((obs, oldDoc, newDoc) -> {
        JSObject window = (JSObject) webEngine.executeScript("window");
        window.setMember("app", this);          
    });
    webEngine.load(Solution.class.getResource("/mypage.html").toString());
There are a number of other problems with your code:
- JFrames must be constructed on the AWT event dispatch thread (the same rule also applies to modifying components displayed in a- JFrame). You can do this by wrapping the call to- createUI()in- SwingUtilities.invokeLater(...).
- It is unclear why you made Solutiona subclass ofJFrame, as well as creating a newJFrameincreateUI(). Since you never use the fact thatSolutionsubclassesJFrame, you should remove that.
- PlatformImplis not part of the public API: consequently it would be perfectly OK for the JavaFX team to remove that class in a later release. You should use methods in the- Platformclass.
- You almost certainly want the Javascript callback to interact with the current Solutioninstance, not some arbitrary instance you create. (If you're in an inner class, useSolution.thisto access the current instance of the surrounding object.)
A working version of your code is
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.SwingUtilities;
import javafx.application.Platform;
import javafx.embed.swing.JFXPanel;
import javafx.scene.Scene;
import javafx.scene.layout.BorderPane;
import javafx.scene.web.WebEngine;
import javafx.scene.web.WebView;
import netscape.javascript.JSObject;
public class Solution  {
    private JFXPanel jfxPanel;
    public static void main(String[] args) {
        SwingUtilities.invokeLater(new Solution()::createUI);
    }
    private void createUI() {
        JFrame f = new JFrame("panel");
        f.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        JPanel p = new JPanel();
        jfxPanel = new JFXPanel();
        createScene();
        p.add(jfxPanel);
        f.add(p);
        f.setSize(300, 300);
        f.setVisible(true);
    }
    private void createScene() {
        Platform.setImplicitExit(false);
        Platform.runLater(() -> {
            BorderPane borderPane = new BorderPane();
            WebView webComponent = new WebView();
            WebEngine webEngine = webComponent.getEngine();
            webEngine.documentProperty().addListener((obs, oldDoc, newDoc) -> {
                JSObject window = (JSObject) webEngine.executeScript("window");
                window.setMember("app", this);          
            });
            webEngine.load(Solution.class.getResource("/mypage.html").toString());
            borderPane.setCenter(webComponent);
            Scene scene = new Scene(borderPane, 300, 300);
            jfxPanel.setScene(scene);
        });
    }
    public void onClick() {
        System.out.println("Invoked from JS");
    }
}