I'm working on a Java application with Swing-based GUI. The application uses JTextPane to output log messages as follows: 1) truncate existing text to keep total text size under the limit; 2) append new text; 3) scroll to the end (actual logic is slightly different, but it's irrelevant here).
I was using Eclipse with JVM Monitor to determine reasonable text size limit and found significant memory leak. I tried to remove UndoableEditListeners from the underlying document and disable automatic caret position updates (by changing the position explicitly with DefaultCaret.NEVER_UPDATE and JTextPane.setCaretPosition(int)), but without success. Finally, I decided to disable changing caret position completely, and this fixed the leak.
I have two questions:
Is there an issue with my code? If yes, how can I change it to accomplish the task?
Is it a Swing/JVM bug? If yes, how can I report it?
Details:
Here is SSCCE: GUI with textPane and two buttons, for small and stress tests. FIX and FIXXX flags correspond to my attempts to fix memory leak.
package memleak;
import java.awt.*;
import java.awt.event.*;
import javax.swing.*;
import javax.swing.event.UndoableEditListener;
import javax.swing.text.*;
class TestMain
{
  private JTextPane textPane;
  // try to fix memory leak
  private static final boolean FIX = false;
  // disable caret updates completely
  private static final boolean FIXXX = false;
  // number of strings to append
  private static final int ITER_SMALL = 20;
  private static final int ITER_HUGE = 1000000;
  // limit textPane content
  private static final int TEXT_SIZE_MAX = 100;
  TestMain()
  {
    JFrame frame = new JFrame();
    JPanel panel = new JPanel();
    textPane = new JTextPane();
    textPane.setEditable(false);
    if (FIX)
    {
      tryToFixMemory();
    } // end if FIX
    JScrollPane scrollPane = new JScrollPane(textPane);
    scrollPane.setPreferredSize(new Dimension(100, 100) );
    panel.add(scrollPane);
    JButton buttonSmall = new JButton("small test");
    buttonSmall.addActionListener(new ButtonHandler(ITER_SMALL) );
    panel.add(buttonSmall);
    JButton buttonStress = new JButton("stress test");
    buttonStress.addActionListener(new ButtonHandler(ITER_HUGE) );
    panel.add(buttonStress);
    frame.add(panel);
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
    frame.pack();
    frame.setVisible(true);
  } // end constructor
  public static void main(String[] args)
  {
    @SuppressWarnings("unused")
    TestMain testMain = new TestMain();
  } // end main
  private void append(String s)
  {
    Document doc = textPane.getDocument();
    try
    {
      int extraLength = doc.getLength() + s.length() - TEXT_SIZE_MAX;
      if (extraLength > 0)
      {
        doc.remove(0, extraLength);
      } // end if extraLength
      doc.insertString(doc.getLength(), s, null);
      if (FIX && !FIXXX)
      {  // MEMORY LEAK HERE
        textPane.setCaretPosition(doc.getLength() );
      } // end if FIX
    }
    catch (Exception e)
    {
      e.printStackTrace();
      System.exit(1);
    } // end try
  } // end method append
  private void tryToFixMemory()
  {    
    // disable caret updates
    Caret caret = textPane.getCaret();
    if (caret instanceof DefaultCaret)
    {
      ( (DefaultCaret) caret).setUpdatePolicy(
          DefaultCaret.NEVER_UPDATE);
    } // end if DefaultCaret
    // remove registered UndoableEditListeners if any
    Document doc = textPane.getDocument();
    if (doc instanceof AbstractDocument)
    {
      UndoableEditListener[] undoListeners = 
          ( (AbstractDocument) doc).getUndoableEditListeners();
      if (undoListeners.length > 0)
      {
        for (UndoableEditListener undoListener : undoListeners)
        {
          doc.removeUndoableEditListener(undoListener);
        } // end for undoListener
      } // end if undoListeners
    } // end if AbstractDocument
  } // end method tryToFixMemory
  private class ButtonHandler implements ActionListener
  {
    private final int iter;
    ButtonHandler(int iter)
    {
      this.iter = iter;
    } // end constructor
    @Override
    public void actionPerformed(ActionEvent e)
    {
      for (int i = 0; i < iter; i++)
      {
        append(String.format("%10d\n", i) );
      } // end for i
    } // end method actionPerformed
  } // end class ButtonHandler
} // end class TestMain
JVM was from official Oracle Java SE Development Kit 8u45 for Linux x64. All tests were done with -Xmx100m limit.
Both flags are false:
Small test
Works as expected:

Stress test
GUI freezes at an intermediate point:

Memory is leaking:

At some point there is no memory left and I got the following error:
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
  at java.util.Formatter.parse(Formatter.java:2560)
  at java.util.Formatter.format(Formatter.java:2501)
  at java.util.Formatter.format(Formatter.java:2455)
  at java.lang.String.format(String.java:2928)
  at memleak.TestMain$ButtonHandler.actionPerformed(TestMain.java:117)
  at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
  at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2346)
  at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
  at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
  at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
  at java.awt.Component.processMouseEvent(Component.java:6525)
  at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
  at java.awt.Component.processEvent(Component.java:6290)
  at java.awt.Container.processEvent(Container.java:2234)
  at java.awt.Component.dispatchEventImpl(Component.java:4881)
  at java.awt.Container.dispatchEventImpl(Container.java:2292)
  at java.awt.Component.dispatchEvent(Component.java:4703)
  at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4898)
  at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4533)
  at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4462)
  at java.awt.Container.dispatchEventImpl(Container.java:2278)
  at java.awt.Window.dispatchEventImpl(Window.java:2750)
  at java.awt.Component.dispatchEvent(Component.java:4703)
  at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
  at java.awt.EventQueue.access$500(EventQueue.java:97)
  at java.awt.EventQueue$3.run(EventQueue.java:709)
  at java.awt.EventQueue$3.run(EventQueue.java:703)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
  at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:86)
  at java.awt.EventQueue$4.run(EventQueue.java:731)
  at java.awt.EventQueue$4.run(EventQueue.java:729)
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
  at javax.swing.text.GlyphPainter1.modelToView(GlyphPainter1.java:147)
  at javax.swing.text.GlyphView.modelToView(GlyphView.java:653)
  at javax.swing.text.CompositeView.modelToView(CompositeView.java:265)
  at javax.swing.text.BoxView.modelToView(BoxView.java:484)
  at javax.swing.text.ParagraphView$Row.modelToView(ParagraphView.java:900)
  at javax.swing.text.CompositeView.modelToView(CompositeView.java:265)
  at javax.swing.text.BoxView.modelToView(BoxView.java:484)
  at javax.swing.text.CompositeView.modelToView(CompositeView.java:265)
  at javax.swing.text.BoxView.modelToView(BoxView.java:484)
  at javax.swing.plaf.basic.BasicTextUI$RootView.modelToView(BasicTextUI.java:1509)
  at javax.swing.plaf.basic.BasicTextUI.modelToView(BasicTextUI.java:1047)
  at javax.swing.text.DefaultCaret.repaintNewCaret(DefaultCaret.java:1308)
  at javax.swing.text.DefaultCaret$1.run(DefaultCaret.java:1287)
  at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311)
  at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:756)
  at java.awt.EventQueue.access$500(EventQueue.java:97)
  at java.awt.EventQueue$3.run(EventQueue.java:709)
  at java.awt.EventQueue$3.run(EventQueue.java:703)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:75)
  at java.awt.EventQueue.dispatchEvent(EventQueue.java:726)
  at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
  at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
  at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
  at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
  at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
  at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: GC overhead limit exceeded
  at java.lang.Thread.getName(Thread.java:1135)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:677)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
Detailed memory statistics show very high counts for java.awt.event.InvocationEvent, sun.awt.EventQueueItem, and javax.swing.text.DefaultCaret$1 (absent in the fixed version):

Setting FIX = true did not improve the situation.
Both flags are true:
Small test
Now shows that the caret position is not updated:

Stress test
Works and has no indication of memory leak:
