Approach - Variable width with WrapLayout
The GridLayout solution presumes the GUI requires 6 components per row. 
For as many cols as needed to fill the width, & then show the components in as many rows as required, look to WrapLayout.
 

Approach - Variable width with JList
A JList can also be used here, since it seems all the components consist of a single (GUI) object.
 
  
 
import java.awt.*;
import java.awt.event.ActionEvent;
import javax.swing.*;
import javax.swing.border.*;
public class ListComponents {
    public static void main(String[] args) {
        Runnable r = new Runnable() {
            @Override
            public void run() {
                String[] listData = {
                    "Component 1", "Component 2", "Component 3",};
                final DefaultListModel<String> model
                        = new DefaultListModel<String>();
                for (String datum : listData) {
                    model.addElement(datum);
                }
                JList list = new JList(model);
                list.setLayoutOrientation(JList.HORIZONTAL_WRAP);
                list.setVisibleRowCount(-1);
                list.setCellRenderer(new ObjectCellRenderer());
                Action addAction = new AbstractAction("Add New") {
                    @Override
                    public void actionPerformed(ActionEvent e) {
                        model.addElement("New Component");
                    }
                };
                JButton addNew = new JButton(addAction);
                JPanel ui = new JPanel(new BorderLayout(3, 3));
                ui.setBorder(new EmptyBorder(4, 4, 4, 4));
                ui.add(new JScrollPane(list), BorderLayout.CENTER);
                ui.add(addNew, BorderLayout.PAGE_START);
                JFrame f = new JFrame("Component List");
                f.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);
                f.setContentPane(ui);
                f.pack();
                f.setLocationByPlatform(true);
                f.setVisible(true);
            }
        };
        SwingUtilities.invokeLater(r);
    }
}
class ObjectCellRenderer extends DefaultListCellRenderer {
    Border border = new EmptyBorder(20, 5, 20, 5);
    public Component getListCellRendererComponent(
            JList list,
            Object value,
            int index,
            boolean isSelected,
            boolean cellHasFocus) {
        JLabel label = (JLabel) super.getListCellRendererComponent(
                list, value, index, isSelected, cellHasFocus);
        label.setBorder(border);
        return label;
    }
}
Approach - Fixed width with GridLayout
Using a GridLayout, when we know we always want a fixed number per row, regardless of width.
JPanel mainPanel = new JPanel(new GridLayout(0,6));
See new GridLayout(rows,cols):
Creates a grid layout with the specified number of rows and columns. All components in the layout are given equal size.
One, but not both, of rows and cols can be zero, which means that any number of objects can be placed in a row or in a column.
See this code for an example using a GridLayout(0,2) for the labels beneath Add Another Label
