Simple answer
Basically, "BoxLayout can't be shared" in this situation means that you're trying to make your JFrame and its contentPane() share the same BoxLayout object.
Advanced Explanation
When setting the Layout for a JFrame, it implicitely calls getContentPane().setLayout(manager) so you are actually setting the layout for the contentPane() and not for the frame itself.
This brings us to how the containers are checked within BoxLayout and FlowLayout.
FlowLayout
This layout does not have a constructor that will take the container as a parameter and thus will not, at the making of the object, consider the container. Neither has this class a container instance variable.
BoxLayout
BoxLayout on the contrary has a constructor that takes the container as a parameter and stocks it within the instance variable target. This is done to check it later within the layoutContainer(container) method. It has a checkContainer(container) method that will verify if the instance variable is equals to the container given in parameter. It throws throw new AWTError("BoxLayout can't be shared"); if it is not the case.
This was an introduction to the following explaination.
As said within the first paragraph, JFrame.setLayout(LayoutManager) will call JFrame.getContentPane().setLayout(LayoutManager) and thus the Layout is set on the contentPane() which is a JPanel by default.
Check out the constructor of BoxLayout(container, int) and ask yourself :
Now I know that the Layout is set on the JPanel (contentPane()) and not the JFrame itself, which parameter will I give to this constructor?
Is it a good idea to give it the JFrame itself? Knowing that it is not the Component which on the Layout has been set?
The answer is : of course it is not a good idea.
Here is the correct way to achieve this :
JFrame frame = new JFrame();
frame.setLayout(new BoxLayout(frame.getContentPane(), ...);
Why?
Because we know now that the container is the contentPane() and that eventually, when adding a component for example, a check is going to happen and the parameter within the constructor must be the exact same object as the component where on the layout is set which does not happens with FlowLayout for example.
Sources
BoxLayout
FlowLayout