I have a framework with a universal binary that contains x86_64 and arm64 which I merge with lipo with a custom script at framework build time. I encountered this same issue for Xcode 12.3 and have created a work around for now. Hopefully this will get fixed in Xcode quickly, but until then, one quick fix would be to thin the architectures and use the framework that you need.
Please see my answer here on how to start producing .xcframeworks which is the long term solution for framework authors
For instance, let's assume I'm in a terminal in the working directory where my universal framework some_framework.framework is. If I want to run on an actual physical device, I execute the following command:
lipo -thin arm64 some_framework.framework/some_framework -output some_framework
With the above command, you extract the arm64 binary. Afterwards, replace the current some_framework.framework/some_framework with the newly generated arm64 only binary
mv some_framework some_framework.framework
If you have a universal framework built only from Objective-C sources, your job is done. But if you've got Swift code too, then you would need to update some_framework.framework/Modules/some_framework.swiftmodule so that it does not contain any references to architectures that are not arm64.
You would follow a similar process for running on the simulator except that you need x86_64. I'm currently now maintaining two versions of my framework until this is fixed. Whenever I switch between the simulator and the device, I simply switch out which framework is in my project.