The error tells you that both your frameworks in build/simulator and build/device folders have been built for the same architecture (arm64, which is a device architecture). You can verify this by yourself, by looking inside the .framework file: FrameworkName.framework/Modules/FrameworkName.swiftmodule.
It is possible, that one of the folders (or both) contain more than one architecture like this:

Personally, I like to build my 'fat' frameworks outside the Xcode.app folders (just to make sure I have complete control over what is where). First, run your framework for simulator (select any simulator as build target). After the process has completed, go to Products folder in Xcode Navigator, click Show in Finder on FrameworkName.framework file. Copy the shown .framework somewhere more convenient (e.g. Desktop/simulator folder)

Then, build the framework again, only this time for device (select Any iOS device as build target). Copy second .framework somewhere like Desktop/iphone folder.
Create empty Desktop/universal folder for output framework. Copy .framework file there from Desktop/iphone folder and remove Desktop/universal/FrameworkName.framework/Framework executable file. This file will later be replaced by lipo.
Next, do the lipo magic:
lipo -create ~/Desktop/iphone/FrameworkName.framework/FrameworkName ~/Desktop/simulator/FrameworkName.framework/FrameworkName -output ~/Desktop/universal/FrameworkName.framework/FrameworkName
Last step, go to Desktop/simulator/FrameworkName.framework/Modules/FrameworkName.swiftmodule copy all files that start with x86_64 prefix, and paste them to Desktop/universal/FrameworkName.framework/Modules/FrameworkName.swiftmodule. Now your Desktop/universal/FrameworkName.framework contains both device and simulator architectures. Congrats, you've got your 'fat' library!
Disclaimer: Yes, I realise there are easier ways to do this with various scripts and terminal commands, but all of them do pretty much the same thing. Once you try to do this manually step by step, it will help you understand what goes where, and what are architectures and how they can be combined.
Disclaimer 2: Starting from Xcode 12, Apple insists you build .xcframeworks instead of 'fat' libraries. See here