If you're set on operating over BOOLs, then instead of:
for (BOOL flagA = NO; flagA <= YES; flagA++)
for (BOOL flagB = NO; flagB <= flagA; flagB++)
// ...
You should really be doing something this (though it is not what you want):
for (BOOL flagA = NO; flagA != YES; flagA = !flagA)
for (BOOL flagB = NO; flagB != flagA; flagB = !flagB)
// This is the only safe way to 'iterate' BOOLs
The behaviour, (BOOL)++ is not well-defined* as a BOOL can only be YES or NO. What you really should be doing is casting your BOOL to an int, and iterating over that, or refactoring your loop entirely to use int types.
The problem with casting your BOOL values to ints is, as you have pointed out, BOOL is typedef'd to something with only 8 bits of information*, therefore it only makes sense to have 255 iterations. In fact in more recent times, BOOL is not cast-able at all because it is defined as a compiler intrinsic (objc_bool, which can have values __objc_yes and __objc_no). __objc_no++ has no meaning.
TL;DR My (strong) suggestion would be to refactor your code so you are iterating over integers, and inspecting BOOLs within each iteration. Whether you cast your BOOL values, or refactor your loop is up to you, but iterating over BOOL values in the way you have indicated is both unsafe and (now, because of that) unsupported.
* In past years, the implementation details of BOOL were obvious (namely a cast to an unsigned char). With the advent of compiler intrinsics, the details are hidden (though they are likely the same). The reason they are now hidden is because you're really not supposed to rely on them, and the easiest way to stop people relying on them is to hide them from the compiler altogether.