This doesn't work because SpecialController isn't a single type. You can think of associated types as a kind of generics. A SpecialController with its SpecialValueType being an Int is a completely different type from a SpecialController with its SpecialValueType being an String, just like how Optional<Int> is a completely different type from Optional<String>.
Because of this, it doesn't make any sense to cast to SpecialValueType, because that would gloss over the associated type, and allow you to use (for example) a SpecialController with its SpecialValueType being an Int where a SpecialController with its SpecialValueType being a String is expected.
As compiler suggests, the only way SpecialController can be used is as a generic constraint. You can have a function that's generic over T, with the constraint that T must be a SpecialController. The domain of T now spans all the various concrete types of SpecialController, such as one with an Int associated type, and one with a String. For each possible associated type, there's a distinct SpecialController, and by extension, a distinct T.
To draw out the Optional<T> analogy further. Imagine if what you're trying to do was possible. It would be much like this:
func funcThatExpectsIntOptional(_: Int?) {}
let x: Optional<String> = "An optional string"
// Without its generic type parameter, this is an incomplete type. suppose this were valid
let y = x as! Optional
funcThatExpectsIntOptional(y) // boom.