The major thing that stops you from doing this, is that this is final. You can't reassign this. There are good reasons for this restriction.
However, even if you could reassign this, your method doesn't make much sense, because there are only two instances of Entry, ever - Entry.X and Entry.O:
§8.9
An enum type has no instances other than those defined by its enum constants. It is a compile-time error to attempt to explicitly instantiate an enum type
So if you did:
Entry.X.switchEntry();
You would not be able to access the object originally referenced by Entry.X from that point onwards! Because switchEntry sets Entry.X to Entry.O! Now both Entry.X and Entry.O refers to the same object!
And what about if I have a class that contains a getEntry method, that returns a private field entry, without a setter for entry. What if you called (and this is perfectly valid Java code):
getEntry().switchEntry();
Would that set my private entry variable? If so you've just broken encapsulation! Or would it do something stupid like getEntry() = Entry.O;?
Do you see the problem with allowing your implementation of switchEntry?
Anyway, you should instead write a method that returns an instance of Entry. You can then use = at the call site to assign to whatever you like.