This might not be a solution for a mock of useNavigation, but I found this helpful to achieve my own intent (spying), and it looks like it would achieve the OP's intent (expecting on calls)
import { CommonActions } from '@react-navigation/routers';
jest.spyOn(CommonActions, 'navigate');
This seems to be where the actions that surface out of the useNavigation hook are defined, too. I was unsuccessful to create a mock on this, but the spy works excellently.
Caveat
I'm using react-navigation 6.
I am calling render with a NavigationContainer thus I am not mocking anything in react-navigation, hence I can spy. If you don't do that, and you're looking to truly mock navigation, this won't work.
FWIW, I would recommend trying this approach first, as you can then allow much more of your code to be tested properly, as it would get used.
e.g. For my CaptureLocationScreen which is in a Stack inside a modal.
render(
  <NavigationContainer>
    <RootStack.Navigator>
      <RootStack.Group screenOptions={{ presentation: 'modal', }}>
        <RootStack.Screen
          name="CaptureLocation"
          component={CaptureLocationScreen}
          options={({ navigation }: RootStackScreenProps<'CaptureLocation'>) => ({
            headerTitle: 'Add new location',
            headerLeft: () => (<CloseButton onPressed={navigation.goBack} />),
            headerRight: () => (<AcceptButton />)
          })} />
      </RootStack.Group>
    </RootStack.Navigator>
  </NavigationContainer>);
References
https://reactnavigation.org/docs/navigation-actions/