When using a Popup, neither PlacementTarget nor CustomPopupPlacementCallback is used after the popup has originally appeared. So any use of these properties will not allow the popup to track the button as it moves.
Several ways occur to me of achieving what you desire:
Attach a custom Adorner to the button and put the popup inside it. Disadvantage: Popup is not connected to Button or surrounding elements, so it won't inherit properties & DataContext from them.
Attach a custom Adorner to the button. This adorner will get measure and arrange calls when the button moves relative to the AdornerLayer, allowing you to manually update the Popup position. As long as your AdornerDecorator doesn't move relative to your Window (eg if it is the direct child of the Window), you can easily detect the AdornerLayer being moved by monitoring changes to Window size. Disadvantage: Complex to code & get right.
Don't use a Popup at all. Instead wrap the button in a <Grid> alongside a <Canvas> with zero width and height and the desired position. Inside the <Canvas> add the UserControl for the popup with an appropriate ZIndex. It will extend past the edge f the Canvas, which is just fine in WPF. Instead of using a Popup control just control the visibility of the UserControl. Disadvantage: Will not really be totally on top of all other objects, can't extend off edge of window (may not be an issue for XBAP, though).