At the end of the first loop, $value is pointing to the same place as $variable[3] (they are pointing to the same location in memory):
$variable  = [1,2,3,4];
foreach ($variable  as $key => &$value) 
    $value ++;
Even as this loop is finished, $value is still a reference that's pointing to the same location in memory as $variable[3], so each time you store a value in $value, this also overwrites the value stored for $variable[3]:
foreach ($variable as $key => $value);
var_dump($variable);
With each evaluation of this foreach, both $value and $variable[3] are becoming equal to the value of the iterable item in $variable.
So in the 3rd iteration of the second loop, $value and $variable[3] become equal to 4 by reference, then during the 4th and final iteration of the second loop, nothing changes because you're passing the value of $variable[3] (which is still &$value) to $value (which is still &$value).
It's very confusing, but it's not even slightly idiosyncratic; it's the code executing exactly as it should.
More info here: PHP: Passing by Reference
To prevent this behavior it is sufficient to add an unset($value); statement after each loop where it is used. An alternative to the unset may be to enclose the foreach loop in a self calling closure, in order to force $value to be local, but the amount of additional characters needed to do that is bigger than just unsetting it:
(function($variable){
   foreach ($variable  as $key => &$value) $value++;
})($variable);