Rust has a lot of operations that work with both references and objects. These include things like the comparison operators (x < y is the same as &x < &y), method calls ((&x).func() is the same as x.func()), and certain functions (println!("{}", &x) is the same as println!("{}", x)). However, there are only three different ways in Rust for references to be treated the same as objects.
The first way is trait implementations. Some traits are implemented for a type T, but they also include a blanket implementation impl<T> Trait for &T that just defers to the Trait implementation for T. One example of this is the standard comparison operators, which use the std::cmp::Ord trait. This trait has a blanket implementation, impl<A> Ord for &A, which just defers to the Ord implementation on A. This is why you can compare two i32s (since i32 implements Ord), and you can compare two &i32s (since &i32 implements Ord because of the blanket implementation), but you can't compare an i32 and &i32 (since they're different types).
The second way is the dot operator (.). When you call a method on a value, (&3).abs(), Rust does a whole bunch of magic to automatically coerce the type into one that applies to the method. The full specification is here, but the important part is that Rust will try to implicitly dereference the value if it doesn't detect a valid method call on a reference. Rust even does this recursively, meaning that you can write (&&&&&&&&&&&3).abs() and still have it compile. This is why method calls can take references along with objects.
The third way is the std::borrow::Borrow trait. Borrow<T> is satisfied by both T and &T, meaning that certain functions can state that they will accept either references or objects. The following code compiles and prints your number is 3 twice:
use std::borrow::Borrow;
fn f<T: Borrow<i32>>(num: T) {
println!("your number is {}", num.borrow());
}
fn main() {
f(3);
f(&3);
}
The reason why vector[&idx] doesn't magically dereference the argument is because the [] indexing operator is handled by the std::ops::Index trait, and the index function from that trait only accepts the exact type. The std::ops::Index::index function could be changed to use Borrow, but for all the Rust devs know, someone might want to implement a data structure that handles data[&x] and data[x] independently, and they want to facilitate that.
So why doesn't Rust do this dereferencing for primitive types implicitly?
Rust attempts to treat primitive types the exact same as user-provided types: they don't have any extra special behavior that can't be recreated on a user-provided type. This is why crates like ux can provide more 'primitive' types like i7, u4 and i42: because there's nothing that primitive types can do that user-defined types can't do as well.
Since we'd want to extend this auto-dereference behavior to user-provided types, we might go ahead and say "okay, all Copy types will have auto-dereference behaviour." The problem is that types like [u8; 1024*1024*1024*1024] (i.e. a 1GB sized type) implement Copy, and you definitely don't want to autodereference them (imagine messing up in your code and accidentally creating a 1GB-sized copy... yikes.) So then you might say that "okay, let's make a trait AutoDeref that you #derive to get autoderefence behaviour." Now the other problem comes into play: this would increase compile time and compiler complexity significantly, as Rust now needs to dereference juggle all primitive operations in any context.
TL;DR: It would increase compiler complexity and compile time significantly.