I'm looking for the most elegant, fastest and most efficient way to store several objects of the class Point with an index/key in any kind of collection.
Point contains attributes like index, posX, posY, area, neighborPoints[], AdjacentTriangles[], etc. and are used for creating a Delaunay-Triangulation and a Voronoi-Mesh.
For reasons of selection the single points should have an index in the collection, but don't need to be ordered.
While triangulating, I'm creating and deleting points. Also I want to loop through all points in the collection. Hence, the type of collection should be optimized for these operations.
In the beginning I used a List<Point> for storing the points and the occurence in the list was identical with the index of the point. In case of deleting one point of the list, I had to decrease all indices of the higher points. This sounds pretty inconvenient and time-consuming to me.
This is why I tried Dictionary<int, Point> afterwards. Here the indices are fixed and if I delete for example the second point, then all higher points, maybe Point[5] stays Point[5], just Point[1] is not there anymore (return is null). However, my runtimes are now even longer (see picture), although I don't have to do the index-shifting anymore. (Why is this?)

Using a Hashmap makes no sence to me, since then I would not be able to use calls like Point.posX (error code CS1061), because in a Hashmap the datatype is not set.
Do you have any suggestion for a efficient type of collection with faster performance?