The problem seems to be in the pointee type, which has to be the same in both pointer declarations (map key type and the get_attached_value's argument). 
OP's code uses const A*, which is a pointer to a const instance of class A (an alternative spelling is A const *). Leaving this const in both map declaration and in get_attached_value' argument almost works, but reset_all does not allow you to assign a new value to *p.first, because the resulting type is A const& (which cannot be assigned into).
Removing both consts works as well, but OP wants to keep a const in get_attached_value.
One solution for OP's requirements, keeping as many consts as possible, seems to be to change the pointer type to a const pointer to a non-const instance of A. This will keep reset_all working, while allowing to use a const pointer in both map declaration and get_attached_value's argument:
#include <map>
using namespace std;
struct A {};
map<A * const, int> data;
int get_attached_value(A * const p) {
    return data.at(p);
}
void reset_all() {
    for (const auto &p : data)
        *p.first = A();
}
Another possible solution, with map's key as non-const but the get_attached_value's parameter const, could use std::lower_bound with a custom comparator to replace the data.at() call:
#include <map>
#include <algorithm>
using namespace std;
struct A {};
map<A*, int> data;
int get_attached_value(A const * const p) {
    auto it = std::lower_bound(data.begin(), data.end(), p,
        [] (const std::pair<A* const, int>& a, A const* const b) {
            return a.first < b;
        }
    );
    return it->second;
}
void reset_all() {
    for (const auto &p : data)
        *p.first = A();
}
However, this solution will be significantly less efficient than one that would use map's native search functions - std::lower_bound uses linear search when input iterators are not random access.
To conclude, the most efficient solution in C++11 or lower would probably use a const pointer as the map's key, and a const_cast in the reset_all function.
A bit more reading about const notation and pointers can be found here.