On Xcode 10.2, Swift 5, using $0.load(as:) didn't work for me, both when reading from the pointer or writing to it.
Instead, using $0.baseAddress?.assumingMemoryBound(to:) seems to work well.
Example reading from the pointer buffer (code is unrelated to the question):
var reachability: SCNetworkReachability?
data.withUnsafeBytes { ptr in
    guard let bytes = ptr.baseAddress?.assumingMemoryBound(to: Int8.self) else {
        return
    }
    reachability = SCNetworkReachabilityCreateWithName(nil, bytes)
}
Example writing to the buffer pointer (code is unrelated to the question):
try outputData.withUnsafeMutableBytes { (outputBytes: UnsafeMutableRawBufferPointer) in
    let status = CCKeyDerivationPBKDF(CCPBKDFAlgorithm(kCCPBKDF2),
                                      passphrase,
                                      passphrase.utf8.count,
                                      salt,
                                      salt.utf8.count,
                                      CCPseudoRandomAlgorithm(kCCPRFHmacAlgSHA1),
                                      rounds,
                                      outputBytes.baseAddress?.assumingMemoryBound(to: UInt8.self),
                                              kCCKeySizeAES256)
    guard status == kCCSuccess else {
        throw Error.keyDerivationError
    }
}
The code from the question would look like:
let value = data.withUnsafeBytes { 
    $0.baseAddress?.assumingMemoryBound(to: UInt32.self)
}
In cases where the 'withUnsafeBytes' is deprecated: use withUnsafeBytes<R>(…) warning persists, it seems like the compiler can get confused when the closure has only one line. Making the closure have two or more lines might remove the ambiguity.