.wav files may be compressed, if so the sample values won't be stored as simple bytes. You would need to use a library (as mentioned in the comments).
If your file is uncompressed then you are almost correct. Uncompressed .wav files contain 16bit samples, not 8bit. So ReadAllBytes isn't very helpful.
Instead use a BinaryReader and reader.ReadInt16() to read each sample, eg.
List<Int16> sampleBuffer = new List<Int16>();
using(var inputStream = File.Open(inputFile)){
using (var reader = new BinaryReader(inputStream))
{
while (inputStream.Position < inputStream.Length)
sampleBuffer.Add(reader.ReadInt16());
}
}