0

I'm capturing audio with AVCaptureSession. In my callback function for processing the data captured, I put the stream in the Data structure (byte buffer). It appears Data is UInt8 (makes sense for a byte buffer), but I believe the stream data is UInt32.

I'm not sure which of the following I should be doing, but I can't get any of them to work. Do I:

  1. convert Data to be UInt32 instead of UInt8?
  2. When reading from data, take 4 bytes to make a UInt32?
  3. Change my capture session to UInt8?
  4. Give up on Data structure and make my own?

My callback function is:

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

    var audioBufferList = AudioBufferList()
    var data = Data()
    var blockBuffer: CMBlockBuffer?

    // Put the sample buffer in to a list of audio buffers (audioBufferList)
    CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, bufferListSizeNeededOut: nil, bufferListOut: &audioBufferList, bufferListSize: MemoryLayout<AudioBufferList>.size, blockBufferAllocator: nil, blockBufferMemoryAllocator: nil, flags: 0, blockBufferOut: &blockBuffer)
    // Extract the BufferList in to an array of buffers
    let buffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers, count: Int(audioBufferList.mNumberBuffers))
    // for each buffer, extract the frame.  There should only be one buffer as we are recording in mono!
    for audioBuffer in buffers {
        assert(audioBuffer.mNumberChannels == 1)        // it should always be 1 for mono channel
        let frame = audioBuffer.mData?.assumingMemoryBound(to: UInt8.self)
        data.append(frame!, count: Int(audioBuffer.mDataByteSize) / 8)
    }

    // limit how much of the sample we pass through.
    viewDelegate?.gotSoundData(data.prefix(MAX_POINTS))
}

All the gotSoundData goes from the view to a number of subviews for processing

func addSamples(samples: Data) {
    //if (isHidden) { return }

    samples.forEach { sample in
        [...process each byte...]
    }
}

I can see that Data.append has the definition:

mutating func append(_ bytes: UnsafePointer<UInt8>, count: Int)
TPot
  • 340
  • 3
  • 10
  • Are you sure it's 8bit and not 16bit? – meggar Oct 18 '18 at 13:28
  • I've added to the description the Data being Uint8. I'm not sure what the stream is in. Can't find it now. Wondering if I was getting confused with the flag on CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer which is UInt32. The old AudioUnit code I am porting was SInt16. – TPot Oct 18 '18 at 20:00
  • ok - I'd say you're right that it's 16 bit. I haven't found anything to say that, but the frame is 1024 samples and it's size is 2048 bytes... 16 bits per sample. The question still stands though... how do I read from Data 2 bytes at a time. – TPot Oct 19 '18 at 09:11
  • So it could be like `let pData = UnsafeMutablePointer(yourBuffer.data)` and then `let sampleArray = UnsafeMutableBufferPointer( start: pData, count: Int(yourBuffer.mDataByteSize)/sizeof(Int16))`. and then `sampleArray` is your array of 16bit values. – meggar Oct 19 '18 at 12:58
  • The first line fails because yourBuffer is UnsafeMutableRawPointer (returned from UnsafeBufferPointer) which can't become an UnsafeMutablePointer. – TPot Oct 19 '18 at 21:05
  • Appears to be working now. Thanks for the support Meggar. I'll add an answer for those that are interested. – TPot Oct 19 '18 at 21:50

1 Answers1

1

Meggar helped me concentrate on option 4 - using my own structure [Int16]. If anyone is interested in option 1, then check out this link I found later that extends Data for more data types: round trip Swift number type to/from Data

callback function:

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    var audioBufferList = AudioBufferList()
    var blockBuffer: CMBlockBuffer?

    // Put the sample buffer in to a list of audio buffers (audioBufferList)
    CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer(sampleBuffer, bufferListSizeNeededOut: nil, bufferListOut: &audioBufferList, bufferListSize: MemoryLayout<AudioBufferList>.size, blockBufferAllocator: nil, blockBufferMemoryAllocator: nil, flags: 0, blockBufferOut: &blockBuffer)
    // Extract the BufferList in to an array of buffers
    let audioBuffers = UnsafeBufferPointer<AudioBuffer>(start: &audioBufferList.mBuffers, count: Int(audioBufferList.mNumberBuffers))
    // For each buffer, extract the samples
    for audioBuffer in audioBuffers {
        let samplesCount = Int(audioBuffer.mDataByteSize) / MemoryLayout<Int16>.size
        let samplesPointer = audioBufferList.mBuffers.mData!.bindMemory(to: Int16.self, capacity: samplesCount)
        let samples = UnsafeMutableBufferPointer<Int16>(start: samplesPointer, count: samplesCount)
        //convert to a "safe" array for ease of use in delegate.
        var samplesArray:[Int16] = []
        for sample in samples {
            samplesArray.append(sample)
        }
        viewDelegate?.gotSample(samplesArray)
    }        
}

and the consuming function stays pretty much the same:

func addSamples(samples: [Int16]) {
    samples.forEach { sample in
        [...process each Int16...]
    }
}
TPot
  • 340
  • 3
  • 10