Superpowered: real time pitch shift with timestretcher not working

Question

I am using Superpowered for various real-time FX and they all work very straightforward. However the pitch shifting is a whole other story, I think in fact because it's based on the time-stretching algorithm that of course has to deal with output that changes in time which is a lot more complex than applying FX like EQ or reverb. However I'm only interested in change the pitch of my mic input.

I looked at the only example I could find on GitHub and I slightly adapted it to fit my work:

static bool audioProcessing(void *clientdata,
                            float **buffers,
                            unsigned int inputChannels,
                            unsigned int outputChannels,
                            unsigned int numberOfSamples,
                            unsigned int samplerate,
                            uint64_t hostTime) {
    __unsafe_unretained Superpowered *self = (__bridge Superpowered *)clientdata;

    SuperpoweredAudiobufferlistElement inputBuffer;

    inputBuffer.startSample = 0;
    inputBuffer.samplesUsed = 0;
    inputBuffer.endSample = self->timeStretcher->numberOfInputSamplesNeeded;
    inputBuffer.buffers[0] = SuperpoweredAudiobufferPool::getBuffer(self->timeStretcher->numberOfInputSamplesNeeded * 8 + 64);
    inputBuffer.buffers[1] = inputBuffer.buffers[2] = inputBuffer.buffers[3] = NULL;

    self->outputBuffers->clear();
    self->timeStretcher->process(&inputBuffer, self->outputBuffers);
    int samples = self->timeStretcher->numberOfInputSamplesNeeded;
    float *timeStretchedAudio = (float *)self->outputBuffers->nextSliceItem(&samples);
    if (timeStretchedAudio != 0) {
        SuperpoweredDeInterleave(timeStretchedAudio, buffers[0], buffers[1], numberOfSamples);
    }

    //self->outputBuffers->rewindSlice();

    return true;
}

I have removed most of the code that I thought wasn't necessary. For example there was a while loop that seemed to deal with time-stretch scenarios, I'm just outputting the same time as I input.

Some observations:

• If I don't clear the outputBuffers my memory usage goes through the roof
• If I use self->outputBuffers->rewindSlice(); the app becomes silent, probably meaning the buffers are getting overwritten with silence
• If I do not use self->outputBuffers->rewindSlice(); I can hear my own voice coming back, but timeStretchedAudio is always 0 except the very first time

Answer 1

I finally got it working:

static bool audioProcessing(void *clientdata,
                                float **buffers,
                                unsigned int inputChannels,
                                unsigned int outputChannels,
                                unsigned int numberOfSamples,
                                unsigned int samplerate,
                                uint64_t hostTime) {
    __unsafe_unretained Superpowered *self = (__bridge Superpowered *)clientdata;

    //timeStretching->setRateAndPitchShift(realTimeRate, realTimePitch);
    SuperpoweredAudiobufferlistElement inputBuffer;
    inputBuffer.startSample = 0;
    inputBuffer.samplesUsed = 0;
    inputBuffer.endSample = numberOfSamples;
    inputBuffer.buffers[0] = SuperpoweredAudiobufferPool::getBuffer((unsigned int) (numberOfSamples * 8 + 64));
    inputBuffer.buffers[1] = inputBuffer.buffers[2] = inputBuffer.buffers[3] = NULL;

    // Converting the 16-bit integer samples to 32-bit floating point.
    SuperpoweredInterleave(buffers[0], buffers[1], (float *)inputBuffer.buffers[0], numberOfSamples);
    //SuperpoweredShortIntToFloat(audioInputOutput, (float *)inputBuffer.buffers[0], (unsigned int) numberOfSamples);

    self->timeStretcher->process(&inputBuffer, self->outputBuffers);

    // Do we have some output?
    if (self->outputBuffers->makeSlice(0, self->outputBuffers->sampleLength)) {
        while (true) { // Iterate on every output slice.
            // Get pointer to the output samples.
            int numSamples = 0;
            float *timeStretchedAudio = (float *)self->outputBuffers->nextSliceItem(&numSamples);
            if (!timeStretchedAudio || *timeStretchedAudio == 0) {
                break;
            }

            // Convert the time stretched PCM samples from 32-bit floating point to 16-bit integer.
            //SuperpoweredFloatToShortInt(timeStretchedAudio, audioInputOutput,
            //                            (unsigned int) numSamples);
            SuperpoweredDeInterleave(timeStretchedAudio, buffers[0], buffers[1], numSamples);
            self->recorder->process(timeStretchedAudio, numSamples);

            // Write the audio to disk.
            //fwrite(audioInputOutput, 1, numSamples * 4, fd);
        }
        // Clear the output buffer list.
        self->outputBuffers->clear();
        // If we have enough samples in the fifo output buffer, pass them to the audio output.
        //SuperpoweredFloatToShortInt((float *)inputBuffer.buffers[0], audioInputOutput, (unsigned int) numberOfSamples);
    }

    return true;
}

I am not sure if changing the rate also works, but I don't care for this application. YMMV.

Answer 2

Implement the part marked with TODO. That's the point where you need to provide input for the timeStretcher. Also take care of separating the output from the input. Output could be written before the input is consumed.