How can I write Open CV frames to an RTSP stream with Gstreamer and C++?

Question

I'm trying to take a video frame into OpenCV, do some processing on it (to be exact, aruco detection) and then package the resultant frame into a RTSP stream with GStreamer.

I've seen a Python solution to this problem, but I'm having trouble translating it to C++.

Here's my attempt at recreating the SensorFactory class:

#include <glib-object.h>
#include <iostream>
#include "SensorFactory.h"

SensorFactory::SensorFactory(std::string launch) {
    launchString = launch;
    cap = cv::VideoCapture(0);
    // should be incremented once on each frame for timestamping
    numberFrames = 0;

    // simple struct with only the cap (int*), lastFrame (cv::Mat*) and numberFrames (int* again) fields
    CVData cvData;
    cvData.cap = &cap;
    cvData.lastFrame = &lastFrame;
    cvData.numberFrames = &numberFrames;
}

GstFlowReturn SensorFactory::on_need_data(GstElement *src, CVData *datum) {
    if (datum->cap->isOpened()) {
        if (datum->cap->read(*(datum->lastFrame))) {
            std::string data = std::string(reinterpret_cast<char * > (datum->lastFrame->data));
            GstBuffer *buf = gst_buffer_new_allocate(nullptr, data.max_size(), nullptr);
            gst_buffer_fill(buf, 0, &data, data.max_size());
            buf->duration = static_cast<GstClockTime>(duration);
            GstClockTimeDiff timestamp = *(datum->numberFrames) * duration;
            buf->pts = buf->dts = static_cast<GstClockTime>(timestamp);
            buf->offset = static_cast<guint64>(timestamp);
            int *numf = datum->numberFrames;
            *numf += 1;
            g_signal_emit_by_name(src, "push-buffer", buf);
            gst_buffer_unref(buf);
            return GST_FLOW_OK;
        }
    }
    // never reached
    return GST_FLOW_NOT_LINKED;
}

GstElement *SensorFactory::create_element(const GstRTSPUrl *url) { return gst_parse_launch(launchString.c_str(), nullptr); }

void SensorFactory::configure(GstRTSPMedia *rtspMedia) {
    numberFrames = 0;
    GstElement *appsrc;
    appsrc = gst_rtsp_media_get_element(rtspMedia);
    g_signal_connect(appsrc, "need-data", (GCallback) on_need_data, &cvData);
}

The header for SensorFactory is nothing special:

#include <gst/rtsp-server/rtsp-media-factory.h>
#include <gst/rtsp-server/rtsp-media.h>
#include <gst/app/gstappsrc.h>
#include <opencv2/videoio.hpp>

class SensorFactory : public GstRTSPMediaFactory {
public:
    typedef struct _CVData {
        cv::VideoCapture *cap;
        cv::Mat *lastFrame;
        int *numberFrames;
    } CVData;

    CVData cvData;
    std::string launchString;
    cv::VideoCapture cap;
    cv::Mat lastFrame;

    int numberFrames = 0;
    const static int framerate = 30;
    const static GstClockTimeDiff duration = 1 / framerate * GST_SECOND;

    explicit SensorFactory(std::string launch);

    static GstFlowReturn on_need_data(GstElement *src, CVData *datum);

    GstElement *create_element(const GstRTSPUrl *url);

    void configure(GstRTSPMedia *media);
};

And then main.cpp looks like so:

#include <gst/gst.h>
#include "src/SensorFactory.h"

int main() {
    gst_init(nullptr, nullptr);

    GstRTSPServer *server;
    server = gst_rtsp_server_new();

    SensorFactory sensorFactory("appsrc name=source is-live=true block=true format=GST_FORMAT_TIME"
                                "caps=video/x-raw,format=BGR ! "
                                "videoconvert ! video/x-raw,format=I420 ! "
                                "x264enc speed-preset=ultrafast tune=zerolatency ! rtph264pay name=pay0");

    g_print("setting shared\n");
    gst_rtsp_media_factory_set_shared(&sensorFactory, true);
    g_print("set shared\n");
    GstRTSPMountPoints *mounts;
    mounts = gst_rtsp_server_get_mount_points(server);
    gst_rtsp_mount_points_add_factory(mounts, "/test", &sensorFactory);

    GMainLoop *loop;
    loop = g_main_loop_new(nullptr, false);
    g_main_loop_run(loop);
}

The program compiles fine, and will even start running, but segfaults on gst_rtsp_media_factory_set_shared(&sensorFactory, true);. There isn't any other hacky memory management in this program.

`gst_rtsp_media_factory_set_shared` is expecting a type `GstRTSPMediaFactory*`, whereas you pass `SensorFactory*`. — theUser, Mar 30 '19 at 13:32
@theUser ok, can I cast it or something (`SensorFactory` does extend `GstRTSPMediaFactory`...)? Bit of a C++ noob still. — c-x-berger, Mar 30 '19 at 13:35
I'm a C++ noob myself tbh. I can't see you calling `gst_rtsp_media_factory_new()` anywhere though ? — theUser, Mar 30 '19 at 14:12
@theUser Sorry, it's in a header I didn't add. I'll add it now... — c-x-berger, Mar 30 '19 at 14:31

Bahramdun Adil · Answer 1 · 2019-03-29T12:43:56.517

You can try the steps below to write the stream as RTMP.

 if (platform is "Windows") {
    // if the platform is windows, then add the head data of the video
    // otherwise it will not work on the HTML flash player
    headData = " !  video/x-h264,profile=high";
 }
 // to rtmp (media server e.g: NGINX)
 rtmpUrl = "appsrc ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency "+headData+" ! flvmux ! rtmpsink location=rtmp://192.168.1.25/mylive/test";
 // using UDP broadcast to all 1~255 IPs
 rtmpUrl = "appsrc ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency "+headData+" ! flvmux ! udpsink host=192.168.1.255 port=5000";

 // using UDP broadcast specific IP 
 rtmpUrl = "appsrc ! videoconvert ! x264enc speed-preset=ultrafast tune=zerolatency "+headData+" ! flvmux ! udpsink host=192.168.1.25 port=5000";    

// give the FPS and the size of the video
VideoWriter writer = new VideoWriter(rtmpUrl, Videoio.CAP_GSTREAMER, FOURCC, currentFps, new Size(width, height));

// then you can write the video using writer

NOTE: Make sure you build OpenCV with GStreamer.

Well, that helps (at least I have *a* stream) ... Still targeting RTSP though — c-x-berger, Mar 29 '19 at 12:35

score 0 · Answer 2 · answered Mar 30 '19 at 14:39

Here is an alternative approach. Seperate your SensorFactory from the rtsp code for now.

Start your SensorFactory with the pipeline.

appsrc name=source is-live=true block=true format=GST_FORMAT_TIME caps=video/x-raw,format=BGR,width=640,height=480,framerate=30/1 ! videoconvert ! video/x-raw,format=I420 ! x264enc speed-preset=ultrafast tune=zerolatency ! udpsink port=5050

We end that pipeline by piping the h264 over a udpsink on port 5050.

Then compile the gstreamer rtsp server example here And launch that with pipeline

./test-launch "( udpsrc port=5050 ! rtph264pay name=pay0 pt=96 )"

Assuming your SensorFactory works as you intend, this should get you an RTSP Stream serving at rtsp://localhost:8554/test

This looks like a viable solution (and I can possibly cut out `test-launch` as well.) How would I start the factory in `main`? — c-x-berger, Mar 30 '19 at 14:44
Yeap, I'd just take the rtsp server specific stuff out of main.cpp for now. — theUser, Mar 30 '19 at 14:50

How can I write Open CV frames to an RTSP stream with Gstreamer and C++?

2 Answers2

Linked