Converting Texture2D into a video

Question

I've did a lot of research, but I can't find a suitable solution that works with Unity3d/c#. I'm using a Fove-HMD and would like to record/make a video of the integrated camera. So far I managed every update to take a snapshot of the camera, but I can't find a way to merge this snapshots into a video. Does someone know a way of converting them? Or can someone point me in the right direction, in which I could continue my research?

public class FoveCamera : SingletonBase<FoveCamera>{

 private bool camAvailable;
 private WebCamTexture foveCamera;
 private List<Texture2D> snapshots;

 void Start ()
 {
     //-------------just checking if webcam is available
     WebCamDevice[] devices = WebCamTexture.devices;
     if (devices.Length == 0)
     {
         Debug.LogError("FoveCamera could not be found.");
         camAvailable = false;
         return;
     }

     foreach (WebCamDevice device in devices)
     {
         if (device.name.Equals("FOVE Eyes"))
             foveCamera = new WebCamTexture(device.name);//screen.width and screen.height 
     }

     if (foveCamera == null)
     {
         Debug.LogError("FoveCamera could not be found.");
         return;
     }
     //-------------camera found, start with the video

     foveCamera.Play();
     camAvailable = true;
 }

 void Update () {
     if (!camAvailable)
     {
         return;
     }

     //loading snap from camera
     Texture2D snap = new Texture2D(foveCamera.width,foveCamera.height);
     snap.SetPixels(foveCamera.GetPixels());
     snapshots.Add(snap);
 }
}

The code works so far. The first part of the Start-Method is just for finding and enabling the camera. In the Update-Method I'm taking every update a snapshot of the video.

After I "stop" the Update-Method, I would like to convert the gathered Texture2D object into a video.

Thanks in advance

Answer 1

1. Create MediaEncoder

using UnityEditor; // VideoBitrateMode
using UnityEditor.Media; // MediaEncoder

var vidAttr = new VideoTrackAttributes
{
    bitRateMode = VideoBitrateMode.Medium,
    frameRate = new MediaRational(25),
    width = 320,
    height = 240,
    includeAlpha = false
};

var audAttr = new AudioTrackAttributes
{
    sampleRate = new MediaRational(48000),
    channelCount = 2
};
                
var enc = new MediaEncoder("sample.mp4", vidAttr, audAttr);

2. Convert each snapshot to Texture2D

3. Call consequently AddFrame to add each snapshot to MediaEncoder

enc.AddFrame(tex);

4. Once done call Dispose to close the file

enc.Dispose();

Answer 2

I see two methods here, one is fast to implement, dirty and not for all platforms, second one harder but pretty. Both rely on FFMPEG.

1) Save every frame into image file (snap.EncodeToPNG()) and then call FFMPEG to create video from images (FFmpeg create video from images) - slow due to many disk operations.

2) Use FFMPEG via wrapper implemented in AForge and supply its VideoFileWriter class with images that you have. Image sequence to video stream?

Problem here is it uses System.Bitmap, so in order to convert Texture2D to Bitmap you can use: How to create bitmap from byte array?

So you end up with something like:

Bitmap bmp;
Texture2D snap;
using (var ms = new MemoryStream(snap.EncodeToPNG()))
{
    bmp = new Bitmap(ms);
}
vFWriter.WriteVideoFrame(bmp);

Both methods are not the fastest ones though, so if performance is an issue here you might want to operate on lower level data like DirectX or OpenGL textures.