Put each image inside a div. In each div add an onmouseenter and onmouseout attribute. The id will be used later. These attributes "listen" for when the user's mouse enters or leaves the div element:
<div id = "0" onmouseenter = "makeVideo(this)" onmouseout = "makeImage(this)">
<img>
</div>
We then need to define the makeVideo and makeImage functions:
<script>
   pairs = [["<img src = 'example.png'>","<video autoplay><source src='example.mp4' type='video/mp4'></video>"]] // create an array where each element pairs an image element string with a video. You will need a sub-array for each image and video pair
makeVideo = function(element){ //take the div as an input
   element.innerHTML = pairs[element.id][1] // take the appropriate video string based on the div id, and put it inside the div
}
makeImage = function(element){ 
   element.innerHTML = pairs[element.id][0] // this time use the img
}
</script>
Then just scale up to however many images you need.
Some things to keep in mind:
Make sure you set the width and height attributes of your videos to be the same.
Remember to give each div an id corresponding to its index in the pairs array
The most time-consuming part if you choose to use this method will probably be creating the pairs array with all the html strings.
Hope you get this working!