You can make
  $(window).height()
  $(window).width()
a condition for your animation.
Like..
 // check if window size changed
   $(window).resize(function() {
    if($(window).height() < 800 && $(window).width() < 600){
      $('.tlt').textillate();
    }
   });
And it gets only executed on your condition.
Hope that is what you asked for.
Some more informations: How can I detect window size with jQuery?
Edit: 
<h1 class="tlt">Thr brown fox jumped over the grizzely</h1> 
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jqueryui/1.11.4/jquery-ui.min.js"></script>
<script type="text/javascript"> 
    if($(window).height() > 600 && $(window).width() > 800) 
        $('.tlt').textillate();
$(window).resize(function() { 
    if($(window).height() > 600 && $(window).width() > 800) 
        $('.tlt').textillate(); 
    else
        $('.tlt').textillate('stop');
}); 
</script>
First, it initially checks the height/width of the window and starts or don't starts the textillate function. Then, everytime the window resized it checks again. If it doesn't is in height/width area, it stops, otherwise, it will go start the thing/go on. I've tried it and it worked for me. You have to include your textillate-file of course.