Have a good look at The definitive guide to form-based website authentication and ask yourself if you really need a captcha.
Besides that, you can use Callbacks :after_validation, before_save, around_save, after_save, before_create, around_create, after_create, before_update, around_update, after_update to handle stuff still inside your transaction. 
The way to call one of these callbacks is to simply declare them in your model
If you need to use a captcha however, I would do this with javascript and ajax, to append it to your form before the user sends it. 
You should not do this in the controller after recieving a post of the form, since you will have to:
- Store the filled form values in the session after validation (dont save)
- Redirect the user to a captcha page (which will make any user confused) 
- Check the captcha multiple times before it passes (they are quite unreadable) 
- Get the model out of the session (which you have no idea of which one it is)
- Call save on the model to actually write it to your DB.
So basically you avoid starting a transaction before the captcha is passed.