Here is one possible solution
class Quota < ApplicationRecord
  belongs_to :domain, optional: true
  belongs_to :project, optional: true
  validate :present_domain_or_project?
  validates :domain, presence: true, unless: Proc.new { |q| q.project_id.present? }
  validates :project, presence: true, unless: Proc.new { |q| q.domain_id.present? }
  private
  def present_domain_or_project?
    if domain_id.present? && project_id.present?
      errors.add(:base, "Specify a domain or a project, not both")
    end
  end
end
In the first block, we define the associations and specify optional: true so we overpass the new Rails 5 behavior of validating the presence of the association. 
belongs_to :domain, optional: true
belongs_to :project, optional: true
Then, the first thing we do is just simply eliminating the scenario of both the association attributes (project_id and domain_id) are set. This way we avoid hitting the DB twice, in reality, we would only need to hit the DB once.
validate :present_domain_or_project?
...
private 
def present_domain_or_project?
  if domain_id.present? && project_id.present?
    errors.add(:base, "Specify a domain or a project, not both")
  end
end
The last part is to check if one of the association is present(valid) in the absence of the other 
validates :domain, presence: true, unless: Proc.new { |q| q.project_id.present? }
validates :project, presence: true, unless: Proc.new { |q| q.domain_id.present? }
Regarding:
Is there a way to get Rails to raise an error if the ID is invalid
  instead of setting it to nil? (while still allowing a nil value)
When using the create! method, Rails raises a RecordInvalid error if validations fail. The exception should be caught and handled appropriately.
begin
  q = Quota.create!(domain_id: nil, project_id: 'invalid_id')
rescue ActiveRecord::RecordInvalid => invalid
  p invalid.record
  p invalid.record.errors
end
The invalid object should contain the failing model attributes along with the validation errors. Just note that after this block, the value of q is nil since the attributes were not valid and no object is instantiated. This is normal, predefined behavior in Rails.
Another approach is to use the combination of new and save methods. Using the new method, an object can be instantiated without being saved and a call to save will trigger validation and commit the record to the database if valid.  
q = Quota.new(domain_id: nil, project_id: 'invalid_id')
if q.save
  # quota model passes validations and is saved in DB
else 
  # quota model fails validations and it not saved in DB
  p q
  p q.errors
end
Here the object instance - q will hold the attribute values and the validation errors if any.