You have to let your view model implement INotifyDataErrorInfo MSDN. Example. Example from MSDN (Silverlight). 
Since .Net 4.5 this is the recommended way to introduce validation to your view models and will help you to solve your propblem.
When implementing this interface you will have to provide a HasErrors property that you can bind to. INotifyDataErrorInfo replaces the obsolete IDataErrorInfo.
Binding to the Validation.HasError directly, as you did in your triggers, will not work since Validation.HasError is a read-only attached property and therefore doesn't support binding. To prove this I found this statement on MSDN:
... read-only dependency properties aren't appropriate for many of the scenarios for which dependency properties normally offer a solution (namely: data binding, directly stylable to a value, validation, animation, inheritance).
How INotifyDataErrorInfo works
When the ValidatesOnNotifyDataErrors property of Binding is set to true, the binding engine will search for an INotifyDataErrorInfo implementation on the binding source to subscribe to the ErrorsChanged event.
If the ErrorsChanged event is raised and HasErrors evaluates to true, the binding will invoke the GetErrors() method for the actual property to retrieve the particular error message and apply the customizable validation error template to visualize the error. By default a red border is drawn around the validated element.
How to implement INotifyDataErrorInfo
The CustomVertex class is actually the ViewModel for the DataGrid columns since you are binding to it's properties. So it has to implement the INotifyDataErrorInfo. It could look like this:
public class CustomVertex : INotifyPropertyChanged, INotifyDataErrorInfo
{
    public CustomVertex()
    {
      this.errors = new Dictionary<string, List<string>>();
      this.validationRules = new Dictionary<string, List<ValidationRule>>();
      this.validationRules.Add(nameof(this.X), new List<ValidationRule>() {new DoubleValidationRule()});
      this.validationRules.Add(nameof(this.Y), new List<ValidationRule>() {new DoubleValidationRule()});
    }
    public bool ValidateProperty(object value, [CallerMemberName] string propertyName = null)  
    {  
        lock (this.syncLock)  
        {  
            if (!this.validationRules.TryGetValue(propertyName, out List<ValidationRule> propertyValidationRules))
            {
              return;
            }  
            // Clear previous errors from tested property  
            if (this.errors.ContainsKey(propertyName))  
            {
               this.errors.Remove(propertyName);  
               OnErrorsChanged(propertyName);  
            }
            propertyValidationRules.ForEach(
              (validationRule) => 
              {
                ValidationResult result = validationRule.Validate(value, CultuteInfo.CurrentCulture);
                if (!result.IsValid)
                {
                  AddError(propertyName, result.ErrorContent, false);
                } 
              }               
        }  
    }   
    // Adds the specified error to the errors collection if it is not 
    // already present, inserting it in the first position if isWarning is 
    // false. Raises the ErrorsChanged event if the collection changes. 
    public void AddError(string propertyName, string error, bool isWarning)
    {
        if (!this.errors.ContainsKey(propertyName))
        {
           this.errors[propertyName] = new List<string>();
        }
        if (!this.errors[propertyName].Contains(error))
        {
            if (isWarning) 
            {
              this.errors[propertyName].Add(error);
            }
            else 
            {
              this.errors[propertyName].Insert(0, error);
            }
            RaiseErrorsChanged(propertyName);
        }
    }
    // Removes the specified error from the errors collection if it is
    // present. Raises the ErrorsChanged event if the collection changes.
    public void RemoveError(string propertyName, string error)
    {
        if (this.errors.ContainsKey(propertyName) &&
            this.errors[propertyName].Contains(error))
        {
            this.errors[propertyName].Remove(error);
            if (this.errors[propertyName].Count == 0)
            {
              this.errors.Remove(propertyName);
            }
            RaiseErrorsChanged(propertyName);
        }
    }
    #region INotifyDataErrorInfo Members
    public event EventHandler<DataErrorsChangedEventArgs> ErrorsChanged;
    public System.Collections.IEnumerable GetErrors(string propertyName)
    {
        if (String.IsNullOrEmpty(propertyName) || 
            !this.errors.ContainsKey(propertyName)) return null;
        return this.errors[propertyName];
    }
    public bool HasErrors
    {
        get { return errors.Count > 0; }
    }
    #endregion
    protected virtual void OnPropertyChanged([CallerMemberName] string propertyName = null)
    {
      this.PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName));
    }
    private double x;
    public double X 
    { 
      get => x; 
      set 
      { 
        if (ValidateProperty(value))
        {
          this.x = value; 
          OnPropertyChanged();
        }
      }
    }
    private double y;
    public double Y 
    { 
      get => this.y; 
      set 
      { 
        if (ValidateProperty(value))
        {
          this.y = value; 
          OnPropertyChanged();
        }
      }
    }
    private Dictionary<String, List<String>> errors;
    // The ValidationRules for each property
    private Dictionary<String, List<ValidationRule>> validationRules;
    private object syncLock = new object();
}
The View:
<DataGrid CanUserSortColumns="False" CanUserAddRows="True" ItemsSource="{Binding Vertices}" AutoGenerateColumns="False">
    <DataGrid.Columns>
        <DataGridTextColumn x:Name="YColumn" 
                            Width="*" 
                            Header="Latitude" 
                            Binding="{Binding Y, ValidatesOnNotifyDataErrors=True}" 
                            Validation.ErrorTemplate="{DynamicResource ValidationErrorTemplate}" />
        <DataGridTextColumn x:Name="XColumn" 
                            Width="*" 
                            Header="Longitude" 
                            Binding="{Binding X, ValidatesOnNotifyDataErrors=True}" 
                            Validation.ErrorTemplate="{DynamicResource ValidationErrorTemplate}" />            
    </DataGrid.Columns>
</DataGrid>
The following is the validation error template, in case you like to customize the visual representation (optional). It is set on the validated element (in this case the DataGridTextColumn) via the attached property Validation.ErrorTemplate (see above):
<ControlTemplate x:Key=ValidationErrorTemplate>
    <StackPanel>
        <!-- Placeholder for the DataGridTextColumn itself -->
        <AdornedElementPlaceholder x:Name="textBox"/>
        <ItemsControl ItemsSource="{Binding}">
            <ItemsControl.ItemTemplate>
                <DataTemplate>
                    <TextBlock Text="{Binding ErrorContent}" Foreground="Red"/>
                </DataTemplate>
            </ItemsControl.ItemTemplate>
        </ItemsControl>
    </StackPanel>
</ControlTemplate>
The Button that will be disabled when the validation fails (since I don't know where this button is located in the visual tree I will assume that it shares the DataContext of a DataGrid column, the CustomVertex data model):
<Button>
    <Button.Style>
        <Style TargetType="Button">
            <Setter Property="IsEnabled" Value="True" />
            <Style.Triggers>
                <DataTrigger Binding="{Binding Path=HasErrors}" Value="True">
                    <Setter Property="IsEnabled" Value="False" />
                </DataTrigger>
            </Style.Triggers>
        </Style>
    </Button.Style>
</Button>
There are many examples on the web. I updated the links to provide some content to start with. 
I recommend moving the implementation of INotifyDataErrorInfo into a base class together with INotifyPropertyChanged and let all your view models inherit it. This makes the validation logic reusable and keeps your view model classes clean.
You can change the implementation details of INotifyDataErrorInfo to meet requirements.
Remarks: The code is not tested. The snippets should work, but are intended to provide an example how the INotifyDataErrorInfo interface could be implemented.