As you have already said, setting the Background of the button will not work, because the default style and control template will override the background in states like Mouse Over, Pressed or Disabled.
A more lightweight variant than creating a custom template would be to create a Button style that displays the color as Content using a content template. A Border would display the color if the content type is a Brush. A trigger ensures that the Opacity is adapted depending on IsEnabled of the parent Button.
<Style x:Key="ColorSwatchButtonStyle" TargetType="{x:Type Button}" BasedOn="{StaticResource {x:Type Button}}">
<Setter Property="HorizontalContentAlignment" Value="Stretch"/>
<Setter Property="VerticalContentAlignment" Value="Stretch"/>
<Setter Property="ContentTemplate">
<Setter.Value>
<DataTemplate DataType="{x:Type Brush}">
<Border Background="{Binding}">
<Border.Style>
<Style TargetType="{x:Type Border}">
<Setter Property="Opacity" Value="1.0"/>
<Style.Triggers>
<DataTrigger Binding="{Binding IsEnabled, RelativeSource={RelativeSource AncestorType={x:Type Button}}}" Value="False">
<Setter Property="Opacity" Value="0.3"/>
</DataTrigger>
</Style.Triggers>
</Style>
</Border.Style>
</Border>
</DataTemplate>
</Setter.Value>
</Setter>
</Style>
If you do not want to put a style in the Border, use data template triggers instead.
<Style x:Key="ColorSwatchButtonStyle" TargetType="{x:Type Button}" BasedOn="{StaticResource {x:Type Button}}">
<Setter Property="HorizontalContentAlignment" Value="Stretch"/>
<Setter Property="VerticalContentAlignment" Value="Stretch"/>
<Setter Property="ContentTemplate">
<Setter.Value>
<DataTemplate DataType="{x:Type Brush}">
<Border x:Name="ColorBorder" Background="{Binding}"/>
<DataTemplate.Triggers>
<DataTrigger Binding="{Binding IsEnabled, RelativeSource={RelativeSource AncestorType={x:Type Button}}}" Value="False">
<Setter TargetName="ColorBorder" Property="Opacity" Value="0.3"/>
</DataTrigger>
<DataTrigger Binding="{Binding IsMouseOver, RelativeSource={RelativeSource AncestorType={x:Type Button}}}" Value="True">
<Setter TargetName="ColorBorder" Property="Opacity" Value="0.8"/>
</DataTrigger>
<DataTrigger Binding="{Binding IsPressed, RelativeSource={RelativeSource AncestorType={x:Type Button}}}" Value="True">
<Setter TargetName="ColorBorder" Property="Opacity" Value="0.5"/>
</DataTrigger>
</DataTemplate.Triggers>
</DataTemplate>
</Setter.Value>
</Setter>
</Style>
Apply the style explicitly or remove the x:Key to make the style implicit and apply it to all buttons that are in scope. Set the color as Content of the Button.
<Button Style="{StaticResource ColorSwatchButtonStyle}"
Content="{x:Static Brushes.Red}"/>
As a note on this approach, since the content Border occludes the background of the button, you will not see the effects in the aforementioned states. However, you can add triggers to this style to display your colors in the Mouse Over and Pressed states appropriately as well.
<DataTrigger Binding="{Binding IsMouseOver, RelativeSource={RelativeSource AncestorType={x:Type Button}}}" Value="True">
<Setter Property="Opacity" Value="0.8"/>
</DataTrigger>
<DataTrigger Binding="{Binding IsPressed, RelativeSource={RelativeSource AncestorType={x:Type Button}}}" Value="True">
<Setter Property="Opacity" Value="0.5"/>
</DataTrigger>
This is just an example, of course, you could also change the BorderBrush and BorderThickness or add other elements to the template as well. You are not limited to change Opacity only.
Alternatively, you can always copy the default style and control template using Blend or Visual Studio. Then you can adapt the affected states of the button to display the Background in a different manner.