Your custom composition brush is not a UIElement and that's why it cannot be placed onto the XAML visual tree directly.
Try adding it as a brush for an element -
<Grid>
    <Grid.Background>
        <local:BackdropBlurBrush BlurAmount="5" />
    </Grid.Background>
</Grid>
You normally want to place your blur brush on top of your background image like this -
<Grid x:Name="Root">
    <Grid.Background>
        <ImageBrush Stretch="UniformToFill" ImageSource="background.jpg"/>
    </Grid.Background>
    <Rectangle>
        <Rectangle.Fill>
            <local:BackdropBlurBrush BlurAmount="5" />
        </Rectangle.Fill>
    </Rectangle>
</Grid>
