I'm trying to add some fancy stuff to my desktop, and the thing that I'm dealing with now is setting shades to all windows. I'm using compton composite manager, and everything is fine, but just look at the following images:

If a window has normal corners, everything is ok, but when I'm trying to round them, the strange effect occurs.
I'm using debian testing disto, standalone openbox, and nvidia graphic (7600GT) + proprietary drives. I have installed the following packages:
$ dpkg -l | grep -i mesa
ii glx-alternative-mesa 0.4.1 amd64 allows the selection of MESA as GLX provider
ii libegl1-mesa:amd64 9.2.2-1 amd64 free implementation of the EGL API -- runtime
ii libegl1-mesa-drivers:amd64 9.2.2-1 amd64 free implementation of the EGL API -- hardware drivers
ii libgl1-mesa-dri:amd64 9.2.2-1 amd64 free implementation of the OpenGL API -- DRI modules
ii libgl1-mesa-glx:amd64 9.2.2-1 amd64 free implementation of the OpenGL API -- GLX runtime
ii libglapi-mesa:amd64 9.2.2-1 amd64 free implementation of the GL API -- shared library
ii libglu1-mesa:amd64 9.0.0-2 amd64 Mesa OpenGL utility library (GLU)
ii libopenvg1-mesa:amd64 9.2.2-1 amd64 free implementation of the OpenVG API -- runtime
ii libtxc-dxtn-s2tc0:amd64 0~git20121227-2 amd64 Texture compression library for Mesa
ii mesa-utils 8.1.0-2 amd64 Miscellaneous Mesa GL utilities
$ dpkg -l | grep -i xorg
ii xorg 1:7.7+4 amd64 X.Org X Window System
ii xorg-docs-core 1:1.7-1 all Core documentation for the X.org X Window System
ii xserver-xorg 1:7.7+4 amd64 X.Org X server
ii xserver-xorg-core 2:1.14.3-5 amd64 Xorg X server - core server
ii xserver-xorg-input-all 1:7.7+4 amd64 X.Org X server -- input driver metapackage
ii xserver-xorg-input-evdev 1:2.8.2-1 amd64 X.Org X server -- evdev input driver
ii xserver-xorg-input-mouse 1:1.9.0-1+b1 amd64 X.Org X server -- mouse input driver
ii xserver-xorg-input-synaptics 1.7.1-2+b1 amd64 Synaptics TouchPad driver for X.Org server
ii xserver-xorg-input-vmmouse 1:13.0.0-1+b1 amd64 X.Org X server -- VMMouse input driver to use with VMWare
ii xserver-xorg-video-all 1:7.7+4 amd64 X.Org X server -- output driver metapackage
ii xserver-xorg-video-ati 1:7.2.0-1+b2 amd64 X.Org X server -- AMD/ATI display driver wrapper
ii xserver-xorg-video-cirrus 1:1.5.2-1+b1 amd64 X.Org X server -- Cirrus display driver
ii xserver-xorg-video-fbdev 1:0.4.4-1 amd64 X.Org X server -- fbdev display driver
ii xserver-xorg-video-intel 2:2.21.15-1+b2 amd64 X.Org X server -- Intel i8xx, i9xx display driver
ii xserver-xorg-video-mach64 6.9.4-1+b1 amd64 X.Org X server -- ATI Mach64 display driver
ii xserver-xorg-video-mga 1:1.6.2-1+b1 amd64 X.Org X server -- MGA display driver
ii xserver-xorg-video-modesetting 0.8.1-1 amd64 X.Org X server -- Generic modesetting driver
ii xserver-xorg-video-neomagic 1:1.2.8-1 amd64 X.Org X server -- Neomagic display driver
ii xserver-xorg-video-nouveau 1:1.0.10-1 amd64 X.Org X server -- Nouveau display driver
ii xserver-xorg-video-nvidia-legacy-304xx 304.116-2 amd64 NVIDIA binary Xorg driver (304xx legacy version)
ii xserver-xorg-video-openchrome 1:0.3.3-1 amd64 X.Org X server -- VIA display driver
ii xserver-xorg-video-qxl 0.1.0-2.1 amd64 X.Org X server -- QXL display driver
ii xserver-xorg-video-r128 6.9.1-1 amd64 X.Org X server -- ATI r128 display driver
ii xserver-xorg-video-radeon 1:7.2.0-1+b2 amd64 X.Org X server -- AMD/ATI Radeon display driver
ii xserver-xorg-video-savage 1:2.3.7-2 amd64 X.Org X server -- Savage display driver
ii xserver-xorg-video-siliconmotion 1:1.7.7-2 amd64 X.Org X server -- SiliconMotion display driver
ii xserver-xorg-video-sisusb 1:0.9.6-2 amd64 X.Org X server -- SiS USB display driver
ii xserver-xorg-video-tdfx 1:1.4.5-1 amd64 X.Org X server -- tdfx display driver
ii xserver-xorg-video-trident 1:1.3.6-2 amd64 X.Org X server -- Trident display driver
ii xserver-xorg-video-vesa 1:2.3.3-1+b1 amd64 X.Org X server -- VESA display driver
ii xserver-xorg-video-vmware 1:13.0.1-2 amd64 X.Org X server -- VMware display driver
$ dpkg -l | grep -i compton
ii compton 0.1~beta2-1 amd64 compositor for X11, based on xcompmgr
$ dpkg -l | grep -i nvidia
ii glx-alternative-nvidia 0.4.1 amd64 allows the selection of NVIDIA as GLX provider
ii libgl1-nvidia-legacy-304xx-glx:amd64 304.116-2 amd64 NVIDIA binary OpenGL libraries (304xx legacy version)
ii nvclock-gtk 0.8b4+cvs20100914-5 amd64 Overclock an NVIDIA card (GTK+ interface)
ii nvidia-detect 319.76-1 amd64 NVIDIA GPU detection utility
ii nvidia-installer-cleanup 20131102+1 amd64 cleanup after driver installation with the nvidia-installer
ii nvidia-kernel-common 20131102+1 amd64 NVIDIA binary kernel module support files
ii nvidia-legacy-304xx-alternative 304.116-2 amd64 allows the selection of NVIDIA as GLX provider
ii nvidia-legacy-304xx-driver 304.116-2 amd64 NVIDIA metapackage (304xx legacy version)
ii nvidia-legacy-304xx-kernel-dkms 304.116-2 amd64 NVIDIA binary kernel module DKMS source (304xx legacy version)
ii nvidia-settings-legacy-304xx 304.116-1 amd64 tool for configuring the NVIDIA graphics driver (304xx legacy version)
ii nvidia-support 20131102+1 amd64 NVIDIA binary graphics driver support files
ii xserver-xorg-video-nvidia-legacy-304xx 304.116-2 amd64 NVIDIA binary Xorg driver (304xx legacy version)
Kernel version:
$ uname -a
Linux morfikownia 3.11-2-amd64 #1 SMP Debian 3.11.10-1 (2013-12-04) x86_64 GNU/Linu
Does anyone know why this happens and how to fix it?
