I want the background to be clear..
Transparent? Yes, android does have a color defined for that. You can access it as:
@android:color/transparent
Or, in code:
Color.TRANSPARENT
@android:color/transparent is defined in res/values/colors.xml:
<color name="transparent">#00000000</color>
The alpha bits (first and second) are zero.
But, your definition for color clear:
<color name="clear">#ffffff00</color>
is not transparent. You can visualize transparent as any color with its alpha set to zero. In your definition of clear, the alpha bits are full-blown to ff - 255 - opaque.
Your color definition produces this:

Is there any way to keep the toggle and change the background color without rolling my own toggle button?
The thing is: the background color and the toggle is one drawable. The whole 'on' state is represented by one single drawable, and so is the 'off' state. You cannot simply change the color without losing the toggle feedback. To change anything about the default ToggleButton's background, you will have to provide alternate drawables for each state.
I want the background to be clear, like in the default alarm app for the days of the week..
Setting the background to transparent will not work then. I would suggest that you go through the source code and resources involved in the making of a ToggleButton. For example, the on and off states of a ToggleButton are represented by drawable resources. So, if you decide to change the background, you will need to provide ToggleButton with at least two drawables: one for each state.
Look at how the default alarm app does it. The ToggleButtons being used for days are defined as:
<ToggleButton
android:layout_width="wrap_content"
android:layout_height="48dp"
android:layout_gravity="center"
android:padding="0dp"
style="@style/body"
android:textColor="@color/clock_gray"
**android:background="@drawable/toggle_underline"**
android:clickable="false"
android:singleLine="true"/>
The drawable toggle_underline is a state-selector:
<selector xmlns:android="http://schemas.android.com/apk/res/android">
<item android:state_pressed="true" android:state_window_focused="true"
android:drawable="@drawable/toggle_underline_activated"/>
<item android:state_checked="true"
android:drawable="@drawable/toggle_underline_activated"/>
<item
android:drawable="@drawable/toggle_underline_normal"/>
</selector>
As you can see, when the ToggleButton is set to on or checked ( or when it is pressed), @drawable/toggle_underline_activated is set to the background. Otherwise, @drawable/toggle_underline_normal is used - state is off.
toggle_underline_activated and toggle_underline_normal are both 9-patch drawables.
toggle_underline_activated.9.png:

toggle_underline_normal.9.png:

You can get these drawables (and more) here: Link.
So, you can either create your own 9 patch drawables with transparent background, and use them with a state-selector drawable - or you can look at the default alarm clock project and use the drawables from its drawable-XXXX folders.