This currently accepted answer (May 2012) is mostly correct, except for when you are using session-based authentication. It's also worth mentioning the role of CORS.
The simple scenario is that you visit foo.com and the website executes JavaScript to make an AJAX-based DELETE request to api.com/users/123 and ends up deleting the user on your behalf. Now this isn't always possible because of CORS -- browsers will prevent foo.com from making a request to api.com unless api.com explicitly whitelists foo.com.
This also assumes that you are using session-based authentication for your APIs as opposed to token-based authentication. In session-based authentication, any user who is logged in to api.com can execute requests while they remain logged in. If you have token-based authentication (each request must be crafted with an HTTP Authorization header containing the auth token) then you are safe. Session-based authentication implicitly sends the auth token via cookies.
A slightly worse scenario is if one of your trusted CORS domains becomes compromised - say you have a form which doesn't sanitize JavaScript and a user manages to inject JS onto your site through that form. If you are using session-based authentication, then an authenticated user visiting the page will see the JavaScript run and make an API request. This could be disastrous and a very real possibility if you are using session-based authentication for your API.