The easiest way to do this is to think about it in a bit of a different way: "how do I get a list of all customers who have no transaction history?"
Simple! You get a list of all of the customers, join it against their transactions and filter out any customers who have a non-empty list of transactions. Or, in SQL:
SELECT 
    customer.customer_id 
FROM customer
LEFT JOIN transaction
    ON transaction.customer_id = customer.customer_id
WHERE
    transaction.transaction_id IS NULL
Note that you cannot simply use the transaction table like you're attempting. It is not a complete list of customer_id but rather it contains only IDs of customers who have an order.
Instead of operating on transaction and finding customers with no transactions (which you literally cannot do), you must find all customers and then filter by those who have no transactions. Similar concept, just opposite order.