First, create a vector with the letters in the desired order. Then match* the vector with the variable to be sorted. match returns indices of (first) matches, which can be plugged into slice:
library(dplyr)
# create a vector with letters in the desired order
x <- c("C", "A", "B")
DT %>%
slice(match(x, category))
# category b
# 1 C 3
# 2 A 1
# 3 B 2
Another way would be to convert "category" to a factor, set levels to the desired order, and use arrange:
DT %>%
mutate(category = factor(category, levels = x)) %>%
arrange(category)
# category b
# 1 C 3
# 2 A 1
# 3 B 2
*The match method is inspired by this answer.