I would like to produce a completely customized legend for a ggplot. For complicated reasons that are beyond a MWE, I do not want to specify color within my various aes arguments. 
Here's an example of what I want to work:
p = ggplot() + 
  # First plot some data from one set
  geom_point(aes(x = rnorm(10), y = rnorm(10)), col = "blue") +
  # Then plot some data from another set
  geom_point(aes(x = rnorm(10) + 1, y = rnorm(10) + 1), col = "orange") + 
  # Just to make the plot mildly cleaner
  labs(x = "X", y = "Y") +
  # Now want to manually come up with my own legend
  # However ggplot seems to ignore this
  scale_colour_manual(name = "Group", 
                    values = c("Group 1" = "blue", "Group 2" = "orange"))
# Printing the plot
p
But the resulting plot ignores my scale_colour_manual call as we can see below: 
How can I independently force a custom legend?

