I have this data
set.seed(28100) 
my_data <- data.frame(cat = factor(sample(c('cat1','cat2'), 50, replace = T)),
                      x = sample(1:100, 50, replace = T),
                      y = sample(1:100, 50, replace = T),
                      value = sample(1:100), 50, replace = T)
And I want to color my observations based on two dimensions. First, I want to distinguish between the categories by colouring based on the factor variable cat. Second, I want to scale the colour based on the continuous variable value.  
This doesn't give the expected result:
require(ggplot2)
ggplot(my_data, aes(x=x, y=y, colour=cat, fill=value)) +
  geom_point(size=5) +
  theme_bw()
Is it possible by tweaking the arguments of ggplot() or geom_point()?