There are several options you could try, actually custom deserializer/serializer would probably make sense, but you also can achieve this with @JsonIdentityInfo(for deserialization) + @JsonIdentityReference(if you need serialization as integer) annotations.
Deserialization
Work both for 
{ "category":1 }
{ "category":{ "id":1 }
So you need to annotate every class that can be deserialized from its id with @JsonIdentityInfo
@JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class,
        property = "id", 
        scope = Product.class,  // different for each class
        resolver = MyObjectIdResolver.class)
The hard part here is that you actually have to write custom ObjectIdResolver that can resolve objects from your db/other source. Take a look at my simple reflection version in MyObjectIdResolver.resolveId method in example below:
private static class MyObjectIdResolver implements ObjectIdResolver {
    private Map<ObjectIdGenerator.IdKey,Object> _items  = new HashMap<>();
    @Override
    public void bindItem(ObjectIdGenerator.IdKey id, Object pojo) {
        if (!_items.containsKey(id)) _items.put(id, pojo);
    }
    @Override
    public Object resolveId(ObjectIdGenerator.IdKey id) {
        Object object = _items.get(id);
        return object == null ? getById(id) : object;
    }
    protected Object getById(ObjectIdGenerator.IdKey id){
        Object object = null;
        try {
            // todo objectRepository.getById(idKey.key, idKey.scope)
            object = id.scope.getConstructor().newInstance(); // create instance
            id.scope.getField("id").set(object, id.key);  // set id
            bindItem(id, object);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return object;
    }
    @Override
    public ObjectIdResolver newForDeserialization(Object context) {
        return this;
    }
    @Override
    public boolean canUseFor(ObjectIdResolver resolverType) {
        return resolverType.getClass() == getClass();
    }
}
Serialization
Default behavior
{ "category":{ "id":1 , "name":null} , secondaryCategories: [1 , { { "id":2 , "name":null} ]}
Default behavior is described here: https://github.com/FasterXML/jackson-databind/issues/372
and will produce object for the first element and id for each element after. An ID/reference mechanism  in Jackson works so that an object instance is only completely serialized once and referenced by its ID elsewhere.
Option 1. (Always as id)
Works for 
{ "category":1 , secondaryCategories:[1 , 2]}
Need to use @JsonIdentityReference(alwaysAsId = true) above each object field(can uncomment in demo at the bottom of the page)
Option 2. (Always as full object representation)
Works for 
{ "category" : { "id":1 , "name":null} , secondaryCategories: [{ "id":1 , "name":null} , { "id":2 , "name":null}]}
This option is tricky because you will have to remove all the IdentityInfo for serialization somehow. One option could be to have 2 object mappers. 1 for serialization and 2-nd for deserialization and configure some sort of mixin or @JsonView
Another approach that is easier to implement is to use SerializationConfig to ignore @JsonIdentityInfo annotations completely
@Bean
public ObjectMapper objectMapper() {
    ObjectMapper mapper = new ObjectMapper();
    SerializationConfig config = mapper.getSerializationConfig()
            .with(new JacksonAnnotationIntrospector() {
        @Override
        public ObjectIdInfo findObjectIdInfo(final Annotated ann) {
            return null;
        }
    });
    mapper.setConfig(config);
    return mapper;
}
Probably the better approach would be to actually define @JsonIdentityInfo for deserializerconfig the same way and remove all annotations above classes. Something like this 
At this point you probably wish you just wrote custom serializer/deserializer 
Here is working (simple Jackson without spring) demo: 
import com.fasterxml.jackson.annotation.*;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.util.Set;
public class Main {
    @JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class,
            property = "id",
            resolver = MyObjectIdResolver.class,
            scope = Category.class)
    public static class Category {
        @JsonProperty("id")
        public int id;
        @JsonProperty("name")
        public String name;
        public int getId() {
            return id;
        }
        public void setId(int id) {
            this.id = id;
        }
        @Override
        public String toString() {
            return "Category{" +
                    "id=" + id +
                    ", name='" + name + '\'' +
                    '}';
        }
    }
    @JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class,
            property = "id",
            resolver = MyObjectIdResolver.class,
            scope = Product.class)
    public static class Product {
        @JsonProperty("id")
        public int id;
        @JsonProperty("name")
        public String name;
        // Need @JsonIdentityReference only if you want the serialization
        // @JsonIdentityReference(alwaysAsId = true)
        @JsonProperty("category")
        Category category;
        // Need @JsonIdentityReference only if you want the serialization
        // @JsonIdentityReference(alwaysAsId = true)
        @JsonProperty("secondaryCategories")
        Set<Category> secondaryCategories;
        public int getId() {
            return id;
        }
        public void setId(int id) {
            this.id = id;
        }
        @Override
        public String toString() {
            return "Product{" +
                    "id=" + id +
                    ", name='" + name + '\'' +
                    ", category=" + category +
                    ", secondaryCategories=" + secondaryCategories +
                    '}';
        }
    }
    private static class MyObjectIdResolver implements ObjectIdResolver {
       private Map<ObjectIdGenerator.IdKey,Object> _items;
        @Override
        public void bindItem(ObjectIdGenerator.IdKey id, Object pojo) {
            if (_items == null) {
                _items = new HashMap<ObjectIdGenerator.IdKey,Object>();
            } if (!_items.containsKey(id))
                _items.put(id, pojo);
        }
        @Override
        public Object resolveId(ObjectIdGenerator.IdKey id) {
            Object object = (_items == null) ? null : _items.get(id);
            if (object == null) {
                try {
                    // create instance
                    Constructor<?> ctor = id.scope.getConstructor();
                    object = ctor.newInstance();
                    // set id
                    Method setId = id.scope.getDeclaredMethod("setId", int.class);
                    setId.invoke(object, id.key);
                    // https://github.com/FasterXML/jackson-databind/issues/372
                    // bindItem(id, object); results in strange behavior
                } catch (NoSuchMethodException | IllegalAccessException
                        | InstantiationException | InvocationTargetException e) {
                    e.printStackTrace();
                }
            }
            return object;
        }
        @Override
        public ObjectIdResolver newForDeserialization(Object context) {
            return new MyObjectIdResolver();
        }
        @Override
        public boolean canUseFor(ObjectIdResolver resolverType) {
            return resolverType.getClass() == getClass();
        }
    }
    public static void main(String[] args) throws Exception {
        String str = "{ \"name\": \"name\", \"category\": {\"id\": 2 }, " +
                "\"secondaryCategories\":[{\"id\":3},{\"id\":4},{\"id\":5}]}";
        // from  str
        Product product = new ObjectMapper().readValue(str, Product.class);
        System.out.println(product);
        // to json
        String productStr = new ObjectMapper().writeValueAsString(product);
        System.out.println(productStr);
        String str2 = "{ \"name\": \"name\", \"category\":  2, " +
                "\"secondaryCategories\": [ 3,  4,  5] }";
        // from  str2
        Product product2 = new ObjectMapper().readValue(str2, Product.class);
        System.out.println(product2);
        // to json
        String productStr2 = new ObjectMapper().writeValueAsString(product2);
        System.out.println(productStr2);
    }
}