Assuming you're already using the JavaTimeModule (with WRITE_DATES_AS_TIMESTAMPS as false that produces the output in your first snippet), you simply need to provide an appropriate date pattern in the @JsonFormat. For example,
@JsonFormat(pattern = "yyyy-MM-dd'T'HH:mm:ssZZZZZ")
where pattern is
Datatype-specific additional piece of configuration that may be used to further refine formatting aspects. This may, for example, determine low-level format String used for java.util.Date serialization; however, exact use is determined by specific JsonSerializer
The "Datetype-specific" here is a serializer for JsonSerializer. Since you're using JavaTimeModule (assuming!!!), that's OffsetDateTimeSerializer. That @JsonFormat pattern is interpreted as a DateTimeFormatter pattern which states
Offset Z: This formats the offset based on the number of pattern letters. One, two or three letters outputs the hour and minute, without a colon, such as '+0130'. The output will be '+0000' when the offset is zero. Four letters outputs the full form of localized offset, equivalent to four letters of Offset-O. The output will be the corresponding localized offset text if the offset is zero. Five letters outputs the hour, minute, with optional second if non-zero, with colon. It outputs 'Z' if the offset is zero. Six or more letters throws IllegalArgumentException.