It looks like there is not one single answer to this question.
Of course, for JPA repositories, Lore answer is the best : use @DataJpaTest (or @JdbcTest for my use case). But be also sure to use "@AutoConfigureTestDatabase(replace = Replace.NONE)" if your test data is in your database and not in some in-memory one.
Also there is a special chapter talking about this in Spring doc :
Spring Boot’s auto-configuration system works well for applications
  but can sometimes be a little too much for tests. It often helps to
  load only the parts of the configuration that are required to test a
  “slice” of your application.
source : https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-testing.html#boot-features-testing-spring-boot-applications-testing-autoconfigured-tests
But it doesn't show all you can/need to do.
For example, I had a smtpClientService to test.
To test this service, alone in its own layer, I had to do these specific adaptations (if I omit "@AutoConfigureWebClient", I won't get RestTemplateBuilder injected) :
@RunWith(SpringRunner.class)
@SpringBootTest
@AutoConfigureWebClient
public class smtpClientServiceTest {
  @Autowired
  SmtpClientService service;
  @Configuration
  @Import(SmtpClientConfig.class)
  @ComponentScan(basePackageClasses = SmtpClientService.class)
  static class TestConfiguration {
  }  
  @Test
  public void testSendMessage() {
    (...)
  }
}