Having an inline item next to an inline-flex with a nested flex box messes with the vertical alignment (it ignores the top margin), one solution I found was to put a ::before in the inline-flex item, but I'm not really sure why this fixes it.
The first one ignores the top margin on the label, the top margin works on the second one, because of the ::before.
label {
  margin: 20px 5px 0 0;
}
.input-container {
  display: inline-flex;
}
.with-before::before {
  content: '';
}
.buttons {
  display: flex;
  flex-direction: column;
}
<div>
  <label>Top margin ignored:</label>
  <div class="input-container">
    <div class="buttons">
      <button><</button>
      <button>></button>
    </div>
    <input type="text"/>
  </div>
</div>
<br/>
<div>
  <label>Top margin works:</label>
  <div class="input-container with-before">
    <div class="buttons">
      <button><</button>
      <button>></button>
    </div>
    <input type="text"/>
  </div>
</div>