After looking into the code, realize I just need to save the value, as what @Parth mentioned in the comment above.
savedStateHandle.set(KEY, textLiveData.value)
Explanation
If we look into the set of SavedStateHandle.java
@MainThread
public <T> void set(@NonNull String key, @Nullable T value) {
    validateValue(value);
    @SuppressWarnings("unchecked")
    MutableLiveData<T> mutableLiveData = (MutableLiveData<T>) mLiveDatas.get(key);
    if (mutableLiveData != null) {
        // it will set value;
        mutableLiveData.setValue(value);
    } else {
        mRegular.put(key, value);
    }
}
This shows that it will put the data into the mutableLiveData that we store in mLiveDatas if there is one. 
To ensure that our liveData is in mLiveDatas, we just need to use textLiveData = savedStateHandle.getLiveData(KEY) at the start. Checkout the getLiveData of 
SavedStateHandle.java
@NonNull
private <T> MutableLiveData<T> getLiveDataInternal(
        @NonNull String key,
        boolean hasInitialValue,
        @Nullable T initialValue) {
    MutableLiveData<T> liveData = (MutableLiveData<T>) mLiveDatas.get(key);
    if (liveData != null) {
        return liveData;
    }
    SavingStateLiveData<T> mutableLd;
    // double hashing but null is valid value
    if (mRegular.containsKey(key)) {
        mutableLd = new SavingStateLiveData<>(this, key, (T) mRegular.get(key));
    } else if (hasInitialValue) {
        mutableLd = new SavingStateLiveData<>(this, key, initialValue);
    } else {
        mutableLd = new SavingStateLiveData<>(this, key);
    }
    mLiveDatas.put(key, mutableLd);
    return mutableLd;
}
It will create one and put into mLiveDatas when we request for one, if there isn't one already.