If people might have a significant amount of entries, performance should be considered. So, instead of searching with an n^2 algorithm, you should make use of Swifts dictionary and the corresponding hash-search to find items. 
If Id is unique for people then I would store them in a dictionary like:
    var peopleDict: [String:PeopleSelectItem] = [:]()
You can easily convert from the array you have to this dictionary:
    people.foreach {peopleDict[$0.id] = $0}
With this dictionary it's very easy to delete single entries:
    selectedPeople.foreach {peopleDict.remove($0.id)}
Optionally to switch back to an array for people you just say:
    let filteredPeople = peopleDict.values as [PeopleSelectItem]
Remarks
- I assumed, that 
selectedPeople is smaller then the base of all people. If this is not the case, you should pu selectedPeople in a dictionary.  
- did I say I like this Spark like api? I think I do so. 
 
- I just wrote that code from top of my mind. If it is not completely syntactically correct let me know and I correct it.