I/O space / ports is a x86-specific legacy creation. Back in the good old says, there were multiple different address spaces (you may have seen the recent trend towards shared memory address space to better share CPU/GPU resources, such as in the PlayStation 4 and Xbox One) for addressing memory and addressing "I/O peripherals".
The I/O space was used to talk to "real" hardware peripherals, like your serial port or parallel port. Conventionally, at I/O space address 0x3F8, the first serial port on a system would be located there. To talk to this device, rather than use conventional memory access instructions, you'd issue I/O access instructions instead; on Linux for example you would use outb(), outl() or outw(), depending on the width you intend to use. Again, this is a completely different memory space than the system memory map.
As PCs and x86 matured (read: we could now natively address 32-bits of memory and more), memory-mapped I/O become more common. Now, an arbitrary memory location in the system memory map (say from 0 to 0xFFFFFFFF) could be mapped to a specific device dynamically. A PCI device will have BAR (base address registers) that require a certain amount of memory OR I/O space. Therefore, a simple read/write to say memory address 0x80000000 may actually map to a physical PCI device that flips a LED when you write a '1' to that location. This is a gross over-simplification that neglects the existence of virtual memory / per-process address space and even user- vs kernel-space addresses.
So to summarize, in my opinion and view as an electrical / computer engineer, an "IO Port" refers to a location in the x86-architecture IO Memory space that maps to some specific device. A physical port would be some of real, tangible connector on your computing device be it USB, serial, whatever.
As a fun example, if you remember playing old DOS games like Star Wars: Dark Forces, you'll recall setting up say your SoundBlaster Pro. You'll set parameters like Port 220, IRQ 5, DMA 1. This is telling the game that your soundcard is located in I/O space at 0x220, using IRQ channel 5 (remember, you didn't have a high-end APIC at this point, you probably had an Intel 8259 or two), and DMA channel 1.