According to the RFC 793 Reset Generation rules:
As a general rule, reset (RST) must be sent whenever a segment arrives
which apparently is not intended for the current connection. A reset
must not be sent if it is not clear that this is the case.
There are three groups of states:
- If the connection does not exist (CLOSED) then a reset is sent
in response to any incoming segment except another reset. In
particular, SYNs addressed to a non-existent connection are rejected
by this means.
Since the port is closed (not listening or communicating) there is no connections and because of that TCP is supposed to reply with a RST package.
RFC 768 for UDP does not specify any action on a closed port but the ICMP RFC 792 specifies a message Type 3 Code 3, Destination Unreachable: Destination port unreachable that may be sent.
However, ports only actually do this if they are unfiltered. Filtered connections do not reply at all and simply drop the packet. Filtering is usually done by any firewall worthy of the name since it makes attackers jobs harder by providing less information.