RST is one of the eight flags in the TCP-header that identifies that the TCP connection should be reset; colloquially, it's a small TCP packet that resets the connection, sent when the connection gets terminated with the close(2) system call.
Other common TCP header flags include SYN, ACK and FIN. Flag validity depends on the state of the TCP connection.
References: