In the past I used a consumer grade Gigabit Switch and had 100MBit as well as Gigabit cards connected to it. Some of my Gigabit cards were prone to "drop" the Ethernet connection and "reconnect" with 100MBit, sometimes persistently sometimes "switching" back to Gigabit speed after some time. After removing all 100MBit NICs the problem vanished for good.
Can somebody explain to me what happened here? Were the Switch's buffers too small? How do 100MBit and Gigabit Cards coexist on a switch?
When gigabit links fall back to 100 Mbit the usual cause is bad cabling. 10, 100, 1000 and even 10,000 Mbit/s coexist nicely on the same switch (or possibly even faster but faster switches support 10 and 100 Mbit/s less and less). Check the NIC statistics for FCS errors, runts or other drops.
1000BASE-T requires all four twisted pairs to work while 100BASE-TX only uses two of them. Also, 1000BASE-T is slightly more picky on the cable as the line encoding is a bit more delicate. Quite a few devices fall back to 100BASE-TX when gigabit negotiation fails. The link may also fail altogether as well.
Everything else that's been described here - buffer overflow or flow control has NO impact on the negotiation link speed (physical layer L1) and will NEVER cause a link drop or fallback.
A switch always receives a frame completely before forwarding it (store-and forward) - most do anyway, across different link speeds all switches use store-and forward. It's no problem at all to receive a frame on one 10 Mbit/s port and forward it out another 100 Gbit/s port or vice versa.
Flow control might interfere with the effective throughput rate but never changes the physical layer link rate.
When a gigabit port tries to send a full rate flow to a 100 (or 10) Mbit/s device and flow control is active on all devices, the pause frames sent from the low-speed device will throttle the gigabit port of the sender even if another receiver might want to receive full rate - this is called head-of-line blocking and is a design flaw.
Legacy flow control should generally not be used unless you understand its function and it works in your scenario. Flow control is much better left to the transport layer (esp. TCP) or application layer protocols.
The article When Flow Control is not a Good Thing describes the case when there is a mixture of gigabit and Fast Ethernet (100 Mbps) clients in non-managed gigabit switches, where readers have reported gigabit links being forced to Fast Ethernet speeds.
The article lays the blame on 802.3x Flow Control and says :
Unfortunately, it seems (at least in small networks) that 802.3x does more harm than good. This may be partly because it duplicates the loss-based flow control mechanism already built into the TCP protocol. But whatever, the reason, I was able to confirm that the throughput loss that some people were attributing to "defective" or "low performance" switches, was in fact, due to Flow Control.
You may find a good treaty on the subject in the article To flow or not to flow?. The article gives three reasons for disabling it :
Flow Control is disabled by default on many switches, but check your switch. If enabled, try disabling it. You may need to disable it on all endpoints in some rare cases.
Intermittent results sounds a bit like CAT5e (instead of CAT6a or CAT6), or, more likely (and worse yet), CAT5 (instead of CAT5e) cabling. Although removing the 100MBit NICs wouldn't likely help much with that, so...
The switch's buffers being too small sounds like a possibility.
Unlike a "hub", a "switch" has communications that are independent per port. In other words, if you have a 24 port switch, then those are 24 independent connections. If one connection is 100Mbit, that shouldn't prevent another connection from being gigabit.
I could see some potential benefit from a switch being able to get rid of any packet data that it might be remembering. If it is successfully using gigabit connections, then it can complete conversations more quickly, and just be done with a job sooner. This may lower buffer usage. This could also reduce "overheating" issues, if the jobs get done sooner so heat generated by work can stop existing sooner.
