I'm using ConnectX5s for most, and some ConnectX4s in the older servers. Both of those cards have really come down in price in the used/ebay market. I have been playing around with some different optics - I have a bunch of CWDM4s which are very inexpensive and use a single SingleMode pair.... but of course they run hot so if you have them in servers without good air flow you might have problems.
I'm using mostly fiber just because the servers are connected to Cisco 9305 with 72 100g ports.
Oh they are both loud and eat a ton of power. I think the 9305 is at least 800 watts at idle. That is the biggest downside of the retired datacenter gear... yo u really need a dedicated room with power, cooling, and sound isolation.
I'm using mostly fiber just because the servers are connected to Cisco 9305 with 72 100g ports.