NBASE-T vs SFP+

So, I've been thinking about this for a few days now and wanted to suss out some items...

So, now that AP's are coming out with 2.5gbps Ethernet ports on them and finally coming in under $300 price points it's a bit more feasible.

I dug around and found several single port 2.5G cards in the $30 range:
SD-PEX24065
EP-9635

As well there are several 2.5G USB options as well:
$20-$35
UGREEN
CableCreation
FOXMM
NT-S25G

Then the middle price point of $60 for 5G options:
TUC-ET5G
NT-SS5G
QNA-UC5G1T ($79)
(there weren't any 5G cards w/o bumping to a 10G multi-speed)

10G/Multi-Speed options single port ~$90-$110
SD-PEX24055
QXG-10G1T
XG-C100C
TEG-10GECTX
ST5GPEXNB

Dual port 10G multi-speed jump to:
SIIG Dual 2.5G $110
QXG-10G2T-107 $179 (2-port)
ST10GSPEXNDP $207 (2 port)

POE Injectors that hit 2.5G+
TPE-215GI $55 @ 2.5G
PT-PSE104GB-60-10 $69 @ 10G

So, the cheapest option would be:
2.5 PCI / USB @ $30
POE 2.5G @ $55
AP @ ~$300
-------------------------
~$400 w/ tax/shipping

AP's currently selling under $300 and 2.5G Port for backhaul to router/switch
NWA210AX
EWS377AP (cheaper on e-bay)

This led to looking at dual port or more 2.5+ cards and wondering if there was a cheaper way to get more port density than paying ~$200 and being stuck with 2 ports that are static with Copper.

I started looking at the SFP+ cards and found several SFP+ modules reporting multi-speed options for copper in the $35-$50 range. I couldn't find documentation though on cards stating they would indeed move data at sub 10G speeds if the proper module were to be used in place of a normal 1G/10G module.

GW-FIBER
10G-SFP-T-Ubiquiti
Q-SFP-10G-T-Cisco
Cisco 10G-SFP-T

SFP+ cards with 2-4 ports are anywhere from $35 on e-bay up to $225+ for newer releases including some PCI3 options, while PCI3 opens using an x1 slot vs a minimum of x4 with PCI2 it's just something to consider.

The big question is if anyone's using these cards w/ the multi-speed modules and if so which card chips would you opt for over another. Intel is the most expensive at almost double some of the lesser known chips like Aquantia.

I'm currently running a 8700K / Linux / router/WAP (hostapd) / FW and some other functions off this box Temps aren't an issue but space is with a mATX form factor limiting the slots I can expand into though. While right now I have a single slot open and can juggle cards from one slot to another as needed this type of step up to an AP externally used would free up a couple of slots to put the new gear into.

Leaving 3 open slots since there's a DVR card in there as well. Ideally leaving an open slot for troubleshooting anything that might come up would be a consideration which lead me to looking at some 4-port SFP cards in case I need to add some ports. That led to some questionable 4-port options on e-bay but, they were also dirt cheap.

Riverbed @ $99 4-ports
While the Riverbed + 2 SFP's would be slightly more up front than a 2-port it's effectively a 3 port switch option @ 10G for the same price as a 2 port fixed copper card like QXG-10G2T-107. It would be nice to maintain the 4-port density while upgrading from 1G to 10G at the same time.

Some other 2-port models:
IBM @ $70
Broadcom @ $135
HP @ $136

Optimal / slightly more $$:
Riverbed @ $99
4 SFP+ @ 10G/5G/2.5G/1G // $140 (($60/10G port))
vs.
2-port 10G Copper ~$185 ($45 difference / twice the ports) $95/10G port and can't swap for fiber later on)
POE 10G @ $69
AP @ ~$300
---------------------------------------------
~$650



Another thing prompting this is Netgear released a 2.5G port on their new CM2000 @ $249 as the increase from 1gbps plans are coming down from the cable providers in the near future as the XB7 from Comcast also has a 2.5G port on it as well.

I also stumbled upon a dual-port 2.5G @ $42 card that kind of makes sense for streamline things from a 4-port 10G card to just swapping out my existing wifi card for it and then adding a 2-port 10G might work for internal traffic.

Motorola is also pending a release of MB8611 which will be direct competition with the XB7 / CM200 in the coming months.

 

The best trancievers for SFP+ to RJ45 are 10GTek ($50-60) and Wiitek ($45-47) both equally good for peak throughput and relatively lower (slightly) wattage, they also support Multi-Gig operation vs the Cisco which is not that good compared to them and if I recall limited to 1/10 Gbps operation with no Multi-Gig support.

For ethernet cards if using in a firewall/router regardless of RJ45 or SFP+ I'd prefer a used Chelsio or Intel over other brands like Broadcom, Aquantia etc in terms of reliability and support, Intel for example puts in work in FreeBSD and Linux, with relatively responsive reactions to bug reports from my experience. They also usually have properly working hardware offloads with less buggy drivers. Realtek is probably the worst, disliked by most firewall/router distros I'v seen and also FreeNAS. Mellanox I've heard is also decent, now owned by nVidia. One can find used/server pulled Quad ports for ~$100-140 for the Chelsio T540 SO-CR. Also SFP+ to RJ45 is something to be avoided unless absolutely needed like a modem to adapter link. For RJ45 cards I know the Intel X550 supports Multi-Gig on Linux and I think recently Windows Multi-Gig support was added for the card, the X710-T2L on the other hand has Multi-Gig support on all OSs at the outset but is at a steep price.

RJ45 to SFP+ transceivers run quite hot and draw a lot of power, running multiple side by side would be kinda bad at higher speeds like 5/10 Gbps due to heat output. Thats why Cisco and others recommend spacing those type of transceivers with one open port between on their switches for higher than 1 Gbps links.

Comcast gives upwards of 1.2 Gbps peak on my gig profile while averaging around 1.1 Gbps, I have an Intel X710-T2L card in my pfSense box to make use of that in terms of single target peak speeds. Along with the pfSense box I'm using a Zyxel XS1930-10 (8x 10Gb Multi-Gig & 2x SFP+ Ports) switch (got it at a good discount from a Zyxel reseller), with a Netgear RAX120 now converted as AP. I was fortunate enough to receive the CAX80 2.5 Gbps modem from Netgear for evaluation.

 

Good info....

I guess the primary thing I'm trying to wrap my head around is whether the card controls the speed or the SFP+ controls the link negotiation and then just passes the data through the card.

X710 is going to have to wait since they average ~$500

I dug through some more forums and such and found all of what you mentioned as well as some Broadcom based cards that were able to do 1/2.5/10G.

My focus is to get a 2.5G port / POE / AP

Secondary is to probably do a 5G USB > Server for backups .... since a 5G card doesn't exist in the wild it's either have a dongle off the back or up it to 10G card. If I'm going to future proof things a bit more then why not do 2 ports 10G but then cost per port comes into play and quad port cards aren't that much more.

I don't personally need a switch but stumbled upon some cheap 2.5G switches if the desire to mux the connection vs use the server as the switch as I have been doing with the 4 port gig cards in the past.

Compatibility between card & SFP is my main concern before dealing with the OS compatibility side of it. If there's a restriction on the chipset of the card providing the middle ground sync rates then it wouldn't make sense to attempt it and end up with a quad 1G setup for 2-3X the price I paid for the quad port gig I350 I'm using now. SFP+ just seems more sensible than static copper ports even though both theoretically they both achieve the same result other than port cost.

When it comes to the OS side there's almost always a hack to make things work if you think outside the box. I run Ubuntu but use it more as a headless through command line.. really most distros are defined by their GUI and have the same foundation. Commands vary slightly between maybe 3 branches of *nix. I'm already running kernel 5.9 after backing down to 5.8 while the RC's were being pushed due to some oddities popping up but the final release seems to have resolved those.

 

That dual port 2.5G card is a SD-PEX24066 and it mux's the signal for sharing through an ASM1182e PCIe switch / chip to provide 2 ports that exceed the throughput of the PCI interface

Less interesting now finding the shared bandwidth going on.

Found a 5Gx2 card though for $88.50 that would be a good intermediate step between 10Gx2 price wise. QXG-5G2T-111C

Discovered the Engenius AP doesn't support 160mhz channels as it should prior to pulling the trigger on a couple of them on e-bay under $200... reviews though show decent speeds considering the lack of 160 but for a couple bucks more the Zyxel has it.

 

I guess an update is due here. I ended up going with the following based on disk speed / costs.

Testing my Raid showed a max spinner speed of ~400MB/s in a R10 configuration using 8TB 5400's. This puts me just below 5GE for network speed.

I picked up:
QXG-5G4T-111C - 4 port 5GE / temps are good on this one with 3 ports in use (2 to CM / 1 LAN/AP)
-- 40C on the idle port and up to 50C on the ones in use
NT-SS5G - works well for bulk transfers between laptop / server | gets full speed from Raid to NVME on laptop
PT-PSE104GB-60-10 - went with this to streamline the # of things being plugged in
ZADA37MC-06 - added one of these to split the PC/POE power into a single cable from the surge protector

NWA210AX - ultimately this had more features than the other one, working with Zyxel to get a firmware update that limits the speeds to 1GE over wifi and potentially have them add 80+80 vs the 160mhz option to split but get higher spectrum usage in cases when 160mhz just won't work with density congestion of other AP's

 

Quick question regarding, the SFP+ to RJ45 10GBASET modules. You listed the Ipolex below, for example as multi-speed:

https://www.amazon.com/dp/B01M5LIUK...olid=31WCNUVA7YOTT&psc=1&ref_=lv_ov_lig_dp_it

It doesn't seem to say so explicitly. Can it be assumed that 10GBASET automatically supports lower speeds, 5, and 2.5, or is it not always the case. It would seem reasonable that lower speeds are supported, however, the NBASE standard is newer than 10GBASET hence a potential for issues I guess.

Many thanks.

 

The potential issue is a device not supporting 802.3bz which is the NBASE-T standard. https://en.wikipedia.org/wiki/2.5GBASE-T_and_5GBASE-T

 

No some don’t support NBASE-T, 10GTek and Wiitek are probably the best ones, power use and throughput wise slightly better than ipolplex and others. Wiitek ironically was also among the cheapest at one time.

Also don’t take it for granted your switch / adapter will like these types of modules. Some people have stated high retransmission rates / errors regardless of which RJ45 to SFP+ tranciever module used and having more than one next to each other if ports are too close can be bad as they can generate a lot of heat.


Here’s test data from ServeTheHome, they later tested the 10GTek as well which is a teeny bit better than the Wiitek.
https://www.servethehome.com/sfp-to-10gbase-t-adapter-module-buyers-guide/

 

Thanks. I figured dealing with these SFP+ transceivers looks like pain (non-trivial extra cost, heat, potential compatibility issues, some people reported modules getting stuck inside the switch lol) that would primarily be justified if RJ45 route is not viable. Therefore I provisionally shortlisted a Netgear switch with 8 10G/Mulitmode with just one SFP+ dual-mode just in case. There are cheaper SFP+ options, but once you add the transceiver cost, it works out about the same.

 

Yeah that’s why I said look at a switch with both SFP+ and RJ45. If going only SFP+ there are cheap options like a microtik 4 port 10G model for like $100 but obviously the average person doesn’t have SFP+ only clients.

SFP+ to SFP+ DAC cables can be relatively cheap at like $15 though assuming you have SFP+ on client and switch/adapter side.

 

Internal LAN PC <> Server copy speed over WIFI using the NWA210AX


Going with the QXG-5G4T-111C serves a couple of purposes.
a. it exceeds my drives speeds using Raid 10
b. it allows for the 2.5GE port on the AP to run full speed
c. hardline copies with the NT-SS5G run at full speed as 10GE adapters for laptops are insanely expensive in comparison by about 5X



Full system inventory


5GE temps: