What's new
  • Please do not post any links until you have 3 posts as they will automatically be rejected to prevent SPAM. Many words are also blocked due to being used in SPAM Messages. Thanks!

Gallery Home lab (pic heavy)

WolfBane

Well-known member
Joined
Jun 28, 2008
Messages
1,240
Location
new bruinswick
are you like a super cat scientists that has a cat ?
i love all the gear!!! but man am i ever lost in that server world you guys play in !!
but i will say what a fantastic looking beasts you have there :)
 

doup93

Well-known member
Joined
Mar 11, 2008
Messages
216
Location
Montreal, QC
Love this thread. Waiting on delivery of my Ubiquiti US-16-XG that I bought through a member of another forum that is in the US. From what I've read, Ubiquiti won't even sell if you're not using a US credit card with address in the US. How do you like that switch so far? I wasn't too thrilled about buying Beta stuff but for the price, it's hard to pass up.

Also bought some Intel X520-DA2 for my Hyper-V servers (2) and my file server. I'll start my own thread since my home lab has evolved quite a bit.
 

zoob

Well-known member
Joined
Aug 21, 2007
Messages
1,757
Location
Toronto, ON
Thanks guys, glad you are enjoying these posts :)

Ubiquiti allowed two separate orders of mine go through. I recently added the UAP-AC-HD to my setup, and am testing the 2 port LAG just out of curiosity even though I'll never have enough devices to max out a gig connection. US-8 and US-8-60W switches are also on order (from a Canadian retailer) so I can clean up my POE setup and get everything setup on to separate VLANs and reduce the number of power adapters required for cameras/phones/etc.

The US-16-XG is nice but I'm running into weird issues where the link speed will show 10Gbps but actual throughput is more like 100Mbps up/400Mbps down. It's really odd, and some ports are worse than others. Port 11 tested at 33Mbps using the Unifi controller 'download' speed test. I eventually found two ports that let my servers stretch their legs and max out 10Gbps.

The fan is really annoying, so if I were to have put one in my office I'd need to swap them out with something larger (stock is 30mm, I'd put in 40mm with 3D printed adapters). I'm a little upset that I didn't get to snag a second one before it was removed from the beta store. 10GBase-T cards are much more expensive than 10GBe SFP+ cards which is my motivation for getting another one.

This leads me to another issue I've been having. I'm just going to cut+paste the same post I made over on STH forums.

--

Running into some weird problems with Chelsio T420-BT cards and X99 setups. I'm not getting full speed in a PCI-E 3.0 x8 lane slot even though the card is a PCI-E 2.0 x8 card. The same card in a Z170 x8 slot works fine.

E5-1650 V3 (40 lanes)
ASROCK X99 Taichi
+ Graphics card
+ Chelsio T420-BT
+ Any other PCI-E add-in card

Motherboard spec page shows:
- 3 x PCI Express 3.0 x16 Slots (PCIE2 @ x16 mode; PCIE4 @ x16 mode; PCIE5 @ x0 mode) (PCIE2 @ x16 mode; PCIE4 @ x8 mode; PCIE5 @ x8 mode)*
*If you install CPU with 28 lanes, PCIE2/PCIE4/PCIE5 will run at x16/x0/x8 or x8/x8/x8.

In this configuration I get full line speed as expected
PCIE2 (x16) : GPU
PCIE4 (x16): Chelsio T420-BT
PCIE5 (x0): Empty

In this configuration I get 33% line speed (iperf around 3.35 Gbps)
PCIE2 (x16): GPU
PCIE4 (x8): Chelsio T420-BT
PCIE5 (x8): Any other card. Even a 1x Gigabit nic.

PCIE2 (x16): GPU
PCIE4 (x8): Any other card. Even a 1x Gigabit nic.
PCIE5 (x8): Chelsio T420-BT

In this configuration, I get full line speed
PCIE2 (x16) : Chelsio T420-BT
PCIE4 (x8): GPU
PCIE5 (x8): Any other card. Even a 1x Gigabit nic.

The Chelsio T420-BT is supposed to be PCI-E 2.0 x8 card. Why does it not like PCI-E 3.0 x8 lanes?
I tried forcing the slot to be 2.0 in BIOS and no help.

I tested the same in an ASUS X99 Deluxe and ran into the same type of performance problems when I manually switched the SLI/CF PCI-E selector to make it x8/x8/x8 or x16/x16/x8.

Similar weird performance issues with a ConnectX-2 in my E5-2670 V1 S2600CP2J Freenas server.
All of the slots in the S2600CP2J are x8 electrically. What is strange is in my Freenas server, I get 50% line speed, iPerf around 4.7 Gbps.

To throw another wrench in the mix, I have a ConnectX-2 in my E5-2670 V1 R2312 S2600GZ4GC server and can get 100% speed off of an x8 slot. The only difference is the R2312 has the latest BIOS installed, whereas the Freenas server as 1.06 to keep the PCI-E 3.0 from being restricted to whitelisted devices.


Now, look at this testing on Z170 platform!

i7 6700K
Gigabyte GA-Z170X-Gaming 7
PCIEX16: GPU
PCIEX8: 1 gig nic
PCIEX4: Chelsio T420-BT
iperf shows around 6.6-8.5 Gbps with the T420-BT on a x4 slot!

PCIEX16: GPU
PCIEX8: Chelsio T420-BT
PCIEX4: 1 gig nic
Full line speed. 9.4 Gbps, 99.93% network utilization.

This is really frustrating, since I wanted X99 for >4 cores, PCI-E lanes for NVME storage and also 10GBe networking.

I ordered some ConnectX-3 PCI-E 3 cards to do more testing. Chelsio T520/other 10GBase-T cards are a bit cost prohibitive, so this may be an incentive to finally pull fibre through the walls.
 

zoob

Well-known member
Joined
Aug 21, 2007
Messages
1,757
Location
Toronto, ON
The "Ubiquiti" compatible DACs I ordered from China arrived. Took a bit of wrangling but finally setup LACP on 2 ports between my HP 1950 and US-16-XG. The US kept only taking one out of two ports for the aggregation until I removed and reapplied the settings.

The "OEM" ones are the new cables from olightcom.

GiktVoTl.jpg

FXr29jJl.jpg

DACs

Code:
(UBNT) >show fiber-ports optics-info all

                         Link Link                                 Nominal
                       Length Length                                   Bit
                         50um 62.5um                                  Rate
Port     Vendor Name      [m] [m]  Serial Number    Part Number     [Mbps] Rev  Compliance
-------- ---------------- --- ---- ---------------- ---------------- ----- ---- ----------------
0/5      CISCO-MOLEX      0   0    MOC16070KNX      74752-9520       10300 08   DAC
0/6      CISCO-MOLEX      0   0    MOC1606030M      74752-9520       10300 08   DAC
0/8      CISCO-MOLEX      0   0    MOC160604ZW      74752-9520       10300 08   DAC
0/11     OEM              0   0    HL161026024      SFP-H10GB-CU1M   10300      DAC
0/12     OEM              0   0    HL161026017      SFP-H10GB-CU1M   10300      DAC

Code:
(UBNT) #show lacp partner all

        Sys       System        Admin Prt    Prt     Admin
 Intf   Pri         ID          Key   Pri    Id      State
------- ----- ----------------- ----- ----- ----- -----------
0/1     0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/2     0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/3     0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/4     0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/5     0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/6     0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/7     0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/8     0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/9     0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/10    0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/11    32768 5C:8A:38:xx:xx:xx 2     32768 25    ACT|AGG|LTO
0/12    32768 5C:8A:38:xx:xx:xx 2     32768 26    ACT|AGG|LTO
0/13    0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/14    0     00:00:00:00:00:00 0     0     0     PSV|IND|LTO
0/15    32768 08:BD:43:xx:xx:xx 18    128   5     ACT|AGG|LTO
0/16    32768 08:BD:43:xx:xx:xx 18    128   6     ACT|AGG|LTO

Ports 11/12 connect to my HP
Ports 15/16 connect to my Netgear
The state for each port in an aggregate group shows "AGG".

MJF8esQh.png

US-16-XG ports as viewed in UniFi controller

X97BiH9h.png

BAGG10 is the aggregate to UAP-AC-HD
BAGG20 is the aggregate to US-16-XG

Ez3n4xmh.png

Netgear view showing which ports are active in each LAG

HUQBzesl.jpg
 

zoob

Well-known member
Joined
Aug 21, 2007
Messages
1,757
Location
Toronto, ON
Verified that the PCI-E lane width negotiation is what's giving me problems.

It links at x1 width when all three PCIE slots are populated.
Code:
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	[b]Speed 5GT/s, Width x1[/b], TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

It links at x8 width when only two slots are populated.
Code:
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	[b]Speed 5GT/s, Width x8[/b], TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

I've opened a ticket with ASROCK. Fingers crossed they'll do something about it.
 

zoob

Well-known member
Joined
Aug 21, 2007
Messages
1,757
Location
Toronto, ON
So I did more testing and now realized that my ASUS X99 Deluxe 10GBe performance problems was likely due to the US-16-XG port speed issues I described above where certain ports performed worse than others. Confirmed x8 width with the Chelsio T420-BT card and ran an iperf in Ubuntu that gave me basically the full 10GBe performance. The X99 Deluxe is set to 3-way SLI mode so that I get a lane setup of x16/x16/x8.

Also confirmed that my Freenas really is linking x8 width with my network card and all of the SAS controllers
Code:
                LnkCap: Port #8, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited                          
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-                                                            
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-                                                              
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-                                                              
                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-

There's something else specific to Windows 10 screwing up my speed, and there's something specific to my Freenas PC that's screwing up its speed.
I've already tried disabling QoS and setting "Limit reservable bandwidth" to 0 on my Win10 PC.

kstk4qY.png

"Only" 450 MB/s.


Win 7 test PC with T420-BT in x16 slot (not X99 computer):
iperf to Proxmox = 9.4 Gbps
iperf to Freenas = 5 Gbps

X99 workstation with T420-BT in x8 slot:
iperf to Proxmox = 5.0-5.7 Gbps (in Win10)
iperf to Freenas = 5.1 Gbps (in Win10)

iperf to Proxmox = 9.4 Gbps (in Ubuntu)
iperf to Freenas = ?? (in Ubuntu, need to re-test this but not optimistic)
 

zoob

Well-known member
Joined
Aug 21, 2007
Messages
1,757
Location
Toronto, ON
It's definitely some TCP tuning that needs to be done. If I run iperf with 4 parallel threads, 10G connection to Freenas is saturated.

I was playing around with tuning variables and managed to briefly get 10Gbps on iperf tests. The last setting I modified that finally gave me full speed iperf was:
kern.ipc.soacceptqueue=1028

SMB speeds were negatively impacted. Sometimes I could hit 1GB/s, other times it would crawl around 50 MB/s so I figured a reboot was in order...

Now I can't get back to where I was.
 

lowfat

Moderator
Staff member
Joined
Feb 12, 2007
Messages
11,096
Location
Grande Prairie, AB
SMB doesn't perform great in FreeNAS since Samba is single threaded. Have you tried to see if iSCSI was any better?
 

zoob

Well-known member
Joined
Aug 21, 2007
Messages
1,757
Location
Toronto, ON
I tried iSCSI but it only performed better because my Win10 box was caching everything (sync writes off).

There definitely is some FreeNAS TCP tuning required. Look at this from my Win7 box:
deY1Sr9.png


and there's something definitely up with my Win10 box based on the iperf diff on the same hardware using Ubuntu. Time for a re-install methinks, or at least a dualboot.
 

zoob

Well-known member
Joined
Aug 21, 2007
Messages
1,757
Location
Toronto, ON
This Chelsio T420 just doesn't like to negotiate link widths properly. Using a ConnectX-2 in the same setup links x8 width every time.

Now the issue is putting SFP+ connectivity in my office, or finding alternative Base-T NICs...
I've only got 2x copper cards, so really only need max 4 copper ports (2 for LACP to office, 2 to connect the cards).

Edge switch (Basement):
HP 1950 in basement - 2 copper & 2 SFP+ ports

One of these will need to be the core and one would be a 10G edge switch in the office:
XS708T - 8 ports total: 6 copper and 2 copper/SFP+ combo ports
US-16-XG - 16 ports total: 12 SFP+ and 4 copper

Both will be the loudest items in my office, so I would need to modify them for silence.


XS708T:
1x Nidec W40S12BS4A5-07 40mm fan, 26.5 CFM
Runs at ~20% duty = 3300 rpm = ~5.5CFM
Rated noise: 27.4 dBA Measured @ 23-30 dBA @ STH
Max power: 49.5 W
Has a bit of a whiny/grindy noise character
Switch PCB has a second unused fan header
Chassis has only 1x fan cutout
Code:
Slot	FAN	Description	Type	Speed	Duty level(%)	State
1	1	Fan-1	Fixed	3292	21	Operational


US-16-XG:
2x Sunon MC30151V2-0000-G99 30mm fans, 5 CFM/each
Runs at ~50% duty = 7200rpm = ~10 CFM
Rated noise: ? Measured @ ~36-38 dBA @ STH (ES-16-XG model, which is more or less identical)
Max power: 36W (excluding SFP modules)
Fan noise has a lot of whine to it
Switch PCB has two fan headers (duh)
Fan cutouts are only for 30mm size, need to print out 40mm->30mm shrouds

Code:
Fans:
Unit Fan Description    Type      Speed         Duty level    State
---- --- -------------- --------- ------------- ------------- --------------
1    1   FAN-1          Fixed     7200          51%           Operational
1    2   FAN-2          Fixed     7171          51%           Operational
This doesn't even look right because spec sheet says that's max RPM of the fans but the switch thinks it's running them at 50% speed?

Netgear seems like best for office use in stock form.
Unifi seems like best for fan modding.

Regardless of the final layout, it looks like 2x quiet 40mm fans will be on the shopping list.
 

Latest posts

Twitter

Top