Description

So the new Broadwell Xeon E5-2600 v4 series just came out, and I decided to get my hand on a few of them. A lot of them are really good value upgrades over the older Haswell chips. This machine will be used for highly parallel numerical work (Fortran / C++).

Notes on parts:

  • E5-2680v4; This offers both a lot of cores and decent clock speeds, and going with two of these saved me $600 over the 2690 v4’s without a huge performance hit.

  • SuperMicro X10DAX; I didn’t need a full server board for this, and this has everything I needed at a decent price, including the ability to add high-end GPUs if I decide to get into GPGPU work. The only thing I had to do with this board was flash a new BIOS to support the E5-26xx v4 chips. Luckily, I had an old E5-26xx v3 chip I could use for this.

  • 256GB ECC DDR4 RAM; this is slight overkill for my current uses (I haven’t had to use more than 200GB in a long time), but it fills in the RAM slots on the board and gives some future-proofing.

  • GeForce GT 610; I’m running this headless, I just needed a basic card for install Debian and / or do any on-hand troubleshooting.

  • Phanteks Enthoo Pro; I absolutely love this case. It’s priced well, has tons of features, and makes cable management a breeze. It’s also one of the few cases that explicitly supports SSI-EEB boards. I normally go for much smaller form factors, but you don’t have many choices with a dual-socket build.

Comments

  • 41 months ago
  • 4 points

When just the ram is more expensive than your whole build....

me realizing its 256 gb of ram

Wow!!!! +1

  • 41 months ago
  • 2 points

Thanks! And yeah, ECC DDR4 is really pricey, and I didn't even go for LRDIMMs (which are even more expensive).

  • 41 months ago
  • 2 points

It will definitely last you!!

  • 41 months ago
  • 1 point

That's definitely the hope. I've been waiting for these Broadwell Xeons for a while; they're a huge upgrade in value over the older Haswell models.

  • 41 months ago
  • 3 points

Gets dual 14 core Xeons...

Has a GT 610

Yea, I know you don't need a High-End GPU, but it was funny to me to realize that lol

  • 41 months ago
  • 2 points

**dual 14-core Xeons

And I might invest in higher-end GPUs later on if I decide to get more into CUDA / GPGPU computing. Definitely waiting at least for Pascal for that, though.

  • 41 months ago
  • 1 point

Oh, that's nice!

  • 41 months ago
  • 2 points

Thanks!

  • 41 months ago
  • 2 points

siiiiick!

  • 41 months ago
  • 1 point

Thanks! It was a super fun build.

  • 41 months ago
  • 2 points

Another awesome Linux build. :D +1

  • 41 months ago
  • 1 point

Thanks! I think the last time I used Windows outside a VM was something like a decade ago.

  • 41 months ago
  • 2 points

That's some nice Ram.

  • 41 months ago
  • 2 points

Great build and great choices of hardware! Only thing cames to mind is to add another WD red 4tb for raid 1. Your data is more valuable than the hardware itself i belive...

Edit: And some more case fans (and to replace stock fans on the case), i would go for noctuas again, not just for better temps, but also for making it quieter. More fans are less louder than fewer ones when optimised, in my experience.

  • 41 months ago
  • 1 point

Only thing cames to mind is to add another WD red 4tb for raid 1. Your data is more valuable than the hardware itself i belive...

I have a fairly large NAS for that at the moment. Right now the workflow I use is: Run individual jobs on SSDs Batch job results, store on local HDD * cron/rsync HDD to NAS and / or cloud storage

That said, I have more than enough room in this case and the motherboard to add a ton more storage if needed.

And some more case fans (and to replace stock fans on the case), i would go for noctuas again, not just for better temps, but also for making it quieter. More fans are less louder than fewer ones when optimised, in my experience.

Yeah, something I'm considering. Right now temps are fine (55 deg C under extended full load). It's also sitting in the basement where I don't have to hear it, but quieter is always better :)

  • 41 months ago
  • 2 points

since you are using this as a server of some sort.

any plans to add redundancy with dual PSU and Phanteks Power Combo or just the plain tried and tested UPS?

so far i yet to see one VM eating up beyond 64GB

what sort of VM you run?

  • 41 months ago
  • 1 point

any plans to add redundancy with dual PSU and Phanteks Power Combo or just the plain tried and tested UPS?

Probably just the UPS at the moment. The Corsair AX / AXi series are very reliable in my experience, but I'll definitely be testing it fairly regularly.

so far i yet to see one VM eating up beyond 64GB what sort of VM you run?

Not doing any virtualization. Just a ton of scientific computing / going through large amounts of data (and BOINC during its spare cycles).

  • 40 months ago
  • 1 point

with that insane amount of RAM and CPU

i think you can run like 20+ VMs and still have spare horsepower to run your own games

points at Linus Xeon 7 VM gaming rig

JK

on serious note

Broadwell-E 6950X looks like will be a serious workhorse for gaming and work related jobs like 4K render jobs.

  • 40 months ago
  • 1 point

points at Linus Xeon 7 VM gaming rig

Except I don't have nearly enough PCI slots for that many GPUs .... D: (or nearly enough thermal management ....)

Broadwell-E 6950X looks like will be a serious workhorse for gaming and work related jobs like 4K render jobs.

I think that chip in particular is serious overkill for pretty much any gaming right now. As a workstation chip, it looks very nice (expensive, but nice). I'm guessing Intel will also release Broadwell-E Xeon E5-16xx v4 chips that are similar to the enthusiast chips soon as well (like they did with the Xeon E5-16xx v3 chips). Hopefully they'll keep up the trend of leaving those Xeons unlocked as well.

  • 40 months ago
  • 1 point

interesting stuff about the Xeon

you can imagine how its like to use a LGA775 rig for photo editing.

and the HD6850 cant help much as well

so processing RAW file is pretty much a torture for me.

I do wish i could afford something decent. But the inner weeb in me wants the spanking new 10 core Broadwell E for real use and also as bragging rights.

  • 40 months ago
  • 1 point

What size of RAW files are you working with? Photo processing / editing performance is very much a CPU-bound application, not a GPU-bound application.

Also, the HD6850 is a perfectly fine card. Plenty capable for many things.

  • 40 months ago
  • 1 point

any plans to add redundancy with dual PSU and Phanteks Power Combo or just the plain tried and tested UPS?

And now I'm definitely doing this ....... just had the AX860i fail on me after 4 days of 24/7 stress testing across all 28 cores, which is really annoying. Ordered the Phanteks Combo, another power supply and RTV/RMA'ed the first one, and another UPS. Pricey, but should be worth it :/

Now I just need to figure how to do this in the case. Probably going to have to rig something with the empty 5.25'' bays to hold the other power supply.

  • 40 months ago
  • 1 point

Yea with such a expensive rig. Failsafe system like the power combo and ups should keep your system running.

  • 12 months ago
  • 1 point

Any pictures of the outcome?

  • 41 months ago
  • 2 points

Uhhhh....

Wow.

  • 41 months ago
  • 1 point

Thanks :D

  • 41 months ago
  • 2 points

+1, very nice, would make a fairly nice DC box... perhaps BOINC

  • 41 months ago
  • 1 point

Thanks! I'm using it for BOINC in its spare cycles when I'm not running other jobs on it.

  • 26 months ago
  • 1 point

I want to build a monster exclusively for processing BOINC data, leaning toward dual xeon setup and was hoping to see someone doing this here. I know it sounds crazy, but if you can afford to build something along the lines of this as a toy...why not? I'd love to get very specific recommendations from someone currently doing this. This would be my first foray into both multi chip and the world of Xeon, but I've had several Linux systems. Advice, recommendations, what to specifically avoid...

  • 41 months ago
  • 2 points

I haven’t had to use more than 200GB in a long time

u wot m8

  • 41 months ago
  • 3 points

He's doing serious number crunching and deep learning, which is much more demanding and specialized than any game

  • 41 months ago
  • 2 points

+1 But can it run minesweeper? XD

  • 41 months ago
  • 1 point
  • 41 months ago
  • 2 points

Wow, I'm speechless.

This inspires me more to build my Sand Bridge EP Xeon E5 2670 build later this year, they go for $60 on eBay :D

+1

  • 41 months ago
  • 2 points

publicstatic void main string args{

system.out.println("+1 m9");

} //I think i messed up the first declaration, but still

  • 41 months ago
  • 2 points

This thing is a beast! Hope you enjoy it!! :)

  • 41 months ago
  • 1 point

Thanks! So far it's working great, but I'm still running stress tests for a few more days before I throw actual jobs at it.

  • 41 months ago
  • 2 points

I don't know why, but I looked up the mobo. Although the physical board and BIOS are ugly. I find it beautiful because of what it supports.

  • 41 months ago
  • 2 points

Yep. SuperMicro aren't really known for awesome-looking PCBs but they're really nice quality.

If you want to see an awesome looking dual-socket board, check out the Asus Z10PE-D16 WS -- https://www.asus.com/us/Motherboards/Z10PED16_WS/

Main reason I went with that SuperMicro board is I've had much better experiences with them as a company than ASUS (especially with respect to customer support).

  • 41 months ago
  • 2 points

If you have good experience with a company's quality, there is little reason to abandon their products unless their competition offers something drastically better.

  • 41 months ago
  • 2 points

Definitely agree with that!

  • 40 months ago
  • 2 points

Wouldn't it be more efficient to aim the cpu coolers up and not having one blowing into the other? Not that it doesn't cool fine, just seems logical.

  • 40 months ago
  • 1 point

Yeah, you're right about that. I'll probably change it when I'm done stress testing it in a couple days.

  • 40 months ago
  • 2 points

Could you please comment on your choice of CPU cooler? Is air cooling preferable to water cooling for long-term reliability? Also, are the three SSDs for RAID 5?

  • 40 months ago
  • 1 point

Is air cooling preferable to water cooling for long-term reliability?

There's not really a huge difference, I've just always stuck with air cooling. They're also just simpler than water cooling - just a hunk of metal and some fans. Under 100% extended load these CPUs are running at around 55 deg C, which isn't too bad (and they idle at under room temp).

Also, are the three SSDs for RAID 5?

Nope. One is the boot drive (mounted at /), another is the home drive (mounted at /home), and the third is what I run actually run jobs in (mounted at /scratch -- this one by far gets the most write cycles because some jobs write out a lot of data).

  • 40 months ago
  • 2 points

HOLY RAM

  • 41 months ago
  • 1 point

With 2 CPUs like that, I would expect you to have a MUCH better GPU...lol

  • 41 months ago
  • 3 points

Well, considering it boots to a plain terminal and I access it via SSH ....

  • 41 months ago
  • 3 points

Haha I feel like the real use of dual XEONS and a SERVER motherboard just isnt really understood by some people on here... All they think of is overkill builds on linustechtips or something where he uses over the top hardware for gamesss :)

I realllllyyyy want an excuse to buy expensive server hardware... what kind of work/study/hobby do you do for this number crunching?

  • 41 months ago
  • 2 points

Haha I feel like the real use of dual XEONS and a SERVER motherboard just isnt really understood by some people on here... All they think of is overkill builds on linustechtips or something where he uses over the top hardware for gamesss :)

Yeah, server hardware requirements are much different (and much more specific) than the typical enthusiast build.

what kind of work/study/hobby do you do for this number crunching?

Work / study: Computational physics

Hobby: Machine learning / data analytics

Also running BOINC on this during spare cycles

I realllllyyyy want an excuse to buy expensive server hardware...

Before you drop a ton of money, I highly recommend you figure out exactly what you want to use that hardware for.

  • 41 months ago
  • 1 point

For BOINC and intense simulation programs

  • 32 months ago
  • 1 point

I plan on doing a monster server at some point with a barebones Linux/BSD server installation that serves as nothing more than as a secure OS for I/O and spinning up/running VMs.

I'd then have 1 VM solely for a server (yes, a server running a VM for the actual server stuff) with a GPU passed through for extra compute. This would serve as my renderfarm and file server.

1 VM for my main Linux workstation. Because I do so much tinkering and messing around in Linux, doing so in a VM with a dedicated GPU would make my life so much easier, especially for rolling back borked changes and whatnot.

1 VM for my Windows Creative/Gaming workstation (possibly with separate virtual drives and separate launch scripts, although both VMs would use the same GPU)

1 VM for outdated software running XP or something with a weak, outdated GPU passed through.

My real dream with this is to have it use the OpenPower9 CPUs from IBM, since those are fully-open source, Copyleft-licensed, powerful and allow for 4 threads per CPU core with virtually no performance loss, however this is banking on full x86 virtualization with minimal overhead being a thing for my VMs...

Also, lifegoal: have a fully-populated server rack (at home) that I actually use, complete with battery backup and backup power generation for 99.9% uptime.

  • 41 months ago
  • 1 point

Thinking that too...

  • 41 months ago
  • 1 point

But can it run crysis?

EDIT: Yes I know its for numba kruncking ;)

  • 41 months ago
  • 1 point

Let me just pull out some SLI 980Ti's and get back to you .... :P

  • 41 months ago
  • 1 point

I would assume the 28 core setup wouldnt do as well then if you had an i5, graphics aside. Do you have a gpu lying around that you can through in and see if it makes a difference?

  • 41 months ago
  • 1 point

Do you have a gpu lying around that you can through in and see if it makes a difference?

Not really, unfortunately :/

And the gaming performance probably wouldn't be horrible (and it would be good for any games with very good parallelization), but the much higher single-core clock speeds of enthusiast processors definitely make them much better choices for most games. The max clock speeds of this processor range from 3.3 GHz (very few cores under load) to 2.9 GHz (all cores under load). Also worth noting that even games which support multithreading may not be designed to scale to 2 sockets and 56 threads, and trying to scale to that many threads may actually result in decreased performance due to inter-thread communication (it all depends on how specifically things are parallelized).

  • 40 months ago
  • 1 point

Yea thats what I thought. Less cores is still better nowadays.

  • 41 months ago
  • 1 point

I'm clueless.

  • 41 months ago
  • 1 point

Hm?

  • 41 months ago
  • 1 point

What?

  • 12 months ago
  • 1 point

It's a very powerful computer.

  • 41 months ago
  • 1 point

So it's a really expensive calculator? I don't get it

  • 41 months ago
  • 2 points

Most computers are really just expensive calculators in a certain sense.

But yes, I'm mainly using this for parallel simulations (computational physics) and analysis of very large data sets. Very different than what most people on this site use a computer for.

  • 41 months ago
  • 2 points

True, but what you're doing is just as cool, if not more so. +1

  • 41 months ago
  • 1 point

Thanks!

  • 40 months ago
  • 1 point

You must be enjoying opening like 100 tabs all at once with no lag XD

  • 40 months ago
  • 6 points

Nah, you need at least 1TB RAM to handle 100 Chrome tabs.

  • 40 months ago
  • 1 point

When you installed the operating system, how much space did you allocate for swap? In the good old days (say, 10 years ago) the swap partition was typically double the amount of RAM in the machine. Obviously you didn't do that with 256 GB of RAM. If I remember correctly, the default installer for Debian 8 automatically creates a swap partition of double your RAM, so you presumably overrode that. One might make the case for zero swap when you have more RAM than you think you're ever going to use. But for various reasons the operating system does make some use of swap even when RAM usage isn't maxed out, so no swap at all might not be optimal.

I'd be interested to hear how you managed this issue. Hope I'm not off-topic for this site, but this sort of question could influence people's hardware choices.

  • 39 months ago
  • 1 point

Yes, that's the default behavior of the Debian installer, and yes, I definitely overrode it (I also do non-default things like mounting / and /home on separate drives, etc. so I have to declare swap manually anyway). I have a 10GB swap partition on this right now (it's just a smallish roundish number), and that's honestly overkill for my needs. The jobs I run on this (and similar machines) either (a) tend to max out threads well before they max out RAM and / or (b) if they are RAM intensive, allow me to have very tight control over both allocated memory and data access patterns.

One might make the case for zero swap when you have more RAM than you think you're ever going to use.

My view on that is that unless you're extremely limited on drive space (which is cheap), it's worth having a small swap partition to avoid processes being killed randomly, etc. if, for whatever reason, you do happen to run out of RAM. Better safe than sorry and all that.

In the good old days (say, 10 years ago) the swap partition was typically double the amount of RAM in the machine.

I'm honestly surprised Debian still does this by default. The amount of swap that's necessary is highly workload-dependent, especially for a server environment, and the swap = 2 * RAM guideline is ancient.

  • 39 months ago
  • 1 point

Not sure how I missed this build. Smashing work though. Do love a pair of xeons. Awsome stuff sir.

  • 39 months ago
  • 1 point

Thanks! It's been a great machine so far. Running calculations pretty much 24/7 with no issues.

  • 37 months ago
  • 1 point

What's this machine crunching exactly? You mentioned simulations and data analysis. Can you give more details? I'm into statistics and also learning about this kind of build. :)

  • 35 months ago
  • 1 point

Hey, I just saw this message (I haven't been on PCP for a long time) -- I'm a computational physics grad student, and I mainly simulate ultrafast laser-molecule interactions (by solving the time-dependent Schrodinger equation). So it's a lot of linear algebra (since you numerically solve most differential equations by converting them to linear algebra problems). I also have a personal interest in big data / machine learning, so I dabble in that in my spare time.

  • 35 months ago
  • 1 point

Haha, it's been awhile since I've solved a differential equation by hand! I guess your software can use many cores as you throw at it. Kinda like Prime95?

If you weren't getting into the big data scene, would you say that you overspent?

And, if you mind my asking, what attracts you to big data / machine learning? (I'm wondering if I'm missing out on something really useful).

Thanks for sharing, man. I love to learn! :)

  • 35 months ago
  • 2 points

I guess your software can use many cores as you throw at it. Kinda like Prime95?

Yes, numerical linear algebra problems tend to parallelize very well.

If you weren't getting into the big data scene, would you say that you overspent?

Yes, overspent on RAM, not on anything else.

And, if you mind my asking, what attracts you to big data / machine learning?

It's just a fascinating subject in computer science for me (both from an algorithmic perspective and in terms of what kind of problems you can work with). It's also a really marketable skill right now, and since I'm not sure if I want to stick with physics forever, developing additional tech skills is important to me.

  • 35 months ago
  • 1 point

Cool, man. Thanks for sharing.

If you're really into the big data scene, then why not switch over now... or get the PhD in Physics and then switch over? In either field, you get to research, and since the former seems more interesting, then why not?

  • 35 months ago
  • 2 points

I have 3-4 years left in my PhD program (atomic / molecular / optical physics PhDs typically take 6-7 years at my school), and I've already put 2 years into it already, so it would be a bit of a shame to leave it now. Besides, I can't really predict at this point what I'll actually want to do once I get my PhD, so I'm trying to keep my options open :)

  • 32 months ago
  • 1 point

Did all the holes on the motherboard line up with the case? Did you need to drill at all, or left any unscrewed? Supermicro lists the board as EATX, but I'm suspecting it's 100% SSI-EEB.

  • 28 months ago
  • 1 point

god dam

  • 14 months ago
  • 1 point

How long did it take you to put the ram into the slots?

g e e e e e e e e z

  • 12 months ago
  • 1 point

Nice PC, which framerate on GTA V maximum graphics settings?

[comment deleted]
  • 41 months ago
  • 3 points
program thanks
  implicit none
  write(*,*) "Thanks!"
end program thanks
[comment deleted]
  • 41 months ago
  • 2 points

Not for a bit at least; I've thought about getting more into CUDA work, but I'm waiting to see what Pascal offers regarding double precision performance / price before investing in that further. I'm really hoping NVIDIA doesn't continue its trend of kneecapping double precision GTX performance as a way of charging ridiculous Quadro prices.

[comment deleted]
  • 41 months ago
  • 1 point

Guess what... People buy the Quadros so they can price them at whatever they want.

Still not as bad as the Tesla cards. They can really price gouge on those.

And there's definitely enough single precision CUDA stuff that I can do that getting a good consumer Maxwell card down the line is probably worth it.

[comment deleted]
[comment deleted by staff]
  • 41 months ago
  • 1 point

Last month. Distributors are still having issues keeping some of them in stock.

[comment deleted by staff]
  • 41 months ago
  • 1 point

Thanks! It's all I use.

[comment deleted by staff]
  • 41 months ago
  • 1 point

Thanks! My wallet definitely took a hit, but it's something I've wanted to build for a while so I decided to pull the trigger when the Broadwell E5 Xeons came out.

[comment deleted by staff]
  • 40 months ago
  • 1 point

Not sure how long that will continue, though. Most of the upcoming Skylake Xeon's aren't rumored to be huge upgrades over Broadwell (in terms of single-chip specs -- the rumored 8 socket support will be a huge overall upgrade), Intel's given up its tick/tock dev model, and it seems like the fastest growing part of Intel's data center business are the ultra-dense, power-efficient Xeon D units (which are very impressive little systems).

[comment deleted by staff]
  • 40 months ago
  • 1 point

Man, I really hope AMD can get back into the server market. Or really, I'm just hoping that Zen isn't a Bulldozer 2.0-level fiasco.