PC Water Cooling for High Performance Computing in Science Research

April 2022

My water cooled high performance computing rig for astronomy research – 10 fans, 3 radiators, AMD Ryzen 9 3900XT CPU, Nvidia RTX3090 GPU and RTX2060 Super GPU

TOC

 1. Intro  2. The Benefits of Water Cooling for High Performance Computing  3. Isn’t This Some Kind of Cryptocurrency Mining Rig?  4. Risks, Cost, Emissions and Waste  5. Water Cooling Loop Design and Components  6. Water Cooling Parts  7. High Performance System Parts List  8. But Why Don't You Just...  9. Single versus Dual Loop  10. How Many Radiators Do You Need?  11. AIO vs Custom Loop  12. Flushing all parts out with Distilled Water  13. Temperature and Flow Monitor  14. Tubing  15. Fans and Airflow  16. VRM, RAM, PCH and NVMe cooling  17. Reservoir and Pump  18. Coolant  19. RGB Lighting  20. Coil Whine

1. Intro

Earlier this year I wrote about how you can contribute your PC's computing power (and part of your electricity bill) to distributed volunteer cloud computing for science research https://blog.ajflynn.io/become-part-of-a-global-super-computer-network-for-astronomy-research

This kind of high performance computing can place extreme demands on a PC's cooling system. The CPU and GPU may be run at the limits of their power consumption, 24/7, producing far more heat than standard gaming or business usage ever would.

In this article I'll share some thoughts and experiences from my recently completed project to install water cooling in my home PC, which I use for day to day gaming, as well as for volunteer cloud computing when it is idle.

While the following advice is focused on a system designed for high performance computing for science research applications, the information may still be useful for those looking to add water cooling to their PC for gaming use.

For more detail about water cooling setups and components in general, check out this excellent guide by EKWB https://www.ekwb.com/guides

2. The Benefits of Water Cooling for High Performance Computing

Water cooling has a number of advantages over air cooling for high performance science research applications:

Image by https://www.build-gaming-computers.com Water cooling parts. Image by https://www.build-gaming-computers.com

3. Isn’t This Some Kind of Cryptocurrency Mining Rig?

Water cooling doesn’t make sense for cryptocurrency mining. An open-air rig that is cooled by fans alone is much more cost effective than water cooling. For that kind of computational work, you’re trying to maximise output while minimising cost, and you don’t care if it’s noisy. A mining rig is easier to position in a garage or other out-of-the-way spot where the noise it produces won’t be so much of an issue.

4. Risks, Cost, Emissions and Waste

Before rushing in to contributing your PC’s idle time to cloud computing for research, or water cooling your PC, you should consider the following:

My water cooling loop, which cools 2x GPUs, 1x CPU using 3 radiators.

5. Water Cooling Loop Design and Components

Some thoughts on loop design and component choice:

My custom loop is a single combined loop with 3 radiators (one thick and two thin radiators). My high performance computing setup needs to dissipate 800W+ of heat when the CPU and GPUs are overclocked. The loop cools an AMD Ryzen 9 3900XT CPU, an Nvidia RTX3090 GPU and RTX2060 Super GPU.

As I run two GPUs and a high-end CPU at full power 24/7, I purchased a new 1200W Power Supply Unit (PSU) to replace my old 850W PSU. This allows about 50% headroom over the max combined wattage of all the components running at full power. I understand that as little as 25% headroom is still fine.

Image by https://www.performance-pcs.com Image by https://www.performance-pcs.com Serial versus parallel loop setups. Radiators and reservoir not shown. In my setup, the radiators are in series along the red lines/arrows between the waterblocks and pump. Image by https://www.performance-pcs.com

6. Water Cooling Parts

The basic parts needed for a custom water cooling loop are:

7. High Performance System Parts List

The following is a list of the components I purchased for my system, to give you an idea of what is required in a custom high performance water cooling loop, to cool a one CPU and two GPUs, that are expected to run 24/7 at maximum wattage. I didn't use every single part listed, as I got some extra fittings (such as barbs or right angle adapters), just in case.

I didn't purchase it all at once. I purchased the basics first (and started out with just one radiator), then had to get additional fittings and additional radiators as I worked out what I needed to improve cooling performance to an acceptable point, and to make some of the connections and pipe routing more efficient and elegant.

I did not need to purchase new NVMe drives, CPU, RAM, GPUs or motherboard, as the ones I already owned were adequate.

The total cost for the parts I required to make this cooling loop, including the extras such as a new PSU, ended up being around $3,300.

For a more traditional single loop cooling 1 GPU and 1 CPU, with one radiator, the cost for the water cooling parts alone would be more like $1,500.

Prices are in AUD and include GST, but exclude shipping.

From MWave https://www.mwave.com.au/

AC14686 Thermaltake Pacific CL360 360mm Radiator (64mm thickness) x1 $159.00

AC19474 Thermaltake Pacific C360 360mm Radiator (27mm thickness) x2 @ $129.00ea $258.00

AC50178 Thermaltake Pacific PR12-D5 Plus Pump & Reservoir Combo w/ RGB Software Control x1 $339.00

AC38261 Thermaltake TOUGHFAN 12 120mm High Static Pressure Radiator Fan [SINGLE FAN] x3 @ $39.00ea $117.00

AC49086 be quiet! Light Wings ARGB 120mm PWM Black Case Fan – [TRIPLE PACK] x2 @ $105.00ea $210.00

AC37179 Thermaltake Pacific TF2 Temperature and Flow Indicator – CL-W275-CU00SW-A x1 $79.00

AC11819 EKWB Filling Bottle (1000mL) x1 $13.99

AB61483 Thermaltake V-Tubler 3T Flexible Tubing 3/8” (9.5mm) – 2m x2 @ $15.00ea $30.00

AC22192 Thermaltake TT Premium T1000 1L Transparent Coolant – Blue x4 @ $20.00ea $80.00

AC38919 EKWB EK-Quantum Vector Strix RTX 3080/3090 D-RGB Nickel + Plexi GPU Water Block x1 $299.00

AC38928 EKWB EK-Quantum Vector Strix RTX 3070/3080/3090 Backplate – Black x1 $99.00

AC52280 Thermaltake Pacific MX1 Plus CPU Water Block x1 $169.00

AB61445 Thermaltake Pacific G1/4 Stop Plug w/ O-Ring – Black x4 @ $6.00ea $24.00

AB61458 Thermaltake Pacific G1/4 Female to Male 30mm extender – Black x3 @ $10.00ea $30.00

AB61450 Thermaltake Pacific 3/8” ID Barbs – Black x18 @ $7.00ea $126.00

AB61465 Thermaltake Pacific G1/4 90 Degree Adapter – Black x10 @ $15.00 $150.00

AC10340 Noctua Chromax.Black 30cm 4Pin PWM Power Extension Cables – 4 Pack x1 $15.00

AC11822 EKWB EK-Cable Y-Splitter 3-Fan PWM (10cm) x3 @ $7.99ea $23.97

From Amazon https://www.amazon.com.au/

PH-GB2080ASDEVO_CR01 Phanteks Glacier Dual Evo, Full GPU Water Block for ASUS RTX 2080/2070/2060 Super Dual EVO Series x1 $236.93

From PCCaseGear https://www.pccasegear.com/

3830046991737 EK M.2 NVMe Heatsink Black x2 @ $19.00ea $38.00

BN816 be quiet! Dark Power Pro 12 Titanium Modular 1200W Power Supply x1 $599.00

PC-O11DXL-A Lian Li PC-O11 Dynamic XL ROG Certified Case Silver x1 $319.00

8. But Why Don't You Just...

People suggest all sorts of extreme cooling loop designs online, such as massive external radiators, refrigerated coolant systems, running cooling pipes outside the house, running pipes into their swimming pool or underground through the cold earth, and so on. Undoubtedly many of these suggestions would improve the cooling capacity of the system. However, the aim of a standard water cooling loop is to fit within a PC case, be self-contained, to use readily available parts, to be cost effective and to be relatively easy to set up. If you want to experiment with crazy ideas after getting the basic cooling loop working, I say good luck and let me know how it goes! :D

Image by https://www.performance-pcs.com A dual loop setup, with a separate cooling loop each for the CPU and GPU. Image by https://www.performance-pcs.com

9. Single versus Dual Loop

There may be advantages to having two separate cooling loops, one for the GPU(s) and one for the CPU. I have not experimented with this, but advice I have read suggests this may be better for CPU cooling, to keep the CPU temperatures lower than may be possible when the CPU shares a loop with the GPUs. A dual loop will increase the costs as you will need an additional pump and reservoir, at least one more radiator, and more fittings.

I got lucky as the AMD Ryzen 3900XT CPU is designed to run at high temperatures, so it is happy sharing a cooling loop with two GPUs. In my loop, the CPU runs at between 80C – 90C while at max processing load (this is within its max operating temperature range), with coolant water temperatures at about 35C while room temp is around 22C. These are not great numbers for CPUs that need to run cooler than 80C (apparently for ideal CPU cooling, the water temp should be closer to 5C – 10C over room temp while the CPU is at the load you expect to run it at). Keep in mind that the temperatures my CPU and GPUs are reaching are during high performance computing loads. Gaming use will rarely push components to such extremes.

10. How Many Radiators Do You Need?

This is a difficult question to answer as it depends on the radiator size/thickness, wattage of your CPU and GPU, fan type and speed, and so on. It's best to read some of the guides available online, and to do your own research once you know the max wattage of the devices you want to cool. There's various bits of advice out there and even water cooling wattage vs radiator size calculators available online. The EKWB system configurator can provide some estimates and parts suggestions (though of course, it's designed to sell you their gear, so just use it to get an idea of the size and kinds of parts you'll need).

I should note that my first try at the wattage and radiator size calculations suggested I'd get away with using a single Thermaltake Pacific CL360 360mm Radiator (64mm thickness, the thickest 120mm x 320mm radiator they make). It's a huge radiator, but I found it just couldn't cool my system by itself when the CPU and GPUs were running continuously at high load. However the guides I followed are all for gaming use, not for running your devices at max wattage 24/7. I found I had to effectively double the amount of radiator surface area (by adding two more 27mm thickness radiators) to sufficiently cool my system for research computing use.

When it comes to radiators, bigger is is better (in all dimensions), as surface area determines the rate at which a radiator can exchange energy with the air. Make sure your case can accommodate the planned number of radiators, plus the thickness of the fans installed on top of them.

11. AIO vs Custom Loop

An alternative to a custom water loop is the “AIO” or All-In-One cooler. AIOs are a radiator, tubing and pump, all pre-assembled and filled with coolant. You attach the module to your CPU or GPU, mount the radiator, and then don't have to buy or assemble any other parts. These can provide a quick and easy alternative to creating your own custom cooling loop.

The disadvantages of AIO coolers are that they may not suit high performance computing where the CPU And GPU are pushed to extremes, and you will need space to mount a separate radiator per device needing to be cooled, rather than having the option to cool a number of devices with a single cooling loop. You may also find the length of the pre-attached non-removable coolant tubes isn’t sufficient to reach between your CPU or GPU, and the mount point you have chosen for the radiator.

It may also be difficult to find an AIO cooler that will suit your particular GPU, especially if it is an older model. Separate water blocks are available for a far wider selection of GPUs. CPU AIO coolers tend to be more universal and fit a wide range of CPU socket types, so are easier to fit to older CPUs.

12. Flushing all parts out with Distilled Water

It is recommended to flush all water cooling components and tubing with distilled water before connecting them to your cooling loop and filling the system with coolant. This clears out any fine metal and plastic that may be left over from the manufacturing process, that might interfere with your water blocks or pump impeller, etc. Distilled water is usually available from supermarkets, as it is used in fabric cleaning and filling clothes irons, etc.

13. Temperature and Flow Monitor

As I touched on earlier, I believe a temperature and flow sensor are essential for a water cooling loop. The model I use is the Thermaltake Pacific TF2 Temperature and Flow Indicator – CL-W275-CU00SW-A.

This lets you keep track of exactly what is happening in your loop, and lets you work out precisely the pros and cons of any changes or additions you make to the loop.

These sensors are best combined with measurements from a simple room thermometer, and software that can track temperature, watts and load over time. If you record temperature values as you make changes to your system's configuration (fan speed, direction, cooling loop setup, etc) then you can see what changes are really improving your system's performance and temperature.

These monitors also protect against future failures, when they have an inbuilt alarm. You can then instantly be warned of low flow rates or high coolant temperatures.

The combined reservoir and pump, with temperature and flow monitor.

14. Tubing

Some advice on the tubes that carry the coolant through your coolant loop:

15. Fans and Airflow

16. VRM, RAM, PCH and NVMe cooling

17. Reservoir and Pump

18. Coolant

19. RGB Lighting

20. Coil Whine

Ashley Flynn – Games and Simulations Software Engineer Portfolio and contact – https://ajflynn.io/

Back to Latest Posts