ajflynn-io

April 2022

My water cooled high performance computing rig for astronomy research – 10 fans, 3 radiators, AMD Ryzen 9 3900XT CPU, Nvidia RTX3090 GPU and RTX2060 Super GPU

TOC

 1. Intro  2. The Benefits of Water Cooling for High Performance Computing  3. Isn’t This Some Kind of Cryptocurrency Mining Rig?  4. Risks, Cost, Emissions and Waste  5. Water Cooling Loop Design and Components  6. Water Cooling Parts  7. High Performance System Parts List  8. But Why Don't You Just...  9. Single versus Dual Loop  10. How Many Radiators Do You Need?  11. AIO vs Custom Loop  12. Flushing all parts out with Distilled Water  13. Temperature and Flow Monitor  14. Tubing  15. Fans and Airflow  16. VRM, RAM, PCH and NVMe cooling  17. Reservoir and Pump  18. Coolant  19. RGB Lighting  20. Coil Whine

1. Intro

Earlier this year I wrote about how you can contribute your PC's computing power (and part of your electricity bill) to distributed volunteer cloud computing for science research https://blog.ajflynn.io/become-part-of-a-global-super-computer-network-for-astronomy-research

This kind of high performance computing can place extreme demands on a PC's cooling system. The CPU and GPU may be run at the limits of their power consumption, 24/7, producing far more heat than standard gaming or business usage ever would.

In this article I'll share some thoughts and experiences from my recently completed project to install water cooling in my home PC, which I use for day to day gaming, as well as for volunteer cloud computing when it is idle.

While the following advice is focused on a system designed for high performance computing for science research applications, the information may still be useful for those looking to add water cooling to their PC for gaming use.

For more detail about water cooling setups and components in general, check out this excellent guide by EKWB https://www.ekwb.com/guides

2. The Benefits of Water Cooling for High Performance Computing

Water cooling has a number of advantages over air cooling for high performance science research applications:

• Air cooling alone may be unable to sufficiently cool parts inside an enclosed PC case with the CPU and GPU(s) running at 100% power load for extended periods of time. This is a situation that would rarely be encountered while gaming or during normal computer use.
• Systems running more than one GPU may have insufficient space to allow the GPU stock air coolers to operate efficiently, and the GPUs may even block each others fan inlets entirely.
• Air cooling may be extremely loud, as a large number of fans may need to be running at high speed. GPU onboard fans are especially loud as they are quite small but must move large volumes of air. Noise isn't a big issue in a data center or dedicated server room, but is a nuisance when the PC is in a home or office environment.
• It is important to note that water cooling will not cause your PC to produce less heat. It will simply move the heat away from the hot internal components and dissipate it into the air in your room more efficiently. This can allow components to operate at the peak of their power capability without overheating, and operate more quietly, which may not be possible with air cooling alone.

Water cooling parts. Image by https://www.build-gaming-computers.com

3. Isn’t This Some Kind of Cryptocurrency Mining Rig?

Water cooling doesn’t make sense for cryptocurrency mining. An open-air rig that is cooled by fans alone is much more cost effective than water cooling. For that kind of computational work, you’re trying to maximise output while minimising cost, and you don’t care if it’s noisy. A mining rig is easier to position in a garage or other out-of-the-way spot where the noise it produces won’t be so much of an issue.

4. Risks, Cost, Emissions and Waste

Before rushing in to contributing your PC’s idle time to cloud computing for research, or water cooling your PC, you should consider the following:

• Everything we do consumes resources, creates waste and produces emissions. Aiming to reduce these factors helps to ensure we have a sustainable society, where we can keep doing the things we want to do, with minimum negative consequences in our lifetime or the lives of future generations.
• We don't absolutely need to investigate the universe around us through the science of astronomy. Life would continue without it. But science research, art, recreation such as gaming and any number of other “non essential” human activities arguably make life better for everyone. Astronomy is contributing to our knowledge and understanding of the universe around us, and the knowledge gained is shared with all mankind. Astronomy also leads to technological advances and benefits in other aspects of human life. This is arguably a more beneficial pursuit than burning fuel, creating carbon emissions and e-waste to support the digital mining of Cryptocurrency, which serves no other purpose than increasing some individuals’ wealth, for instance.
• Running your PC on high performance computing tasks 24/7 will increase your power bill, and your carbon emissions footprint, whether you use water cooling or not.
• If you have access to a renewable energy source, such as solar power, the environmental impact and cost of 24/7 high performance computing may be greatly reduced.
• On the positive side, simply adding water cooling will have minimal impact on power consumption on its own. You're just adding a few extra fans and a small water pump. Water cooling won't change the power consumption of your PC components by itself, but it may allow you to run those components at a higher wattage for gaming or science research purposes.
• Water cooling is difficult, costly, may void the warranty on your GPU. In general these days, GPU manufacturers will usually consider returns for faulty GPUs that have had water cooler installs done correctly, as long as installing the cooler isn’t what caused the GPU to malfunction. But there is no guarantee this will be the case.
• Water cooling also carries risks such as the potential for a water leak to destroy expensive electronic components. While leaks are unlikely when the hardware is installed correctly, and leaks are unlikely to actually destroy your equipment (usually it just creates the need for a tedious cleanup), they are still a risk factor.
• Are you considering water cooling just because it looks awesome or is a fun hobby? I don't think there's anything wrong with doing things just because they are aesthetically pleasing or fun to do, but you should carefully weigh up the potential risks and cost when it is a hobby rather than a practical need.
• Running your PC on high performance computing tasks 24/7 can contribute to heating your house during winter, which is a benefit. But this will be offset by the need to cool the house more in summer (leading to more use of the air conditioner, for instance).
• The basic laws of thermodynamics dictate that one Watt spent on computing power equals that same Watt of energy being turned into heat that ultimately ends up in the air surrounding the computer. Energy is never “used up”, it just gets moved somewhere else.
• I found that the amount of plastic and cardboard waste from purchasing all the individually boxed and wrapped water cooling parts was alarming. However, I was able to recycle the cardboard and hard plastics which make up the majority of the packaging materials. In Australia you can drop off soft plastics such as bags for recycling at Woolworths or other locations (check with your local council).
• Cardboard and many types of hard plastics can of course be recycled in your usual domestic recycling bins. Please check your local council recycling advice. The specific plastic types that can be recycled in domestic recycling bins and what you need to take to special drop-off points differs greatly by council and suburb, even between neighbouring suburbs within the same city.

My water cooling loop, which cools 2x GPUs, 1x CPU using 3 radiators.

5. Water Cooling Loop Design and Components

Some thoughts on loop design and component choice:

• Consider where everything will fit. Water cooling components take up a lot of space. If your computer case doesn't already have special places to attach radiators, then you will probably need a new case before you can install water cooling. I highly recommend the Lian Li O11 series of cases for their flexibility, modularity, dedicated radiator mount points and large amounts of space. It is possibly the most popular case for new water cooling builds at this time. The case I am using is the Lian Li O11 Dynamic.
• The most likely issue you will have with space is getting the GPU, radiators and the reservoir to fit together in the space you have available, as these are the bulkiest items. Be sure that the placement you have in mind within your case really works with the measurements of all these components. Be prepared to have to change your plans when you get the equipment, and to exchange parts or get a different case, etc. Things simply may still not fit as you expected, even after careful planning.
• GPUs may actually become smaller in some dimensions when switching the coolers to water blocks, as you remove the bulky stock air cooler. Check out the water block dimensions on the manufacturer's site to determine the amount of space your GPU will need once it has a water block installed. Even in the roomy Lian Li O11D case, I was only able to fit the reservoir in the mounting position I chose, because the RTX3090 GPU was made several centimeters shorter when the water block was installed. However, the height of the RTX3090 plus its water inlet and outlet riser was far greater than I expected. The case lid cleared the water block by less than 1mm when closed. I didn't check the card and water block height, as I was focused entirely on its length and potential collision with the reservoir. Thankfully the O11D case is very roomy, so I got away with it.
• Component order in the loop doesn’t matter. Your loop could run coolant from the pump to the GPU, then to a radiator, then to the CPU, then to another GPU, then another radiator. Or any other order you choose. The water is moving so fast through the loop that it’s more like a single pool of liquid that each component happens to be sitting in. CPUs and GPUs add heat to the pool. Radiators take heat out of the pool. The pump keeps the liquid in the pool stirred up and moving.
• A parallel loop setup is better than series, when it comes to high performance computing. Parallel hookup of the cooling blocks allows highest possible flow through each water block. The difference between series and parallel connections is minor in a gaming setup but noticeable for high performance setups. In a series water block setup, the flow rate will be limited by how much water the single most restrictive water block will allow through at any given pump pressure. In a parallel water block setup, the flow rate will be limited by the sum of what each water block will allow through at any given pump pressure. As a result, the parallel setup flow rate will always be higher.
• My cooling loop uses a hybrid of parallel and series connections (see diagram). The CPU and two GPUs are connected in parallel to the loop, as their water blocks are the most restrictive components on water flow. The radiators and pump are connected in series.

My custom loop is a single combined loop with 3 radiators (one thick and two thin radiators). My high performance computing setup needs to dissipate 800W+ of heat when the CPU and GPUs are overclocked. The loop cools an AMD Ryzen 9 3900XT CPU, an Nvidia RTX3090 GPU and RTX2060 Super GPU.

As I run two GPUs and a high-end CPU at full power 24/7, I purchased a new 1200W Power Supply Unit (PSU) to replace my old 850W PSU. This allows about 50% headroom over the max combined wattage of all the components running at full power. I understand that as little as 25% headroom is still fine.

Serial versus parallel loop setups. Radiators and reservoir not shown. In my setup, the radiators are in series along the red lines/arrows between the waterblocks and pump. Image by https://www.performance-pcs.com

6. Water Cooling Parts

The basic parts needed for a custom water cooling loop are:

• Radiators. These try to bring the water back to room temperature by dissipating the heat into the air.
• Fans. These move air through the radiators to cool them.
• Tubing. I recommend using soft tubing at first so you can easily experiment and change your setup. You can get hard tubing too, which looks nicer and can be bent at 90 degrees where required, but it is a lot harder to work with and change.
• Reservoir. A tank that contains extra coolant that typically sits directly above the pump.
• Pump. This moves the coolant around the system.
• Barbs. If you choose soft tubing for your loop, barbs are fittings that you can squeeze the ends of tubes on to that will hold the tubes without the need for clamps or collars. But I recommend you use clamps on all barbs, as I found that if you bump a tube connected to a barb fitting, it can leak a bit of fluid. I used standard household cable ties.
• Right angles. These fittings allow you to connect tubes at right-angles to the coolant sockets on water blocks, pumps, etc. Mostly useful when used with soft tubing, which doesn’t like to be bent at 90 degrees.
• Water blocks. These are the coolers that go on the GPU or CPU to run coolant past a special water cooling heat sink attached to the core of the hot device. Water blocks completely replace the air coolers that are normally attached to these devices. NOTE: Be sure to get exactly the right water block for your GPU or CPU type, or it won't fit. Make sure your exact CPU socket type or GPU model are listed as compatible with the water block on the manufacturer's website!
• Flow and temperature meter. Not essential, but I highly recommend one as it lets you see exactly what is happening in your cooling loop in response to changes you make to the layout, or the loads and stresses you put it under. Many include an audible alarm to warn you when water temperature or coolant flow are at unacceptable levels.
• Coolant. Ideally a pre-made mixture usually made of distilled water, anti-rust and anti-algal agents that is used to fill your water cooling loop. Pre-mixed coolant is readily available from MWave and similar stores. See more info in the Coolant section below.
• Cables. You will likely need extra cables to allow for the multiple fans the radiator(s) will need (such as fan cable splitters and extenders). RGB fans and RGB water blocks add to this requirement twofold, as they each have double the cables leading to them (one for power and one for RGB) and you need to somehow connect potentially dozens of cables to just a few fan and RGB connections on a motherboard, that they may not reach on their own.
• RGB controllers & fan controllers. May not be required, depending on your setup, but it's something to consider. Many water cooling parts, fans, etc will come with controllers, so check what is included before buying separate controllers.

7. High Performance System Parts List

The following is a list of the components I purchased for my system, to give you an idea of what is required in a custom high performance water cooling loop, to cool a one CPU and two GPUs, that are expected to run 24/7 at maximum wattage. I didn't use every single part listed, as I got some extra fittings (such as barbs or right angle adapters), just in case.

I didn't purchase it all at once. I purchased the basics first (and started out with just one radiator), then had to get additional fittings and additional radiators as I worked out what I needed to improve cooling performance to an acceptable point, and to make some of the connections and pipe routing more efficient and elegant.

I did not need to purchase new NVMe drives, CPU, RAM, GPUs or motherboard, as the ones I already owned were adequate.

The total cost for the parts I required to make this cooling loop, including the extras such as a new PSU, ended up being around $3,300.

For a more traditional single loop cooling 1 GPU and 1 CPU, with one radiator, the cost for the water cooling parts alone would be more like $1,500.

Prices are in AUD and include GST, but exclude shipping.

From MWave https://www.mwave.com.au/

AC14686 Thermaltake Pacific CL360 360mm Radiator (64mm thickness) x1 $159.00

AC19474 Thermaltake Pacific C360 360mm Radiator (27mm thickness) x2 @ $129.00ea $258.00

AC50178 Thermaltake Pacific PR12-D5 Plus Pump & Reservoir Combo w/ RGB Software Control x1 $339.00

AC38261 Thermaltake TOUGHFAN 12 120mm High Static Pressure Radiator Fan [SINGLE FAN] x3 @ $39.00ea $117.00

AC49086 be quiet! Light Wings ARGB 120mm PWM Black Case Fan – [TRIPLE PACK] x2 @ $105.00ea $210.00

AC37179 Thermaltake Pacific TF2 Temperature and Flow Indicator – CL-W275-CU00SW-A x1 $79.00

AC11819 EKWB Filling Bottle (1000mL) x1 $13.99

AB61483 Thermaltake V-Tubler 3T Flexible Tubing 3/8” (9.5mm) – 2m x2 @ $15.00ea $30.00

AC22192 Thermaltake TT Premium T1000 1L Transparent Coolant – Blue x4 @ $20.00ea $80.00

AC38919 EKWB EK-Quantum Vector Strix RTX 3080/3090 D-RGB Nickel + Plexi GPU Water Block x1 $299.00

AC38928 EKWB EK-Quantum Vector Strix RTX 3070/3080/3090 Backplate – Black x1 $99.00

AC52280 Thermaltake Pacific MX1 Plus CPU Water Block x1 $169.00

AB61445 Thermaltake Pacific G1/4 Stop Plug w/ O-Ring – Black x4 @ $6.00ea $24.00

AB61458 Thermaltake Pacific G1/4 Female to Male 30mm extender – Black x3 @ $10.00ea $30.00

AB61450 Thermaltake Pacific 3/8” ID Barbs – Black x18 @ $7.00ea $126.00

AB61465 Thermaltake Pacific G1/4 90 Degree Adapter – Black x10 @ $15.00 $150.00

AC10340 Noctua Chromax.Black 30cm 4Pin PWM Power Extension Cables – 4 Pack x1 $15.00

AC11822 EKWB EK-Cable Y-Splitter 3-Fan PWM (10cm) x3 @ $7.99ea $23.97

From Amazon https://www.amazon.com.au/

PH-GB2080ASDEVO_CR01 Phanteks Glacier Dual Evo, Full GPU Water Block for ASUS RTX 2080/2070/2060 Super Dual EVO Series x1 $236.93

From PCCaseGear https://www.pccasegear.com/

3830046991737 EK M.2 NVMe Heatsink Black x2 @ $19.00ea $38.00

BN816 be quiet! Dark Power Pro 12 Titanium Modular 1200W Power Supply x1 $599.00

PC-O11DXL-A Lian Li PC-O11 Dynamic XL ROG Certified Case Silver x1 $319.00

8. But Why Don't You Just...

People suggest all sorts of extreme cooling loop designs online, such as massive external radiators, refrigerated coolant systems, running cooling pipes outside the house, running pipes into their swimming pool or underground through the cold earth, and so on. Undoubtedly many of these suggestions would improve the cooling capacity of the system. However, the aim of a standard water cooling loop is to fit within a PC case, be self-contained, to use readily available parts, to be cost effective and to be relatively easy to set up. If you want to experiment with crazy ideas after getting the basic cooling loop working, I say good luck and let me know how it goes! :D

A dual loop setup, with a separate cooling loop each for the CPU and GPU. Image by https://www.performance-pcs.com

9. Single versus Dual Loop

There may be advantages to having two separate cooling loops, one for the GPU(s) and one for the CPU. I have not experimented with this, but advice I have read suggests this may be better for CPU cooling, to keep the CPU temperatures lower than may be possible when the CPU shares a loop with the GPUs. A dual loop will increase the costs as you will need an additional pump and reservoir, at least one more radiator, and more fittings.

I got lucky as the AMD Ryzen 3900XT CPU is designed to run at high temperatures, so it is happy sharing a cooling loop with two GPUs. In my loop, the CPU runs at between 80C – 90C while at max processing load (this is within its max operating temperature range), with coolant water temperatures at about 35C while room temp is around 22C. These are not great numbers for CPUs that need to run cooler than 80C (apparently for ideal CPU cooling, the water temp should be closer to 5C – 10C over room temp while the CPU is at the load you expect to run it at). Keep in mind that the temperatures my CPU and GPUs are reaching are during high performance computing loads. Gaming use will rarely push components to such extremes.

10. How Many Radiators Do You Need?

This is a difficult question to answer as it depends on the radiator size/thickness, wattage of your CPU and GPU, fan type and speed, and so on. It's best to read some of the guides available online, and to do your own research once you know the max wattage of the devices you want to cool. There's various bits of advice out there and even water cooling wattage vs radiator size calculators available online. The EKWB system configurator can provide some estimates and parts suggestions (though of course, it's designed to sell you their gear, so just use it to get an idea of the size and kinds of parts you'll need).

I should note that my first try at the wattage and radiator size calculations suggested I'd get away with using a single Thermaltake Pacific CL360 360mm Radiator (64mm thickness, the thickest 120mm x 320mm radiator they make). It's a huge radiator, but I found it just couldn't cool my system by itself when the CPU and GPUs were running continuously at high load. However the guides I followed are all for gaming use, not for running your devices at max wattage 24/7. I found I had to effectively double the amount of radiator surface area (by adding two more 27mm thickness radiators) to sufficiently cool my system for research computing use.

When it comes to radiators, bigger is is better (in all dimensions), as surface area determines the rate at which a radiator can exchange energy with the air. Make sure your case can accommodate the planned number of radiators, plus the thickness of the fans installed on top of them.

11. AIO vs Custom Loop

An alternative to a custom water loop is the “AIO” or All-In-One cooler. AIOs are a radiator, tubing and pump, all pre