Questions about cluster computing

Cluster computing, to my knowledge, is when a program distributes the same task over multiple devices; which results in it being done faster. “I’ll mow the lawn, and you clean the bathrooms.” You’re limited by the speed of the network connecting them and the sophistication of the distribution program, but otherwise the computation power is improved additively.

If that was true, then why isn’t clustering more common? Practical considerations aside, wouldn’t it be worlds cheaper to frankenstein a bunch of older hardware from a surplus store instead of buying shinier parts?

I am just learning now about how clustering works, and I had this fantasy of duct taping a bunch of raspberry pi units to my tower instead of dropping five grand on a graphics card. I’m sure things aren’t that easy, but I am curious about what is possible.

It would not be cheaper to put together a bunch of old hardware. Short term, maybe. But long term, no. The reason for this is the processing power/kWh of electricity used. If some unscrupulous jerk were to sneak a bitcoin mining program on your computer it could easily raise your electric bill $20 a month. Processing take power. The more processing you do, the more power it takes.

Older equipment is much slower and much less efficient. What this means is that you get less return on your investment of electricity than with newer equipment. And the difference can be massive. To understand why you have to know a little about how processors work.

The heart of any processor is the clock. That’s the part where they tell you the speed in, today, GHz, that the processor works at. The Intel i5-9600K processor has a clock speed of 3.7GHz with a burst speed of up to 4.6GHz. At 3.7GHz the clock “ticks” 3,700,000,000 times in one second. The way electronics work is that an electrical signal can “trigger” on the rising edge or falling edge of a clock tick. So, every time the clock ticks, one thing can happen (I don’t think any electronic circuitry uses both edges at the same time). The much earlier 8086 processor originally ran at 5MHz (5,000,000 clock ticks per second, nearly 1,000 times slower than the burst speed of the modern processor) with successors which ran at up to 8MHz.

Every processor also has an instruction set. The instruction set is the list of “single things” the a processor can do in one clock tick. The x86 instruction set is very popular beginning in the '80s (remember the “486” computer? That computer use a “80486” processor, the third successor of the “8086” processor). The number of instructions in an instruction set is very hard to count because it depends on how you look at it, but that number has gone from around 100 with the 8086 to “just under 1,000 - a little over 2,300” (again, depending on how you count) with the modern x86 instruction set. The bigger the instruction set, the greater the number of things you can do in a single clock tick because you don’t have to use 2 or more, each one using up one clock tick, to get a task done. (Note that even in the early days, some instructions used 2 clock ticks, but we’re ignoring that for simplicity).

Earlier processor were single core as well while modern processors are 2 or more. Once more, I once heard it said (though I never bothered to check it) that you shouldn’t bother to use SLI, where you connect 2 graphics cards together to use as one, unless you used the best graphics card on the market (this was over a decade ago) because the latest graphics card would be faster than 2 of the next one down put together.

And then there’s Moore’s Law, which states that the number of transistors in a processor doubles every 2 years. This actually held true for decades. Assuming a direct correlation between number of transistors and processing power (it’s not quite that easy) then 1 brand new computer would be the equivalent of 4x 4 year old computers, 8x 6 year old computers, 16x 8 year old computers, etc.

Of course it’s not quite that simple. Some years there are massive leaps, some generations are nearly identical to the previous generation of processors. But the difference is certainly large enough that you replace your computer, usually, about every 6 years because, due to the increase in size and complexity of the operating system, the programs you run and the data you process (like web pages), it has slowed to an intolerable speed. It cannot process fast enough to handle modern programs and data sets at reasonable speeds.

One last thing, while you are limited by the speed of the network, this is generally not as huge an issue as you might think. Distributed computing usually involves sending a a small amount of data to be processed and then receiving the results of the processing. They don’t send every single command over the network. Generally the program doing the processing will be specific enough that they can send a simple mathematical operation (formula to be solved) or a block of numbers to be processed by a formula the program already has (as in “You process for variables 100,000 to 199,999 and tell me what you find”). It’s not like, “Tell me what 2+6 is. Okay, now calculate what this pixel should be.” It’s more like “Perform mathematical operation X on variable Y” where the program already has a list of the mathematical formulas it will be using.