Server Specs - Lots Of Smaller Servers Or 1 Big One?

MySQL & MariaDB Other MySQL® Questions
1

Hi,

Sorry to as such a question on the first post - but a post on MySQLPerformanceBlog describing 512GB in a server as relatively common (or something like that) got me thinking people may already have tested in environments we are working up to.

We’re currently testing a server for performance on MySQL - it’s a Xeon system with 48GB of ram and a FusionIO 80GB Solid State Drive.

We’re just in development so we can do all sorts of strange things to it and not worry about users, but are curious about the performance of systems with more RAM.

I have seen servers that allow 2TB of ram (e.g. HP Itanium servers with 64 processors) - but they are very expensive. What we’re likely to need is around 2TB of almost instant access storage (so disks are out - must be DRAM or enterprise SSDs).

Does anyone have any experience with either splitting tasks over several smaller machines (similar to the development server we are working with - but with bigger SSDs) or using one of the huge memory-capable machines. What sort of issues start to show up (e.g. network speed when multiple machines are involved, but they would be passing reduced datasets that have been mostly processed; or, are there limits to the actual memory space that a single core can address - ignoring table locking issues due to read only operation most of the time).

Any pointers on what problems there may be would be appreciated. I know I’ll probably be asked to describe our requirements - the best way to put this is there will probably be many smaller tables that can answer individual queries - hence multiple machines would be possible - but I’m thinking maintenance would be a nightmare. Having said that I can’t see how these servers with massive amounts of ram could possibly cope with 64 processors (times many cores) could keep memory busses from getting overloaded very quickly - am I right that such machines are more suited to applications that only/mainly use the memory that’s 'associated. with that processor (or am I completely wrong).

Thanks for any help - absolutely loving using MySQL and want to see how far I can push it.

2

A lot of this depends on your access patterns. Are you doing mainly key lookups or are you performing aggregations? What’s your read-write ratio? What kind of reliability/redundancy are you looking for? What is your expected throughput? Your desired response time and tolerance?

Personally, I think it’s generally cheaper and “safer” to run on a larger number of smaller machines than on one (really two) big machines.

I don’t have any experience running on hardware with TBs of memory, but I do know that the Percona guys are always blogging about how it can be a serious issue warming a large buffer pool. You think you can reboot or restart MySQL without much downtime, but all of the sudden you need to load 2 TBs of data into the buffer pool before your database is performing at the level your customers expect. That can be a serious issue.