The End Of Software - Anibal's Blog

• 36 data centers across the globe ... 150 racks per data center ... more than 200,000 servers .. clusters of 1,800 servers are pretty routine
• an ordinary Google search query that involves 700 to 1,000 servers
• “Our view is it’s better to have twice as much hardware that’s not as reliable than half as much that’s more reliable”
• “You have to provide reliability on a software level. If you’re running 10,000 machines, something is going to die every day.”
• ... first year, it’s typical that 1,000 individual machine failures will occur ... ... thousands of hard drive failures will occur...... one power distribution unit will fail, bringing down 500 to 1,000 machines for about 6 hours ... 20 racks will fail, each time causing 40 to 80 machines to vanish from the network ...... 5 racks will “go wonky,” with half their network packets missing in action; and the cluster will have to be rewired once, affecting 5 percent of the machines at any given moment over a 2-day span...there’s about a 50 percent chance that the cluster will overheat, taking down most of the servers in less than 5 minutes and taking 1 to 2 days to recover

Also, recently, Google talked about one of its own designed servers:

• each server has its own 12-volt battery to supply power if there's a problem
• Google's designs supply only 12-volt power, with the necessary conversions taking place on the motherboard
• Google's PUE (energy efficiency) scores are enviably low, but the company is working to lower them further.
• the core of the company's data centers are composed of standard 1AAA shipping containers packed with 1,160 servers each, with many containers in each data center.