Management Server: Common Six Reasons for Servers to Crash

If you don't have any experience, learning how to extend a website is quite difficult. Suppose now that you have a lot of sites like highscalability.com, you need some good solutions to extend them, but there is no “all-purpose medicine” in the world, and no solution can adapt to the needs of all websites. You have to do it yourself and think through constant thinking to find a solution that meets your needs. I did the same.

A few years ago, my boss came to me and said to me: "We have a new project that we want to give you. It is mainly a website refactoring, within a month, this site I already have 1 million users. You have to rebuild this website to make sure we can handle the increasing number of users in the future." I am already an experienced programmer, but not good at these, so I can't Don't start learning how to extend a website – the whole process has gone through hardships. (Related articles recommended: Refactoring: "Why" and "How to do it")

The background software for this site is a PHP content management system based on Smarty and MySQL. The first task is to find a suitable hosting company that needs to have extensive experience to manage servers for us. After some research, we found a company like this and then told them our needs. The recommended configuration they gave us is as follows:

Load Balancer (+Fallback)
2 Web Servers< BR>MySQL Server (+Fallback)
Developing Machines
They said that this is all we need - we are convinced. The last configuration we got was:

load balancer (single core, 1GB RAM, Pound)
2 web servers (dual core, 4GB RAM, Apache)
MySQL server (quad core, 8GB Memory)
Development machine (single core, 1GB memory)
This configuration is very basic and has not been further optimized. In order to synchronize files (PHP and media files), they built an active-active DRBD. Finally, the refactoring began - of course, we are very excited. Early in the morning, we switched the domain name to a new IP, ran our monitoring script, and stared at the screen. We immediately saw traffic on these machines and everything seemed to work very well. The page loads very quickly, and MySQL is burdened with a lot of query tasks, and we are all very happy.

Then, suddenly our phone started to ring: "We can't visit your website. What's going on?" We looked at our monitoring software, and it turned out to be the case - the server Being frozen, the site is offline! Of course, the first thing we did was to call our hosting provider: "All our servers are dead. What is going on?" They promised to check the machine. I will call again later. The call came: "Your system simply can't intervene. What have you done? It's completely messed up." They stopped the load balancer and let me observe one of the web servers. I was shocked to see the index.php file. It contains some weird C code snippets, error messages and something that looks like a log file. After further investigation, we found that the DRBD caused the accident.

One way to "kill" your server

Put Smarty compile and template cache on a highly loaded active-active DRBD cluster, then your server will hang When our hosting provider fixed the web server, I rewrote some of the CMS code in order to store Smarty cache files on the local file system of these servers. We are online again!

It is afternoon. This site usually peaks in the evening to late evening. At night, there is almost no traffic. We have been staring at the monitoring software, and we are all very nervous. This website can be loaded, but later, the higher the system load, the slower the response. I added the lifetime of the Smarty template cache, hopefully this will work - but unfortunately this didn't work! Soon, the server started giving timeouts, blank pages and error messages. There are two machines that cannot handle the load.

Our customers are a bit nervous at this time, but he said: OK, refactoring usually causes some problems. As long as you can fix it quickly, it's okay!

We need a plan to reduce the load, and then we discussed this with our hosting provider. One of their system administrators put forward a good idea: "Man, your server is now running on a very common Apache+mod_php architecture. How about replacing your web server with Lighttpd? It's a pretty small project, but Wikipedia is using it." We agreed. (Related articles recommended: Better choices for the four alternatives to the Apache server)

"Methods to kill" your server

Get an out-of-the-box Web The server is set up on your machine and it is not optimized at all, then your server will hang! The administrator did his best to reconfigure all the web servers as soon as possible. He abandoned Apache and switched to Lighttpd+FastCGI+Xcache. Later, when we re-launched, we barely felt the pressure again. How long will these servers last?

These servers are surprisingly good at running. The load is much lower than before and the average response time is good. We were completely relieved, and then we all went home to sleep. It’s already late, and we don’t think there is anything else we need to do. The next day, the site was running quite well, but during peak hours it was close to the edge of the crash. We found MySQL to be the bottleneck and we called our hosting provider again. They recommend using MySQL to perform MySQL master-slave synchronization from the server on each web server.

The third method of killing your server

The powerful database server also has its limits. When you reach its limit, your server will hang! In this case, sometimes your database will become so slow that a large number of network connections in the queue will "kill" our web server again. Unfortunately this problem is difficult to fix. The content management system is very simple in this respect, and it does not itself support reading and writing SQL queries individually. It took a long time to rewrite this, but it is quite worthwhile to encounter hibernation every minute.

The MySQL sync was really successful, and the site was finally stable! In the next few weeks and months, the site was a success and the number of users began to increase. Traffic again exceeds our resource limits, which is only a matter of time.

The fourth method of killing your server

No planning in advance, your server may hang!

Fortunately, we have been thinking And has been planning. We optimized the code and reduced the number of SQL queries needed to load each page. We unexpectedly found MemCached as a good stuff. First, we added support for MemCached on some core features, and we also added support for MemCached on some heavyweight (slowly running) features. When we deploy these changes, we can't believe the result - it feels a bit like discovering the Holy Grail. Our number of queries per second has been reduced by at least 50%. We decided to use MemCached more than to buy another web server.

Five ways to kill your server

Forgetting to do cache, you will waste a lot of money, and your server will hang! It turns out that MemCached helps us reduce 70%-80% of the load on the MySQL server, meanwhile, on the Web server, also produced a huge performance boost. The page loads quite quickly.

In the end, our configuration seems to be perfect. Even during peak hours, we don't have to worry about crashes or slow page responses. Have we got it? No! A blue web server started to respond a bit slowly. Then there are some error messages, blank pages, and so on. This system has a very good load capacity, and in most cases the server is working, but only in most cases.

Six ways to kill your server

Put hundreds of small files in a folder. When the index node is exhausted, your server will Will hang!

Yes, you are not mistaken. We used to focus on MySQL, PHP and the web server itself, and didn't pay much attention to the file system. Smarty cache files are stored in the local file system - all cache files are stored in the same directory. The solution is to put Smarty in a dedicated ReiserFS partition. In addition, we have also turned on Smarty's "use_subdirs" option.

In the past few years, we have been optimizing the page. We put the Smarty cache in memcached. To handle static files more quickly, we installed Varnish to reduce I/O load. We also switched to Nginx (Lighttpd will randomly generate error 500 messages), install more memory, buy better hardware, more hardware... This list will never end.

Summary

Extending a website is a process that never ends. When you solve a bottleneck, you are likely to encounter the next bottleneck. Never think like this: "That's it, we're done" and then stand by. This will "kill" your server, or even your business. Planning and learning is an ongoing process. If you can't do this yourself because of lack of experience or resources, then you can find a partner who is capable of doing the job and is very reliable, and work with it to do the job. Never stop communicating with your team and partners about some of the problems you are currently experiencing and some of the problems you will encounter. Think before you can get ahead.