Last January 16th, Funio took over the domain management systems, hosting infrastructures and customer service. Today, this transition which is not always very easy, is ongoing. We have faced many challenges in the last month, but our team has dedicated itself to the task and has been able to catch up with the accumulated lateness that was due to a high volume of requests. We see today that our efforts have borne fruit.
Here is article 2 out of 3 articles that include a Post-mortem of the last month’s activities, as well as a descriptive of certain projects on which we will continue to work on in the next few weeks. Your comments are appreciated and will help us take the right decisions.
Our Infrastructures
1. Post-Mortem on infrastructure stability
In the last few months, we have lived through certain problematic situations on our infrastructures. First, new security exploits on CMS (WordPress, Joomla, and others) saw many sites hosted on our servers hacked. The primary consequence of this was that many of ours customers’ sites were affected and many backup restores were necessary to get the content of their sites back up. This operation requires the restoration of a great quantity of data and it is not possible to restore the contents of multiple sites at the same time on our backup servers. This explains the longer delays to perform backup restores.
A secondary consequence was that the majority of the hackers use these exploits to send out spam via scripts, which leads us into our second problematic issue. When spam is sent out, the servers are identified by anti-spam organizations as behaving with malevolent activity, which in turn causes several issues. In fact, the organizations in question would “blacklist” automatically the IP addresses used for the outbound spam, which translates, for certain customers on these servers, an occasional incapacity to send certain emails to certain recipients. We always intervene quickly on these problems and have a good relationship with these organizations that know well that we denounce spam activities, but the process can take sometimes several days before an IP is delisted.
A third consequence to these hacking episodes is a larger utilization of disk capacity. Hackers use an account’s space to include a multitude of files that can then be downloaded from multiple sources. A large quantity of files is then uploaded in a very short time, which makes the space available on the disks to load up quickly. This prevents our legitimate customers from uploading/downloading their files correctly while the hacked account is identified and that the files are erased.
Finally, the fourth consequence to this is a general degradation of server performance. Usually, when a customer abuses server resources, we intervene quickly and the intervention is, in most cases, seamless. In certain cases, when a customer uses too much resource, the server has a hard time allocating its resources correctly because it is too solicited, and our team has more interventions to perform. When a server is highly solicited, the first service to have difficulties is the email service. This is the reason why, in the past few weeks, the servers have encountered slow downs and intermittent interruptions during peak activity hours of the day.
2. Infrastructure stability projects
Additionally to our usual interventions, we have begun 3 improvement projects to prevent situations as mentioned above.
First, our team installed “mod_security” on certain servers. This application allows the prevention of certain problems mentioned above by blocking a large part of hacker attacks on our servers, even on CMS (WordPress, Joomla and others) that our customers use that would not be up to date. We know that this installation will have a positive impact on reducing hacking attempts and we will assuredly gain greater server stability and performance. However, we know that this might block certain legitimate scripts used by our customers and we always attempt to intervene quickly to correct these cases. The objective is not to cut off functionalities, but to improve stability and performance for all our customers on the server.
Furthermore, we usually plan a certain quantity of “free” resource on the server (disk space, RAM, CPU) in order to assume peak traffic times that must not consume more than a certain quantity of the total server’s resources. We are going to augment this quantity of free resource by adding new servers and migrating certain customers to these new servers. We normally add new servers to our infrastructures, but these new machines will be used uniquely to reduce the load of other servers from our park. Additionally, we have identified that disk i/o was the origin of certain performance problems, so these new servers will be equiped with faster RAID cards and disks, which should, as per the tests performed, help us gain in performance.
Finally, we are currently using “CloudLinux” on our servers. This platform has a great potential that has not been exploited until today. We are currently working in collaboration with CloudLinux specialists to put into place configurations that would allow us a larger control over resource over-usage by certain customers, as well as put a prevention process to improve our proactivity on problem resolution.
The objective is to put into place these modifications and add-ons by early April. Already, last week, we have noticed a clear improvement of server performances with the start of the implementation of certain of these measures. We are confident that the infrastructure stability will reach new heights once the projects are completed.
The next article will appear in the next few days : Post-Mortem and improvement on customer service
You can also consult the previous article about our new domain name management.
