It seems like not a month goes by without another code vulnerability being discovered. One only hopes that the people who find the vulnerabilities are good guys. This time it is an L1 Terminal Fault (L1TF) in the Intel chip that it used by web hosts running virtual machines. That includes Digital Ocean, the web host that we use.
The vulnerability exposes data to anyone running on the same processor core as another domain’s data. It’s like one tenant in an apartment block being able to look in on the next apartment.
Thankfully, Intel has learned from its earlier mistake of keep its problems to itself. Instead, it shared information about the problem with web hosts and that means that Digital Ocean has already started working on a fix.
They say that it’s going to take a few weeks to complete the work, but they do not anticipate any downtime for their users (Quillcards, in our case) as a result of their efforts to fix the problem.
What I do expect is that they will share any information about anyone who actually has used the vulnerability to look where they shouldn’t.
This is a follow-up to our last newsletter about Quillcards being affecting by our web host working to patch vulnerabilities in Intel chips.
Our hosting company, Digital Ocean, which hosts thousand of websites, began a server reboot at 2pm UK time today.
Our site went down at 2:02pm, and it has not come back up automatically. Therefore we are going to have to manually restart it, which will not be until tonight.
Then we will also need to do some work on our setup, and we will do that after the site is running again.
As we said before, Digital Ocean is doing the reboot because of the industry-wide security vulnerabilities known as Meltdown and Spectre that affect Intel chips that are deep at the heart of many web host servers.
Update Tuesday Evening
The site is now running normally.
You may have read in the news that there are industry-wide security vulnerabilities, known as Meltdown and Spectre, in the Intel chips used in literally millions of systems and thousands of web hosts.
Because of these vulnerabilities in the chips, web hosts need to patch and reboot their systems.
This is not a problem in Quillcards itself. It is a problem with the Intel chips used by web hosts all over the world, including the chips used by our web host.
We have been expecting to hear from our web host about this and we have now received an email from them telling us when they are going to start work to patch their system to mitigate the vulnerabilities.
Our web host is beginning the work now and will start with data centers in the USA. Then they will move to deal with data centers elsewhere.
When Will We Be Affected
Our data center is located in London in the United Kingdom, so it will be affected at some point later this week or next week.
This is an unknown. Rebooting their system and our system may be easy and you may not notice it, or it could take us down and we don’t know how quickly we will be able to restart the site.
Our web host has promised us at least 24 hours ahead of scheduled maintenance windows for our data center.
We can’t do anything about the maintenance schedule but we will make every effort to keep you updated.
We will add updates here and on the Server Updates page on this site.
We suggest you bookmark this page so that you can reach it easily.
We track Quillcards and get an email alert if it goes down for any reason. It went down for about 15 minutes less than an hour ago.
Our web host posted a notice immediately and I signed up to be alerted with updates to the notices.
The first notice said:
We are seeing some heavy packet loss at our Dallas datacenter right now. We are investigating.
I signed in to a chatroom at the web host and was told that the packet loss (loss of data) was at the Dallas data center where our site is hosted.
So I knew that the problem affected our web host and was not Quillcards itself that was having a problem internally.
The next notice said:
It looks like the loss has subsided now, this affected our entire Dallas network, and appears to have been something from upstream. Working with the datacenter NOC team to find out what occurred.
The thing is that the Dallas data center handles data for various web hosts, and this was the final notice:
We have been advised another customer at the datacenter was subject to a very large DOS attack. That has been blocked and the network has returned to normal.
So what is a DOS attack?
It’s a malicious attack by outsiders who send a query with forged credentials to the website they are attacking. The web server responds and the attacker asks the same question again. Except the attacker doesn’t just ask once or twice: It asks many, many times a second and eventually overwhelms the service.
The way to stop the attacks varies with the kind of attack, but one way is to query the credentials and get the attacker to prove it is who it claims to be. When it fails to do so, the attack is blocked.