Following several weeks of investigation and updates with S4labour, we have have summased the fixes that have been put in place inorder to support system stability and performance.
The Login Procedure
On Monday morning, the system was being hit with 600 – 1000 login attempts per second, over 100 times more than any typical day of the week. This led to hundreds of thousands of calls to the server every minute, resulting in slower system performance and on occasions causing the system to crash.
We are still investigating the cause for this volume of login, given the recent announcements from other SaaS businesses in recent month, it is likely to be a bot attempting to breach our security procedures. The good news is that, while the system has suffered performance and speed issues, all of the security procedures did their job and kept S4labour secure.
However, we have and are continuing to implement a number of significant architectural changes to both improve the security of S4labour and ensure that system performance is not impacted in the future.
Server Duplication
As well as increasing the server capacity by 50% for the main system, we have added an additional server, dedicated to the login process. This will mean that any impact that login demand may have on S4labour, will not impact on anyone who has already logged in. This will also mean that, if demand remains high, and the login server needs resetting, users already logged in will be unaffected.
CAPTCHA
The introduction of the CAPTCHA on Thursday the 16th of December will significantly reduce the impact a bot could have of draining server capacity going forward.
Login Control Procedures
The login process is controlled to allow up to 30 logins per second. This combined with the CAPTCHA will mean that the login server cannot reach maximum capacity, but there is a possibility of a 3 second delay between logging in and being let into the system at peak demand times. We will be monitoring server load closely and should the system near CPU capacity, we will update this control appropriately. This may slightly increase the time it takes to get into the system by a few more seconds. However, we now have the ability to do this very quickly and with no disruption to anyone already logged into the system.
