View topic - Serious site issues: 11/07/2012 (see newest posts)

Hi all, I'm logged into the admin account right now because it carries a bit more formality with it. It's still me, honest.

If you tried to access the site at around 5pm last night you would have been met with a 403 error, which translates to "access forbidden". Quite simply, our friends at Site5 decided to block access to the part of the site that powers almost everything.

Apparently, the MediaWiki end of MSO was using so many resources that other Site5 customers were experiencing issues. After some investigating, they found we were to blame.

Basically, if we don't want to get kicked off their system (which we don't, not yet anyway, because we've got nowhere else to go to), we need to make sure we're living within our means. I have made quite a few behind the scenes changes to improve the performance, but at this stage it's hard to judge how successful they've been.

Until we know how big a problem we've got for sure, I have locked the site so that no-one can make any edits or uploads, as editing a page strains the server by much more than just loading one. Hopefully this will only be necessary for a few days whilst I bring other aspects of the site under control, but it's too early to say for sure. They haven't mentioned these forums, which run independently to the rest of the site, so I've kept quiet about them.

I have also cleared all the edit logs and removed the view count that each page has. These are likely to be permanent changes as it is things like updating the view count every time a page is loaded that create unnecessary strain. The photo on the home page won't update as often. Although I doubt people will really miss these things, you might be looking for them and wonder what's happened.

----------------------------------

Looking forward a bit, the obvious solution is to look for another provider. The main attraction with Site5 is that they have given us a lot of resources for very little cost. While we could look for a similar package (with the benefit of hindsight that I didn't have five years ago), it's not worth it if we're going to have the same problems a few months later. Right now I think our options are downsize a bit or make a big investment - any middle ground option is likely to land us in the same problems a few months later.

Site5 can move us on to our own server (where there are no other websites to upset or steal the duvet from), but if we choose this the costs of running the site will increase by - prepare yourself - 5000%. Obviously there is no way I can support this myself, so we will have to head down the path of donations and advertising. I will be interested to hear what other people have to say about this. It could only be a last resort, surely.
(Other providers will offer virtual severs too, but I'm quoting Site5's prices because I've just been negotiating with them)

If we have any further problems this week, then I'm sorry but it's the same issue coming back to haunt us.

Finally, while I'm trying my hardest, I don't really know what I'm talking about and would be happy to be put right by any IT experts.

_________________
This account is used only for administrative issues.

Update: Having made all the panic-stricken changes over the past 24 hours, I've been told that we are now "well within reasonable limits". I'm going to wait for a second report before lightening up and unlocking the website.

Something else I've noticed: the biggest pages, the ones which use the most power when loading or editing them, are as follows:
1. ‎A1 ‎[18,954 bytes]
2. ‎50 Years of Motorway Services ‎[16,240 bytes]
3. ‎Little Chef Timeline ‎[15,801 bytes]
4. ‎A30 ‎[15,452 bytes]
5. ‎Areas Without Services ‎[12,936 bytes]
6. ‎Welcome Break ‎[10,452 bytes]
7. ‎MSA Policy ‎[9,789 bytes]
8. ‎Moto ‎[9,061 bytes]
9. ‎Little Chef ‎[8,379 bytes]
10. ‎Watford Gap ‎[8,249 bytes]

Beyond this they aren't too bad. What this might call for is more page splitting - particularly on the A1 and A30 where larger services can be moved aside. This means that people won't be loading and refreshing data they're not interested in.

Update: Most pages are now open to editing again. The site is still unstable so please be gentle!

This graph gives you an idea of the scale of the problem we're stood on top of:

Now feel free to shoot me down in flames here, as it may give you lots of extra work.

What about if the sites gets split into two different sites. Motorway Services Online stays with all the motorway content but it gets a sister site that deals with trunk roads and all that they encompass.

MSO could still refer to any trunk road services that are owned by MSA operators but direct people to the new site (TRSO?)for any more specific detail.

I know it would not cut the size of the site in half, as there is obviously far more motorway content than TRSA content, but things like Little Chef, Happy Eater, AJ's and all the trunk roads which list services could be moved. The above brands could still be listed as former Motorway Brands too but if you clicked on them on MSO it could tell you to follow the link to TRSO instead! Even the forum-specific stuff could be moved if needs be.

Update: Editing disabled again whilst several other issues are ironed out.

I hadn't thought of that idea. While it would still double the running costs (which is far from desirable), it's a not half as bad as paying for an upgrade. I'm wondering if there is going to be any catch to stop us doing it, otherwise it would have been done before, surely? I think the problems will come when you have to decide how interlinked the two sites should be. If you wanted integrated log ins, for instance, you'll be looking at sharing databases which might make the problems worse. I'll have to find out a lot more about what our problem actually is before deciding to move things apart.

Update: The site is currently running as normal and we are waiting for the all-clear from Site5.

There has been some confusion because Site5 is American, so what they consider to be a typical morning is our peak time. They have said they are now going to measure our resource usage in the evening, and don't tell them, but that's 3am our time!

I like the cut of your jib son

And we're clear!

Go easy now, folks.

Bump.

It's exactly a year on, and we've got the same problem, but this time we've really annoyed them. Due to the extent of the problem, I cannot guarantee that by the end of this week this website will actually still exist. Apparently we are using 30x the amount of power that the websites we share with do, and that's quite an achievement when you think about it!

It will be back - I have copies of all the information and I own the website - but this is basically a warning for regular readers that we could well be in for a bumpy ride.

The website was partially disabled (for the same reasons as outlined in the first post) at 10:50 this morning. The difference between this time and last year is that last year I could implement space-saving measures. I don't have many left!

They have asked us to halve our resource usage with immediate effect. I am not convinced that is possible.

_________________
This account is used only for administrative issues.