Wuxiaworld – Servers, Caches, WordPress, & More

Since it has been a somewhat rocky month for Wuxiaworld and there’s always quite a few posts here as soon as anything goes wrong, I felt I really owed it to you guys to get everything on the record! This will be a long post…but you guys are LN novels, you are okay with reading, right? xD. This post will specifically be on ‘caches’ and our current issues with them.

Okay, so here goes, in a simplified manner, and bear in mind, I’m not a programmer, so I might be wrong on some details. First, the nature of self-hosted WordPress. WordPress websites are dynamic websites. What does that mean? Simply put, in the ‘old days’, websites were static; you created a ‘.html’ page, modified each page, then ‘uploaded’ them. Tadaa! Nice and simple. Buuut, that also meant that every time you wanted to edit a page or to the style, you had to do it all manually (or with the help of certain software). Booohoo!

With WordPress and other dynamic CMS came the rise of ‘dynamic’ websites. With WordPress, there is actually no such thing as a persistent ‘page’; rather, all the information for each ‘link address’ is stored inside a database, which then automatically generates the page on the fly when people come visit it. This is awesome in a number of ways, because then, with the proper software, you can just make modifications to the database, which will then automatically change the pages. It makes being a website manager infinitely easier.

For ‘static’ pages, the only thing which really matters is bandwidth; because all the pages are pre-created, there is very little load on the CPU. For ‘dynamic’ pages, things are different; out of the box, each time a user visits a WordPress website, the CPU of the server has to pull the necessary data out of the database and ‘build’ it into a page for you. Thus, if you have a lot of viewers, that puts a huge strain on the CPU.

Now, let’s talk numbers. Wuxiaworld has 5537 registered users, and around chapter release, we have up to 2200 viewers on simultaneously. Wow! Awesome right? Yes, it absolutely is! Buuuut, here’s the trick; if the pages were all ‘static’ html, the only thing which matters is bandwidth, which is no problem. But since the pages are being generated on the fly, that means that my CPU is being hit with hundreds or thousands of requests (just earlier, we had 9000+ simultaneous connections open to my server!). Eeek! 8-core CPU choking! 8-core CPU choking!

So the solution? Caching. Caching, simply put, is a process by which some software plugins will create a ‘fake’ static page and ‘save’ it; when a request comes in, instead of having the CPU ‘build’ the page, the CPU instead just loads up the prebuilt ‘fake’ static page and sends it to the user. Yaaay!

But caching isn’t a perfect solution either. First of all, you can’t cache people who are logged in, because then everyone would start to see everyone else’s saved information. This, btw, is why logging in fixes the ‘caching’ issues. The second, and more important issue is ‘purging’; what happens when the information on the page changes? If the cache isn’t ‘purged’, then for example, even though I have updated the ‘preview’ chapter in the database to a ‘full’ chapter, the CPU is still serving the ‘preview’ cache for that link, instead of destroying it and rebuilding a new, updated ‘cache’.

So, a critical part of caching plugins is the ability to ‘purge’ and get rid of those ‘fake static pages’ that are now expired. My caching plugin (Supercache) normally does that. The problem is, for some reason, some of the backend changes we made aren’t playing nicely with this ‘caching’ software, causing the cache to either not purge correctly (which happened today), or purge after 15-30 minutes of a new post instead of right away (like for mobile).

So what we are trying to do, going forward, is find another caching plugin that will play nicely with our backend and purge correctly, instead of either delaying 15-30 minutes, or necessitating me manually going in and purging the files by hand like what happened today. Once that happens, then we will be 100% set for good.

There’s a lot of other issues, like the DDOS, CDN implementation, Linux server tuning, etc., but that’s for another day. I just thought I would share with you guys a small part of running a self-hosted WordPress-powered website, some of the challenges, and some of the solutions 🙂

Posted by RWX

70 thoughts on “Wuxiaworld – Servers, Caches, WordPress, & More” - NO SPOILERS and NO CURSING

  1. did i mention that with log-in through wordpress (ie you have a wordpress account); the server doesn’t remember your choice (even if you don’t clear your browsers cache & cookies) and each time I come here I have to re-click the log in button>log-in via wordpress (even I’m already logged into wordpress…).

    Oh did I mention that log-in via wordpress (in the log-in page) is horrendously slow….!

  2. You know, I had a problem the other day. I tried to read a chapter that wasn’t translated yet so I got the page “Oops! That page can’t be found.” but later when the translated chapter was already uploaded I still kept getting the “Oops! That page can’t be found.”

    I ended up fixing the problem with Ctrl + F5 (instead of the normal F5 that we, from the “F5 army”, normally use).

    Is that a problem with your cache or mine alone? I don’t know much about cache.

  3. So you run this website off your personal PC? I don’t really see a way to make that work :0

    Granted, I’m no computer engineer, but I did run a few servers back in my day. There’s no third party hosting options? Once I got off running the server from my PC it simplified things a bit. I suppose if you wanted a third party hoster, you may need to change the website, however… also, it would cost dat $$$ so I can see why you wouldn’t want to.

    Upgrading your PC also would cost dat $$$ and it would only be a temporary solution as well… hm I think I need to do research on these dynamic sites. I want to host my own website one day so this would be helpful to know 0.0

  4. Are you using a specific WP plugin for your mobile site that integrates with Supercache? I use Supercache on large multisite installs that I manage (running 3000+ sites) and know it caches on a domain basis. So when updating http://www.wuxiaworld.com, unless using a supported WP plugin for you mobile site with the correct settings saved for supercache, m.wuxiaworld.com wont get cleared out. Haven’t ever had an issue with supercache not clearing out the primary domain cache though.

    1. Hi Ellcrys, nope, just Supercache with a custom nginx configuration. Supercache DOES still purge mobile when configured correctly, but on a ‘time’ basis, whereas for desktop, it will purge it as soon as I update a page.

      I’m not sure what the problem was with the primary domain not clearing either; I could literally see the expired files, but even when I tried to delete them manually, it wouldn’t go through. I had to go in and manually delete through cPanel. It was very, very strange! I think it might have had something to do with the permissions, since we updated some of them earlier today.

  5. When you read it, it all sounds so simple, but if one actually tries to do all of that himself without prior knowledgle… Hell! 😛 Good job Ren!

    PS: Sine we all got used to being your proofreaders, more of less…

    ‘but you guys are LN novels’

    I believe you meant ‘LN lovers’ or something like that 😛

  6. ever thought about doings static pages just for the chapters?
    with the high number of users you have here, it would solve many problems… the extra grunt work can be streamlined after everything is setup.

    it’s not an unheard method, just tougher for those without any coding skills..

    1. Hi senthiel, welcome to the community, and thanks for the suggestion! I’ve thought about it, but it would just introduce so much variables that my head would spin, especially if I were to ever migrate again or change configurations. I think I’m going to have to stick with CMS!

  7. sooooo I begun read this as a courtesy for our Ren-sama……. buuut the more it went the less i understood *sigh* letters seemed to have turned Greek and my forehead had a hole after sooo much facepalming! …….. T.T
    I’m so sorry for having such a stupid brain, I always skipped IT lessons cuz it’s boring! now I’m reaping what I saw! *sob*
    aaanyway I think now all the issues have been dealt with? right??
    hm, then that’s good! good luck with the rest of the day Ren-sama!!

    1. I also do not know much about all of that, but i understood most of it with how Ren explained it… Soo, what were you doing recently that your brain fried, TsunTsun? 🙂 Lack of sleep is showing xD

  8. I’m not too familiar with WordPress plugins but does Supercache support fragment caching? Essentially what it does is caches parts of your page separately instead of caching the whole thing. It will solve problems like your donation bar being consistent across pages. Also you can then cache the chapters for logged in users also.

    1. Supercache doesn’t, no, it’s full page caching. We played around with memcache and some other solutions, but thus far, full page caching really does seem to have by far the better performance!

      1. Well yes, because it’s caching the whole page as if it’s static. As long as you’re fine with parts like donation bar or forum last post dates being outdated it’s fine.
        Also like you said, you can’t cache pages for users this way, so I assume the majority of traffic is from people not logged in?

        1. Supercache is set up so that every 60 minutes, a cache is automatically invalidated and set to be purged/recached, so it doesn’t persist for too long! And yeah, we have around 5500 registered users, while according to WordPress IP’s, we have around 500,000 unique IP visitors each month. Granted, a lot of those are cell phone users who have constantly shifting IP’s, but the point remains; we have a LOT of non-registered people.

          1. A solution that could discourage people from constantly refreshing during peak times is using captchas. You could set something to trigger if a page is refreshed too many times within a certain time frame.
            Though this will make some people upset.

    1. DDos is usually used through infected web, as also Ren explained, so quite a few innocent, yet stupid people would have their PC’s fired, too 😛

      1. Well, I’m wising it on the guilty so it may end for the innocent ‘-‘

        Though I say that, I actually know nothing about it.. Just that it’s causing a lot of problems XP

  9. Nice, looks like you have done your research. I ran ( for a client ) a worpress news portal while ago with around 80-200 ppl online the configuration was nginx+wp+W3 total cache I had no issues with caching + pages were set to rebuild upon changes ( ex. new comment added ) the server itself was amazone ec3 micro instance tough to make it work nicely it took me quite a while, so what i would recommend is to newer give up since there are lots of things that can be improved as you experience grows.

    1. Thanks for joining the community and for sharing your experiences, rakweb! I earned every bit of experience the hard way, and have the bruises to prove it, hehe.

      1. Hard way is the most rewarding one once you do it that knowledge will stay around and surprisingly it will help in other areas aswell and if you feel like you have reached a dead end take five minutes and picture the web without any search engine….. boss: we need to improve this … me: ok ill do it this week & maybe abit of next ( nowdays 2h work )

  10. I find it ironic that I was not able to read this post yesterday due to the site not working :'(

    Hang in there Ren, it will get easier as you get you gain further insights into the profound mysteries of modern computing 🙂

  11. Hi Ren,

    Hopefully you will find a solution to your problems (which I currently have no experience about it to help). I know how hard it is to manage a high traffic website.

    When you changed your hosting to arvixe last time, I thought that since you use them, they should be good. That’s why I also decided to use their hosting service. Of course with some difference, since I’m just using their cheap hosting package since I’m just starting out yet. But so far, since it was also the first time for me to handle quite a high traffic website (compared to wuxiaworld, my traffic is so lowwwww hehehe), I have encounter many problems with not too much help from the support (well, I guess it comes with the price LOL).

    I hope in the future, I could discuss or ask for your advice regarding the hosting matter when the server couldn’t handle the traffic anymore and I have to change to another plan or another provider.

    And as for those DDos or hackers, yeah, they could go to hell. Few days ago, my site was also attacked heavily, luckily not DDos but just a brute force attempt to crack the admin login. It’s solved now, but at that time, since I never had any experienced before, it sent me to panic 🙂 I just don’t understand what they are trying to achieve in doing all that stuffs.

    Anyway, I wish all the best for your server to run smoothly, and THANK YOU SO MUCH for your translations.

    1. szeimartin, RSS stops working when I turn on anti-DDOS, which happens every so often when I get spikes of problematic traffic from Cloudflare!

  12. ♥Thank you for the chapters my wonderful friends♥
    Written by: I Eat Tomatoes (我吃西红柿)
    Translated by: Ren
    Hosted by: WuxiaWorld

    Do not give up! I hope Bebe would swallow up those meanies DDOS’ers!!

  13. Not that I have anything against WordPress but wouldn’t a simple solution be to just get yourself a simple html/php based page?

    What exactly are the advantages that WordPress offers against an ordinary website?

    1. Quite a lot if you ask me.

      – Takes care of the security issues.
      – Is actively mantained.
      – It’s been tested millions of times and it’s both stable and production ready by the time you get an update. So no testing is required.
      – It works seamlessly with a rich ecosystem of high performance tools. IE. Databases, web servers, php releases.
      – There is a very big community that can help you with it.
      – Allows dynamic content without having to code a single line.
      – Works out of the box for the vast majority of the sites.
      – Can be expanded with plugins and has a very big library of both free and premium ones.
      – It has support for themes, and allows changing the visual aspect with a few clicks.
      – Allows for modularity in the design of the site.
      – A lot of people has experience working with it.
      – It’s cheap to deploy and maintain.
      – An “ordinary website” sounds like custom. And custom has many shortcomings that do not make it worth the effort.
      – It helps automating common tasks to less experienced users, like updates or installation of new content.

      I don’t know what’s an ordinary website for you. It’s not like if you get a simple web page rolling with PHP is going to be any better or any faster than the core of wordpress properly tunned. And it’s way cheaper to buy resources, than it is to hire a programmer to make you anything remotely similar.

      Plus, WP fits all of the user cases Wuxiaworld should be concerned. The problem here is the high load the site gets at certain times of the day. I presume that it is arround the time Ren posts a new chapter.

      I’m just trying to explain, I hope this post is understood as such.

  14. This post reminds me the time I first stumbled to this site…
    My first thought was “Another site uses WordPress”.
    In my time(Before cloud *ahem*ancient, I know*ahem*) I hosted a website…and I too searched for an easier tool to use…
    In the end I’ve skipped quiet a few options and choose IPS (Invasion power board) with quite a few plugins…I know not the first choice for a portal…BUT I personally think, it’s more ‘light’ than other systems(more so than WordPress).
    I don’t remember the exact statistics, but I’m pretty sure I had more traffic…and it was smoother until I’ve changed hosting(which corrupted the database)
    And so it still is (my thoughts “oh, WordPress”)

    P.s: I’m not sure but something feels odd with my writing(I.E, Grammar), if I’m right, please correct me?
    [My third language, and…not the last.]

  15. I have some experience with WordPress and in my honest opinion best way is to not use WordPress accounts at all (If your page isn’t highly customized service based on WordPress you don’t really need to use them). There are better solutions.

    WordPress comments suck, Disqus is a whole lot better, it offers great features (those you have now and many more) and is easy to manage and customize via their dashboard and admin panel, and it’s totally free. The best thing about Disqus is that they store and process all the comments on their side (for example when user adds comment nothing happens on your server cause everything is processed by them). You just have to create your Disqus group and then add plugin to WordPress which will add Javascript/AJAX code to load the comments box from their servers (it will automaticly create new comment thread in your group for any new or existing post) and that’s all. Ah, and you can use the same Disqus account on any other page using this comment system.

    As for the forum, plugins offered for WordPress are very limited and can cause much frustration. Individual script is the only right choice. Theres a lot of great, free scripts like phpBB, myBB or SMF.

    Finally, talking about caching plugins I can recommend ZenCache (former Quick Cache). Pro version for $40 can cache logged in users properly, but with Disqus, free, lite version is all you need (If you’re not going to use CDN). I think it’s worth trying.

    If you have any further questions I will be happy to answer them.

    1. No. I don’t like Disqus. Their tracking is way too much.
      It’s like Facebook without privacy controls and everything default to worldwide.
      Their terms and conditions are also too broad.

      1. I’m not sure what type of privacy controls do you want. There is “Keep your profile activity private” option in profile settings and the only thing you have to provide to create account is name (nickname) and e-mail, no one forces you to login using Facebook, Google or Twitter. – http://i.imgur.com/VjYAQXs.png

        I don’t know what world you live in, but nowadays every service has got similar ToS.

Leave a Reply