LL       III  NNN  NN KK KK  EE      RR   RR RR   RR OO   OO RR   RR
LL       III  NN  NNN KK KK  EE      RR  RR  RR  RR  OO   OO RR  RR 
Posted on: Thursday, April 16th, 2009 at 00:21.
Filed under: All, ServerAdmin.
RSS 2.0 feed for comments.
You can leave a response, or trackback from your own site.

If you have an account on the donationcoder.com member server, you might be aware that I have recently been working on the server intensively. The mysql hostname has changed, and the word ‘jail’ was dropped here and there.In this post I’ll attempt to explain in more detail what is going on.

Quite some time ago now, we had moved the member server accounts from our old, now discontinued vps server to a new dedicated server.

This move had to happen as quickly as possible, since all the websites of our members would be completely down during the transition.
Unfortunately this caused me having to implement certain things that I normally would/should have implemented beforehand later on.
Also, some of the unique problems associated with the member server only came to really stand out the more I was working with it.
The problems can be summarized as following:

  • There are many accounts on the server.
  • Many people are running custom code, or third party code that is not always upgraded to the latest version. This poses a great security risk. Even though we ask that they do, many of our users fail to keep track of security updates for say, wordpress, or other popular software. Another problem is people using code they found on the internet, that may not always be secure. (One example is a script to make pretty directory indexes. It allowed you to pass a ?dir=/some/folder parameter. It had no bounds checking, so it effectively exposed all files the user running the script could access (which is currently the apache www user.), effectively exposing most files on the server. Some of these things are very subtle, and with 100 some accounts, I cannot possibly police them all.
  • Some things are more subtle, such as a malfunctioning php script of a user writing to the database in an infinite loop, thus filling the hard drive. To prevent one user from being able to fill the drive on error, we have a quota system. Unfortunately, by default the mysql binary files are not owned by the user in question, so a database was not protected by this system. This has now been fixed by chowning the database files to the user in question, but it still serves as a great example of the complexities and subtle nature of problems when you’re administring a server shared by many users.

A few realities:

  • Giving a user the ability to run custom php/ruby/perl/cgi scripts pretty much equals giving them shell access for the user running the script.
  • Control over what code runs on the server becomes more and more impossible as the amount of user accounts grows. Leading to reality:
  • Server becomes a hostile environment, and should be treated as such.

Ok, so what can be done to at least try to contain the situation a bit:

  • Visualization, chroot, or jails: Each user gets a virtual system. -> Not advisable because 1) This way you effectively manage n systems instead of just 1, where n is the number of users you have. This leads to maintenance nightmares. Instead of having to apply a patch or security update once, it has to be performed on all the virtual machines simultaniously. In addition the maximum capacity of the server would be greatly reduced due to the additional overhead of virtualization and extra disk-space needed for each self-contained system. It’s possible to pull this off if you build scripts for replicating updates across different jails or virtual machines and you have more time to implement these things and resources(money) for exponentially adding hardware as needed – but we don’t, so this is not a realistic option for us.
  • suphp :  Instead of running all php and cgi scripts as the www user, scripts run as the user who owns the file. This way it is possible to use plain unix permissions to prevent users from accessing eachother’s files or other system files that don’t have to be accessible. A valid uid range can be entered to prevent files owned by root to be run as root (that would be bad.) -> This approach only works as long as users assign proper permissions to their files. Many of them are not familiar with file permissions, let alone the various nuances in security problems of the things they install. If chmod 777 is the easyer way to accomplish something, some may probably just do that, defeating the purpose of suexec. Also, many many files still need to be shared between users to even be able to run php, rails, or cgi scripts. A compromised script is still only a local exploit away from gaining access to the full server.
  • MAC (mandatory access control). With mandatory access control, security policies can be set up for each application. Access is restricted at the kernel level to certain system calls etc. SELinux uses this approach. The TrustedBSD project brought this to the FreeBSD kernel. -> The downside here is that MAC is very time consuming to set up, and tends to lead to a very complex security setup. Arguably, a complex security setup is a security risk in it’s own, since it becomes harder to clearly oversee the big picture. Also, you’re still only a kernel exploit away from being pwned. (There is no real defense against kernel exploits other than keeping the kernel patched and up to date for known exploits and hope that nobody has an unpublished 0-day on hands.)

So, it is clear that each approach is not without it’s problems. As is usually is the case with computer security, the best approach is a layered model. (ie, combining several methods.), there is not one magical perfect solution.

The plan is to put each service exposed to the internet in a FreeBSD jail. (eg, a jail for MySQL, one for Apache, one for E-mail, etc… ). Then, inside the Apache jail, use suphp. This way there is a controlable number of systems to maintain, and users are still able to protect their files from other users. A compromise of a web script is contained within the apache jail, and will not necesarily compromise mysql or the e-mail services, for example. Perhaps later a MAC layer can be added, if I can figure out  a way to not make it overly complex. And all this has to happen with minimal downtime while the system is live.

I have already moved MySQL and Apache into jails. Bind DNS was already jailed before we went live.

Jails each need an IP address assigned with them. For the sake of taking advantage of the jails concept and virtual interfaces, I am not running all of the jails on the public interface (which would be a really bad idea in the case of MySQL to begin with.) – Instead each jail has it’s own virtual LAN ip. (eg: for apache, for mysql) – It is for this reason that I have contacted users to now use mysql.dcwing.com as MySQL server instead of localhost. Each jail having it’s own IP address is handy, for example, if you want to tcpdump (sniff) traffic to/from a specific service, or run stats on it, etc. It’s all nicely isolated. It also allows you to prevent net access on jails that don’t need it, and to prevent certain jails to have network connectivity to services that they shouldn’t have network connectivity to.

In order to redirect traffic from the internet to the public WAN interface, to the virtual LAN interface of the apache jail, I had to add some port forwarding rules to the firewall:

00001 fwd,80 tcp from any to dst-port 80
00001 fwd,443 tcp from any to dst-port 443

The only problem now is that there is no reliable way to redirect traffic from inside, say the apache jail, to the internet, other than using NAT.

add 2 divert natd ip from any to not

The above rule works great, internet access from inside jails works. However it seems to introduce a problem I haven’t quite been able to debug yet. Traffic from the internet to the server becomes very slow. For example when downloading a file, it starts at 3 KiB/sec and then gradually slows down to 0, until the connection stalls and dies. Clearly there is something going wrong in the firewall, and I haven’t quite figured out yet what. For this reason, internet access from inside php scripts is currently not working. (I have to leave the nat rule disabled to prevent the slowdown.)
All this will be a lot easier when FreeBSD8 is out. Jails will then be able to be assigned multiple IP addresses, so no NAT is required.
I could just go and apply the jails patch to enable this feature, but I’m puzzled by this NAT problem, and would prefer to figure it out instead of going for a quick fix.

Given the length of this post, now I realize why I haven’t been wanting to talk to people about what I do, it’s just too friggin’ much to explain :D

pixelstats trackingpixel

5 Responses to “Some notes on the dc member server, aka administering a server shared with many users”

  1. Alex says:

    Interesting post, though I wont pretend that I understand all of it. I guess I’ll have to read your web log on a regular basis to improve my knowledge. How about some more ascii art?
    You’re theme is tres funky by the way, makes me want to say “SHALL WE PLAY A GAME?” ;)

  2. Tom says:

    Nice ‘Web Log’. I like it – and am very interested in administration topics anyway. One other thing I am very interested in is Virtualization. Have you considered container type virtualization such as found in Solaris? Parallels offers a good version for Macs, Windows, & Linux called Virtuosso and they sponsor an open source project called OpenVZ. The key here is that one patch to the core system patches ALL virtualized systems. The main drawback (if you consider it one, since most people utilize more traditional virtualization schemes this way anyway) is that you can not run multiple different OS’s. Because it only creates a ‘container’ or a virtualized instance of the base OS it is running on, it allows for very high densities while maintaining patching manageability. Just a thought.

    • Tom says:

      BTW – that wasn’t a suggestion, but rather a question. I am curious if you did look into it why you decided against it.


      • John says:

        Of course we considered it! :)

        I think it was f0dder on IRC who suggested it before we even got the server. (don’t kill me if it was someone else :D)

        The answer, again is: simply because of scalability issues.

        You have to ask yourself, what are you virtualizing.
        For now, all our users are using the server to run websites. So, we would be virtualizing apache and their php/cgi/rails processes etc… This means that you’d have to virtualize an apache instance for each user.

        So, the problem isn’t as much the overhead of the virtual machines themselves. FreeBSD jails, much like OpenVZ are not an emulation but a security separation at the kernel level.

        The problem is that you would have to run an apache instance in each jail, AND an apache instance with mod_proxy on the main OS.
        Thus, the overhead is not in the virtualization but in the additional apache instances you run.

        The dc server is donation funded, and there is no telling how many more users we’ll get. I can’t imagine 800 virtual machines (and apache instances) running on this one rather low-end dedi box ;)

        As far as maintenance issues, what you describe of patches in the base system migrating to the vm’s, I can achieve with FreeBSD jails by mounting the base filesystem files via nullfs (so any change in the base filetree would automatically happen in the jails too).


Leave a Reply