Obsolete:Ms4
Ms4 is a Sun Fire X4540 running Solaris 10.
Ms4 is our primary server for thumbnails.
Web server set up
install
We are running the Sun Java Web Server 7. To get this:
- Make sure you have a login at sunsolve (I wonder if you need this); you can create it for free.
- One the left where it says "Change Contract" click that, and add our contract number. You are supposed to log out and log back in for it to take effect.
- Go to Oracle and download the software. We just need the plain web server, not all the rest. To do this, click through to the point where you actually click to download; when the download starts, interrupt it, right click on the link in the page to copy it and then use wget on ms4 to retrieve the file. (Uploading it form the office is going to be too slow.) The session token is only good for a few minutes.
- Install info: [1]. In a nutshell: unpack the tarball someplace convenient for you, cd into the directory and run ./setup --console
- The install root will be /opt/oracle/webserver7, hostname ms4, port 80, runtime id apache, /export/thumbs for the document root. Make sure you install all of parts 1,2,3,4 (you need them all so you can build and run dtrace later). On new hosts we do a 64-bit build.
config files
If you need to go fool with the configuration later, the file you want is server.xml and it's in /opt/webserver7/https-ms4/config.
If you wind up copying the /opt/webserver7/https-*/config files to another host, you will have to move the directory to reflect the new hostname and then edit all of the scripts under /opt/webserver7/https-*/bin to change the hostname as well. If you're not using /export/thumbs on the new host, edit that too. Also, edit /opt/webserver7/https-*/config/server.xml to change the hostname there. You may also need to tweak /opt/webserver7/https-*/obj.conf to change directory paths and so on.
Copies of the config files are on /home/wikipedia/conf/webserver7 in the directory with the hostname. (It would be nice to have a repo!)
You need to manually create the directory /var/log/http/ (where errors go); also chown and chgrp it to apache.
startup with legacy scripts
On ms4 the following is true.
If you need the web server to start on boot and you didn't specify it at install time, you're going to go looking for /etc/init.d/webserver7 and it won't be there. Steal one from another Sun box running Solaris (if anyone knows the right way, please add it here :-P) Then cd into /etc/rc2.d and
ln -s ../init.d/webserver7 K05webserver7
and cd into /etc/rc3.d and
ln -s ../init.d/webserver7 S75webserver7
The new services management system under Solaris 10 uses svcs but this is a legacy app so it uses scripts in init.d. You should have a look at the scripts bin/startserv and bin/stopserv because they may, depending on how the webserver was installed, contain a lot of cruft about SERVER_FMRI, SERVER_SVC_START_CMD, SERVER_SVC_STOP_CMD without there actually being an svc instance for the webserver. Run svcs -a to see if there is an instance (probably not). You can toss that junk. If you don't, the script will claim to succeed but actually no webserver will start.
(If someone has the info on how to add the legacy scripts to svcs... and whether we want that... please add it here. It seems to want an xml file in /var/svc/manifest, couldn't find one to steal from; how is it running on ms1??)
startup without legacy scripts
On ms5 the web server is started using:
svcadm enable svc:/network/http:https-ms5
To stop it do
svcadm disable svc:/network/http:https-ms5
To find out its status do
svcs -l svc:/network/http:https-ms5
This will also tell you the location of the error log if it fails to start up.
The manifest file of instances is in /var/svc/manifest/network/http.xml
If you change this file you either need to reboot (to get it to be active) or, much better, to
svccfg -s network/http:https-ms5
and at the prompt say
validate http.xml
(to make sure there are no typos or other gotchas) and then
import http.xml
You can give the command
exit
to leave the services configuration manager.
other files
You need the /opt/webserver7/https-ms4/cgi-bin directory (it contains oggz-chop). This in turn needs /opt/ts/lib/libogg* which can be installed from the TS repo (audio/TSlibogg.spec) and /opt/ogzz/lib/liboggz* (don't know where we build this from, I steal it off of other hosts).
If you want to be able to look at the access logs live (hint: yes, you do!), you need some dtrace stuff: go to /home/wikipedia/conf/sunwebserver/optlocalshare and put the files over to /opt/local/share on the thumb server. Also put dtrace.so from /home/wikipedia/conf/sunwebserver/optwebserver7/plugins and put it into /opt/webserver7/plugins on the thumb server.
You need to have a place for the thumbs; assuming we are running Solaris, and that we put them in /export/thumbs, you need:
- zpool create export raidz1 /dev/......
- creates the top level zfs pool (see Zfs for more information).
- zfs create export/thumbs
- creates the zfs filesystem where the thumbs and the scripts will live
The filesystem needs to be exported everywhere:
- zfs set sharenfs=sec=sys,rw=@10.0.0.0/8:@208.80.152.0/22,root=@10.0.0.0/8:@208.80.152.0/22 export/thumbs
It needs to be mounted everywhere (apaches, scalers, anywhere else?):
- prestuff 'list' with apaches, scalers, zwinger, whatever else needs it mounted, then
- dsh -f -w $list -e 'echo "ms4.wikimedia.org:/export/thumbs /mnt/thumbs nfs bg,soft,tcp,rsize=8192,wsize=8192,timeo=14 0 0" >> /etc/fstab'
For snapshots, replication, and some metrics you need the files in /home/wikipedia/conf/zfs-tools/; scp over the ones in opttsbin to /opt/ts/bin/ and in optlocalbin to /opt/local/bin. Then look at one of the crontabs on the /home/wikipedia/conf/zfs-tools/crontab directory and ad what you need to the crontab on the thumb host. (Please make a subdir for it and scp it back over to /home/wikipedia/conf/zfs-tools/crontab/server-name if this is a new host.)
We need rsync to be in a standard place, right now it/s in /opt/ts/bin, so symlink it into /opt/local/bin and /usr/bin on your host.
You need to make the directory sync-from-home in /export/thumbs, and you should symlink it to "scripts" in that same directory. Then push over the contents: from /home/wikipedia/upload-scripts check the sync script, add your server to it if needed, and run (hey, it might be nice if it could sync to just one server, huh? Maybe later.)
Additional things you need/want:
- hdtool
This is in the sun package SUNWhd and you get it by grabbing the cd of firmware and tools for the X4500 or X4540 or whatever yourmodel is. The X4540 software is here [2] and the X4500 is here [3] (needs a login with a Sun contract attached to it). When you get to the point where you are downloading in Firefox, you can stop the download, go to your host and wget there; you'll need to do wget -O filename url because otherwise it will complain that the filename is too long. Once retrieved, install it with pkgadd -d ./SUNWhd-version-number-here.pkg It installs in /usr/bin/hd with doc in /opt/SUNWhd/hd/bin/hd.html
- php
You get this from the toolserver repo. Check out the spec files (you need a bunch so just get them all):
svn co https://svn.toolserver.org/svnroot/toolserver/trunk/ts-specs
You need to build packages as a non root user. To do this make sure you have a *local* user defined, with home directory /export/home/username-here, and then add the following to /etc/user_attr:
username-here::::type=normal;profiles=Primary Administrator
Now you can do the build as that user by going into the ts-repo/ts-specs directory and running:
pkgtool -v build-only --autodeps --download www/TSphp52.spec www/TScurl security/TSlibmcrypt.spec devel/TSautoconf devel/TSautomake libs/TSlibtool
Do the install of packages that get built (some need to be installed before the rest will build) (as root) by:
pkgadd -d /home/my-username-here/packages/PKGS TSpackagename-here
NOTE: check that /opt/webserver7/https-nnn/obj.conf has the right path to php-cgi. It may need to be /opt/ts/php/5.2/php-cgi or /opt/php/bin/php-cgi depending on which repo build you have.
Monitoring: You need ganglia. It's in the toolserver repo. You need to edit the heck out of the spec file: get rid of all references to librrd and gmetada, or it will try to install X as well as all its libraries, and fail horribly. Then...
pkgtool --download build-only libs/TSlibconfuse.spec pkgadd -d /home/ariel/packages/PKGS TSlibconfuse0 TSlibconfuse-dev pkgtool --download build-only graphics/TSganglia.spec pkgadd -d /home/ariel/packages/PKGS TSganglia TSganglia312
(Please let me verifiy these commands tomorrow when I get back in.)
changing config files
If you wind up editing the config files on a running instance of the webserver, you don't kill -HUP to get it to reread the config files. Instead, you run /opt/webserver7/https-servername/bin/reconfig to force the reread.
This doesn't work in all cases; see [4] (primarily if you touch certain things in the server.xml file). In this case you will have to /etc/init.d/webserver7 restart (hard stop and restart of the server), for ms4 or hosts that use legacy startup. On ms5 or hosts that use svcadm for startup, you can run wadm (found in /opt/oracle/webserver7/bin/) and at the prompt tell it pull-config --user=admin --config=ms5 ms5 (for example).
More info:
- [5]: Sun FAQ on installing