php-Blogger

nginx tips: adding arbitrary output to a response

A while ago i wrote about how to use nginx as a proxy to do cookie based redirects. We use this functionality at work to provide easy access and view the progress of development of each developer.

I thought it would be nice to have the information, of which developers machine you currently have access to, right on the website. But i always disliked to put functionality to accomplish this inside the framework or even the app itself or have to install / modify some special configuration on each developers machine. I always wanted my proxy to do this kind of work. ... And nginx can.

It's the substitution module of nginx, that can replace arbitrary text in a http response. Nginx must be compiled with the option --with-http_sub_module configured.

The following rows show how to fill a variable $name with the name of the developer we are accessing the machine of. The statement sub_filter defines the search pattern as first parameter and the replace string as second parameter -- very easy, isn't it?


set $name ""
if ($http_cookie ~* "(; )?devredirect=([^;]+)") {
    set $name $2;
}
sub_filter      "</body>"       "<div style='position: fixed; 
    left: 0; top: 0; font-weight: bold; padding: 5px; color: #000; 
    background-color: rgb(235,58,0);'>devredirect: ${name}</div>
    </body>";

The following rows show the part of the proxy configuration with the inserted substitution filter:


server {
    listen          80;
    server_name     .clipdealer.devcenter.int;

    proxy_redirect          off;
    proxy_set_header        Host            $host;
    proxy_set_header        X-Real-IP       $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;

    location / {
        set $name ""
        if ($http_cookie ~* "(; )?devredirect=([^;]+)") {
            set $name $2;
        }
        sub_filter      "</body>"       "<div style='position: fixed; 
            left: 0; top: 0; font-weight: bold; padding: 5px; 
            color: #000; background-color: rgb(235,58,0);'>
            devredirect: ${name}</div></body>";

        if ($http_cookie ~* "(; )?devredirect=harald") {
            proxy_pass              http://10.0.0.20;
            break;
        }
        ...
    }
    ....
}

OSX 10.5.x + PHP + pecl_newt

I always wanted to provide nice user interfaces for some of my commandline tools written in PHP, but was not able to solve one problem until recently. The big problem when writing user interfaces for commandline utilities written in PHP is: ... what library to use for actually building the user interface?

You could try to write your own library using ANSI escape sequences -- but your user interfaces would either be very limited or it would be a hell of work to write an extensive library providing more than just the basics.
You could try to get the ncurses extension to work -- i failed.
You could try to get the newt extension to work -- i failed ...

... until recently. Every once in a while i tried to dig up more information of how to get newt to work, because this is the library i would prefer over the other solutions. However, it did not compile on my system. Recently i searched again and was able to dig up a patch for newt-0.5.22.11. So, here are the steps to get things work:

Download newt 0.52.11 and extract it.
Download the patch for newt from the page above and apply it as described in the short tutorial provided on that page.
Download slang, which is required by newt. I decided to download the latest snapshot (pre2.2.3-60) and things worked just fine with it. Extract and build it -- i did not use any special flags for this.
Download popt, which is required for newt. I decided to download popt-1.16, which seems to be the latest release and can be found at the bottom of the download page. Extract and build it -- i did not use any special flags for this.
After installing slang and popt and patching newt, you should now be able to build and install newt.
Download pecl_newt, or install it using pecl install .... Don't forget to add newt.so to your php.ini.

pecl_newt provides some examples which should be executed to verify, that installation succeeded.

OSX 10.5.x + PHP + pecl_ncurses

After several days playing around with newt i am quite disappointed to come to a point where i have to say, that newt has not only a very limited documentation (i think there's only documentation of about 50% of the newt functionality) -- which would not be that of a problem, though. The bigger problem is, that things are very unstable. "Segmentation faults" reproducable on different operating systems and platforms -- not sure though, if the problem is newt or pecl_newt. However: this is definitly not what i want. So, let's move on to something better.

I had a look at the apple's opensource repository the other day and stumbled over a ncurses package. I had problems with the normal gnu ncurses package before, it would not install but hang when building the terminfo database. With the ncurses package i found at apple's opensource repository, i get it to install. Here is the howto:

Download ncurses from apple's opensource repository. You can find it by clicking on the link of the Mac OSX version you are running, eg. 10.5.8.
Unpack the package und you get a directory called something like "ncurses-21" or so. Next cd into this direktory and run make and sudo make install. If make complains about a missing directory /tmp/ncurses/Build, just create it mkdir -p /tmp/ncurses/Build and start again.
Now an intermediate build should have been generated. Next cd /tmp/ncurses/Build, make and sudo make install. Now ncurses should have been successfully installed.
Ready to install the ncurses php extension by either executing sudo pecl install ncurses or downloading ncurses from pecl.php.net and building manually.
Don't forget to add ncurses.so to your php.ini

Now everything should be ready to start writing ncurses apps -- however: for me it sill did not work. The simplest ncurses app resultet in the following error message:

Error opening terminal: vt100.

What's going on? dtruss might be your friend to figure this out. Execute the following command, where example.php is your ncurses app you want to run: sudo dtruss php example.php. You should see output like:

[...]
lstat("/Users/harald\0", 0xBFFFE41C, 0xFFFFFFFFBFFFEFC8)                 = 0 0
lstat("/Users\0", 0xBFFFE2EC, 0xFFFFFFFFBFFFEFC8)                = 0 0
ioctl(0x3, 0x4004667A, 0xBFFFF17C)               = -1 Err#25
ioctl(0x3, 0x402C7413, 0xBFFFF150)               = -1 Err#25
fstat(0x3, 0xBFFFF1B0, 0xBFFFF150)               = 0 0
mmap(0x0, 0x226, 0x1, 0x2, 0x3, 0x100000000)             = 0x17FA000 0
lseek(0x3, 0x0, 0x1)             = 518 0
munmap(0x17FA000, 0x1F3)                 = 0 0
close_nocancel(0x3)              = 0 0
ioctl(0x1, 0x4004667A, 0xBFFFECEC)               = 0 0
stat("/Users/harald/.terminfo\0", 0xBFFFEC00, 0xBFFFECEC)                = 0 0
access("/Users/harald/.terminfo/73/vt100\0", 0x4, 0xBFFFECEC)              = -1 Err#2
stat("/usr/local/share/terminfo\0", 0xBFFFEC00, 0xBFFFECEC)              = -1 Err#2
write_nocancel(0x2, "Error opening terminal: vt100.\n\0", 0x24)            = 36 0

You can see, that ncurses is looking for a file vt100 in a terminfo directory and fails looking at /usr/local/share/terminfo. I indeed had no directory /usr/local/share/terminfo but a directory /usr/share/terminfo. I was not able to figure out, where i could specify the correct directory configuration for this, so i just created a symlink: sudo ln -snf /usr/share/terminfo /usr/local/share/terminfo. After creating the symlink, i was able to execute my example ncurses application.

proftpd + mod_sql: solving "slow login" problem

I had a very annoying problem with proftpd, which seems a common one at first sight: slow login and the problem, that a lot of ftp clients out there have a low timeout setting configured. The problem is that googling "slow connection" or "slow login" in combination with "proftpd" led me in a totally wrong direction. A lot of people seem to have a problem with DNS lookups, which can be easily fixed by adding ...

UseReverseDNS off
IdentLookups  off

... to the configuration file, to turn of any DNS lookups. But this did not change anything for me. Running a ftp client in debug mode it turned out, that the authorization itself took a very long time, which led to a timeout with most ftp clients:

air:~ harald$ ftp -d ftp.xxxxxxxxxx.de
Connected to ftp.xxxxxxxxxx.de.
220 xxxxxxxxxx FTP Server
ftp_login: user `' pass `' host `ftp.xxxxxxxxxx.de'
Name (ftp.xxxxxxxxxx.de:harald): 
---> USER harald
331 Password required for harald
Password: 
---> PASS XXXX 
...

The password was send, and than the ftp client had to wait 10 seconds and longer for a respone. Lot's of ftp clients have a timeout of less than 10 seconds, which results in a timed out connection for such a long response time.

After googling for quite some time without finding anything useful on this topic -- besides the DNS lookup problem -- i delved deeper into to the proftpd documentation and found a howto which gave me some hints of how to speed up ftp login.

As it turned out the problem was my SQLAuthenticate directive, which i just copied from the example configuration file of mod_sql. The configuration was set to:

SQLAuthenticate users userset

The problem with this configuration is, that the userset switch seems to be very, very expensive. I still don't know, why this switch is set in the configuration -- the documentation contains no useful examples of when to use / when to avoid this switch, but eventually i found a forum post of a proftpd maintainer, where he tells, that the userset switch is not necessary to be configured. After changing above configuration to ...

SQLAuthenticate users

... login is fast as hell. I'm still curious why the switch was there ...