Your server got hacked by an attack like this ?

POST /cgi-bin/php4?%2D%64+%61%6C%6C%6F%77%5F%75%72%6C%5F%69%6E%63%6C%75%64%65%3D%6F%6E+%2D%64+%73%61%66%65%5F%6D%6F%64%65%3D%6F%66%66+%2D%64+%73%75%68%6F%73%69%6E%2E%73%69%6D%75%6C%61%74%69%6F%6E%3D%6F%6E+%2D%64+%64%69%73%61%62%6C%65%5F%66%75%6E%63%74%69%6F%6E%73%3D%22%22+%2D%64+%6F%70%65%6E%5F%62%61%73%65%64%69%72%3D%6E%6F%6E%65+%2D%64+%61%75%74%6F%5F%70%72%65%70%65%6E%64%5F%66%69%6C%65%3D%70%68%70%3A%2F%2F%69%6E%70%75%74+%2D%64+%63%67%69%2E%66%6F%72%63%65%5F%72%65%64%69%72%65%63%74%3D%30+%2D%64+%63%67%69%2E%72%65%64%69%72%65%63%74%5F%73%74%61%74%75%73%5F%65%6E%76%3D%30+%2D%6E HTTP/1.1"
This translates to:
-d allow_url_include=on -d safe_mode=off -d suhosin.simulation=on -d disable_functions="" -d open_basedir=none -d auto_prepend_file=php://input -d cgi.force_redirect=0 -d cgi.redirect_status_env=0 -n
Afterwards you will notice (or maybe not), that you server slowly sends spam. The process which is responsible for this is a perl script, which lives in /tmp and /dev/shm. It camouflages itself as as a legit process, like cron, apache, syslogd, klogd.
The attacker scans IP addresses, and if something is found which looks interesting he tries to execute /cgi-bin/php4 and some other variants, until he gets a 200 ok. Having done this, he is able to upload some files in /tmp and execute these with the limited rights of the webserver. But this is enough to abuse your mailserver and pollute the internet with spam and damaging your mailservers reputation.
Not all, but many php versions built as cgi have this bug. Unless you are able to fix your php or run an apache filter, you are out of luck.
Denying execution of programs in /tmp and /dev/shm is a good idea, but it wont stop someone to execute a command like "/usr/bin/perl /tmp/shz". What you really need to patch this hole is a wrapper for perl which drops execution if the script lives somwhere where anyone has write permissions. A regulary cleanup of /tmp might stop this attack for a short time, but is not a real solution.
Unfortuantely, on most systems you cant wrap up perl with a perl wrapper. The wrapper needs to be an executable. I have written this as a simple C Program, which can be used as an example wrapper:

Rename your perl binary /usr/bin/perl to something else like "perl.50009.random", Copy the code, customize the #include statements to match your system:
replace: "#define PERLTOEXECUTE "/usr/local/bin/perl.real.5.0001" with the path and executable to your real perl.
replace: "#define MYEXECNAME "perl" with the name the compiled executable will have if it is not "perl".
build it with "gcc perl.c -o perl".
Copy the perl wrapper to /usr/bin.

Create a logfile "/var/log/hacklog.log" in /var/log/ and make it world writable with "chmod a+w /var/log/hacklog.log" (otherwise, the program will fail to write logs if it is executed from a process which is not root).
Test the wrapper by executing "perl /tmp/test.pl" and "perl /somedir/test.pl", and look in the logfile. You should see something like:
2014-01-26 10:39 bad /dev/shm/test.pl
2014-01-26 11:16 good /root/test.pl
Code of the wrapper:

#include <libgen.h>
#include <unistd.h>
#include <stdio.h>
#include <time.h>
#include <string.h>
#include <stdlib.h>


#define MYEXECNAME "perl"
#define PERLTOEXECUTE "/usr/local/bin/perl.real.5.0001"
#define DEBUUG 0

int main (int argc, char *argv[]) {

int i=0;
char mytime[128];
time_t rawtime;
struct tm * timeinfo;
char *myname;
char *mybasename;    
char *argname_d;
char *argname_p;
char *argpathname;
char *argbasename;
char *bad1="/tmp";
char *bad2="/dev/shm";
char *logfilename="/var/log/hacklog.log";
FILE *logfile;
 
time (&rawtime);
timeinfo = localtime (&rawtime);
strftime(mytime, 127, "%F %R",timeinfo);

#ifdef DEBUG
printf("Time: %s\n", mytime);
printf("\ncmdline args count=%d", argc);
printf("\nexe name=%s", argv[0]);
#endif

myname = strdup(argv[0]);
if ( myname == NULL ) {
 printf("cannot allocate memory for string duplication, exit\n");
 exit(-1);
}
                          
mybasename = basename(myname);

#ifdef DEBUG
printf("basename: %s\n", mybasename );
#endif


logfile = fopen(logfilename,"a");
if ( logfile == NULL) {
 printf("could not open logfile %s for writing\n",logfile);
 exit(-1);

}

 if ( strcmp (mybasename, MYEXECNAME ) == 0)
{

       #ifdef DEBUG
       printf("I was called as perl. ok, to continue\n");
       #endif
       
       for ( i=1; i<argc; i++ ) {
        if ( i > 32766 ) {
         printf("number of arguments exceeded int size, terminating\n");
         exit(-1);
        }
        argname_d = strdup(argv[i]);
        if ( argname_d == NULL ) {
         printf("cannot allocate memory for string duplication, exit\n");
         exit(-1);
        }
        argname_p = strdup(argv[i]);
        if ( argname_p == NULL ) {
         printf("cannot allocate memory for string duplication, exit\n");
         exit(-1);
        }
        argpathname = dirname(argname_d);
        argbasename = basename(argname_p); 
#ifdef DEBUG
        printf("%s %s %s\n", argv[i], argpathname, argbasename);
#endif
        if ( strcmp( argpathname, bad1) == 0 || strcmp( argpathname, bad2) == 0 ) {
         fprintf(logfile,"%s bad %s\n",mytime, argv[i]);
         exit(-1);
        }
        else {
         fprintf(logfile, "%s good %s\n",mytime, argv[i]);
        }
        free(argname_d);
        free(argname_p);

       }
       fclose(logfile);
       execv(PERLTOEXECUTE,argv);
       
}
else {
       printf("I was called as a different program as expected: %s, terminating\n", mybasename);
       exit(-1);
}


}


you may also download it as a tar.gz file from here. Works great for me. Also easily customizable to stop similar attacks which might be done using other script langugages like python or bash.

comments ? Send them to "safeperl AT biermann dot org".