%! % PS WEB SITE LOG 500 ACCUMULATING REPORTER % ============================================== % Copyright c 2003/2010 by Don Lancaster and Synergetics, Box 809, Thatcher, AZ, 85552 % (428) 428-4073 don@tinaja.com http://www.tinaja.com % Consulting services available per http://www.tinaja.com/info01.html (C:\\Documents and Settings\\Don\\Desktop\\Gonzo\\gonzo.ps) run % use internal gonzo /guru { gonzo begin ps.util.1 begin printerror nuisance begin} def guru % activate gonzo utilities %%%%%%%%% Utility - BUBBLE SORT %%%%%%%%%%%%%%%%%%%%%%%% % Sorts a defined array by the numeric value of its SECOND field in descending % order. Speed seems acceptable even without optimization. Minimal extra storage used. /swapbub { /bshold bsarray bspos get store % temporary stash for rollover bsarray bspos 1 add get % move next to previous bsarray exch bspos exch put bsarray bspos 1 add bshold put} store % move previous hold to next /bubblesort { /bsarray exch store bsarray length { % n trips max - can shorten later 0 1 bsarray length 2 sub {/bspos exch store % for all but last entry bsarray bspos get 1 get % compare present with next bsarray bspos 1 add get 1 get lt {swapbub} if } for % swap if next is bigger } repeat } store % repeat till done %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % makestring converts a stack top array into a string... /makestring {dup length string dup /NullEncode filter 3 -1 roll {1 index exch write} forall pop} def %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% /sourcefilenameprefix (C:\\Documents and Settings\\Don\\Desktop\\log_play\\) store % the path of the input file /listoflogstoprocess [ (u_ex100318.log) (u_ex100323.log) (u_ex100324.log) (u_ex100325.log) (u_ex100326.log) (u_ex100329.log) (u_ex100330.log) (u_ex100401.log) (u_ex100402.log) (u_ex100403.log) (u_ex100405.log) (u_ex100406.log) (u_ex100407.log) (u_ex100408.log) (u_ex100409.log) (u_ex100410.log) (u_ex100411.log) ] store % this is a modified log file reporter /addtothe404list { %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% modified logfile reporter starts here %%%%%%%%%%%%%%%% /sourcefilename sourcefilenameprefix shortsourcefilename mergestr store /workstring 20000 string def % Assumed log file format is preceeded by four # comment lines. Each live entry consists % of 16 space delimited printable ASCII strings in the following sequence... % date % time % s-ip % cs-method % cs-uri-stem % cs-uri-query % s-port % cs-username % c-ip % cs(User-Agent) % cs(Referer) % sc-status % sc-substatus % sc-win32-status % sc-bytes % cs-bytes % time taken % Dynamic array allocation is probably too slow, so a scanned size system is used here. % Present size should handle log files of 6 megs or less, depending on host. /arraysize 0 store % variable to be counted for data array lengths /definesubarrays { /datearray arraysize array store % date /timearray arraysize array store % time /s-iparray arraysize array store % s-ip /cs-method arraysize array store % cs method /cs-uri-stem arraysize array store % cs uri stem /cs-uri-query arraysize array store % cs-uri-query /s-port arraysize array store % s-port /cs-username arraysize array store % cs-username query /c-ip arraysize array store % c-ip query /cs-useragent arraysize array store % cs-useragent /cs-referer arraysize array store % cs-referer /sc-status arraysize array store % sc-status /sc-substatus arraysize array store % sc-substatus /sc-win32ss arraysize array store % sc-substatus /sc-bytes arraysize array store % sc-bytes /cs-bytes arraysize array store % cs-bytes /time-taken arraysize array store % time-taken } store %%%%%%%%% (A) Strip log file to individual arrays %%%%%%%%%%%%%%%% /linecount -1 store % gets bumped to zero on first pass /filelength 0 store /logfileerror { (----> ) print curline == (\n\n) print flush (Format error on line ) linecount % hang on bad format 20 string cvs (.\n\n) print flush formaterror } store % Note that PostScript string dereferencing is essential, as reuse of a string will % trash pointers in the arrays. (\nBeginning analysis of web log file ) sourcefilename mergestr (.) mergestr print flush % add header to log file % this should list files at beginning /splitintoarrays {curline ( ) search {exch pop % dump space dup length string cvs % dereference!!!! datearray exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! timearray exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! s-iparray exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! cs-method exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! cs-uri-stem exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! cs-uri-query exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! s-port exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! cs-username exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! c-ip exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! longer string possible here cs-useragent exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! longer string possible here cs-referer exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! sc-status exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! sc-substatus exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! sc-win32ss exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! sc-bytes exch linecount exch put ( ) search {exch pop % dump space dup length string cvs % dereference!!!! cs-bytes exch linecount exch put dup length string cvs % dereference!!!! time-taken exch linecount exch put % default remainder }{logfileerror} ifelse % cs-bytes }{logfileerror} ifelse % sc-bytes }{logfileerror} ifelse % sc-win32ss }{logfileerror} ifelse % sc-substatus }{logfileerror} ifelse % sc-status }{logfileerror} ifelse % cs-referrer }{logfileerror} ifelse % cs-useragemt }{logfileerror} ifelse % c-ip }{logfileerror} ifelse % cs-username }{logfileerror} ifelse % s-port }{logfileerror} ifelse % cs-uri-query }{logfileerror} ifelse % cs-uri-stem }{logfileerror} ifelse % cs-method }{logfileerror} ifelse % s-iparray }{logfileerror} ifelse % time }{logfileerror} ifelse % date } store % processline strips out short lines or those starting with a # comment delimiter /processline { /curline exch store curline length 5 gt { % skip very short lines as errors. curline 0 get (#)0 get ne % skip lines starting with # { % skip lines starting with # /linecount linecount 1 add store % save for later % linecount 1 add 2500 mod 0 eq {(Processed ) linecount 1 add 20 string cvs mergestr % ( log lines\n) mergestr print flush} if /filelength workfile fileposition store % save for later - posn only if valid splitintoarrays } if % commentskipper } if % shortline skipper } def % /findarraylength makes a preliminary pass thorugh the log file to determine the % number of live log lines that need actual processing. This avoids a fixed array % size definition or the overhead of dynamic array size allocation. /findarraylength {sourcefilename (r) file /workfile exch def % make a file to read {mark workfile workstring readline % read one line at a time {/curline0 exch store curline0 length 5 gt { % skip very short lines as errors. curline0 0 get (#)0 get ne { % skip lines starting with # /arraysize arraysize 1 add % one count per useful store } if % commentskipper } if % shortline skipper }{cleartomark exit} ifelse % test lines till done cleartomark % just in case sloppy } loop % arraysize == % optional check of array length definesubarrays } def % /striplogfile reads the logfile one line at a time and passes the results to % processline for array allocation. This is the main reading loop /striplogfile {sourcefilename (r) file /workfile exch def % make a file to read {mark workfile workstring readline % read one line at a time {processline}{cleartomark exit} ifelse % test lines till done cleartomark % just in case sloppy } loop } def % (\nStarting array size count..\n) print flush % stopwatchon findarraylength % stopwatchoff % (\nStarting array splitout..\n\n) print flush % stopwatchon striplogfile % this does it % stopwatchoff % (\nLogfile length was approximately ) filelength 68 add 20 string cvs mergestr % ( characters.\n\n) mergestr print flush % (Total hits for session are ) linecount 1 add 20 string cvs mergestr % (.\n) mergestr print flush %%%%%%%%% (I) FIND AND RANK 500 SERVER ERRORS %%%%%%%%%%%%%%%%%%%%%%%% % This feature finds all the 500 file not found errors and ranks them. % Similar procs can be applied to other error codes. % This differs from file reporters in that the 500 array position needs % discovered so that the actual error request can be separately read. % Note that your 500 error rate may be grossly inflated by piracy attempts. % Be sure to separate the inside correctable errors from the outside ones. % Scan sc-status for 500 errors... /e404list mark % start array 0 1 sc-status length 1 sub % look for 404 errors {/scposn exch store sc-status scposn get (500) eq { cs-uri-stem scposn get } if % find 404 requestor } for ] store % /e404hl [ [e404list 0 get 0 ] ] store % define array % e404hl == (\n\n\n\n\n) print flush % temp debug if uncommented % /stuffunique404 creates a new unsorted array of 404 file not found hits % each array element is of form [[(pagename) hitcount ][(nexpagename) hitcount ]] /stuffunique404 {/curpage exch store % save the current postscript util url /neednewentry true store % assume an update is needed 0 1 e404hl length 1 sub {/curpos exch store % scan the existing hit list e404hl curpos get 0 get curpage eq % already present? {e404hl curpos get 1 get 1 add % increment count e404hl curpos get exch 1 exch put /neednewentry false store }if % clear newentry flag } for neednewentry { mark e404hl aload pop % if new needed, put old on stack [ curpage 1 ] ] % and dynamically expand list /e404hl exch store } if } store e404list {stuffunique404 % expand array as forall loop } forall % e404hl == % optional check (File Not Found 500 errors by severity:\n\n) print flush %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% modified logrile reporter ends here %%%%%%%%%%%%%%%%% } store % this is the main high level routine /accumulate404s { (\n\n\n) print flush /e404hl [ [( temp marker ) 0 ] ] store % define 404 accumulated array listoflogstoprocess { /shortsourcefilename exch store addtothe404list } forall (\n\nAccumulated Server 500 errors by severity:\n\n) print flush e404hl bubblesort e404hl {( ) print ==} forall % indent list (\nTotal accumulated unique 500 server errors = ) e404hl length 20 string cvs mergestr (.\n) mergestr print flush (\n\n\n\n) print flush } store % demo -- remove or alter before reuse stopwatchon accumulate404s stopwatchoff % EOF