%!ps % ====== % LOG_AN1.PSL Postscript demo to generate web log arrays for further analysis. % ====== % Copyright c July 2018 and earlier by Don Lancaster and Synergetics, 3860 % West First Street Box 809 Thatcher AZ. www.tinaja.com (928-428.4073 % Additional support at https://www.tinaja.com . % Personal use permitted. All other rights fully reserved. % Requires an input status report. Such as moo.tinajacom % derived from ???date.gz in the fatcow stats folder and % decompressed via https://www.winzip.com or another .gs reader. % fatcow is reachable at % https://www.fatcow.com/fatcow/special-promo.bml?LinkName=tinaja % Also requires Distiller from Acrobat and Gonzo Utilities from % https://www.tinaja.com/gonzo.psl Must be edited to match resources. % IMPORTANT NOTE: This file DEMANDS that Distiller has its disk file reading % activated! From run accessing the command line, enter //acrodist /F . % Additional PostScript support via https://www.tinaja.com/glib/gonzotut.pdf % https://www.tinaja.com/pssamp1.shtml https://www.adobe.com/content/dam % /acom /en/devnet/actionscript/articles/PLRM.pdf , and % https://www.youtube.com/watch?v=C_tWW560tAE % ======= % Uncomment and modify to use disk based Gonzo. ( or other disk access ) (C:/Users/Don/Desktop/Ghost/gonzo.psl) run % use internal /guru { gonzo begin ps.util.1 begin printerror nuisance begin} def guru % activate gonzo utilities % ============================== % a bubble sort for [[(..)n1] [(..)n2] [(..)n1] [(..)n3] ] ... /popbubblesort2 {/curmat1 exch store curmat1 length 1 sub -1 1 {curmat1 0 get exch 1 exch 1 exch {/posn exch store curmat1 posn get 2 copy 1 get exch 1 get lt {exch} if curmat1 exch posn 1 sub exch put} for curmat1 exch posn exch put } for curmat1 } bind store % attempt at fast bubble... /popbubblesort2 {/curmat1 exch store curmat1 length 1 sub -1 1 { /done true store % short exit marker /maxposition exch store 0 1 maxposition 1 sub { /posn exch store % inner loop curmat1 posn get 1 get curmat1 posn 1 add get 1 get lt { curmat1 posn get % swap here curmat1 posn 1 add get curmat1 exch posn exch put curmat1 exch posn 1 add exch put /done false store }if } for done {exit} if } for % outer loop curmat1 % justreplaced } bind store %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % faster mergestr with bind /mergestr {2 copy length exch length add string dup dup 4 3 roll 4 index length exch putinterval 3 1 roll exch 0 exch putinterval} bind def % initial timing check here??? stopwatchon % start timing for shortarray %%%%%%%%%%%%%% build compact array %%%%%%%%%%%% /fileprefix (C:/Users/don/Desktop/web/) store /infilename (moo.tinajacom) store fileprefix infilename mergestr (r) file /moofile exch store (\n\n/An analysis of chosen web log )infilename mergestr (:) mergestr print /filename (dummy) store /stats ( 40409 ) store /getfilename1 {pop (") search { pop pop getfilename2}{no_quote}ifelse } store /getfilename2 { ( ) search {pop pop getfilename3}{no_space} ifelse } store /getfilename3 { ( ) search { dup length 55 gt { 0 50 getinterval (...) mergestr } if % shorten problem lines /filename exch store pop getstatus1}{no_space_again}ifelse } store /getstatus1 {(") search {pop pop 1 3 getinterval /stats exch store}{no_quote}ifelse } store /extractlinedata { ( ) search { /user exch store getfilename1}{no_spaces!} ifelse mark % curline user filename stats ] % build array }store % moofile line extractor -2 vmreclaim % stop garbage collection? mark 0 1 65000 {/curline exch store % extract moofile lines (shorten for debug!) moofile 10000 string readline { extractlinedata }{exit} ifelse } for ] /shortarray exch store % remove possible last entry ( ) glitch mark shortarray aload pop pop ] /shortarray exch store 0 vmreclaim (\n\n Time to create shortarray ) print stopwatchoff %%%%%%%%%%% addtoarray procs %%%%%%% % user main list /addtoarraylist { mark arrayusers curpos get 2 get % builds user list aload pop xx aload pop] arrayusers curpos get exch 2 exch put} store /addtoarrayusers { /newusers true store % initially assume a different hit /newusershit exch store % save hit to be added 0 1 ausersmax 1 sub {/curpos exch store % get curpos entry arrayusers curpos get 0 get newusershit eq { arrayusers curpos get 1 get 1 add arrayusers curpos get exch 1 exch put addtoarraylist /newusers false store exit } if} for newusers { % only if really new arrayusers ausersmax [ newusershit 1 xx ] put /ausersmax ausersmax 1 add store } if } store %%%%%%%%% % add to array 200 /addtoarray200 { /new200 true store % initially assume a different hit /new200hit exch store % save hit to be added new200hit length 65 gt {new200hit 0 60 getinterval (...) mergestr /new200hit exch store } if % truncate excess 0 1 a200max 1 sub {/curpos exch store % get curpos entry array200 curpos get 0 get new200hit eq { array200 curpos get 1 get 1 add array200 curpos get exch 1 exch put /new200 false store } if } for new200 { % only if really new array200 a200max [ new200hit 1 ] put /a200max a200max 1 add store } if } store %%%%%%%%%% % add to array 206 /addtoarray206 { /new206 true store % initially assume a different hit /new206hit exch store % save hit to be added new206hit length 65 gt {new206hit 0 60 getinterval (...) mergestr /new206hit exch store } if % truncate excess 0 1 a206max 1 sub {/curpos exch store % get curpos entry array206 curpos get 0 get new206hit eq { array206 curpos get 1 get 1 add array206 curpos get exch 1 exch put /new206 false store } if } for new206 { % only if really new array206 a206max [ new206hit 1 ] put /a206max a206max 1 add store } if } store %%%%%%%%%% % add to array 301 /addtoarray301 { /new301 true store % initially assume a different hit /new301hit exch store % save hit to be added new301hit length 65 gt {new301hit 0 60 getinterval (...) mergestr /new301hit exch store } if % truncate excess 0 1 a301max 1 sub {/curpos exch store % get curpos entry array301 curpos get 0 get new301hit eq { array301 curpos get 1 get 1 add array301 curpos get exch 1 exch put /new301 false store } if } for new301 { % only if really new array301 a301max [ new301hit 1 ] put /a301max a301max 1 add store } if } store %%%%%%%%%% % add to array 404 /addtoarray404 { /new404 true store % initially assume a different hit /new404hit exch store % save hit to be added new404hit length 65 gt {new404hit 0 60 getinterval (...) mergestr /new404hit exch store } if % truncate excess 0 1 a404max 1 sub {/curpos exch store % get curpos entry array404 curpos get 0 get new404hit eq { array404 curpos get 1 get 1 add array404 curpos get exch 1 exch put /new404 false store } if } for new404 { % only if really new array404 a404max [ new404hit 1 ] put /a404max a404max 1 add store } if % (\n\n) print array404 0 a404max getinterval {==} forall } store %%%%%%%%%% % add to array 406 /addtoarray406 { /new406 true store % initially assume a different hit /new406hit exch store % save hit to be added new406hit length 65 gt {new404hit 0 60 getinterval (...) mergestr /new406hit exch store } if % truncate excess 0 1 a406max 1 sub {/curpos exch store % get curpos entry array406 curpos get 0 get new406hit eq { array406 curpos get 1 get 1 add array406 curpos get exch 1 exch put /new406 false store } if } for new406 { % only if really new array406 a406max [ new406hit 1 ] put /a406max a406max 1 add store } if % (\n\n) print array406 0 a406max getinterval {==} forall } store %%%%%%%%%% % add to array 407 /addtoarray407 { /new407 true store % initially assume a different hit /new407hit exch store % save hit to be added new407hit length 65 gt {new407hit 0 60 getinterval (...) mergestr /new407hit exch store } if % truncate excess 0 1 a407max 1 sub {/curpos exch store % get curpos entry array407 curpos get 0 get new407hit eq { array407 curpos get 1 get 1 add array407 curpos get exch 1 exch put /new407 false store } if } for new407 { % only if really new array407 a407max [ new407hit 1 ] put /a407max a407max 1 add store } if % (\n\n) print array407 0 a407max getinterval {==} forall } store %%%%%%%%%%%% find status values and percentages %%%%%%%%%%% % find user hits /findhitusers { -2 vmreclaim /arrayusers 10000 array store % create oversize null array arrayusers 0 [() 0 ] put % make dummy zeroth entry /ausersmax 0 store % point to next available cell shortarray {/curval exch store curval 0 get /xx mark curval 1 get ( ) mergestr curval 2 get mergestr ] store % save the url for below addtoarrayusers } forall arrayusers 0 ausersmax getinterval % truncate to valid popbubblesort2 /arrayusers exch store % and sort on count 2 vmreclaim } store /length200 {0 arrayusers {1 get add} forall} store /unique200 {arrayusers length} store stopwatchon % time findhitusers findhitusers (\n Time to create findhitusers ) print stopwatchoff %%% % find 200 hits /findhit200 { /array200 10000 array store % create oversize null array array200 0 [() 0 ] put % make dummy zeroth entry /a200max 0 store % point to next available cell shortarray {/curval exch store curval 2 get (200) eq {curval 1 get addtoarray200} if } forall array200 0 a200max getinterval % truncate to valid popbubblesort2 /array200 exch store % and sort on count } store /length200 {0 array200 {1 get add} forall} store /unique200 {array200 length} store stopwatchon % time findhit200 findhit200 (\n Time to create findhit200 ) print stopwatchoff %%%%%%%%%%% % find 206 hits /findhit206 { /array206 10000 array store % create oversize null array array206 0 [() 0 ] put % make dummy zeroth entry /a206max 0 store % point to next available cell shortarray {/curval exch store curval 2 get (206) eq {curval 1 get addtoarray206} if } forall array206 0 a206max getinterval % truncate to valid popbubblesort2 /array206 exch store % and sort on count } store /length206 {0 array206 {1 get add} forall} store /unique206 {array206 length} store stopwatchon % time findhit206 findhit206 (\n Time to create findhit206 ) print stopwatchoff %%%%%%%%%%% % find 301 hits /findhit301 { /array301 10000 array store % create oversize null array array301 0 [() 0 ] put % make dummy zeroth entry /a301max 0 store % point to next available cell shortarray {/curval exch store curval 2 get (301) eq {curval 1 get addtoarray301} if } forall array301 0 a301max getinterval % truncate to valid popbubblesort2 /array301 exch store % and sort on count } store /length301 {0 array301 {1 get add} forall} store /unique301 {array301 length} store stopwatchon % time findhit301 findhit301 (\n Time to create findhit301 ) print stopwatchoff %%%%%%%%%%%% % find 404 hits /findhit404 { /array404 10000 array store % create oversize null array array404 0 [() 0 ] put % make dummy zeroth entry /a404max 0 store % point to next available cell shortarray {/curval exch store curval 2 get (404) eq {curval 1 get addtoarray404} if } forall array404 0 a404max getinterval % truncate to valid popbubblesort2 /array404 exch store % and sort on count } store /length404 {0 array404 {1 get add} forall} store /unique404 {array404 length} store stopwatchon % time findhit404 findhit404 (\n Time to create findhit404 ) print stopwatchoff %%%%%%%%%%%% % find 406 hits /findhit406 { /array406 10000 array store % create oversize null array array406 0 [() 0 ] put % make dummy zeroth entry /a406max 0 store % point to next available cell shortarray {/curval exch store curval 2 get (406) eq {curval 1 get addtoarray406} if } forall array406 0 a406max getinterval % truncate to valid popbubblesort2 /array406 exch store % and sort on count } store /length406 {0 array406 {1 get add} forall} store /unique406 {array406 length} store stopwatchon % time findhit406 findhit406 (\n Time to create findhit406 ) print stopwatchoff %%%%%%%%%%%% % find 407 hits /findhit407 { /array407 10000 array store % create oversize null array array407 0 [() 0 ] put % make dummy zeroth entry /a407max 0 store % point to next available cell shortarray {/curval exch store curval 2 get (407) eq {curval 1 get addtoarray407} if } forall array407 0 a407max getinterval % truncate to valid popbubblesort2 /array407 exch store % and sort on count } store /length407 {0 array407 {1 get add} forall} store /unique407 {array407 length} store stopwatchon % time findhit407 findhit407 (\n Time to create findhit407 ) print stopwatchoff /findhit407 {0 shortarray {2 get (407) eq {1 add } if } forall /hit407 exch store } store findhit407 (\n\nThere are a total of ) shortarray length dup /hits exch store 10 string cvs mergestr ( hits in this version of ) mergestr infilename mergestr (.\n) mergestr print %%%%%%%%%%%%%%%%%% % begin status reporting /statreport { % {==} forall { dup 0 get ( ) mergestr exch 1 get 5 string cvs mergestr (\n) mergestr print } forall } bind store %%%%%%%%%%%%%%%%%%%%%%%%%% (\n Status (200) OK accounts for ) length200 10 string cvs mergestr ( hits or ) mergestr length200 hits div 100 mul 100 mul cvi 100 div 20 string cvs mergestr ( percent.\n Of these, there are ) mergestr array200 length 20 string cvs mergestr ( unique url's.\n \n) mergestr print array200 popbubblesort2 /array200 exch store % put in high numeric order stopwatchon % report array200 array200 statreport % new reporter (\n Time to report array200 ) print stopwatchoff %%%%%%%%%%%%% (\n\n Status (206) PARTIAL CONTENT accounts for ) length206 10 string cvs mergestr ( hits or ) mergestr length206 hits div 100 mul 100 mul cvi 100 div 10 string cvs mergestr ( percent. \n Of these, there are ) mergestr array206 length 20 string cvs mergestr ( unique url's.\n \n) mergestr print array206 popbubblesort2 /array206 exch store % put in high numeric order stopwatchon % report array206 % array206 {==} forall array206 statreport % new reporter (\n Time to report array206 ) print stopwatchoff %%%%%%%%%%%%% (\n\n Status (301) REDIRECT accounts for ) length301 10 string cvs mergestr ( hits or ) mergestr length301 hits div 100 mul 100 mul cvi 100 div 10 string cvs mergestr ( percent. \n Of these, there are ) mergestr array301 length 20 string cvs mergestr ( unique url's.\n \n) mergestr print array301 popbubblesort2 /array301 exch store % put in high numeric order stopwatchon % report array301 array301 statreport % new reporter (\n Time to report array301 ) print stopwatchoff (\n\n Status (404) NOT FOUND accounts for ) length404 10 string cvs mergestr ( hits or ) mergestr length404 hits div 100 mul 100 mul cvi 100 div 20 string cvs mergestr ( percent.\n Of these, there are ) mergestr array404 length 20 string cvs mergestr ( unique url's.\n \n) mergestr print stopwatchon % report array404 array404 statreport % new reporter (\n Time to report array404 ) print stopwatchoff (\n\n Status (406) NOT ACCEPTABLE accounts for ) length406 10 string cvs mergestr ( hits or ) mergestr length406 hits div 100 mul 100 mul cvi 100 div 20 string cvs mergestr ( percent.\n Of these, there are ) mergestr array406 length 20 string cvs mergestr ( unique url's.\n \n) mergestr print stopwatchon % report array406 array406 statreport % new reporter (\n Time to report array406 ) print stopwatchoff (\n\n Status (407) PROXY REQUIRED accounts for ) length407 10 string cvs mergestr ( hits or ) mergestr length407 hits div 100 mul 100 mul cvi 100 div 20 string cvs mergestr ( percent.\n Of these, there are ) mergestr array407 length 20 string cvs mergestr ( unique url's.\n\n) mergestr print stopwatchon % report array407 array407 statreport % new reporter (\n Time to report array407 ) print stopwatchoff % arrayusers {==} forall % report user sequences { % alternate reporter... stopwatchon arrayusers { aload pop 3 1 roll exch (\n ) print print ( ) print 5 string cvs print (\n) print { == } forall} forall stopwatchoff } pop (\n\n The total number of unique users is ) arrayusers length 10 string cvs mergestr (.) mergestr print (\n The ratio of hits to users is ) hits arrayusers length div 100 mul cvi 100 div 20 string cvs mergestr (.\n) mergestr print (\n Access by these users are reported in sequential time order...\n) print %-2 vmreclaim stopwatchon arrayusers {dup dup 1 get 5 string cvs exch 0 get % should be cou usr (\n ) exch mergestr ( ) mergestr exch mergestr (\n) mergestr print 2 get { (\n) mergestr print % slightly faster than == } forall } forall (\nTime to report user array ) print stopwatchoff % 0 vmreclaim (\n\n) print %%%%%%%%% new bad guy code %%%%%%%%%%%%%%%% /find404s { (404) eq { /count404 count404 1 add store } if % on a 404 hit } store /reportbadguys { /bgcount 0 store arrayusers {dup 0 get /curuser exch store 2 get /curdata exch store /count404 0 store 0 1 curdata length 1 sub {/curerr exch store curdata curerr get ( ) search { pop pop find404s }{missing_space} ifelse } for % single user data scan count404 8 ge { curuser ( ) mergestr % report bad guy (has ) mergestr count404 5 string cvs mergestr ( \(404\)'s.\n) mergestr print /bgcount bgcount 1 add store } if } forall % arrayusers scan } store %%%%%%%%%%%%%%%%%% % actual badguy reporter here %%%%%%%%%%%%%%%%%% true {stopwatchon (\n Based on 8 or more 404's, bad guy malware suspects include...\n\n) print reportbadguys (\n There were a total of ) arrayusers length 5 string cvs mergestr ( unique users of which ) mergestr bgcount 5 string cvs mergestr ( or ) mergestr bgcount arrayusers length div 100 mul 100 mul cvi 100 div 5 string cvs mergestr (\npercent were potential malware. Shown in hit frequency order.\n) mergestr print (\nBadguy processing time: )print stopwatchoff (\n\n) print } if %%%%%%%%%%%%%%%%%% % heavy hitters %%%%%%%%%%%%%%%%%%% arrayusers 0 10 getinterval /heavyusers exch store (\n Here are the top ten heavy users...\n) print 0 1 9 {/hvypos exch store (\n) heavyusers hvypos get 0 get mergestr ( ) mergestr heavyusers hvypos get 1 get 5 string cvs mergestr print 0 0 1 9 {heavyusers exch get 1 get add } for /tenheavies exch store } for (\n\n These represent ) tenheavies 5 string cvs mergestr ( hits out of ) mergestr hits 5 string cvs mergestr ( total or ) mergestr tenheavies hits div 100 mul 100 mul cvi 100 div 5 string cvs mergestr ( percent.\n) mergestr print ( These also are ) 10 arrayusers length div 100 mul 100 mul cvi 100 div 5 string cvs mergestr ( percent of your ) mergestr arrayusers length 5 string cvs mergestr ( total users so far.\n) mergestr print ( Be sure to subtract out your own url!\n) print %%%%%%%%%%%%%%%%%% % .shtml 200 subset %%%%%%%%%%%%%%%%%%% % presently looks for any .xxx (position independent) that gives (200) % home page based on length=1 only. stopwatchon (\n Valid home page hits...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 length 1 eq {array200 curpos get dup 0 get ( ) mergestr print 1 get ==} if } for (\n Valid "default" hits...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (default) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .asp hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.asp) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .bas hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.bas) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .bmp hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.bmp) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .eml hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.eml) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .gif hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.gif) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .gz hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.gz) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .ico hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.ico) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .jpg hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.jpg) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .js hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.js) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .kml hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.kml) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .lnk hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.lnk) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .log hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.log) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .pdf hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.pdf) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .png hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.png) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .psl hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.psl) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .shtml hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.shtml) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .txt hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.txt) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Valid (200) .xml hits by popularity...\n\n) print 0 1 array200 length 1 sub { /curpos exch store array200 curpos get 0 get % grab any 200 (.xml) search {array200 curpos get dup 0 get ( ) mergestr print 1 get == } { pop } ifelse } for (\n Popularity hits ) print stopwatchoff (\n\n) print %EOF