how can I read last line, or last nn bytes from a file?

nnCron and nnCron LITE discussion

how can I read last line, or last nn bytes from a file?

Postby LuckMan212 » Sat, 18 Feb 2006, 21:46

hello,
I have a program logfile I wish to monitor for a specific event. The logfile is quite long (can be many MB) and thus it is inefficient to read entire file every time just to read the last line. Log file is text format, tab separated.

I wish to create a task in nnCron that can read the last line, or the last nn bytes of the file (similar to unix "tail" command...) and then I will use regexp to parse it and test for my specific event. I have read through the help and not found any way to do this.

smth like this: (but this doesnt work... i need new commands!)
Code: Select all
#( test_log
VARIABLE temp_1
CREATE cur_line 256 ALLOT
AsLoggedUser
WatchFile: "C:\eventlog.evl"
Action:
  S" C:\eventlog.evl" R/O OPEN-FILE-SHARED THROW temp_1 !
  BEGIN cur_line 1+ 255 temp_1 @ READ-LINE THROW WHILE cur_line C!
  RE-MATCH: "%cur_line COUNT%" "/(.*)\t(\d+)\t(\d)my_event/"
    IF
      \ do some stuff...
    THEN
REPEAT
)#

any help?
:?
LuckMan212
 
Posts: 133
Joined: Mon, 04 Jul 2005, 11:19

Postby Nicholas_Nemtsev » Mon, 20 Feb 2006, 17:18

You can recede from the end of a file on some guaranteed distance and to read all rest lines. Last read line is required.
Code: Select all
temp_1 @ FILE-SIZE THROW -256. D+
temp_1 @ REPOSITION-FILE THROW
... ( read lines)
Nicholas Nemtsev
User avatar
Nicholas_Nemtsev
Site Admin
 
Posts: 857
Joined: Thu, 01 Jul 2004, 22:25
Location: Псков

Postby LuckMan212 » Mon, 20 Feb 2006, 21:21

thank you Nicholas, more questions:

1) how do I know if cur_line would contain the last line of the file? i.e. how do I know when I am "on" the last line?
2) in line
Code: Select all
temp_1 @ FILE-SIZE THROW -256. D+
does 256 represent bytes or Kilobytes? (bytes I assume).
3) what is command to exit out of current task immediately? is it EXIT? because I tried to put this in my balloonClick handler and it did not force the task to exit. Does it have to be called from "main" action section?

Here is what I have so far, it gives me problems!!
Code: Select all
#( watch_sil3114_logfile
AsLoggedUser
SingleInstance
VARIABLE temp_1
VARIABLE flagExit
CREATE myLogFile 256 ALLOT S" C:\Program Files\Silicon Image\3114-W-I32-R SATARAID5\eventlog.evl" myLogFile ZPLACE
CREATE cur_line 256 ALLOT
WatchFile: "%myLogFile ASCIIZ>%"
Rule: FILE-EMPTY: "%myLogFile ASCIIZ>%" NOT ;
: user_click3
   flagExit ON
;
Action:
   WIN-EXIST: "StorageManagementUtility" 0=
   IF
      flagExit OFF
      myLogFile ASCIIZ> R/O OPEN-FILE-SHARED THROW temp_1 !
      \ temp_1 @ FILE-SIZE THROW -256. D+
      \ temp_1 @ REPOSITION-FILE THROW
      BEGIN cur_line 1+ 255 temp_1 @ READ-LINE THROW WHILE cur_line C!
      REPEAT
      RE-MATCH: "%cur_line COUNT%" "/\x1F(\d+/\d+/\d+)\x1F\x09\x1F(\d+:\d+.\d+ (A|P)M)\x1F\x09\x1E(Controller \d - Sil3114r5)\x1E\x09\x1F([a-zA-Z0-9:,\.\x34]+)\x1F\x09\x1F([a-zA-Z0-9:,\.\x34]+)\x1F\x09\x1F([a-zA-Z0-9:,\.\x34]+)/"
      IF
         MSG: "$1 = %$1%%crlf%$2 = %$2%%crlf%$3 = %$3%%crlf%$4 = %$4%%crlf%$5 = %$5%%crlf%$6 = %$6%"
\ 1=date, 2=time, 3=cntrlr, 4=source, 5=type, 6=message
         ['] user_click3 OnBalloonClick !
         BALLOON: "System Notification" "RAID event detected.  Sil3114 utility will launch in 5 seconds.  Click balloon to cancel."
         PAUSE: 5000
         BALLOON: "" ""
         flagExit @ 0=
         IF
            3 0 DO 175 500 BEEP 175 1000 BEEP LOOP
            StartIn: "C:\Program Files\Silicon Image\3114-W-I32-R SATARAID5"
            START-APP: javaw.exe -jar SATARAID5Manager.jar
         THEN
      THEN
   ELSE
      WIN-ACTIVATE: "StorageManagementUtility"
   THEN
)#

I get stack error or nnCron crashes when I try to run this-- I think it does not like my Regexp. How can I test for smth like [a-zA-Z0-9:\",\.]+ i.e, all letters, numbers, and also comma,colon,quote,and period, as well as special control chars like 1F, 09 etc?

Here is sample line from the logfile:
Code: Select all
02/19/06   03:19.13 AM   Controller 0 - Sil3114r5   RAID   Warning   Group 0 "GRP0" was not shutdown properly.

I also attach portion of actual logfile below, you can maybe download it and try to get the regexp matching?
download here: eventlog.evl
Can you please help fix me up? thank you!! :)
LuckMan212
 
Posts: 133
Joined: Mon, 04 Jul 2005, 11:19

Postby Nicholas_Nemtsev » Tue, 21 Feb 2006, 13:06

LuckMan212 wrote:how do I know if cur_line would contain the last line of the file? i.e. how do I know when I am "on" the last line?

It's simple: read lines to EOF, last read line is required.
LuckMan212 wrote:in line
Code: Select all
temp_1 @ FILE-SIZE THROW -256. D+
does 256 represent bytes or Kilobytes? (bytes I assume).
naturally bytes.
LuckMan212 wrote:what is command to exit out of current task immediately? is it EXIT? because I tried to put this in my balloonClick handler and it did not force the task to exit. Does it have to be called from "main" action section?

EXIT serves for leaving current word, but not task. To terminate task (thread) immediately use
Code: Select all
0 ExitThread
But it isn't good method. Preferred method in use of words CATCH and THROW
LuckMan212 wrote:Here is what I have so far, it gives me problems!!

BEGIN cur_line 1+ 255 temp_1 @ READ-LINE THROW WHILE cur_line C!
REPEAT DROP
Nicholas Nemtsev
User avatar
Nicholas_Nemtsev
Site Admin
 
Posts: 857
Joined: Thu, 01 Jul 2004, 22:25
Location: Псков

Postby LuckMan212 » Tue, 21 Feb 2006, 13:21

Nicholas_Nemtsev wrote:EXIT serves for leaving current word, but not task. To terminate task (thread) immediately use
Code: Select all
0 ExitThread
But it isn't good method. Preferred method in use of words CATCH and THROW
ok excellent, have you some example code to show how this works?

also, how about my RegExp question-- my regexp string seems to be causing problems for nnCron. Is there more efficient way to match special chars like Quotes (") ?

thank you Nicholas, for your help :D

EDIT:
ok more progress.. yes after adding your suggested DROP command after REPEAT and tweaking my Regexp a little I have it almost working 99%. But now I have problem-- if I load this into my nncron.tab file, after several minutes, nnCron will crash (nnCron.exe has generated errors...etc). Also, if I select "Reload Crontabs" it will also crash very soon afterwards. What could be wrong? Any idea??
here is the code now:
Code: Select all
#( watch_sil3114_logfile
AsLoggedUser
SingleInstance
VARIABLE temp_1
VARIABLE flagExit
CREATE myLogFile 256 ALLOT S" C:\Program Files\Silicon Image\3114-W-I32-R SATARAID5\eventlog.evl" myLogFile ZPLACE
CREATE cur_line2 256 ALLOT
WatchFile: "%myLogFile ASCIIZ>%"
Rule: FILE-EMPTY: "%myLogFile ASCIIZ>%" NOT ;
: user_click3
   flagExit ON
;
Action:
   WIN-EXIST: "StorageManagementUtility" 0=
   IF
      flagExit OFF
      myLogFile ASCIIZ> R/O OPEN-FILE-SHARED THROW temp_1 !
      \ temp_1 @ FILE-SIZE THROW -256. D+
      \ temp_1 @ REPOSITION-FILE THROW
      BEGIN cur_line2 1+ 255 temp_1 @ READ-LINE THROW WHILE cur_line2 C!
      REPEAT DROP
      temp_1 @ CLOSE-FILE DROP
      RE-MATCH: "%cur_line2 COUNT%" "/\x1F(\d+/\d+/\d+)\x1F\x09\x1F(\d+:\d+.\d+ .M)\x1F\x09\x1E(Controller \d - Sil3114r5)\x1E\x09\x1F([a-zA-Z0-9:,\ \.\x34]+)\x1F\x09\x1F([a-zA-Z0-9:,\ \.\x34]+)\x1F\x09\x1F([a-zA-Z0-9:,\ \.\x34]+)/"
      IF
         MSG: "$1 = %$1%%crlf%$2 = %$2%%crlf%$3 = %$3%%crlf%$4 = %$4%%crlf%$5 = %$5%%crlf%$6 = %$6%" \ 1=date, 2=time, 3=cntrlr, 4=source, 5=type, 6=message
         ['] user_click3 OnBalloonClick !
         BALLOON: "System Notification" "RAID event detected.  Sil3114 utility will auto-launch in 10 seconds.  Click this balloon to cancel."
         PAUSE: 10000
         BALLOON: "" ""
         flagExit @ 0=
         IF
            3 0 DO 175 500 BEEP 175 1000 BEEP LOOP
            StartIn: "C:\Program Files\Silicon Image\3114-W-I32-R SATARAID5"
            START-APP: javaw.exe -jar SATARAID5Manager.jar
         ELSE
            MSG: "user cancelled"
         THEN
      THEN
   ELSE
      WIN-ACTIVATE: "StorageManagementUtility"
   THEN
)#
LuckMan212
 
Posts: 133
Joined: Mon, 04 Jul 2005, 11:19

Postby Nicholas_Nemtsev » Tue, 21 Feb 2006, 17:50

The matter is that the buffer should be larger at least by two bytes.
CREATE cur_line 258 ALLOT
Nicholas Nemtsev
User avatar
Nicholas_Nemtsev
Site Admin
 
Posts: 857
Joined: Thu, 01 Jul 2004, 22:25
Location: Псков

Postby LuckMan212 » Tue, 21 Feb 2006, 21:08

:D :D :D
and finally, success!
thank you Nicholas for your assistance!

please, can you tell me a little bit about buffers, why does those 2 bytes make such a difference? anywhere in the manual I can read about these buffers?

the finished, working code:
Code: Select all
#( watch_sil3114_logfile
AsLoggedUser
SingleInstance
VARIABLE temp_1
VARIABLE flagExit
CREATE myLogFile 258 ALLOT S" C:\Program Files\Silicon Image\3114-W-I32-R SATARAID5\eventlog.evl" myLogFile ZPLACE
CREATE cur_line2 258 ALLOT
WatchFile: "%myLogFile ASCIIZ>%"
Rule: FILE-EMPTY: "%myLogFile ASCIIZ>%" NOT ;
: user_click3
   flagExit ON
;
Action:
   WIN-EXIST: "StorageManagementUtility" 0=
   IF
      flagExit OFF
      myLogFile ASCIIZ> R/O OPEN-FILE-SHARED THROW temp_1 !
      5120. FILE-SIZE: "%myLogFile ASCIIZ>%" D<
      IF
         \ size is bigger than 5k, so read only last 1k bytes
         temp_1 @ FILE-SIZE THROW -1024. D+
         temp_1 @ REPOSITION-FILE THROW
      THEN
      BEGIN cur_line2 1+ 255 temp_1 @ READ-LINE THROW WHILE cur_line2 C!
      REPEAT DROP
      temp_1 @ CLOSE-FILE DROP
      RE-MATCH: "%cur_line2 COUNT%" "/\x1F(\d+/\d+/\d+)\x1F\x09\x1F(\d+:\d+.\d+ .M)\x1F\x09\x1E(Controller \d - Sil3114r5)\x1E\x09\x1F([a-zA-Z0-9:,\ \.\x34]+)\x1F\x09\x1F([a-zA-Z0-9:,\ \.\x34]+)\x1F\x09\x1F([a-zA-Z0-9:,\ \.\x34]+)/"
      IF
         \ $1=date, $2=time, $3=cntrlr, $4=source, $5=type, $6=message
         S" %$6%" EVAL-SUBST S" Database initialized.." COMPARE 0=
         IF
            \ we ignore `Database initialized` events, since they occur when launching the RAID utility
         ELSE
            ['] user_click3 OnBalloonClick !
            BALLOON: "System Notification" "RAID event detected.  Sil3114 utility will auto-launch in 10 seconds.  Click this balloon to cancel."
            PAUSE: 10000
            BALLOON: "" ""
            flagExit @ 0=
            IF
               3 0 DO 175 500 BEEP 175 1000 BEEP LOOP
               StartIn: "C:\Program Files\Silicon Image\3114-W-I32-R SATARAID5"
               START-APP: javaw.exe -jar SATARAID5Manager.jar
            THEN
         THEN
      ELSE
         MSG: "Error parsing `%myLogFile ASCIIZ>%`."
      THEN
   ELSE
      WIN-ACTIVATE: "StorageManagementUtility"
   THEN
)#
LuckMan212
 
Posts: 133
Joined: Mon, 04 Jul 2005, 11:19

Postby Nicholas_Nemtsev » Tue, 21 Feb 2006, 22:04

please, can you tell me a little bit about buffers, why does those 2 bytes make such a difference? anywhere in the manual I can read about these buffers?

Read this carefully:
How to write data to a file or read from one?
Nicholas Nemtsev
User avatar
Nicholas_Nemtsev
Site Admin
 
Posts: 857
Joined: Thu, 01 Jul 2004, 22:25
Location: Псков

Postby LuckMan212 » Tue, 21 Feb 2006, 22:37

ah ha--
ok so given command
Code: Select all
READ-LINE ( c-addr u1 fileid -- u2 flag ior )

if I have this line
Code: Select all
BEGIN list-contents 1+ 255 list-file @ READ-LINE THROW WHILE

then:

c-addr = list-contents
u1 = 255
fileid = list-file

then buffer will need to be at least 258. But if we have line such as:
Code: Select all
BEGIN list-contents 1+ 511 list-file @ READ-LINE THROW WHILE

then string buffer list-contents will need to be at least 514, is that correct?
some more questions:
1) what is the purpose of the "+1" added to the address of the string buffer?

2) So is it correct that if u1=255 and the line exceeds 256 bytes then the result will be truncated? or does it just cause a stack error? sorry for too many questions, but I am really trying to learn...... :idea:

I had several other tasks with this same error in the code (buffer not large enough during READ-LINE) -- maybe that was causing some of my other problems..hmm!
LuckMan212
 
Posts: 133
Joined: Mon, 04 Jul 2005, 11:19

Postby LuckMan212 » Sun, 26 Feb 2006, 09:45

nicholas?
LuckMan212
 
Posts: 133
Joined: Mon, 04 Jul 2005, 11:19

Postby Nicholas_Nemtsev » Sun, 26 Feb 2006, 10:38

I'm sorry :)
LuckMan212 wrote:if I have this line
Code: Select all
BEGIN list-contents 1+ 255 list-file @ READ-LINE THROW WHILE

then:
c-addr = list-contents
u1 = 255
fileid = list-file
then buffer will need to be at least 258. But if we have line such as:
Code: Select all
BEGIN list-contents 1+ 511 list-file @ READ-LINE THROW WHILE

then string buffer list-contents will need to be at least 514, is that correct?

Both are correct, but you can't store length that exceeds 255 bytes in one byte.
LuckMan212 wrote:1) what is the purpose of the "+1" added to the address of the string buffer?
first byte to store length (counted string). See Which string types are used in SP-Forth?
Also you can use large counted strings (4 bytes for length and XCOUNT to convert c-addr to c-addr+4 u)
Code: Select all
BEGIN list-contents CELL+ 1024 list-file @ READ-LINE THROW WHILE list-contents ! ...
MSG: "%list-contents XCOUNT%"

LuckMan212 wrote:2) So is it correct that if u1=255 and the line exceeds 256 bytes then the result will be truncated? or does it just cause a stack error? sorry for too many questions, but I am really trying to learn......

Yes, line will be truncated. But you can
1. use large counted strings
2. use az-strings
Code: Select all
CREATE list-contents 1026 ALLOT
BEGIN list-contents 1024 list-file @ READ-LINE THROW WHILE
list-contents + 0 SWAP C!
...
MSG: "%list-contents ASCIIZ>%"
Nicholas Nemtsev
User avatar
Nicholas_Nemtsev
Site Admin
 
Posts: 857
Joined: Thu, 01 Jul 2004, 22:25
Location: Псков

Postby LuckMan212 » Sun, 26 Feb 2006, 20:07

thank you Nicholas, this is very helpful information!! :D
the task is working great.

btw, will you ever take nnCron 1.91 out of "RC" phase? :wink: I think it is pretty stable now!
LuckMan212
 
Posts: 133
Joined: Mon, 04 Jul 2005, 11:19

Postby LuckMan212 » Wed, 22 Mar 2006, 10:22

well i guess you answered my question! :D
thank you for releasing this final 1.91 version!!
LuckMan212
 
Posts: 133
Joined: Mon, 04 Jul 2005, 11:19

Postby LuckMan212 » Sat, 01 Jul 2006, 19:40

help!! :o
somehow between versions of nnCron, this task has stopped working. It is not reading lines properly from my log file. I get many single characters, and then nnCron either crashes or gives stack error.

Can you download this same eventlog.evl file:
eventlog.evl
...and try with task below:
Code: Select all
#( watch_sil3114_logfile
AsLoggedUser
SingleInstance
VARIABLE temp_1
VARIABLE flagExit
CREATE myLogFile 256 ALLOT S" eventlog.evl" myLogFile ZPLACE
CREATE cur_line2 258 ALLOT
WatchFile: "%myLogFile ASCIIZ>%"
Rule: FILE-EMPTY: "%myLogFile ASCIIZ>%" NOT ;
Action:
   myLogFile ASCIIZ> R/O OPEN-FILE-SHARED THROW temp_1 !
   5120. FILE-SIZE: "%myLogFile ASCIIZ>%" D<
   IF
      \ MSG: "size is bigger than 5k, so read only last 1k bytes"
      temp_1 @ FILE-SIZE THROW -1024. D+
      temp_1 @ REPOSITION-FILE THROW
   THEN
   BEGIN cur_line2 1+ 255 temp_1 @ READ-LINE THROW WHILE cur_line2 C!
   REPEAT DROP
   temp_1 @ CLOSE-FILE DROP
   RE-MATCH: "%cur_line2 COUNT%" "/\x1F(\d+/\d+/\d+)\x1F\x09\x1F(\d+:\d+.\d+ .M)\x1F\x09\x1E(Controller \d - Sil3114r5)\x1E\x09\x1F([a-zA-Z0-9:,\ \.\x34]+)\x1F\x09\x1F([a-zA-Z0-9:,\ \.\x34]+)\x1F\x09\x1F([a-zA-Z0-9:,\ \.\x34]+)/"
   IF
      \ MSG: "$1 = %$1%%crlf%$2 = %$2%%crlf%$3 = %$3%%crlf%$4 = %$4%%crlf%$5 = %$5%%crlf%$6 = %$6%" \ 1=date, 2=time, 3=cntrlr, 4=source, 5=type, 6=message
      S" %$6%" EVAL-SUBST S" Database initialized.." COMPARE 0=
      IF
         \ ignore  "Database initialized" events...
      ELSE
         BALLOON: "System Notification" "RAID event detected!!"
         PAUSE: 10000
         BALLOON: "" ""
      THEN
   ELSE
      MSG: "Error parsing `%myLogFile ASCIIZ>%`."
   THEN
)#

I dont know what might be wrong.

Currently, I use:
Code: Select all
nncron.exe v 1.91 Build 1083 21.03.2006
tm.exe v 1.91 Build 499 21.03.2006
Directory: C:\Program Files\nnCron
Windows XP (5.1.2600) Service Pack 2
shell32.dll 6.0.2900

thank you Valery and Nicholas!! :P
LuckMan212
 
Posts: 133
Joined: Mon, 04 Jul 2005, 11:19

Postby Nicholas_Nemtsev » Sun, 02 Jul 2006, 13:40

This task with this eventlog.evl works properly on my system.
Nicholas Nemtsev
User avatar
Nicholas_Nemtsev
Site Admin
 
Posts: 857
Joined: Thu, 01 Jul 2004, 22:25
Location: Псков

Next

Return to nnCron forum (English)

Who is online

Users browsing this forum: No registered users and 2 guests

cron