Getting “near time” logfile processing to work

The task: We need to monitor information which is constantly added at the end of a logfile as lines of text. Getting this information ad hoc into our Middleware is not as easy as it seems at first side:

When trying to read the log to parse for added lines, we have to care about automatic log rollovers and file handles. Log files can get real big… So reading the full file every time we check for a change is not an option. This all has to be taken care of a clever Log Tailer.

Instead of parsing the log file we could also try to hook directly into the log writer. E.g. if that is a log4j system we could create our own appender in the configuration and have the appender handle information forwording to BW. Or say the log is written by a Unix process we could enable syslog and then try to receive syslog messages. But here both were technically not possible or had other drawbacks.

The good news is: A very good implementation of a Tailer is available part of Apache Commons IO. So I decided to use that ;) Important aspects for a Tailer are

  • stability and performance: should not parse through the hole file every time file is updated
  • should not reprocess lines of log file when TIBCO BW is restartet
  • must handle log rollovers
  • if log file is missing at start and later gets created, should automatically process the new file

TIBCO added Java Event Source Process Starter in TIBCO BW 5.7.0. Which that it is possible to write your own custom Starter Code in Java to produce start events. These can than be processed in the BW process. Before starting the coding we need to make the Common IO library available to BW. This can be easily done by copying commons-io-2.4.jar to the BW hotfix/lib folder. To get the Log Tailer to work we create our own class “MMListener” by extending the Apache Common IO TailerListenerAdapter: [code language=“java”] private static class MMListener extends TailerListenerAdapter { LogTailerJavaEventSource jevents; public MMListener(LogTailerJavaEventSource j) { jevents = j; } … } [/code] Lines tailed in our TailerListenerAdapter can then be triggered as events. We implement this by overriding the handle(String) method of the TailerListenerAdapter class. [code language=“java”] @Override public void handle(String line) { try { String a=new String(line); jevents.onEvent(a); } catch (Exception e) { logger.warn(e.getMessage()); } } [/code] Our MMListener is then initiated in the Jave Event Source Class init Method. We use a Tailer that checks for updates every 1000ms. [code language=“java”]] public void init() throws Exception { static logger = org.apache.log4j.Logger.getLogger(“bw.logger”); String filename = com.tibco.pe.plugin.PluginProperties.getProperty( “tibco.clientVar.LOGFILE”); if (null==filename) { logger.warn(“missing Logfile parameter”); return; } logger.info(“starting Tailer for file “+filename); MMListener lister = new MMListener(this); File logfile = new File(filename); Tailer tailer = new Tailer(logfile, lister, 1000); thread = new Thread(tailer); } [/code]