Getting Near-Time Logfile Processing to Work

The task: We need to monitor information which is constantly added at the end of a logfile as lines of text. Getting this information ad hoc into our middleware is not as easy as it seems at first sight.

When trying to read the log to parse for added lines, we have to care about automatic log rollovers and file handles. Log files can get really big, so reading the full file every time we check for a change is not an option. All of this has to be taken care of by a clever Log Tailer.

Instead of parsing the log file we could also try to hook directly into the log writer — e.g. if that is a log4j system we could create our own appender in the configuration and have it handle information forwarding to BW. Or if the log is written by a Unix process we could enable syslog and try to receive syslog messages. But here both approaches were technically not possible or had other drawbacks.

The good news: A very good implementation of a Tailer is available as part of Apache Commons IO. So I decided to use that. Important aspects for a Tailer are:

  • Stability and performance: should not parse through the whole file every time the file is updated
  • Should not reprocess lines of the log file when TIBCO BW is restarted
  • Must handle log rollovers
  • If the log file is missing at start and later gets created, should automatically process the new file

TIBCO added Java Event Source Process Starter in TIBCO BW 5.7.0. With that it is possible to write your own custom Starter Code in Java to produce start events. These can then be processed in the BW process.

Before starting the coding we need to make the Commons IO library available to BW. This can be easily done by copying commons-io-2.4.jar to the BW hotfix/lib folder.

To get the Log Tailer to work we create our own class MMListener by extending the Apache Commons IO TailerListenerAdapter:

private static class MMListener extends TailerListenerAdapter {
    LogTailerJavaEventSource jevents;

    public MMListener(LogTailerJavaEventSource j) {
        jevents = j;
    }
    ...
}

Lines tailed in our TailerListenerAdapter can then be triggered as events. We implement this by overriding the handle(String) method:

@Override
public void handle(String line) {
    try {
        String a = new String(line);
        jevents.onEvent(a);
    } catch (Exception e) {
        logger.warn(e.getMessage());
    }
}

Our MMListener is then initiated in the Java Event Source class init method. We use a Tailer that checks for updates every 1000ms:

public void init() throws Exception {
    static logger = org.apache.log4j.Logger.getLogger("bw.logger");
    String filename = com.tibco.pe.plugin.PluginProperties.getProperty(
        "tibco.clientVar.LOGFILE");
    if (null == filename) {
        logger.warn("missing Logfile parameter");
        return;
    }
    logger.info("starting Tailer for file " + filename);
    MMListener lister = new MMListener(this);
    File logfile = new File(filename);
    Tailer tailer = new Tailer(logfile, lister, 1000);
    thread = new Thread(tailer);
}