gms | German Medical Science

Introduction: Complex epidemiologic studies such as the Study of Health in Pomerania (SHIP) generate and integrate data from different origins such as laboratory measurements, imaging, clinical examinations, interviews, and many more [1]. Within EUthyroid, a Horizon 2020 project, web applications were utilized to collect metadata of several European and non-European iodine monitoring studies [2]. A centralized management of the data requires efficient workflows to keep track of any issues occurring during ETL (extract, transform, load) and subsequent data management procedures.

Background: Automatization of procedures is a standard to manage large data quantities. In SHIP for example, medical documentation specialists daily run a large number of scripts to integrate, clean and check data. These scripts, as well as proprietary devices write log-files that may contain error messages and information about technical problems. However, log-files are long, repetitive texts. This makes the extraction of relevant information difficult for human readers, especially when extensive batch jobs are executed daily. Important issues may go unnoticed with potentially serious consequences.

Concept: Conventions for writing log-files are not very strict. Usually, they list one event per row in chronological order amended by time stamps and severity classifications in plain-text format. Therefore, a flexible and easily configurable log-file parser is part of our solution. Relevant information is extracted using configured rules. Since log-files frequently contain confidential information, encryption and access control are necessary as well. The processed log information must trigger some reaction and should therefore not be lost.

Implementation: We developed the software PLIRF ("Parse Logfiles Into Rss Feeds") to automatically filter relevant error information from log-files. PLIRF generates RSS feed files, which can be distributed using methods like encrypted and password protected https-connections. Many reader tools exist for RSS feeds and eligible users can easily subscribe to feeds interesting for them. RSS feeds facilitate a timely response to detected issues which might have gone unnoticed otherwise. PLIRF is fully configurable, reads any plain-text log-file and runs on many operating systems. It filters log-files using search patterns, counts events and recognizes timestamps. It is therefore easily used with any kind of log-files. Even log-files with statistical output, such as data quality indicators, could be handled. PLIRF has been in routine use for several years at SHIP and EUthyroid.

Conclusion: The use of our log-file monitoring software PLIRF has improved our efficiency in dealing with technical problems, configuration problems of proprietary devices as well as with user errors. Especially recurring issues are easier detected and further errors can be prevented. PLIRF is easily integrated with other platforms, and can be shared on request, rendering it a versatile tool for study and data management purposes.

The authors declare that they have no competing interests.

The authors declare that an ethics committee vote is not required.