Logo Logo
Hilfe
Hilfe
Switch Language to English

Kemos, Apostolos; Adel, Heike und Schütze, Hinrich (Juni 2019): Neural Semi-Markov Conditional Random Fields for Robust Character-Based Part-of-Speech Tagging. , Minneapolis, USA, 2. - 7. June 2019. [PDF, 369kB]

[thumbnail of 1808.04208.pdf]
Vorschau
Download (369kB)

Abstract

Character-level models of tokens have been shown to be effective at dealing with withintoken noise and out-of-vocabulary words. However, they often still rely on correct token boundaries. In this paper, we propose to eliminate the need for tokenizers with an end-toend character-level semi-Markov conditional random field. It uses neural networks for its character and segment representations. We demonstrate its effectiveness in multilingual settings and when token boundaries are noisy: It matches state-of-the-art part-of-speech taggers for various languages and significantly outperforms them on a noisy English version of a benchmark dataset. Our code and the noisy dataset are publicly available at http://cistern.cis.lmu.de/semiCRF

Dokument bearbeiten Dokument bearbeiten