Parsing/repp - Revision history

Oe: /* Background */

2019-01-07T22:35:10Z

‎Background

Oe: /* Tokenization */

2019-01-07T20:21:10Z

‎Tokenization

Oe: /* Tokenization */

2019-01-07T20:00:31Z

‎Tokenization

Oe: /* Tokenization */

2019-01-07T19:56:14Z

‎Tokenization

Oe: Created page with "= Background = When parsing naturally-occuring text, it will often be necessary to pre-process input documents to various degrees—depending on what kinds of inputs are supp..."

2019-01-07T19:49:57Z

Created page with "= Background = When parsing naturally-occuring text, it will often be necessary to pre-process input documents to various degrees—depending on what kinds of inputs are supp..."

New page

= Background =

When parsing naturally-occuring text, it will often be necessary
to pre-process input documents to various degrees—depending on what
kinds of inputs are supported by a specific parsing system.
Sentence splitting and tokenization, for example, are traditionally
viewed as mandatory preprocessing steps prior to parsing.
While fully integrated ‘document parsing’ systems have become
more and more common in recent years, some older parsing systems
may further expected their pre-segmented input to be tagged with
parts of speech and maybe even lemmatized.

This NLPL module bundles a selection of preprocessing tools that
the UiO team have repeatedlyfound to provide a good balance of
accuracy and simplicity over a relatively broad range of different
types of input documents.

= Tokenization =

The [http://moin.delph-in.net/ReppTop Regular Expression–Based PreProcessor] (REPP)
implements a cascade of string-level rewrite rules for input normalization
and segmentation.
REPP defines a simple declarative specification language and provides
support for modularization and parameterization of collections of rewrite
rule, as well as limited support for grouping of rules and iteration.
Furthermore, REPP keeps track of character offsets into the original
document for its output segments, including across one-to-many
substitutions, insertions, and deletions.

[http://aclweb.org/anthology/P/P12/P12-2074.pdf Dridan & Oepen (2012; ACL)]
evaluate the performance of the English REPP tokenizer (developed by
Stephan Oepen as part of the [http://lingo.stanford.edu/erg English
Resource Grammar], ERG) against a handful of widely used tokenizers and
observe that REPP delivers state-of-the-art performance in terms of the
segmentation conventions defined by the venerable Penn Treebank (PTB).

← Older revision		Revision as of 22:35, 7 January 2019
Line 1:		Line 1:
−	~~= Background =~~
−
	When parsing naturally-occuring text, it will often be necessary		When parsing naturally-occuring text, it will often be necessary
	to pre-process input documents to various degrees—depending on what		to pre-process input documents to various degrees—depending on what

← Older revision		Revision as of 20:00, 7 January 2019
Line 35:		Line 35:
	observe that REPP delivers state-of-the-art performance in terms of the		observe that REPP delivers state-of-the-art performance in terms of the
	segmentation conventions defined by the venerable Penn Treebank (PTB).		segmentation conventions defined by the venerable Penn Treebank (PTB).
		+
		+	<pre>
		+	module load nlpl-repp
		+	cat /etc/bashrc \| repp -c $LAPROOT/repp/erg/ptb.set
		+	</pre>

@@ Line 32: / Line 32: @@
 Stephan Oepen as part of the
 [http://lingo.stanford.edu/erg English Resource Grammar], ERG)
-against a handful of widely used tokenizers and
+against a handful of widely used tokenizers for English and
 observe that REPP delivers state-of-the-art performance in terms of the
 segmentation conventions defined by the venerable Penn Treebank (PTB).
@@ Line 39: / Line 39: @@
 module load nlpl-repp
 cat /etc/bashrc | repp -c $LAPROOT/repp/erg/ptb.set
 </pre>

@@ Line 30: / Line 30: @@
 [http://aclweb.org/anthology/P/P12/P12-2074.pdf Dridan & Oepen (2012; ACL)]
 evaluate the performance of the English REPP tokenizer (developed by
-Stephan Oepen as part of the [http://lingo.stanford.edu/erg English
+Stephan Oepen as part of the
-Resource Grammar], ERG) against a handful of widely used tokenizers and
+[http://lingo.stanford.edu/erg English Resource Grammar], ERG)
 observe that REPP delivers state-of-the-art performance in terms of the
 segmentation conventions defined by the venerable Penn Treebank (PTB).