eval

Evaluation of (discontinuous) parse trees, following EVALB as much as possible.

Usage: discodop eval <gold> <parses> [param] [options]

where gold and parses are files with parse trees, param is an EVALB parameter file.

Options

--cutofflen=n Overrides the sentence length cutoff of the parameter file.
--verbose Print table with per sentence information.
--debug Print debug information with per sentence bracketings etc.
--disconly Only evaluate discontinuous bracketings (affects bracketing scores: precision, recall, f-measure, exact match).
--goldfmt, --parsesfmt=<export|bracket|discbracket|tiger|alpino>
 Specify corpus format [default: export].
--fmt=<…> Shorthand for setting both --goldfmt and --parsesfmt.
--goldenc, --parsesenc=<utf-8|iso-8859-1|…>
 Specify encoding [default: utf-8].
--la Enable leaf-ancestor evaluation.
--ted Enable tree-edit distance evaluation. NB: it is not clear whether this score is applicable to discontinuous trees.
--headrules=x Specify file with heuristic rules for head assignment of constituents that do not already have a child marked as head. This enables dependency evaluation. NB: this evaluation is affected by the quality of the head markings and heuristics.
--functions=x ‘remove’=default: strip functions off labels, ‘leave’: leave syntactic labels as is, ‘add’: evaluate both syntactic categories and functions, ‘replace’: only evaluate grammatical functions.
--morphology=x ‘no’=default: only evaluate POS tags, ‘add’: concatenate morphology tags to POS tags, ‘replace’: replace POS tags with morphology tags, ‘between’: add morphological node between POS tag and word.

Function tags

If the parses file contains function tags, these are evaluated with the non-null metric of Blaheta & Charniak (2000), which scores function tags of correctly parsed bracketings. Multiple tags on a constituent are scored separately. We additionally consider function tags on preterminals; this does not change the evaluation for the Penn treebank as it does not contain function tags on preterminals. A more stringent metric is to combine phrasal & function tags with the option --functions=add, which incorporates function tags in the bracketing scores.

Parameter file

See the reference documentation on evaluation parameter files.

The commonly used parameter file for Penn-treebank parsing is COLLINS.prm, distributed as part of EVALB. The file proper.prm in the code repository is a version adapted for discontinuous parsing.

Examples

Discontinuous parsing:

$ discodop eval negra-parses.export negra-dev.export proper.prm

Continuous parsing:

$ discodop eval wsj-24.mrg parses.mrg COLLINS.prm --fmt=bracket