Showing 1–2 of 2 results for author: Schakel, A M J

Controlled Experiments for Word Embeddings

Authors: Benjamin J. Wilson, Adriaan M. J. Schakel

Abstract: An experimental approach to studying the properties of word embeddings is proposed. Controlled experiments, achieved through modifications of the training corpus, permit the demonstration of direct relations between word properties and word vector direction and length. The approach is demonstrated using the word2vec CBOW model with experiments that independently vary word frequency and word co-occ… ▽ More An experimental approach to studying the properties of word embeddings is proposed. Controlled experiments, achieved through modifications of the training corpus, permit the demonstration of direct relations between word properties and word vector direction and length. The approach is demonstrated using the word2vec CBOW model with experiments that independently vary word frequency and word co-occurrence noise. The experiments reveal that word vector length depends more or less linearly on both word frequency and the level of noise in the co-occurrence distribution of the word. The coefficients of linearity depend upon the word. The special point in feature space, defined by the (artificial) word with pure noise in its co-occurrence distribution, is found to be small but non-zero. △ Less

Submitted 14 December, 2015; v1 submitted 9 October, 2015; originally announced October 2015.

Comments: Chagelog: Rerun experiment with subsampling turned off; re-interpreted results in light of Schnabel et al. (2015). 15 pages

MSC Class: 68T50 ACM Class: I.2.7

Measuring Word Significance using Distributed Representations of Words

Authors: Adriaan M. J. Schakel, Benjamin J. Wilson

Abstract: Distributed representations of words as real-valued vectors in a relatively low-dimensional space aim at extracting syntactic and semantic features from large text corpora. A recently introduced neural network, named word2vec (Mikolov et al., 2013a; Mikolov et al., 2013b), was shown to encode semantic information in the direction of the word vectors. In this brief report, it is proposed to use the… ▽ More Distributed representations of words as real-valued vectors in a relatively low-dimensional space aim at extracting syntactic and semantic features from large text corpora. A recently introduced neural network, named word2vec (Mikolov et al., 2013a; Mikolov et al., 2013b), was shown to encode semantic information in the direction of the word vectors. In this brief report, it is proposed to use the length of the vectors, together with the term frequency, as measure of word significance in a corpus. Experimental evidence using a domain-specific corpus of abstracts is presented to support this proposal. A useful visualization technique for text corpora emerges, where words are mapped onto a two-dimensional plane and automatically ranked by significance. △ Less

Submitted 10 August, 2015; originally announced August 2015.

Comments: 7 pages, 6 figures