org.nuxeo.ecm.platform.categorization.categorizer.tfidf

Class HashingVectorizer

java.lang.Object

org.nuxeo.ecm.platform.categorization.categorizer.tfidf.HashingVectorizer

All Implemented Interfaces:

Serializable

public class HashingVectorizer
extends Object
implements Serializable

Hashed vector representation of the token unigrams and bigrams of a document provided as a sequence of tokens.

We use a hash representations to be able to maintain low memory requirements by avoiding to store an explicit map from string bigrams to feature vector index in memory.

http://hunch.net/~jl/projects/hash_reps/index.html http://en.wikipedia.org/wiki/Bloom_filter#Counting_filters

Author:

ogrisel

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
HashingVectorizer()

Method Summary

Methods

Modifier and Type	Method and Description
void	addCounts(List<String> tokens, long[] counts)
long[]	count(List<String> tokens)
HashingVectorizer	dimension(int dim) Chain configuration of the number of buckets, which is also the number of the vectors dimensions, small values mean high probabilities of collisions.
HashingVectorizer	probes(int probes) Chain configuration of the number of probes, i.e.
HashingVectorizer	window(int window) Chain configuration of the number of terms to hash together: window = 1 means unigrams and bigrams, window = 3 would add bigrams of distance 2, and so on.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HashingVectorizer

public HashingVectorizer()

Method Detail

dimension

public HashingVectorizer dimension(int dim)

Chain configuration of the number of buckets, which is also the number of the vectors dimensions, small values mean high probabilities of collisions.

window

public HashingVectorizer window(int window)

Chain configuration of the number of terms to hash together: window = 1 means unigrams and bigrams, window = 3 would add bigrams of distance 2, and so on.

probes

public HashingVectorizer probes(int probes)

Chain configuration of the number of probes, i.e. number of distinct hash functions to use for each ngram count.

count

public long[] count(List<String> tokens)

addCounts

public void addCounts(List<String> tokens,
             long[] counts)