Document Token. More...

#include <gtoken.h>

Classes

struct  Search
 Token Search Structure. More...
 

Public Member Functions

 GToken (const R::RString &token=R::RString::Null)
 
int Compare (const GToken &token) const
 
int Compare (const Search &token) const
 
size_t HashCode (size_t max) const
 
R::RString GetToken (void) const
 
tTokenType GetType (void) const
 
GConceptGetConcept (void) const
 
void SetConcept (GConcept *concept)
 
bool IsAlpha (void) const
 
bool IsUsed (GConcept *metaconcept) const
 
bool IsUsed (tConceptCat cat) const
 
size_t GetNbOccurs (void) const
 
virtual ~GToken (void)
 

Private Attributes

R::RString Token
 
tTokenType Type
 
GConceptConcept
 
R::RContainer< GTokenOccur,
false, false > 
Occurs
 
size_t Index
 

Detailed Description

Document Token.

The GToken class represents a token extracted from a document. It is used when a document is analyzed : the tokenizer parses the document and create the corresponding tokens. These tokens are then treated by the different analyzers in a specific order.

It is only used by GDocAnalyze.

Constructor & Destructor Documentation

GToken ( const R::RString token = R::RString::Null)
explicit

Create a token.

Parameters
tokenString containing the token.
virtual ~GToken ( void  )
virtual

Destructor.

Member Function Documentation

int Compare ( const GToken token) const

Compare two tokens. They must match the type and the token.

Parameters
tokenToken to compare with.
Returns
a number usable by R::RContainer.
int Compare ( const Search token) const

Compare a token with a string.

Parameters
tokenSearch to compare with.
Returns
a number usable by R::RContainer.
size_t HashCode ( size_t  max) const

Compute the hash code for a token.

See also
R::RHashContainer
Parameters
maxMaximal size of the hash code.
R::RString GetToken ( void  ) const
Returns
a R::RString representing the token.
tTokenType GetType ( void  ) const
Returns
the type of the token.
GConcept* GetConcept ( void  ) const

Get the concept associated with the token (if any).

Returns
a pointer to a GConcept (may be null).
void SetConcept ( GConcept concept)

Associate a concept to the token.

Parameters
conceptConcept to assign.
bool IsAlpha ( void  ) const

Look if the token contains alphabetic characters only.

Returns
true if it is the case.
bool IsUsed ( GConcept metaconcept) const

Look if the token is associated to a particular meta-concept.

Parameters
metaconceptMeta-concept to search for.
bool IsUsed ( tConceptCat  cat) const

Look if the token is associated to a meta-concept of a particular category.

Parameters
catCategory to search for.
size_t GetNbOccurs ( void  ) const

Get the number of occurrences of the token.

Member Data Documentation

R::RString Token
private

Token.

tTokenType Type
private

Type of the token.

GConcept* Concept
private

Concept associated with that token.

R::RContainer<GTokenOccur,false,false> Occurs
private

Occurrences of the token.

size_t Index
private

Index of the token in the main container.