boostorg/tokenizer
1
0
Fork 0
mirror of https://github.com/boostorg/tokenizer.git synced 2026-07-21 13:33:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

Tokenizer initial commit

Browse Source 
[SVN r10319]
This commit is contained in:
Beman Dawes
2001-06-13 00:46:03 +00:00
commit 0854dc07e9
19 changed files with 2053 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitattributes
+96
View File
@@ -0,0 +1,96 @@
* text=auto !eol svneol=native#text/plain
*.gitattributes text svneol=native#text/plain
# Scriptish formats
*.bat text svneol=native#text/plain
*.bsh text svneol=native#text/x-beanshell
*.cgi text svneol=native#text/plain
*.cmd text svneol=native#text/plain
*.js text svneol=native#text/javascript
*.php text svneol=native#text/x-php
*.pl text svneol=native#text/x-perl
*.pm text svneol=native#text/x-perl
*.py text svneol=native#text/x-python
*.sh eol=lf svneol=LF#text/x-sh
configure eol=lf svneol=LF#text/x-sh
# Image formats
*.bmp binary svneol=unset#image/bmp
*.gif binary svneol=unset#image/gif
*.ico binary svneol=unset#image/ico
*.jpeg binary svneol=unset#image/jpeg
*.jpg binary svneol=unset#image/jpeg
*.png binary svneol=unset#image/png
*.tif binary svneol=unset#image/tiff
*.tiff binary svneol=unset#image/tiff
*.svg text svneol=native#image/svg%2Bxml
# Data formats
*.pdf binary svneol=unset#application/pdf
*.avi binary svneol=unset#video/avi
*.doc binary svneol=unset#application/msword
*.dsp text svneol=crlf#text/plain
*.dsw text svneol=crlf#text/plain
*.eps binary svneol=unset#application/postscript
*.gz binary svneol=unset#application/gzip
*.mov binary svneol=unset#video/quicktime
*.mp3 binary svneol=unset#audio/mpeg
*.ppt binary svneol=unset#application/vnd.ms-powerpoint
*.ps binary svneol=unset#application/postscript
*.psd binary svneol=unset#application/photoshop
*.rdf binary svneol=unset#text/rdf
*.rss text svneol=unset#text/xml
*.rtf binary svneol=unset#text/rtf
*.sln text svneol=native#text/plain
*.swf binary svneol=unset#application/x-shockwave-flash
*.tgz binary svneol=unset#application/gzip
*.vcproj text svneol=native#text/xml
*.vcxproj text svneol=native#text/xml
*.vsprops text svneol=native#text/xml
*.wav binary svneol=unset#audio/wav
*.xls binary svneol=unset#application/vnd.ms-excel
*.zip binary svneol=unset#application/zip
# Text formats
.htaccess text svneol=native#text/plain
*.bbk text svneol=native#text/xml
*.cmake text svneol=native#text/plain
*.css text svneol=native#text/css
*.dtd text svneol=native#text/xml
*.htm text svneol=native#text/html
*.html text svneol=native#text/html
*.ini text svneol=native#text/plain
*.log text svneol=native#text/plain
*.mak text svneol=native#text/plain
*.qbk text svneol=native#text/plain
*.rst text svneol=native#text/plain
*.sql text svneol=native#text/x-sql
*.txt text svneol=native#text/plain
*.xhtml text svneol=native#text/xhtml%2Bxml
*.xml text svneol=native#text/xml
*.xsd text svneol=native#text/xml
*.xsl text svneol=native#text/xml
*.xslt text svneol=native#text/xml
*.xul text svneol=native#text/xul
*.yml text svneol=native#text/plain
boost-no-inspect text svneol=native#text/plain
CHANGES text svneol=native#text/plain
COPYING text svneol=native#text/plain
INSTALL text svneol=native#text/plain
Jamfile text svneol=native#text/plain
Jamroot text svneol=native#text/plain
Jamfile.v2 text svneol=native#text/plain
Jamrules text svneol=native#text/plain
Makefile* text svneol=native#text/plain
README text svneol=native#text/plain
TODO text svneol=native#text/plain
# Code formats
*.c text svneol=native#text/plain
*.cpp text svneol=native#text/plain
*.h text svneol=native#text/plain
*.hpp text svneol=native#text/plain
*.ipp text svneol=native#text/plain
*.tpp text svneol=native#text/plain
*.jam text svneol=native#text/plain
*.java text svneol=native#text/plain
char_delimiters_separator.htm
+148
View File
@@ -0,0 +1,148 @@
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>Boost Char Delimiters Separator</title>
<!--
-- Copyright © John Bandela 2001
--
-- Permission to use, copy, modify, distribute and sell this software
-- and its documentation for any purpose is hereby granted without fee,
-- provided that the above copyright notice appears in all copies and
-- that both that copyright notice and this permission notice appear
-- in supporting documentation. Jeremy Siek makes no
-- representations about the suitability of this software for any
-- purpose. It is provided "as is" without express or implied warranty.
-->
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
vlink="#551A8B" alink="#FF0000">
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
height="86"> <br>
</p>
<h1 align="center">Char Delimiters Separator</h1>
<pre>template &lt;class Char, class Traits = std::char_traits&lt;Char&gt; &gt;
class char_delimiters_separator{
</pre>
<p>The char_delimiters_separator class is an implementation of
the <a href="tokenizerfunction.htm">TokenizerFunction</a> concept
that can be used to break text up into tokens. It is the default
TokenizerFunction for tokenizer and token_iterator_generator. An
example is below.</p>
<h2>Example</h2>
<pre>// simple_example_4.cpp
#include&lt;iostream&gt;
#include&lt;boost/tokenizer.hpp&gt;
#include&lt;string&gt;
int main(){
using namespace std;
using namespace boost;
string s = &quot;This is, a test&quot;;
tokenizer&lt;char_delimiters_separator&lt;char&gt; &gt; tok(s);
for(tokenizer&lt;char_delimiters_separator&lt;char&gt; &gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout &lt;&lt; *beg &lt;&lt; &quot;\n&quot;;
}
}
</pre>
<h2>Construction and Usage</h2>
<p>There is one constructor of interest. It is as follows</p>
<pre>explicit char_delimiters_separator(bool return_delims = false,
const Char* returnable = &quot;&quot;,const Char* nonreturnable = &quot;&quot; )
</pre>
<table border="1">
<tr>
<td><p align="center"><strong>Parameter</strong></p>
</td>
<td><p align="center"><strong>Description</strong></p>
</td>
</tr>
<tr>
<td>return_delims</td>
<td>Whether or not to return the delimiters that have
been found. Note that not all delimiters can be returned.
See the other two parameters for explanation.</td>
</tr>
<tr>
<td>returnable</td>
<td>This specifies the returnable delimiters. These are
the delimiters that can be returned as tokens when
return_delims is true. Since these are typically
punctuation, if a 0 is provided as the argument, then the
returnable delmiters will be all characters Cfor which
std::ispunct(C) yields a true value. If an argument of
&quot;&quot; is provided, then this is taken to mean that
there are noreturnable delimiters.</td>
</tr>
<tr>
<td>nonreturnable</td>
<td>This specifies the nonreturnable delimiters. These
are delimiters that cannot be returned as tokens. Since
these are typically whitespace, if 0 is specified as an
argument, then the nonreturnable delimiters will be all
characters C for which std::isspace(C) yields a true
value. If an argument of &quot;&quot; is provided, then
this is taken to mean that there are no non-returnable
delimiters.</td>
</tr>
</table>
<p>The reason there is a distinction between nonreturnable and
returnable delimiters is that some delimiters are just used to
split up tokens and are nothing more. Take for example the
following string &quot;b c +&quot;. Assume you are writing a
simple calculator to parse expression in post fix notation. While
both the space and the + separate tokens, you only only
interested in the + and not in the space. Indeed having the space
returned as a token would only complicate your code. In this case
you would specify + as a returnable, and space as a nonreturnable
delimiter.</p>
<p>To use this class, pass an object of it anywhere a
TokenizerFunction object is required.</p>
<h3>Template Parameters</h3>
<table border="1">
<tr>
<th>Parameter</th>
<th>Description</th>
</tr>
<tr>
<td><tt>Char</tt></td>
<td>The type of the elements within a token, typically <tt>char</tt>.</td>
</tr>
<tr>
<td>Traits</td>
<td>The traits class for Char, typically std::char_traits&lt;Char&gt;</td>
</tr>
</table>
<h2>Model of</h2>
<p><a href="tokenizerfunction.htm">TokenizerFunction</a> </p>
<p>&nbsp;</p>
<hr>
<p>© Copyright John R. Bandela 2001. Permission to copy, use,
modify, sell and distribute this document is granted provided
this copyright notice appears in all copies. This document is
provided &quot;as is&quot; without express or implied warranty,
and with no claim as to its suitability for any purpose.</p>
</body>
</html>
escaped_list_separator.htm
+166
View File
@@ -0,0 +1,166 @@
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>Boost Escaped List Separator</title>
<!--
-- Copyright © John Bandela 2001
--
-- Permission to use, copy, modify, distribute and sell this software
-- and its documentation for any purpose is hereby granted without fee,
-- provided that the above copyright notice appears in all copies and
-- that both that copyright notice and this permission notice appear
-- in supporting documentation. Jeremy Siek makes no
-- representations about the suitability of this software for any
-- purpose. It is provided "as is" without express or implied warranty.
-->
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
vlink="#551A8B" alink="#FF0000">
<h1 align="left"><img src="../../c++boost.gif" alt="C++ Boost"
width="277" height="86"> </h1>
<h1 align="center">Escaped List Separator</h1>
<div align="left">
<pre>
escaped_list_separator&lt;Char, Traits = std::char_traits&lt;Char&gt; &gt;
</pre>
</div>
<p>The <tt>escaped_list_separator</tt> class is an implementation
of the <a href="tokenizerfunction.htm">TokenizerFunction</a>. The
escaped_list_separator parses a superset of the csv (comma
separated value) format. The examples of this formate are below.
It is assumed that the default characters for separator, quote,
and escape are used.</p>
<p>Field 1,Field 2,Field 3<br>
Field 1,&quot;Field 2, with comma&quot;,Field 3<br>
Field 1,Field 2 with \&quot;embedded quote\&quot;,Field 3<br>
Field 1, Field 2 with \n new line,Field 3<br>
Field 1, Field 2 with embedded \\ ,Field 3</p>
<p>Fields are normally separated by commas. If you want to put a
comma in a field, you need to put quotes around it. Also 3 escape
sequences are supported</p>
<table border="1">
<tr>
<td><p align="center"><strong>Escape Sequence</strong></p>
</td>
<td><p align="center"><strong>Result</strong></p>
</td>
</tr>
<tr>
<td>\&quot;</td>
<td>&quot;</td>
</tr>
<tr>
<td>\n</td>
<td>newline</td>
</tr>
<tr>
<td>\\</td>
<td>\</td>
</tr>
</table>
<p>&nbsp;</p>
<h2>Example</h2>
<pre>// simple_example_2.cpp
#include&lt;iostream&gt;
#include&lt;boost/tokenizer.hpp&gt;
#include&lt;string&gt;
int main(){
using namespace std;
using namespace boost;
string s = &quot;Field 1,\&quot;putting quotes around fields, allows commas\&quot;,Field 3&quot;;
tokenizer&lt;escaped_list_separator&lt;char&gt; &gt; tok(s);
for(tokenizer&lt;escaped_list_separator&lt;char&gt; &gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout &lt;&lt; *beg &lt;&lt; &quot;\n&quot;;
}
}</pre>
<p>&nbsp;</p>
<h2>Construction and Usage</h2>
<p>escaped_list_separator has 1 constructor. It is as follows</p>
<pre>explicit escaped_list_separator(Char escape = '\\', Char c = ',',Char q = '\&quot;')</pre>
<table border="1">
<tr>
<td><p align="center"><strong>Parameter</strong></p>
</td>
<td><p align="center"><strong>Description</strong></p>
</td>
</tr>
<tr>
<td>escape</td>
<td>Specifies the character to use for escape sequences.
It defaults to the C style \ (backslash). However you can
override by passing in a different character. An example
of when you might want to do this is when you have many
fields which are Windows style filenames. Instead of
escaping out each \ in the path, you can change the
escape to something else.</td>
</tr>
<tr>
<td>c</td>
<td>Specifies the character to use to separate the fields</td>
</tr>
<tr>
<td>q</td>
<td>Specifies the character to use for the quote.</td>
</tr>
</table>
<p>To use this class, pass an object of it anywhere in the
Tokenizer package where a TokenizerFunction is required.</p>
<p>&nbsp;</p>
<h2>Template Parameters</h2>
<table border="1">
<tr>
<th><strong>Parameter</strong></th>
<th><strong>Description</strong></th>
</tr>
<tr>
<td><tt>Char</tt></td>
<td>The type of the elements within a token, typically <tt>char</tt>.</td>
</tr>
<tr>
<td>Traits</td>
<td>The traits class for the Char type. This is used for
comparing Char's. It defaults to std::char_traits&lt;Char&gt;</td>
</tr>
</table>
<p>&nbsp;</p>
<h2>Model of</h2>
<p><a href="tokenizerfunction.htm">TokenizerFunction</a> </p>
<p>&nbsp;</p>
<hr>
<p>© Copyright John R. Bandela 2001. Permission to copy, use,
modify, sell and distribute this document is granted provided
this copyright notice appears in all copies. This document is
provided &quot;as is&quot; without express or implied warranty,
and with no claim as to its suitability for any purpose.</p>
</body>
</html>
examples.cpp
+126
View File
@@ -0,0 +1,126 @@
// Boost tokenizer examples -------------------------------------------------//
// © Copyright John R. Bandela 2001.
// Permission to copy, use, modify, sell and distribute this software
// is granted provided this copyright notice appears in all
// copies. This software is provided "as is" without express or
// implied warranty, and with no claim as to its suitability for any
// purpose.
// See http://www.boost.org for updates, documentation, and revision history.
#include <iostream>
#include <iterator>
#include <string>
#include <boost/tokenizer.hpp>
#include <boost/array.hpp>
#define BOOST_INCLUDE_MAIN
#include <boost/test/test_tools.hpp>
int test_main(int, char**)
{
using namespace std;
using namespace boost;
// Use tokenizer
{
const string test_string = "This,,is, a.test..";
const string answer[] = {"This","is","a","test"};
typedef tokenizer<> Tok;
Tok t(test_string);
BOOST_CRITICAL_TEST(equal(t.begin(),t.end(),answer));
}
{
const string test_string = "Field 1,\"embedded,comma\",quote \\\", escape \\\\";
const string answer[] = {"Field 1","embedded,comma","quote \""," escape \\"};
typedef tokenizer<escaped_list_separator<char> > Tok;
Tok t(test_string);
BOOST_CRITICAL_TEST(equal(t.begin(),t.end(),answer));
}
{
const string test_string = "12252001";
const string answer[] = {"12","25","2001"};
typedef tokenizer<offset_separator > Tok;
boost::array<int,3> offsets = {{2,2,4}};
offset_separator func(offsets.begin(),offsets.end());
Tok t(test_string,func);
BOOST_CRITICAL_TEST(equal(t.begin(),t.end(),answer));
}
// Use token_iterator_generator
{
const string test_string = "This,,is, a.test..";
const string answer[] = {"This","is","a","test"};
typedef token_iterator_generator<char_delimiters_separator<char> >::type Iter;
Iter begin = make_token_iterator<string>(test_string.begin(),
test_string.end(),char_delimiters_separator<char>());
Iter end;
BOOST_CRITICAL_TEST(equal(begin,end,answer));
}
{
const string test_string = "Field 1,\"embedded,comma\",quote \\\", escape \\\\";
const string answer[] = {"Field 1","embedded,comma","quote \""," escape \\"};
typedef token_iterator_generator<escaped_list_separator<char> >::type Iter;
Iter begin = make_token_iterator<string>(test_string.begin(),
test_string.end(),escaped_list_separator<char>());
Iter end;
BOOST_CRITICAL_TEST(equal(begin,end,answer));
}
{
const string test_string = "12252001";
const string answer[] = {"12","25","2001"};
typedef token_iterator_generator<offset_separator>::type Iter;
boost::array<int,3> offsets = {{2,2,4}};
offset_separator func(offsets.begin(),offsets.end());
Iter begin = make_token_iterator<string>(test_string.begin(),
test_string.end(),func);
Iter end= make_token_iterator<string>(test_string.end(),
test_string.end(),func);
BOOST_CRITICAL_TEST(equal(begin,end,answer));
}
// Test copying
{
string s = "abcdef";
token_iterator_generator<offset_separator>::type beg, end, other;
boost::array<int,3> ar = {{1,2,3}};
offset_separator f(ar.begin(),ar.end());
beg = make_token_iterator<string>(s.begin(),s.end(),f);
++beg;
other = beg;
++other;
BOOST_CRITICAL_TEST(*beg=="bc");
BOOST_CRITICAL_TEST(*other=="def");
other = make_token_iterator<string>(s.begin(),s.end(),f);
BOOST_CRITICAL_TEST(*other=="a");
}
// Test non-default constructed char_delimiters_separator
{
const string test_string = "how,are you, doing";
const string answer[] = {"how",",","are you",","," doing"};
tokenizer<> t(test_string,char_delimiters_separator<char>(true,",",""));
BOOST_CRITICAL_TEST(equal(t.begin(),t.end(),answer));
}
return 0;
}
include/boost/token_functions.hpp
+273
View File
@@ -0,0 +1,273 @@
// Boost token_functions.hpp ------------------------------------------------//
// © Copyright John R. Bandela 2001.
// Permission to copy, use, modify, sell and distribute this software
// is granted provided this copyright notice appears in all
// copies. This software is provided "as is" without express or
// implied warranty, and with no claim as to its suitability for any
// purpose.
// See http://www.boost.org for updates, documentation, and revision history.
#ifndef BOOST_TOKEN_FUNCTIONS_JRB051801_HPP_
#define BOOST_TOKEN_FUNCTIONS_JRB051801_HPP_
#include <vector>
#include <stdexcept>
#include <cassert>
#include <string>
#include <cctype>
namespace boost{
//===========================================================================
// The escaped_list_separator class. Which is a model of TokenizerFunction
// An escaped list is a super-set of what is commonly known as a comma
// separated value (csv) list.It is separated into fields by a comma or
// other character. If the delimiting character is inside quotes, then it is
// counted as a regular character.To allow for embedded quotes in a field,
// there can be escape sequences using the \ much like C.
// The role of the comma, the quotation mark, and the escape
// character (backslash \), can be assigned to other characters.
struct escaped_list_error : public std::runtime_error{
escaped_list_error(const std::string& what):std::runtime_error(what){}
};
template <class Char, class Traits = std::char_traits<Char> >
class escaped_list_separator{
private:
Char escape_;
Char c_;
Char quote_;
template<class iterator,class Token>
void do_escape(iterator& next,iterator end,Token& tok){
if(++next==end)
throw escaped_list_error(
std::string("cannot end with escape"));
if(Traits::eq(*next,'n')){
tok+='\n';
return;
}
else if(Traits::eq(*next,quote_)){
tok+=quote_;
return;
}
else if(Traits::eq(*next,c_)){
tok+=c_;
return;
}
else if(Traits::eq(*next,escape_)){
tok+=escape_;
return;
}
else
throw escaped_list_error(
std::string("unknown escape sequence"));
}
public:
explicit escaped_list_separator(Char escape = '\\',
Char c = ',',Char q = '\"'):escape_(escape),c_(c),quote_(q){}
void reset(){}
template<class InputIterator,class Token>
bool operator()(InputIterator& next,InputIterator end,Token& tok){
bool bInQuote = false;
tok = Token();
if(next==end)return false;
for(;next != end;++next){
if(Traits::eq(*next,escape_)){
do_escape(next,end,tok);
}
else if(Traits::eq(*next,c_)){
if(!bInQuote) {
// If we are not in quote, then we are done
++next;
return true;
}
else tok+=c_;
}
else if(Traits::eq(*next,quote_)){
bInQuote=!bInQuote;
}
else{
tok += *next;
}
}
return true;
}
};
//===========================================================================
// The offset_separator class, which is a model of TokenizerFunction.
// Offset breaks a string into tokens based on a range of offsets
class offset_separator{
private:
std::vector<int> offsets_;
unsigned int curoffset_;
bool bwrapoffsets_;
bool breturnpartiallast_;
public:
template<typename Iter>
offset_separator(Iter begin,Iter end,bool bwrapoffsets = true,
bool breturnpartiallast = true)
:offsets_(begin,end),curoffset_(0),
bwrapoffsets_(bwrapoffsets),breturnpartiallast_(breturnpartiallast){}
offset_separator()
:offsets_(1,1),curoffset_(),
bwrapoffsets_(true),breturnpartiallast_(true){}
void reset(){
curoffset_ = 0;
}
template<typename InputIterator, typename Token>
bool operator()(InputIterator& next, InputIterator end, Token& tok)
{
assert(!offsets_.empty());
tok = Token();
if(next==end)
return false;
if(curoffset_==offsets_.size())
if(bwrapoffsets_)
curoffset_=0;
else
return false;
int c = offsets_[curoffset_];
int i = 0;
for(; i < c; ++i){
if(next==end)break;
tok+=*next++;
}
if(!breturnpartiallast_)
if(i < (c-1) )
return false;
++curoffset_;
return true;
}
};
//===========================================================================
// The char_delimiters_separator class, which is a model of TokenizerFunction.
// char_delimiters_separator breaks a string into tokens based on character
// delimiters. There are 2 types of delimiters. returnable delimiters can
// be returned as tokens. These are often punctuation. nonreturnable
// delimiters cannot be returned as tokens. These are often whitespace
template <class Char, class Traits = std::char_traits<Char> >
class char_delimiters_separator{
private:
typedef std::basic_string<Char,Traits> string_type;
string_type returnable_;
string_type nonreturnable_;
bool return_delims_;
bool no_ispunct_;
bool no_isspace_;
bool is_ret(Char E)const
{
if(returnable_.length())
return returnable_.find(E) != string_type::npos;
else{
if(no_ispunct_)return false;
using namespace std;
return ispunct(E)!=0;
}
}
bool is_nonret(Char E)const
{
if(nonreturnable_.length())
return nonreturnable_.find(E) != string_type::npos;
else{
if(no_isspace_)return false;
using namespace std;
return isspace(E)!=0;
}
}
public:
explicit char_delimiters_separator(bool return_delims = false,
const Char * returnable = 0,const char * nonreturnable = 0)
:nonreturnable_(nonreturnable?nonreturnable:""),
returnable_(returnable?returnable:""),no_ispunct_(returnable),
no_isspace_(nonreturnable),return_delims_(return_delims){}
void reset(){}
public:
template<class InputIterator,class Token>
bool operator()(InputIterator& next, InputIterator end,Token& tok){
tok = Token();
// skip past all nonreturnable delims
// skip past the returnable only if we are not returning delims
for(;next!=end && ( is_nonret(*next) || (is_ret(*next)
&& !return_delims_ ) );++next){}
if(next == end){
return false;
}
// if we are to return delims and we are one a returnable one
// move past it and stop
if(is_ret(*next) && return_delims_){
tok+=*next;
++next;
}
else
// append all the non delim characters
for(;next!=end && !is_nonret(*next) && !is_ret(*next);++next)
tok+=*next;
return true;
}
};
} //namespace boost
#endif
include/boost/token_iterator.hpp
+111
View File
@@ -0,0 +1,111 @@
// Boost token_iterator.hpp -------------------------------------------------//
// Copyright John R. Bandela 2001
// Permission to copy, use, modify, sell and distribute this software
// is granted provided this copyright notice appears in all
// copies. This software is provided "as is" without express or
// implied warranty, and with no claim as to its suitability for any
// purpose.
// See http://www.boost.org for updates, documentation, and revision history.
#ifndef BOOST_TOKENIZER_POLICY_JRB051801_HPP_
#define BOOST_TOKENIZER_POLICY_JRB051801_HPP_
#include<boost/iterator_adaptors.hpp>
#include<boost/token_functions.hpp>
#include<utility>
#include<cassert>
namespace boost {
namespace detail{
// The base "iterator" for iterator adapter
template<class It>
class token_iterator_base
{
public:
std::pair<It,It> p_;
bool valid_;
token_iterator_base():p_(It(),It()),valid_(false){}
token_iterator_base(const It& b , const It& e )
:p_(b,e),valid_(false){}
operator It(){return p_.first;}
template<class T>
token_iterator_base(const token_iterator_base<T>& other)
:p_(other.p_),valid_(other.valid_){}
};
template<class Type, class TokenizerFunc>
class tokenizer_policy{
private:
TokenizerFunc func_;
Type tok_;
public:
tokenizer_policy(){}
tokenizer_policy(const TokenizerFunc& f):func_(f){};
template<class Base>
void initialize(Base& b){
if(b.valid_) return;
func_.reset();
b.valid_ = (b.p_.first != b.p_.second)?
func_(b.p_.first,b.p_.second,tok_):false;
}
template<class Base>
bool equal(const Base& a, const Base& b) const{
return (a.valid_ && b.valid_)?(a.p_==b.p_):(a.valid_==b.valid_);
}
template<class Reference, class Base>
const Type& dereference(boost::type<Reference>, const Base& a) const{
using namespace std;
assert(a.valid_);
return tok_;
}
template <class Base>
void increment(Base& b){
using namespace std;
assert(b.valid_);
b.valid_ = func_(b.p_.first,b.p_.second,tok_);
}
};
} // namespace detail
template <
class TokenizerFunc = char_delimiters_separator<char>,
class Iterator = std::string::const_iterator,
class Type = std::string
>
class token_iterator_generator {
private:
typedef Type value_type;
typedef detail::tokenizer_policy<Type, TokenizerFunc> policies;
typedef detail::token_iterator_base<Iterator> base;
typedef typename boost::detail::non_bidirectional_category<
Iterator>::type category;
public:
typedef boost::iterator_adaptor<base,policies,value_type,
const value_type&,const value_type*,category,std::ptrdiff_t> type;
};
// Type has to be first because it needs to be explicitly specified
// because there is no way the function can deduce it.
template<class Type, class Iterator, class TokenizerFunc>
typename token_iterator_generator<TokenizerFunc,Iterator,Type>::type
make_token_iterator(Iterator begin, Iterator end,const TokenizerFunc& fun){
typedef typename
token_iterator_generator<TokenizerFunc,Iterator,Type>::type ret_type;
detail::token_iterator_base<Iterator> b(begin,end);
detail::tokenizer_policy<Type,TokenizerFunc> f(fun);
return ret_type(b,f);
}
} // namespace boost
#endif
include/boost/tokenizer.hpp
+95
View File
@@ -0,0 +1,95 @@
// Boost tokenizer.hpp ------------------------------------------------------//
// © Copyright Jeremy Siek and John R. Bandela 2001.
// Permission to copy, use, modify, sell and distribute this software
// is granted provided this copyright notice appears in all
// copies. This software is provided "as is" without express or
// implied warranty, and with no claim as to its suitability for any
// purpose.
// See http://www.boost.org for updates, documentation, and revision history.
#ifndef BOOST_TOKENIZER_JRB051801_HPP_
#define BOOST_TOKENIZER_JRB051801_HPP_
#include <boost/token_iterator.hpp>
#include <cassert>
namespace boost {
//===========================================================================
// A container-view of a tokenized "sequence"
template <
class TokenizerFunc = char_delimiters_separator<char>,
class Iterator = std::string::const_iterator,
class Type = std::string
>
class tokenizer {
private:
typedef detail::tokenizer_policy<Type, TokenizerFunc> Pol;
typedef detail::token_iterator_base<Iterator> TBase;
typedef token_iterator_generator<TokenizerFunc,Iterator,Type> TGen;
// It seems that MSVC does not like the unqualified use of iterator,
// Thus we use iter internally when it is used unqualified and
// the users of this class will always qualify iterator.
typedef typename TGen::type iter;
public:
typedef iter iterator;
typedef iter const_iterator;
typedef Type value_type;
typedef value_type& reference;
typedef const value_type& const_reference;
typedef value_type* pointer;
typedef const pointer const_pointer;
typedef void size_type;
typedef void difference_type;
tokenizer(Iterator first, Iterator last,
const TokenizerFunc& f = TokenizerFunc())
:first_(first), last_(last), f_(f) { }
template<class Container>
tokenizer(const Container& c,const TokenizerFunc& f = TokenizerFunc())
: first_(c.begin()), last_(c.end()), f_(f) { }
void assign(Iterator first, Iterator last){
first_ = first;
last_ = last;
}
void assign(Iterator first, Iterator last, const TokenizerFunc& f){
assign(first,last);
f_ = f;
}
template<class Container>
void assign(const Container& c){
assign(c.begin(),c.end());
}
template<class Container>
void assign(const Container& c, const TokenizerFunc& f){
assign(c.begin(),c.end(),f);
}
iter begin() const { return iter(TBase(first_,last_),Pol(f_)); }
iter end() const { return iter(TBase(last_,last_),Pol(f_)); }
private:
Iterator first_;
Iterator last_;
TokenizerFunc f_;
};
} // namespace boost
#endif
index.htm
+71
View File
@@ -0,0 +1,71 @@
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>Boost Tokenizer Overview</title>
<!--
-- Copyright (c) Jeremy Siek 2000
--
-- Permission to use, copy, modify, distribute and sell this software
-- and its documentation for any purpose is hereby granted without fee,
-- provided that the above copyright notice appears in all copies and
-- that both that copyright notice and this permission notice appear
-- in supporting documentation. Jeremy Siek makes no
-- representations about the suitability of this software for any
-- purpose. It is provided "as is" without express or implied warranty.
-->
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
vlink="#551A8B" alink="#FF0000">
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
height="86"> <br>
</p>
<h1 align="center">Table Of Contents</h1>
<p align="left">&nbsp;</p>
<h2 align="left"><a href="introduc.htm">Introduction</a></h2>
<h2 align="left">Containers and Iterators</h2>
<ul>
<li><h3 align="left"><a href="tokenizer.htm">tokenizer</a></h3>
</li>
<li><h3 align="left"><a href="token_iterator.htm">token
iterator</a></h3>
</li>
</ul>
<h2><a href="tokenizerfunction.htm">TokenizerFunction Concept</a></h2>
<h2>TokenizerFunction Models</h2>
<ul>
<li><h3><a href="char_delimiters_separator.htm">char_delimiters_separator</a></h3>
</li>
<li><h3><a href="escaped_list_separator.htm">escaped_list_separator</a></h3>
</li>
<li><h3><a href="offset_separator.htm">offset_separator</a></h3>
</li>
</ul>
<p>&nbsp;</p>
<hr>
<p>© Copyright John R. Bandela 2001. Permission to copy, use,
modify, sell and distribute this document is granted provided
this copyright notice appears in all copies. This document is
provided &quot;as is&quot; without express or implied warranty,
and with no claim as to its suitability for any purpose.</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
</body>
</html>
introduc.htm
+111
View File
@@ -0,0 +1,111 @@
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>Introduction</title>
</head>
<body bgcolor="#FFFFFF">
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
height="86"> <br>
</p>
<h1 align="center">Introduction</h1>
<p align="left">The boost Tokenizer package provides a flexible
and easy to use way to break of a string or other character
sequence into a series of tokens. Below is a simple example that
will break up a phrase into words.</p>
<div align="left">
<pre>// simple_example_1.cpp
#include&lt;iostream&gt;
#include&lt;boost/tokenizer.hpp&gt;
#include&lt;string&gt;
int main(){
using namespace std;
using namespace boost;
string s = &quot;This is, a test&quot;;
tokenizer&lt;&gt; tok(s);
for(tokenizer&lt;&gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout &lt;&lt; *beg &lt;&lt; &quot;\n&quot;;
}
}</pre>
</div>
<p align="left">You can choose how the string gets broken up. You
do this by specifying the TokenizerFunction. If you do not
specify anything, the default TokenizerFunction is
char_delimiters_separator&lt;char&gt; which defaults to breaking
up a string based on space and punctuation. Here is an example of
using another TokenizerFunction called escaped_list_separator.
This TokenizerFunction parses a superset of comma separated value
(csv) lines. The format looks like this</p>
<p align="left">Field 1,&quot;putting quotes around fields,
allows commas&quot;,Field 3</p>
<p align="left">Below is an example that will break the previous
line into its 3 fields</p>
<div align="left">
<pre>// simple_example_2.cpp
#include&lt;iostream&gt;
#include&lt;boost/tokenizer.hpp&gt;
#include&lt;string&gt;
int main(){
using namespace std;
using namespace boost;
string s = &quot;Field 1,\&quot;putting quotes around fields, allows commas\&quot;,Field 3&quot;;
tokenizer&lt;escaped_list_separator&lt;char&gt; &gt; tok(s);
for(tokenizer&lt;escaped_list_separator&lt;char&gt; &gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout &lt;&lt; *beg &lt;&lt; &quot;\n&quot;;
}
}</pre>
</div>
<p align="left">Finally, for some TokenizerFunctions you have to
pass in something into the constructor in order to do anything
interesting. An example is offset_separator. This class breaks a
string into tokens based on offsets for example</p>
<p align="left">12252001 when parsed using offsets of 2,2,4
becomes 12 25 2001. Below is an example to parse this.</p>
<div align="left">
<pre>// simple_example_3.cpp
#include&lt;iostream&gt;
#include&lt;boost/tokenizer.hpp&gt;
#include&lt;string&gt;
int main(){
using namespace std;
using namespace boost;
string s = &quot;12252001&quot;;
int offsets[] = {2,2,4};
offset_separator f(offsets, offsets+3);
tokenizer&lt;offset_separator&gt; tok(s,f);
for(tokenizer&lt;offset_separator&gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout &lt;&lt; *beg &lt;&lt; &quot;\n&quot;;
}
}</pre>
</div>
<p align="left">&nbsp;</p>
<hr>
<p>© Copyright John R. Bandela 2001. Permission to copy, use,
modify, sell and distribute this document is granted provided
this copyright notice appears in all copies. This document is
provided &quot;as is&quot; without express or implied warranty,
and with no claim as to its suitability for any purpose.</p>
<p align="left">&nbsp;</p>
</body>
</html>
offset_separator.htm
+129
View File
@@ -0,0 +1,129 @@
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>Boost Offset Separator</title>
<!--
-- Copyright © John Bandela 2001
--
-- Permission to use, copy, modify, distribute and sell this software
-- and its documentation for any purpose is hereby granted without fee,
-- provided that the above copyright notice appears in all copies and
-- that both that copyright notice and this permission notice appear
-- in supporting documentation. Jeremy Siek makes no
-- representations about the suitability of this software for any
-- purpose. It is provided "as is" without express or implied warranty.
-->
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
vlink="#551A8B" alink="#FF0000">
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
height="86"> <br>
</p>
<h1 align="center">Offset Separator</h1>
<pre>
class offset_separator
</pre>
<p>The <tt>offset_separator</tt> class is an implementation of
the <a href="tokenizerfunction.htm">TokenizerFunction</a> concept
that can be used with the <a href="tokenizer.htm">tokenizer</a>
class to break text up into tokens. The <tt>offset_separator</tt>
breaks a sequence of <tt>Char</tt>'s into strings based on a
sequence of offsets. For example, if you had the string &quot;12252001&quot;
and offsets (2,2,4) it would break the string into 12 25 2001.
Here is an example.</p>
<h2>Example</h2>
<pre>// simple_example_3.cpp
#include&lt;iostream&gt;
#include&lt;boost/tokenizer.hpp&gt;
#include&lt;string&gt;
int main(){
using namespace std;
using namespace boost;
string s = &quot;12252001&quot;;
int offsets[] = {2,2,4};
offset_separator f(offsets, offsets+3);
tokenizer&lt;offset_separator&gt; tok(s,f);
for(tokenizer&lt;offset_separator&gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout &lt;&lt; *beg &lt;&lt; &quot;\n&quot;;
}
}</pre>
<p>&nbsp;</p>
<h2>Construction and Usage</h2>
<p>The offset_separator has 1 constructor of interest. (The
default constructor is just there to make some compilers happy).
The declaration is below</p>
<pre>template&lt;typename Iter&gt;
offset_separator(Iter begin,Iter end,bool bwrapoffsets = true, bool breturnpartiallast = true)
</pre>
<table border="1">
<tr>
<td><p align="center"><strong>Parameter</strong></p>
</td>
<td><p align="center"><strong>Description</strong></p>
</td>
</tr>
<tr>
<td>begin, end</td>
<td>Specify the sequence of integer offsets.</td>
</tr>
<tr>
<td>bwrapoffsets</td>
<td>Tells whether to wrap around to the beginning of the
offsets when the all the offsets have been used. For
example the string &quot;1225200101012002&quot; with
offsets (2,2,4) with bwrapoffsets to true, would parse to
12 25 2001 01 01 2002. With bwrapoffsets to false, it
would parse to 12 25 2001 and then stop because all the
offsets have been used.</td>
</tr>
<tr>
<td>breturnpartiallast</td>
<td>Tells whether, when the parsed sequence terminates
before yielding the number of characters in the current
offset, to create a token with what was parsed, or to
ignore it. For example the string &quot;122501&quot; with
offsets (2,2,4) with breturnpartiallast set to true will
parse to 12 25 01. With it set to false, it will parse to
12 25 and then will stop because there are only 2
characters left in the sequence instead of the 4 that
should have been there.</td>
</tr>
</table>
<p>To use this class, pass an object of it anywhere a
TokenizerFunction is required. If you default constructruct the
object, it will just return every character in the parsed
sequence as a token. (ie it defaults to an offset of 1, and
bwrapoffsets is true).</p>
<p>&nbsp;</p>
<h2>Model of</h2>
<p><a href="tokenizerfunction.htm">TokenizerFunction</a> </p>
<hr>
<p>© Copyright John R. Bandela 2001. Permission to copy, use,
modify, sell and distribute this document is granted provided
this copyright notice appears in all copies. This document is
provided &quot;as is&quot; without express or implied warranty,
and with no claim as to its suitability for any purpose.</p>
</body>
</html>
simple_example_1.cpp
+15
View File
@@ -0,0 +1,15 @@
// simple_example_1.cpp
#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>
int main(){
using namespace std;
using namespace boost;
string s = "This is, a test";
tokenizer<> tok(s);
for(tokenizer<>::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout << *beg << "\n";
}
}
simple_example_2.cpp
+15
View File
@@ -0,0 +1,15 @@
// simple_example_2.cpp
#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>
int main(){
using namespace std;
using namespace boost;
string s = "Field 1,\"putting quotes around fields, allows commas\",Field 3";
tokenizer<escaped_list_separator<char> > tok(s);
for(tokenizer<escaped_list_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout << *beg << "\n";
}
}
simple_example_3.cpp
+16
View File
@@ -0,0 +1,16 @@
// simple_example_3.cpp
#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>
int main(){
using namespace std;
using namespace boost;
string s = "12252001";
int offsets[] = {2,2,4};
offset_separator f(offsets, offsets+3);
tokenizer<offset_separator> tok(s,f);
for(tokenizer<offset_separator>::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout << *beg << "\n";
}
}
simple_example_4.cpp
+15
View File
@@ -0,0 +1,15 @@
// simple_example_4.cpp
#include<iostream>
#include<boost/tokenizer.hpp>
#include<string>
int main(){
using namespace std;
using namespace boost;
string s = "This is, a test";
tokenizer<char_delimiters_separator<char> > tok(s);
for(tokenizer<char_delimiters_separator<char> >::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout << *beg << "\n";
}
}
simple_example_5.cpp
+20
View File
@@ -0,0 +1,20 @@
/// simple_example_5.cpp
#include<iostream>
#include<boost/token_iterator.hpp>
#include<string>
int main(){
using namespace std;
using namespace boost;
string s = "12252001";
int offsets[] = {2,2,4};
offset_separator f(offsets, offsets+3);
typedef token_iterator_generator<offset_separator>::type Iter;
Iter beg = make_token_iterator<string>(s.begin(),s.end(),f);
Iter end = make_token_iterator<string>(s.end(),s.end(),f);
// The above statement could also have been what is below
// Iter end;
for(;beg!=end;++beg){
cout << *beg << "\n";
}
}
token_iterator.htm
+155
View File
@@ -0,0 +1,155 @@
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>Boost Token Iterator</title>
<!--
-- Copyright © John Bandela 2001
--
-- Permission to use, copy, modify, distribute and sell this software
-- and its documentation for any purpose is hereby granted without fee,
-- provided that the above copyright notice appears in all copies and
-- that both that copyright notice and this permission notice appear
-- in supporting documentation. Jeremy Siek makes no
-- representations about the suitability of this software for any
-- purpose. It is provided "as is" without express or implied warranty.
-->
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
vlink="#551A8B" alink="#FF0000">
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
height="86"> <br>
</p>
<h1 align="center">Token Iterator</h1>
<pre>
template &lt;
class TokenizerFunc = char_delimiters_separator&lt;char&gt;,
class Iterator = std::string::const_iterator,
class Type = std::string
&gt;
class token_iterator_generator </pre>
<pre>template&lt;class Type, class Iterator, class TokenizerFunc&gt;
typename token_iterator_generator&lt;TokenizerFunc,Iterator,Type&gt;::type
make_token_iterator(Iterator begin, Iterator end,const TokenizerFunc&amp; fun)
</pre>
<p>The token iterator serves to provide an iterator view of the
tokens in a parsed sequence.</p>
<h2>Example</h2>
<pre>
/// simple_example_5.cpp
#include&lt;iostream&gt;
#include&lt;boost/token_iterator.hpp&gt;
#include&lt;string&gt;
int main(){
using namespace std;
using namespace boost;
string s = &quot;12252001&quot;;
int offsets[] = {2,2,4};
offset_separator f(offsets, offsets+3);
typedef token_iterator_generator&lt;offset_separator&gt;::type Iter;
Iter beg = make_token_iterator&lt;string&gt;(s.begin(),s.end(),f);
Iter end = make_token_iterator&lt;string&gt;(s.end(),s.end(),f);
// The above statement could also have been what is below
// Iter end;
for(;beg!=end;++beg){
cout &lt;&lt; *beg &lt;&lt; &quot;\n&quot;;
}
}
</pre>
<pre>
</pre>
<p>&nbsp;</p>
<h3>Template Parameters</h3>
<table border="1">
<tr>
<th>Parameter</th>
<th>Description</th>
</tr>
<tr>
<td><tt>TokenizerFunc</tt></td>
<td>The TokenizerFunction used to parse the sequence.</td>
</tr>
<tr>
<td><tt>Iterator</tt></td>
<td>The type of the iterator the specifies the sequence.</td>
</tr>
<tr>
<td><tt>Type</tt></td>
<td>The type of the token, typically string.</td>
</tr>
</table>
<h2>Model of</h2>
<p>The category of Iterator, up to and including Forward Iterator.
Anything higher will get scaled down to Forward Iterator.</p>
<h2>Related Types</h2>
<table border="1">
<tr>
<td><p align="center"><strong>Type</strong></p>
</td>
<td><p align="center"><strong>Remarks</strong></p>
</td>
</tr>
<tr>
<td>token_iterator_generator::type</td>
<td>The type of the token iterator.</td>
</tr>
</table>
<h2>Creation</h2>
<pre>template&lt;class Type, class Iterator, class TokenizerFunc&gt;
typename token_iterator_generator&lt;TokenizerFunc,Iterator,Type&gt;::type
make_token_iterator(Iterator begin, Iterator end,const TokenizerFunc&amp; fun)</pre>
<table border="1">
<tr>
<td><p align="center"><strong>Parameter</strong></p>
</td>
<td><p align="center"><strong>Description</strong></p>
</td>
</tr>
<tr>
<td>begin</td>
<td>The beginning of the sequence to be parsed.</td>
</tr>
<tr>
<td>end</td>
<td>Past the end of the sequence to be parsed.</td>
</tr>
<tr>
<td>fun</td>
<td>A functor that is a model of TokenizerFunction</td>
</tr>
</table>
<p>&nbsp;</p>
<hr>
<p>© Copyright John R. Bandela 2001. Permission to copy, use,
modify, sell and distribute this document is granted provided
this copyright notice appears in all copies. This document is
provided &quot;as is&quot; without express or implied warranty,
and with no claim as to its suitability for any purpose.</p>
</body>
</html>
tokenizer.htm
+204
View File
@@ -0,0 +1,204 @@
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>Boost Tokenizer Class</title>
<!--
-- Copyright © John Bandela 2001
--
-- Permission to use, copy, modify, distribute and sell this software
-- and its documentation for any purpose is hereby granted without fee,
-- provided that the above copyright notice appears in all copies and
-- that both that copyright notice and this permission notice appear
-- in supporting documentation. Jeremy Siek makes no
-- representations about the suitability of this software for any
-- purpose. It is provided "as is" without express or implied warranty.
-->
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
vlink="#551A8B" alink="#FF0000">
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
height="86"> <br>
</p>
<h1 align="center">Tokenizer Class</h1>
<pre> template &lt;
class TokenizerFunc = char_delimiters_separator&lt;char&gt;,
class Iterator = std::string::const_iterator,
class Type = std::string
&gt;
class tokenizer
</pre>
<p>The tokenizer class provides a container view of a series of
tokens contained in a sequence. You set the sequence to parse and
the TokenizerFunction to use to parse the sequence either upon
construction or using the assign member function. Note: No
parsing is actually done upon construction. Parsing is done on
demand as the tokens are accessed via the iterator provided by
begin.</p>
<h2>Example</h2>
<pre>// simple_example_1.cpp
#include&lt;iostream&gt;
#include&lt;boost/tokenizer.hpp&gt;
#include&lt;string&gt;
int main(){
using namespace std;
using namespace boost;
string s = &quot;This is, a test&quot;;
tokenizer&lt;&gt; tok(s);
for(tokenizer&lt;&gt;::iterator beg=tok.begin(); beg!=tok.end();++beg){
cout &lt;&lt; *beg &lt;&lt; &quot;\n&quot;;
}
}
</pre>
<p>&nbsp;</p>
<h3>Template Parameters</h3>
<table border="1">
<tr>
<th>Parameter</th>
<th>Description</th>
</tr>
<tr>
<td><tt>TokenizerFunc</tt></td>
<td>The TokenizerFunction used to parse the sequence.</td>
</tr>
<tr>
<td><tt>Iterator</tt></td>
<td>The type of the iterator the specifies the sequence.</td>
</tr>
<tr>
<td><tt>Type</tt></td>
<td>The type of the token, typically string.</td>
</tr>
</table>
<p>&nbsp;</p>
<h2>Related Types</h2>
<table border="1">
<tr>
<td><p align="center"><strong>Type</strong></p>
</td>
<td><p align="center"><strong>Remarks</strong></p>
</td>
</tr>
<tr>
<td>iterator</td>
<td>The type returned by begin and end. Note: the
category of iterator will be at most ForwardIterator. It
will be InputIterator if the Iterator template parameter
is an InputIterator. For any other category, it will be
ForwardIterator.</td>
</tr>
<tr>
<td>const_iterator</td>
<td>Same type as iterator.</td>
</tr>
<tr>
<td>value_type</td>
<td>Same type as the template parameter Type</td>
</tr>
<tr>
<td>reference</td>
<td>Same type as value_type&amp;</td>
</tr>
<tr>
<td>const_reference</td>
<td>Same type as const reference</td>
</tr>
<tr>
<td>pointer</td>
<td>Same type as value_type*</td>
</tr>
<tr>
<td>const_pointer</td>
<td>Same type as const pointer</td>
</tr>
<tr>
<td>size_type</td>
<td>void</td>
</tr>
<tr>
<td>difference_type</td>
<td>void</td>
</tr>
</table>
<p>&nbsp;</p>
<h2>Construction and Member Functions</h2>
<pre>tokenizer(Iterator first, Iterator last,const TokenizerFunc&amp; f = TokenizerFunc())
template&lt;class Container&gt;
tokenizer(const Container&amp; c,const TokenizerFunc&amp; f = TokenizerFunc())
void assign(Iterator first, Iterator last)
void assign(Iterator first, Iterator last, const TokenizerFunc&amp; f)
template&lt;class Container&gt;
void assign(const Container&amp; c)
template&lt;class Container&gt;
void assign(const Container&amp; c, const TokenizerFunc&amp; f)
iterator begin() const
iterator end() const
</pre>
<table border="1">
<tr>
<td><p align="center"><strong>Parameter</strong></p>
</td>
<td><p align="center"><strong>Description</strong></p>
</td>
</tr>
<tr>
<td>c</td>
<td>A container that contains the sequence to parse. Note:
c.begin() and c.end() must be convertible to the template
parameter Iterator.</td>
</tr>
<tr>
<td>f</td>
<td>A functor that is a model of TokenizerFunction that
will be used to parse the sequence.</td>
</tr>
<tr>
<td>first</td>
<td>The iterator that represents the beginning position
in the sequence to be parsed.</td>
</tr>
<tr>
<td>last</td>
<td>The iterator that represents the past the end
position in the sequence to be parsed.</td>
</tr>
</table>
<p>&nbsp;</p>
<hr>
<p>© Copyright John R. Bandela 2001. Permission to copy, use,
modify, sell and distribute this document is granted provided
this copyright notice appears in all copies. This document is
provided &quot;as is&quot; without express or implied warranty,
and with no claim as to its suitability for any purpose.</p>
</body>
</html>
tokenizer_policy.htm
+142
View File
@@ -0,0 +1,142 @@
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>Boost Tokenizer Policy</title>
<!--
-- Copyright (c) Jeremy Siek 2000
--
-- Permission to use, copy, modify, distribute and sell this software
-- and its documentation for any purpose is hereby granted without fee,
-- provided that the above copyright notice appears in all copies and
-- that both that copyright notice and this permission notice appear
-- in supporting documentation. Jeremy Siek makes no
-- representations about the suitability of this software for any
-- purpose. It is provided "as is" without express or implied warranty.
-->
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
vlink="#551A8B" alink="#FF0000">
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
height="86"> <br>
</p>
<h1><a name="tokenizer-class"></a> </h1>
<pre>
tokenizer_policy&lt;Type, <a href="TokenizerFunction.htm">TkFunc</a>&gt;
detail::tokenizer_base&lt;Iterator&gt;</pre>
<p>&nbsp;</p>
<p>The <tt>tokenizer_policy</tt> class provides a policy class
for <tt>iterator_adapter</tt> that allows it to parse an input
range (typically a range of characters) into a sequece of <tt>Type</tt>
objects (typically string) . The base type for the adapted
iterator is tokenizer_base&lt;Iterator&gt; where Iterator is the
iterator type of the input range. The bulk of the tokenizing work
is not done in this class, but is left up to the <tt>TkFunc</tt>.
The <a href="punct_space_separator.htm"><tt>punct_space_separator</tt></a>
function object breaks up a sequence of characters based on
punctuation and white space</p>
<h3>Example</h3>
<div dir="ltr">
<pre>using namespace std;
using namespace boost;
const string test_string = &quot;This,,is, a.test..&quot;;
// Use the bare iterator adaptor version
punct_space_separator&lt;char&gt; tkfun;
copy(make_token_iterator&lt;string&gt;(test_string.begin(),test_string.end(),tkfun)
,make_token_iterator&lt;string&gt;(test_string.end(),test_string.end(),tkfun),
ostream_iterator&lt;string&gt;(cout,&quot;|&quot;));
cout &lt;&lt; &quot;\n&quot;;
</pre>
</div><div dir="ltr">
<pre>The output is </pre>
</div>
<pre>This|is|a|test|
</pre>
<h3>Template Parameters</h3>
<table border="1">
<tr>
<th>Parameter</th>
<th>Description</th>
</tr>
<tr>
<td><tt>Iterator</tt></td>
<td>The iterator type for the input range. </td>
</tr>
<tr>
<td><tt>TkFunc</tt></td>
<td>The function object that finds the next token. This
type must be a model of <a href="TokenizerFunction.htm">TokenizerFunction</a>.</td>
</tr>
<tr>
<td><tt>Token</tt></td>
<td>The type of the token objects that are produced. The
type must be DefaultConstructible and must have an <tt>assign()</tt>
function that takes an iterator range.</td>
</tr>
</table>
<h3>Model of</h3>
<p>The adapted iterator will be either an Input Iterator or a
Forward Iterator, depending on whether the Iterator type of the
input range is an Input Iterator or not.</p>
<h3>Where Defined</h3>
<p><a href="../boost/tokenizer.hpp"><tt>boost/tokenizer_policy.hpp</tt></a>
</p>
<p>&nbsp;</p>
<h3>Generator</h3>
<pre>template &lt;class Type, class Iterator, class TkFunc&gt;
class token_iterator_generator
</pre>
<h3>Creator Function</h3>
<pre>template&lt;class Type, class Iterator, class TkFunc &gt;
token_iterator_generator&lt;Type,Iterator,TkFunc&gt;::type
make_token_iterator(Iterator begin, Iterator end,const TkFunc&amp; fun){
</pre>
<p>To use this function, you have to specify the Type of the
token object (typically string). For example</p>
<pre>token_iterator_generator&lt;std::string&gt;(...)</pre>
<p>&nbsp;</p>
<p>&nbsp;</p>
<hr>
<h3>See Also</h3>
<p><tt>tokenizer</tt></p>
<p><tt>token_iterator</tt></p>
<p><a href="token_separator.htm"><tt>token_separator</tt></a> </p>
<pre>iterator_adaptor</pre>
</body>
</html>
tokenizerfunction.htm
+145
View File
@@ -0,0 +1,145 @@
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html; charset=iso-8859-1">
<meta name="GENERATOR" content="Microsoft FrontPage Express 2.0">
<title>TokenizerFunction Concept</title>
</head>
<body bgcolor="#FFFFFF" text="#000000" link="#0000EE"
vlink="#551A8B" alink="#FF0000">
<p><img src="../../c++boost.gif" alt="C++ Boost" width="277"
height="86"> </p>
<h1 align="center">TokenizerFunction Concept</h1>
<p>A TokenizerFunction is a functor whose purpose is to parse a
given sequence until exactly 1 token has been found or the end is
reached. It then updates the token, and informs the caller of the
location in the sequence of the next element immediately after
the last element of the sequence that was parsed for the current
token. </p>
<h2>Refinement of</h2>
<p>Assignable, CopyConstructable</p>
<h2>Notation</h2>
<table border="1">
<tr>
<td valign="top"><tt>X</tt> </td>
<td valign="top">A type that is a model of
TokenizerFunction</td>
</tr>
<tr>
<td valign="top"><tt>func</tt> </td>
<td valign="top">Object of type <tt>X</tt> </td>
</tr>
<tr>
<td valign="top"><tt>tok</tt></td>
<td valign="top">Object of Token</td>
</tr>
<tr>
<td>next</td>
<td>iterator that points to the first unparsed element of
the sequence being parsed</td>
</tr>
<tr>
<td>end</td>
<td>iterator that points to the past the end of the
sequence being parsed</td>
</tr>
</table>
<h2>Definitions</h2>
<p>A token is the result of parsing a sequence.</p>
<h2>Valid expressions</h2>
<p>In addition to the expression in Assignable and
CopyConstructable the following expressions are valid</p>
<table border="1">
<tr>
<th>Name </th>
<th>Expression </th>
<th>Return type </th>
</tr>
<tr>
<td valign="top">Functor</td>
<td valign="top"><tt>func(next, end, tok)</tt></td>
<td valign="top"><tt>bool</tt></td>
</tr>
<tr>
<td valign="top">reset</td>
<td valign="top"><tt>reset()</tt></td>
<td valign="top"><tt>void</tt></td>
</tr>
</table>
<h2>Expression semantics</h2>
<p>In addition to the expression semantics in Assignable and
CopyConstructable, TokenizerFunction has the following expression
semantcs</p>
<table border="1">
<tr>
<th>Name </th>
<th>Expression </th>
<th>Precondition </th>
<th>Semantics </th>
<th>Postcondition </th>
</tr>
<tr>
<td>operator()</td>
<td><tt>func(next, end, tok)</tt></td>
<td><tt>next</tt> and <tt>end</tt> are valid iterators to
the same sequence. next is a reference the function is
free to modify. tok is constructed. </td>
<td>The return value indicates whether a new token was
found in the sequence (next,end]</td>
<td>If the return value is true, the new token is
assigned to tok. next is always updated to the position
where parsing should start on the subsequent call.</td>
</tr>
<tr>
<td>reset</td>
<td><tt>reset()</tt></td>
<td><tt>None</tt></td>
<td>Clears out all state variables that are used by the
object in parsing the current sequence.</td>
<td>A new sequence to parse can be given.</td>
</tr>
</table>
<h2>Complexity guarantees</h2>
<p>No guarantees. Models of TokenizerFunction are free to define
their own complexity</p>
<h2>Models</h2>
<p><a href="escaped_list_separator.htm">escaped_list_separator</a></p>
<p><a href="offset_separator.htm">offset_separator</a></p>
<p><a href="char_delimiters_separator.htm">char_delimiters_separator</a></p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<hr>
<p>© Copyright John R. Bandela 2001. Permission to copy, use,
modify, sell and distribute this document is granted provided
this copyright notice appears in all copies. This document is
provided &quot;as is&quot; without express or implied warranty,
and with no claim as to its suitability for any purpose.</p>
</body>
</html>