Navigation

Changeset 754

Timestamp:

01/11/06 17:33:07 (3 years ago)

Author:

kris

Message:

Overhaul of the tokenizers -- now use iterators instead.

Files:

r749	r754
	1
1	2	private import mango.io.Console,
2	3	mango.io.FileConduit;
3	4
4		private import mango.text.Line~~Stream~~;
	5	private import mango.text.LineIterator;
5	6
6	7	/*******************************************************************************
7	8
8	9	Read a file line-by-line, sending each one to the console. This
9		illustrates how to bind a Conduit or a Buffer to a Tokenizer.
	10	illustrates how to bind a conduit to a text iterator. Iterators
	11	also support the binding of buffer and string instances.
10	12
11		Note that ~~Tokenize~~rs are templated for char, wchar and dchar ~
	13	Note that iterators are templated for char, wchar and dchar ~
12	14	this example uses char
13	15
…	…
21	23	auto file = new FileConduit (args[1]);
22	24
23		// create a ~~Tokenize~~r and bind it to the file
24		auto line = new Line~~Stream~~ (file);
	25	// create an iterator and bind it to the file
	26	auto line = new LineIterator (file);
25	27
26	28	// process file one line at a time
27	29	while (line.next)
28		Cout (line.get)~~.newline()~~;
	30	Cout (line.get) .newline;
29	31	}
30	32	else
31	33	Cout ("usage: lineio filename");
32	34	}
33

r741	r754
9	9	private import mango.io.Console;
10	10
11		private import mango.text.Quote~~dToken~~;
	11	private import mango.text.QuoteIterator;
12	12
13	13	void main()
…	…
22	22
23	23	// create quote-aware tokenizer for handling space-delimited tokens
24		auto token = new Quote~~dToken~~ (args, " \t");
	24	auto token = new QuoteIterator (args, " \t");
25	25
26	26	// scan and display trimmed tokens