All new
alle new
This commit is contained in:
erik committed
1 parent
da64472251
commit
5b20adee48
1605 files changed
+396517
-1674
No files matched your search
+248
@@ -0,0 +1,248 @@
|
||||
# lop -- parsing library for JavaScript
|
||||
|
||||
lop is a library to create parsers using parser combinators with helpful errors.
|
||||
|
||||
```javascript
|
||||
|
||||
function parse(tokens) {
|
||||
var parser = lop.Parser();
|
||||
return parser.parseTokens(expressionRule, tokens);
|
||||
}
|
||||
|
||||
// This rule is wrapped inside lop.rule to defer evaluation until
|
||||
// the rule is used -- otherwise, it would reference integerRule
|
||||
// and ifRule, which don't exist yet.
|
||||
var expressionRule = lop.rule(function() {
|
||||
return rules.firstOf("expression",
|
||||
integerRule,
|
||||
ifRule
|
||||
);
|
||||
});
|
||||
|
||||
var integerRule = rules.then(
|
||||
rules.tokenOfType("integer"),
|
||||
function(value) {
|
||||
return new IntegerNode(parseInt(value, 10));
|
||||
}
|
||||
);
|
||||
|
||||
var ifRule = rules.sequence(
|
||||
rules.token("keyword", "if"),
|
||||
rules.sequence.cut(),
|
||||
rules.sequence.capture(expressionRule),
|
||||
rules.token("keyword", "then"),
|
||||
rules.sequence.capture(expressionRule),
|
||||
rules.token("keyword", "else"),
|
||||
rules.sequence.capture(expressionRule)
|
||||
).map(function(condition, trueBranch, falseBranch) {
|
||||
return new IfNode(condition, trueBranch, falseBranch);
|
||||
});
|
||||
```
|
||||
|
||||
lop tries to provide helpful errors where possible. For instance, in `ifRule`
|
||||
as defined above, there is a cut following the keyword `if`. Before the cut,
|
||||
if we fail to match the input, we can backtrack -- in this case, we backtrack
|
||||
and see if another form of expression might match the input. However, after the
|
||||
cut, we prevent backtracking. Once we've see the keyword `if`, there's no doubt
|
||||
about which sort of expression this is, so if parsing fails later in this rule,
|
||||
there's no point in backtracking. This allows informative error messages to be
|
||||
generated: if we try to parse the string `"if 1 42 else 12"`, we get the error:
|
||||
|
||||
Error: File: /tmp/lop-example
|
||||
Line number: 1
|
||||
Character number: 6:
|
||||
Expected keyword "then"
|
||||
but got integer "42"
|
||||
|
||||
## Tokens
|
||||
|
||||
When using a parser built with lop, the input is an array of tokens. A token can be any value so long as it has the property `source`, which must be a `StringSourceRange`.
|
||||
|
||||
### Regex tokeniser
|
||||
|
||||
The easiest way to create a tokeniser is using lop's regex tokeniser.
|
||||
A regex tokeniser can be constructed by calling `new lop.RegexTokeniser(rules)`,
|
||||
where `rules` is a list of token rules.
|
||||
A token rule should have a `name` property that uniquely identifies that rule,
|
||||
and a `regex` property that is an instance of `RegExp` describing the token.
|
||||
|
||||
Calling `tokenise` with a string will return a list of tokens.
|
||||
Each token has three properties:
|
||||
|
||||
* `type`
|
||||
* `value`
|
||||
* `source`
|
||||
|
||||
The tokeniser will apply the regex from each rule in order at the current position.
|
||||
The current position is initially zero, the start of the string.
|
||||
The first rule with a matching regex is used to produce a token,
|
||||
with the token's `value` being the first capture of the regex,
|
||||
or `undefined` if the regex does not define any capture groups.
|
||||
The current position is incremented to the index of the first character unmatched by the regex.
|
||||
If no rule matches at the current position,
|
||||
a single character `unrecognisedCharacter` token is produced,
|
||||
and the current position is incremented by one.
|
||||
|
||||
For instance, to create a simple tokeniser that generates a stream of words tokens separated by whitespace tokens.
|
||||
|
||||
```javascript
|
||||
var lop = require("lop");
|
||||
|
||||
var rules = [
|
||||
{
|
||||
name: "identifier",
|
||||
regex: /(\s+)/
|
||||
},
|
||||
{
|
||||
name: "whitespace",
|
||||
regex: /(\S+)/
|
||||
}
|
||||
];
|
||||
var tokeniser = new lop.RegexTokeniser(rules);
|
||||
tokeniser.tokenise(input);
|
||||
```
|
||||
|
||||
### Custom tokenisers
|
||||
|
||||
You can also create your own tokeniser.
|
||||
For instance, to create a simple tokeniser that generates a stream of words tokens separated by whitespace tokens:
|
||||
|
||||
```javascript
|
||||
var StringSource = require("lop").StringSource;
|
||||
|
||||
function tokeniseString(string) {
|
||||
return tokenise(new StringSource(string, "raw string"));
|
||||
}
|
||||
|
||||
function tokenise(source) {
|
||||
var string = source.asString();
|
||||
var whitespaceRegex = /(\s+)/g;
|
||||
var result;
|
||||
var start = 0;
|
||||
var parts = [];
|
||||
|
||||
while ((result = whitespaceRegex.exec(source)) !== null) {
|
||||
parts.push({
|
||||
type: "word",
|
||||
value: string.substring(start, result.index),
|
||||
source: source.range(start, result.index)
|
||||
});
|
||||
parts.push({
|
||||
type: "whitespace",
|
||||
value: result[1],
|
||||
source: source.range(result.index, whitespaceRegex.lastIndex)
|
||||
});
|
||||
start = whitespaceRegex.lastIndex;
|
||||
}
|
||||
parts.push({
|
||||
type: "word",
|
||||
value: string.substring(start),
|
||||
source: source.range(start, string.length)
|
||||
});
|
||||
parts.push({
|
||||
type: "end",
|
||||
source: source.range(string.length, string.length)
|
||||
});
|
||||
return parts.filter(function(part) {
|
||||
return part.type !== "word" || part.value !== "";
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
lop also defines its own notion of a token. Each instance of `lop.Token` has a type, name, and source, similarly to most of the tokens that would be created by the token above. For instance, instead of:
|
||||
|
||||
{
|
||||
type: "word",
|
||||
value: value,
|
||||
source: source
|
||||
}
|
||||
|
||||
you could use:
|
||||
|
||||
new Token("word", value, source)
|
||||
|
||||
The main advantage of using `lop.Token` is that you can then use the rules `lop.rules.token` and `lop.rules.tokenOfType` (described later). If you don't use `lop.Token`, you must define your own atomic rules, but you can use the other rules without any modifications.
|
||||
|
||||
## Parser
|
||||
|
||||
To parse an array of tokens, you can call the method `parseTokens` on `lop.Parser`, passing in the parsing rule and the array of tokens. For instance, assuming we already have a `tokenise` function (the one above would do fine):
|
||||
|
||||
```javascript
|
||||
function parseSentence(source) {
|
||||
var tokens = tokenise(source);
|
||||
var parser = new lop.Parser();
|
||||
var parseResult = parser.parseTokens(sentenceRule, tokens);
|
||||
if (!parseResult.isSuccess()) {
|
||||
throw new Error("Failed to parse: " + describeFailure(parseResult));
|
||||
}
|
||||
return parseResult.value();
|
||||
}
|
||||
|
||||
function describeFailure(parseResult) {
|
||||
return parseResult.errors().map(describeError).join("\n");
|
||||
|
||||
function describeError(error) {
|
||||
return error.describe();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The result of parsing can be success, failure, or error. While failure indicates
|
||||
that the rule didn't match the input tokens, error indicates that the input
|
||||
was invalid in some way. In general, rules will backtrack when they
|
||||
encounter a failure, but will completely abort when they encounter an error.
|
||||
Each of these results has a number of methods:
|
||||
|
||||
```javascript
|
||||
result.isSuccess() // true for success, false otherwise
|
||||
result.isFailure() // true for failure, false otherwise
|
||||
result.isError() // true for error, false otherwise
|
||||
result.value() // if success, the value that was parsed
|
||||
result.remaining() // if success, the tokens that weren't consumed by parsing
|
||||
result.source() // the StringSourceRange containing the consumed tokens
|
||||
result.errors() // if failure or error, an array of descriptions of the failure/error
|
||||
```
|
||||
|
||||
The final question is then: how do we define rules for the parser, such as the currently undefined `sentenceRule`?
|
||||
|
||||
## Rules
|
||||
|
||||
Each rule in lop accepts an iterator over tokens, and returns a result, as
|
||||
described in the previous section.
|
||||
|
||||
### lop.rules.token(*tokenType*, *value*)
|
||||
|
||||
Success if the next token has type `tokenType` and value `value`, failure
|
||||
otherwise. Value on success is the value of the token.
|
||||
|
||||
### lop.rules.tokenOfType(*tokenType*)
|
||||
|
||||
Success if the next token has type `tokenType`, failure otherwise. Value on
|
||||
success is the value of the token.
|
||||
|
||||
### lop.rules.firstOf(*name*, *subRules*)
|
||||
|
||||
Tries each rule in `subRules` on the input tokens in turn. We return the result
|
||||
from the first sub-rule that returns success or error. In other words, return the
|
||||
result from the first sub-rule that doesn't return failure. If all sub-rules return
|
||||
failure, this rule returns failure.
|
||||
|
||||
### lop.rules.then(*subRule*, *func*)
|
||||
|
||||
Try `subRule` on the input tokens, and if successful, map over the result. For
|
||||
instance:
|
||||
|
||||
```javascript
|
||||
lop.rules.then(
|
||||
lop.rules.tokenOfType("integer"),
|
||||
function(tokenValue) {
|
||||
return parseInt(tokenValue, 10);
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
### lop.rules.optional(*subRule*)
|
||||
|
||||
Try `subRule` on the input tokens. If the sub-rule is successful with the value
|
||||
`value`, then return success with the value `options.some(value)`. If the sub-rule fails, return
|
||||
success with the value `options.none`. If the sub-rules errors, return that error.
|
||||
Reference in new issue
Block a user