_{Official ZPE/YASS documentation}Parser

Introduction

The Parser object provides access to a lightweight token parser based on the Zenith Parsing Engine.

It allows you to:

• Define a custom token map (string → bytecode)
• Load text from a file or directly from a string
• Iterate through tokens and retrieve both bytecodes and token text

The Parser object is intended for building simple parsers, token streams, language tooling, or custom syntax readers inside YASS.

All functions exposed by this object require a permission level of 0.

Parser object functions

The following is a list of internal functions the Parser object exposes. All functions are ZPEObjectNativeFunctions therefore run in native code.

set_tokens(list token_map) ⇒ boolean

Sets the internal token mapping used by the parser. The token map is used by the tokeniser to convert strings into bytecodes during parsing. Returns true.

load(string filename) ⇒ boolean

Loads a file into the parser using UTF-8 encoding. Returns true if the file was successfully loaded, otherwise false.

read(string text) ⇒ boolean

Loads a raw string directly into the parser. Returns true.

parse() ⇒ ParseValue

Parses the next token and returns a ParseValue object containing both the bytecode and text.

has_next() ⇒ boolean

Returns true if there are more tokens available to parse, otherwise false.

ParseValue object functions

The ParseValue object is returned by parse() and represents a single parsed token.

bytecode() ⇒ number

Returns the bytecode associated with the parsed token. If no bytecode is mapped for the token, the parser may return -1.

text() ⇒ string

Returns the raw text of the parsed token.

Examples

This example demonstrates creating a token map, reading text, and iterating through tokens. In this example, certain words are mapped to bytecodes and everything else will default to -1.

$tokens = new Map()
$tokens["print"] = 1
$tokens["let"] = 2
$tokens["="] = 3

$p = new ParserObject()
$p->set_tokens($tokens)

$p->read("let x = 5 print x")

while ($p->has_next())
    $t = $p->parse()
    print("Token: " + $t->text() + "  Bytecode: " + $t->bytecode())
end

This example loads a file and parses through it:

$tokens = new Map()
$tokens["function"] = 10
$tokens["end"] = 11

$p = new ParserObject()
$p->set_tokens($tokens)

if ($p->load("script.yass"))
    while ($p->has_next())
        $t = $p->parse()
        print($t)
    end
else
    print("Failed to load file.")
end

Notes:

• Token map: The token map is used to convert known words into numeric bytecodes. Any word not present in the map will produce a bytecode of -1.
• Delimiters: The parser splits tokens using whitespace (spaces and newlines).
• Comments and quotes: The SimpleTokeniser used by this object does not define comment styles or quote types, therefore it is designed for simple token streams.