Jamie Balfour

Welcome to my personal website.

Find out more about me, my personal projects, reviews, courses and much more here.

ZPE Programming Environment

ZPE Programming Environment

The ZPE Programming Environment or ZPE is a complete language parser, compiler and interpreter framework for the YASS programming language, written in Java. It interprets and compiles YASS (a custom language specific for this) into an abstract syntax tree (AST) and into other languages.

You can download the install package with the configuration script for Mac OS X and Linux here. To run this in your terminal type: sudo ./configure.sh (you need to run with elevated permissions to install it and you will need to be in the directory of the unzipped file) to install all the files including the manual page.

It all started with Operation Foghorn. This was my plan to integrate a macro scripting engine (known as my Macro Scripting Interface Language or MSIL at first) into Painter Pro, Cobweb and Wonderword. Eventually, BlackRabbit evolved into it's own language interpreter and BlackRabbit Script was formed. This was the first programming language I had worked on and I started building it when I turned 17.

As a matter of fact, the project originally called BlackRabbit is actually what ZPE is now and BlackRabbit Script is what YASS used to be called. The Zenith Parsing Engine and being rebuilt around a much faster parser.

The project now known as ZPE was known from May 2015 until July 2018 as the Zenith Parsing Engine due to its original intention as a project but after a rename became known as the ZPE Programming Environment (a recursive acronym).

Feel free to leave comments in comments section below.

PE
1.12

The ZPE Programming Environment features a range of built-in functions. ZPE Programming Environment (ZPE) was also known by its codename Shifting Sands before getting its final name (the name was a reference to shifting from the name and the workings of BlackRabbit).

ZPE was pronounced originally as ZeePeeEee but as of late 2015 it was named Zippy to simplify the pronunciation, the name itself was taken from the nickname for our hamster at the time, Henry (I name a lot of projects after my pets). It was designed specifically for my own enjoyment but also for educational purposes, with the main aim being to help introduce those who find programming difficult to a much easier language to learn. Eventually, it transformed into something more.

This is done using a very flexible compiler, one that will work in many ways, leaving not just one possible way to write a piece of code, but multiple. For example, take a for loop:

YASS
for($i = 3, $i < 10, 1)
	//Do something
end for
			

This loop can be modified to have an end loop instead of an end for ending:

YASS
for($i = 3, $i < 10, 1)
	//Do something
end loop
			

The same for loop can be written using braces rather than end for to terminate the loop and also using the $i++ increment feature rather than specifying just a number to increment by.

YASS
for($i = 3, $i < 10, $i++) {
	//Do something
}
			

It can also be written using semi colons to make the syntax of this identical to PHP:

YASS
for($i = 3; $i < 10; $i++) {
	//Do something
}
			

Or in a syntax closer to VB.NET syntax:

YASS
for($i = 0 to 10)
  //Do something
end for
            			

Actually, there are plenty more variations of this same loop. But crucially, all of them compile to the same thing underneath. The compiler does all of the work checking this. So feel free to write it as if it were using braces like C or just using keywords like VB.NET because remember, the compiler is efficient as it is and it can deal with having these extra checks.

How ZPE works

ZPE is built-up of several core components, namely the YASS syntax and byte-code (or op-code), the Zenith Parsing Engine which parses YASS code to byte-codes, the YASS Compiler which converts the byte-codes to an abstract syntax tree and the runtime which ultimately runs the program.

The following image shows how it works:

ZenithEngine

How ZPE works with plugins and libraries

The YASS Compiler's core principle purpose is to develop intermediate code. This code is then either run or it is compiled.

You can see how the Zenith Parsing Engine itself works with the language by looking at the ZenithJSONParser file here.

Purpose

Prior to ZPE I had already developed a programming language that I began to write when I turned 18 called BlackRabbit Script. The language was slow. But when I began writing a spell checker for Wonderword, I learned more about things like BNF and parsers I began rewriting it. Further, BRS was limited to Windows.

I think it would be fair to say that I began building this down to my own research and knowledge, but that my university course definitely was the push that made the performance almost double after learning about LL(1) parsers where I was converted to remove backtracking almost all over the parser.

ZPE and YASS were started in January 2015 as a programming project I decided to start for fun (as with all of my software). In March I could see it was really coming along as a programming syntax and thought I might as well continue with it. From there on, what was originally called only by it's codename, Shifting Sands and Stratosphere, became something else.

Just as BlackRabbit Script worked on .NET, ZPE works on Java. The two were fundamentally and syntactically different. Both were variations or dialects of my own BlackRabbit Script. YASS as it is known was fully based on the same syntax as BlackRabbit Script started on.

YASS and the ZPE Programming Environment are designed as a small compact programming language that can be used within other applications as scripting languages, but they also both feature stand-alone functionality as well. The core aim of the language was to improve the readability of code to make it easier to learn, therefore the language can also be seen as theoretical and educational languages too.

Technical stuff

ZPE is an LL(1) parser that will compile code from YASS to byte-code. The parser requires the use of backtracking and does this incredibly well. By doing things this way, it can assess mathematical expressions in one simple function rather than needing other methods.

Code is converted from byte-code to a IAST (Inline AST, formerly the Flast AST). The IAST is then run through an optimiser (only if being compiled to a file). The runtime will then run through the IAST and decide what to do based on the byte-code of each AST.

There are very few regular expression evaluations used in the language which is designed to keep the compile time down.

Compiled files will run faster than interpreted files, but interpeting code still runs relatively quickly.

Compared with BRS

BlackRabbit Script is the predecessor to ZPE/YASS and is a scripting language written in C# and VB.NET. It was more of a kind of side project than ZPE became. ZPE is considerably faster due to redevelopment of my parser (now known itself as the Zenith Parsing Engine).

A lot more time and effort went into the development of ZPE than BRS because I had a lot more of a plan when I developed it.

In terms of performance, ZPE is considerably better. In both languages I wrote the same for loop that does the same thing (loops 100,000 times):

YASS/BRS
for($i = 0, $i < 100000, 1)
	print($i)
end for
										

The newly designed ZPE performs nearly twice as fast as it did at the start of 2015, and considerably better than the .NET version of 2014 and the original version of 2011.

The following graph shows system times in milliseconds to perform a loop that prints each number up to 100, 000 times:

Latest version
Oldest version

Scores are calculated by determining time in milliseconds taken to do the same task with better results being the ones that takes less time as shown above. Note, the right two bars are versions of BRS and the left bars are versions of ZPE (with the second bar being ZPE 1.2).

Parser and language features

Many features have been suggested by testers and users of the language, although the majority are my own.

The following core language features have been implemented as of now:

  • LL(1) parser
  • Tree generation
  • Code compilation and exporting
  • Interactive mode
  • A Graphical user interface editor
  • Command line arguments
  • On-the-fly interpreting
  • Input and output
  • Variable assignment and substitution
  • Condition expressions
  • Selection (if statements and select statements)
  • Loops - both fixed and conditional
  • Functions and nested functions, function calling and nested function calls
  • YASS core function overwriting/overriding
  • Optional parameters
  • Finite sets (arrays)/lists and functions
  • Maps (similar to dictionaries or associative arrays) and functions
  • Overloading on functions. Mean, mode, max and min are functions that work on lists and numeric parameters
  • Mathematical and logical parser (LAME)
  • Recursion (since version 1.3.5.60)
  • Native functions written in Java
  • Library support
  • Lambda expressions and anonymous functions with functions as first-class citizens
  • Object-oriented syntax
  • Structures and classes
  • Optional variable and property typing
  • Built-in classes
  • Hundreds of predefined functions
  • ZPE Online cloud storage
  • A choice of strong or weak and dynamic or static typing when needed
  • ZEN networking abilities
  • A web parser, similar to PHP
  • An open source standard library, written itself in YASS.

There is more in the changelog as to what has been added. You can find the changelog here.

Presentation

As I often create slideshows of projects, I have created a slideshow on my website about the ZPE. You can view it from here.

What's good about ZPE?

There are many things that are good about ZPE and YASS, but here are a few.

It's open for extension development

The ZPE is an open for development (closed source for the majority) parser that means you can develop your own functions in Java (one of the most popular programming languages in the world) for it as well as help me develop it.

I maintain the core of the environment, but suggestions are always welcome.

Byte-codes are also available so developing compiler extensions is also very easy.

It's fast and easy to write

ZPE has a huge number of predefined functions within it, often system wrappers that simply perform a more complicated task in a shorter syntax. It allows you to write a program and run it quicker than you could with a lower level language.

It's a learning language

YASS or Yet Another Simple Syntax is really simple and is designed to read that way too. I promise to listen to feedback and to never put in complicated structures where ever I can avoid doing so.

It's multi-platform

Because the underlying Java programming language that powers the parser is Java, it is already platform independent. This means you can run it on a huge number of devices such as Windows PCs, Macs and Linux machines, as well as many others.

But, ZPE builds upon what Java has achieved, by offering platform-specific features that have been streamlined and made seamless.

Build your own functions for it

Whilst it is possible to write programs with multiple functions, compile them into a library and run them as YASS code, it is also possible to write Java based plugins for the language and let them run as libraries. It works really fluidly and well too!

To get started, download the TestPlugin project and compile it to a JAR file. Place this in the plugins/ folder where the zpe.jar resides (if the folder does not exist, simply create it and call it "plugins") which you can find by typing zpe -h config.

Make sure you have the latest ZPE when you compile to ensure full compatibility! This also ensures that the byte-codes are up to date.

It's designed to be intelligent

ZPE is one of the only programming environments I have come across where the print function applied to an array prints the elements of the array. PHP is the closest with it's print_r function. ZPE does the same with objects, associative arrays and much more.

ZEN

ZEN (ZPE Extensions for Networking) is all about accessing a ZPE instance remotely.

This remote mode allows you to setup a server machine that can run more complex tasks on a more powerful computer. Take for instance the limited power of the amazing little Raspberry Pi. Run a ZENServer on a system with a more powerful Core i7 on it and then run a ZENClient on the Raspberry Pi and pass all the complex calculations to the more powerful system. ZPE also supports sending files to the computer using it's own SFTP-like protocol.

I also use the remote mode to execute shutdown and restart commands on my Windows machine rather than installing a heavy SSH server on to the machines.

Contributions

The following software and/or components were used in the production of ZPE:

  • Java BECL (Balfour's Enterprise Class Library) - used for the image effects amongst many things (mostly other helper functions).
  • Google's GSON library - used for parsing JSON to a Java HashMap which is then converted to a ZPEAssociativeArray. This was added in version 1.5.0 (10.12.2016) and replaced by JBSON which is my own JSON parser built on the Zenith Parser in version 1.5.3 (24.02.2017).

The following people are to thank for their contributions to the development of ZPE:

  • Julian Merlin Ertel - major contributor through major testing. Without any doubt the biggest external tester of ZPE, all the way from version 1.3 to around 1.5.

Known bugs

The following list are known bugs in the latest version. Fixes are currently being looked at. These bugs are priority bugs. I will leave all bugs from version 1.5.3 in this list and update them as I manage to fix them. Most of these bugs are found by myself or my test group.

Bug number Description Fixed
#1Modular arithmetic does not always give the correct result2019-01-28
#2Arithmetical evaluation on inner object values does not work2017-03-28
#3MySQL functions no longer work - fixed by moving them to external library2018-03-30
#4Concatenation with value access within object context (e.g. $x -> $y & $x -> $z) does not work2018-08-31
#5ZPE needs to handle errors in code better and not crash when the compiler finds a fault.2017-09-20
#6ZPE Parser needs to work on comments more flexibly, not binding itself to a single style of comments (// and /* */ are currently the only supported options)2018-01-01
#7Concatenation with function access within object context (e.g. $x -> func1() & $x -> func2()) does not work2018-12-29
#8Accessing an object value's index within object context (e.g. $x -> $v[0] = 10) does not work2018-12-29
#9Global traversal of a program without a main function will often call an input function twice2019-09-13
Comments