Design patterns for simple text based scripting language?

Question

In my current application I am trying to determine the best way to implement a simple scripting language. This language would allow a user to create a script that controls a machine with various inputs and outputs. I have defined most of the commands and created a nice editor for the language though I am stuck now trying to figure out how to verify, parse, and "compile" it.

I currently treat the commands as const String and plan to parse the input script text with regex and these strings. Most of the commands have the syntax

COMMAND Arg1 Arg2 Arg3 etc.

The types and numbers of arguments are currently not defined anywhere in my code, only in my head. I need a design pattern that would allow me to take a line from the script and determine if it is valid; check the command versus some list and check its arguments against the match in that list.

Are there any known design patterns for situations like this; scripting languages in general?

I feel like I need a class Command that holds the command string and information about its arguments, also the translation from string to "action" (performing the actual real life action the command describes). Then if I am to encounter the string representing the command I can lookup the command class instance, pass in the line, and get some result if it is valid. Though I feel like I would end up (in my case) with quite a few Command sub classes.

Any ideas or recommendations?

Answer 1

The last time somebody asked this question, I replied: why not just use an existing language?

What you have right now is an API.

You can provide a library in which your commands are function calls, or method calls, or whatever suits you. This saves you from having to design and implement a language, and it saves your users from having to learn yet another little special-purpose language. They still need to learn your API, of course.

I'd say this is the standard design pattern for exposing APIs: as libraries for one or more existing languages.

If you really want to create a new language, you need to specify and implement its syntax and semantics.

For dealing with syntax, using a parser generator is definitely the standard approach. Any language whose programs don't just consist of simple lists of statements will have nested structures and most will allow syntactic recursion (i.e. among the various types of language constructs, some can appear within themselves arbitrarily often). For instance, you may wish to allow a command invocation as an argument to another command invocation, or you may want to have a construct to express iteration or choice that can be arbitrarily nested. In that case, you don't want to parse and process nested constructs with regular expressions, you want grammars, and parser generators are the standard way to work with grammars.

As to the surface syntax, the basic way in which the source code is chopped up into meaningful tokens of your language, I think the most important general lesson is: keep it general, easily readable and context-independent. Make the structure of your programs easy to understand for humans and computers. Examples not to follow are such atrocities as Makefiles or /bin/sh in which the correct application of whitespace and quotes is advanced witchcraft.

As to the semantics, one of the things to decide is what sort of programming paradigm to support. Are all commands of a program executed strictly in sequence without anything else ever interfering? Then, a standard, strictly sequential imperative programming language may be a good option. Can commands set concurrent events in motion, or can multiple programs be running at once? Then, a fundamentally concurrent language may be more appropriate.

Another thing to decide is to what extent you want your language to scale. If scripts can grow large and can be invoked by other scripts, you probably want to provide mechanisms for preventing different pieces of code from biting each other in the leg (e.g. scopes for names, local variables). But this is probably a later concern.

Answer 2

Another possibility, somewhere between API and new language: create a domain-specific language on top of a general-purpose language. Think regular expressions: a mini-language, with its own syntax, built for just one purpose (e.g. text processing), which is usable from within another, more flexible language. Then you get the benefits of a language with a close fit to the domain, but also get a full language that you don't have to implement yourself if things start to get complex.

Lisp, Scala, and Ruby are popular choices for DSLs. Because they're so flexible, you can build a DSL directly inside the host language; this is called an embedded DSL. Interesting examples are Clojure's core.match and Midje, Ruby's rspec unit test library, and Scala's Baysick, a DSL that strongly resembles classic BASIC created using just Scala's own facilities. The designs behind embedded DSLs differ by language; in Clojure and other Lisps, they usually use a lot of macros, while Scala gets a ton of mileage from implicits and Ruby gets a lot of use from blocks. Scala's and Ruby's flexible policies on using parentheses around function arguments are also a big help.

You can also create an external DSL by writing a simple parser and a small interpreter. You can use the parser generator libraries that others have suggested, but if the language is simple enough, you don't actually need to create an abstract syntax tree or a symbol table; you can just parse and execute commands one at a time, possibly holding on to some state as you go. If your syntax is constrained enough, you can just tokenize a line with regular expressions, then loop over the tokens. If there are no user-defined commands, you can just have a class or function for each command; you can store them in a table, and use the command name at the front of each line to look up the appropriate command and pass it arguments. The commands can validate that they received the appropriate type and number of arguments.

If you want to take the external DSL approach, you might be interested in the Interpreter Pattern as a way to organize things.

Answer 3

Expose your API in the form of calls from a well-known language, so you can leverage the development already done in that language, and even combine with modules for other APIs and libraries. I'd probably choose Python or Perl.

Use one of the languages designed as lightweight extension languages, thought as a way to write extensions or control of the program in which it is embedded. Alternatives are lua or tcl/tk, maybe even Scheme (as in GNU guile).

This will be much less work than doing it yourself, gives a much better product faster, gives lots of extra flexibility (you don't have to provide it!), you can use tons of already available documentation (probably the hardest part of getting this into the hand of users), and flattens the learning curve (users can reuse what they learned elsewhere/reuse elsewhere what they learned here).