Fork me on GitHub

Introduction

Cadenza is a template compiler composed of three parts: a lexer, parser and renderer. All three parts were designed to be interchangeable so you could write a replacement for any of these which should integrate with the others seamlessly.

The lexer takes an input file and converts it into a list of useful tokens which the parser can use. The parser will in turn convert these tokens into an Abstract Syntax Tree (AST) which can be used by the renderer. Finally, the renderer uses a Context and an AST to create the rendered template.

Cadenza includes a standard stack of lexer/parser/renderer classes which is what is used in the shorthand rendering method Cadenza#render. This reference will cover the standard compiler stack and later will cover writing custom compiler components in the custom components section.

Contexts

Contexts are managed by the Cadenza::Context class which contains all the information used when rendering the template, including:

Cadenza includes a constant named Cadenza::BaseContext which is an instance of Cadenza::Context that includes the Cadenza standard library of filters, blocks and functional variables.

While you are free to implement a set of these filters, blocks and functional variables yourself by creating your own instance of Cadenza::Context it is suggested you clone Cadenza::BaseContext instead by calling Cadenza::BaseContext.clone and use that as the starting point for creating your own context.

Contexts: Configuration

All configuration for Cadenza is stored on the context object. As such any configuration work you want to do when integrating Cadenza should be done on the Cadenza::BaseContext object (again, simply a suggestion, for maximum control you can always construct your own Context object and configure it from there).

whiny_template_loading (boolean: default = false)
When loading a template using the regular method #load_template instead of the bang method #load_template! Cadenza will still raise a Cadenza::TemplateNotFoundError when no template could be loaded.

Contexts: Variables

Variables are defined on a Cadenza::Context through the context's variable stack, accessible through the #stack method. Each level of the stack is referred to as a scope. When looking up variables Cadenza inspects the stack from the top to the bottom, meaning that if a variable is defined twice on a stack the topmost value will be looked up.

To retrieve the value of a variable given it's name (named it's identifier) you can use the #lookup method. Be sure to include any dots if the identifier uses dot notation to refer to sub-objects:

# returns the value of the variable named "foo"
context.lookup "foo"

# returns the value of the variable named "bar" for the value of the
# variable "foo"
context.lookup "foo.bar"

If the variable can't be found then #lookup will return nil.

Values can also be assigned to the top of the stack by using the #assign, method:

context.assign :foo, "some value for foo"

To add and removes scopes from the variable stack you can call the #push and #pop methods:

# now "bar" will be returned when you call context.lookup("foo")
context.push :foo => "bar"

# now "baz" will be returned when you call context.lookup("foo")
context.push :foo => "baz"

# and again "bar" is returned when "foo" is looked up
context.pop

Contexts: Functional Variables

You can define functional variables by calling the context object's #define_functional_variable method with the name of the variable and an object which responds to #call, such as a Proc, lambda or by passing a block.

context.define_functional_variable :foo do |context|
   # do something here and return the result
end

When evaluating the function the passed callable object will be called with a variable number of arguments. The first argument will always be the context object. The remaining arguments will be the parameters the variable is being evaluated with.

To evaluate a functional variable call the #evaluate_functional_variable method on the context object, passing the name of the variable and an array of the parameters for it:

# calls the "foo" variable proc with "abc" and 123 as the arguments
context.evaluate_functional_variable :foo, ["abc", 123]

Contexts: Filters

You can define filters by calling the context object's #define_filter method with the name of the filter and an object which responds to #call, such as a Proc, lambda or by passing a block.

context.define_filter :foo do |input, parameters|
   # do something here and return the result
end

When evaluating the filter the passed callable object will be called with exactly two arguments. The first argument will always be the input object, which may be any type of ruby object. The remaining arguments will be the parameters the filter is being evaluated with.

It is suggested you do type checking inside your filter to ensure you're not passed unexpected data (duck type checking can be used as well). If you are passed unexpected data you should raise an error which is a subclass of Cadenza::Error.

To evaluate a filter call the #evaluate_filter method on the context object, passing the name of the filter and an array of the parameters for it:

# calls the "foo" filter proc with "abc" and 123 as the arguments
context.evaluate_filter :foo, ["abc", 123]

Contexts: Blocks

You can define blocks by calling the context object's #define_block method with the name of the block and an object which responds to #call, such as a Proc, lambda or by passing a block.

context.define_block :foo do |input|
   # do something here and return the result
end

When evaluating the block the passed callable object will be called with a exactly three arguments. The first argument will always be the context object. The second argument will be an Array of Node objects which are the block's parsed children, it is up to the callable to render these as it sees fit. The final argument will be an array which are the parameters passed to the block.

To evaluate a block call the #evaluate_block method on the context object, passing the name of the block, an array of nodes (the block's children), and an array of the parameters for it:

# calls the "foo" block proc with "abc" and 123 as the arguments and
# a text child node with "xyz" as it's content
context.evaluate_filter :foo, [Cadenza::TextNode.new("xyz")], ["abc", 123]

Contexts: Loaders

Loaders are used by the context to load new template content when needed. For example when you use the "extends" tag or the "render" tag. Loaders are stored on the context inside an array. When loading a template the loaders are searched from the first element of the array to the last until a match is found.

To add a loader onto the list you can call the #add_loader method.

context.add_loader Cadenza::FilesystemLoader.new("path/to/views")

To load the content of a file using the loaders you can call either the #load_source or #load_source! method on the context with the name of the template you want to load. The bang version of the method will raise an exception if the given template could not be loaded.

# should return the file's text (unparsed) or nil if the template
# couldn't be found.
context.load_source "path/to/template.cadenza"

To load the parsed content of a file using the loader you can call either the #load_template or #load_template! in the same fashion as the #load_source methods. These will return an AST if the template was successfully loaded.

# should return an AST for the given template if the template was 
# found, or nil otherwise
context.load_template "path/to/template.cadenza"

Cadenza includes the following loader implementations:

Cadenza::FilesystemLoader
Loads templates directly off of the filesystem from a specified directory.

Scanning

Scanning is the process of converting an input file's text into a stream of tokens. Most users won't have to concern themselves with scanning since the parser will manage constructing and using the default scanner implementation (Cadenza::Lexer).

A token is an array which has two elements in it. The first element is the type of the token and the second will be an instance of Cadenza::Token, which contains the value of a token.

The lexer has an input stream assigned to it by using the #source= method. From here tokens are parsed from the input by continually calling #next_token on the lexer. When there are no more tokens to be retrieved from the input a token [false, false] will be returned instead.

Parsing

Parsing is the process of converting an input token stream into an Abstract Syntax Tree and is managed by the Cadenza::Parser classes. To parse a template call #parse on the parser object like this:

Cadenza::Parser.new.parse "Hello {{ x }}!!"

By default constructing the Parser object will have it use a new instance of Cadenza::Lexer to process the input but by passing the :lexer option to the constructor you can use any custom scanner instead:

Cadenza::Parser.new(:lexer => CustomLexer.new).parse "Hello {{ x }}!!"

Rendering

Rendering is the process of converting an AST to an output template using an instance of Cadenza::Context.

The current version of Cadenza includes two renderer classes intended for two entirely different use cases.

Cadenza::TextRenderer

The most important renderer class which handles the rendering of a Cadenza AST in the fashion discussed in the Syntax section of this manual.

Cadenza::SourceRenderer

An experimental renderer implemented in Cadenza 0.8, this renderer takes a Cadenza AST and outputs template source code. In other words, it outputs a document that can be parsed again into the same Cadenza AST.

Among other use cases you may find this renderer useful when performing upgrades and migrations to templates in your projects.

Renderers must be instantiated with an IO object which will be written to during the rendering process. This object should already be opened and ready for writing when the class is instantiated.

Cadenza::TextRenderer.new File.open("/path/to/file", "w+")

Cadenza::TextRenderer.new StringIO.new

Once constructed the object can render an AST by calling the #render method with the AST and context.

# renders the document using a clone of Cadenza's BaseContext 
# object (the suggested approach)
renderer.render(document, Cadenza::BaseContext.clone)

To facilitate template inheritance an array of Cadenza::BlockNode objects can be passed.

# you likely won't have to do this yourself, the renderer usually
# does this internally to facilitate template inheritance.
context = Cadenza::BaseContext.clone
blocks = [Cadenza::BlockNode.new(:foo), Cadenza::BlockNode.new(:bar)]

renderer.render(document, context, blocks)

All of the renderer classes included with Cadenza also feature a class level shorthand method to render a given AST and context and return the output as a string.

Cadenza::TextRenderer.render(document, Cadenza::BaseContext.clone)
Cadenza::SourceRenderer.render(document, Cadenza::BaseContext.clone)

Writing custom compiler components

Heads up: until Cadenza reaches v1.0.0 the component API is still flexible any may be subject to change. However, the current version should be mostly stable and any changes should be minor.

Custom Scanners

Scanners transform raw text into tokens useable by the parser. Each token is a two element Array object with the first token being a well defined Symbol object (see documentation for specifics). The second object is an instance of Cadenza::Token which will hold the value of the token.

The separation between scanner and parser was intentional to help give Cadenza the ability to parse multiple input languages. This may not always be possible with Lexers alone since the input language might be very different from Cadenza's syntax (example: c++) but may be possible to parse similar languages such as Smarty, Django or Liquid (which were inspirations for Cadenza).

A new lexer must implement two methods to be useable in Cadenza: #source=(io_object) and #next_token

class CustomLexer
   def source=(io_object)
      # reset any internal variables and prepare to read from 
      # the IO object
   end

   def next_token
      # return the next token from the input, if there are no 
      # more tokens then return [false, false]
   end
end

Custom Parsers

Parsers use a Scanner to retrieve tokens from the input template and form them into an Abstract Syntax Tree (AST) of the Cadenza Node classes.

Cadenza's included parser Cadenza::Parser is a look-ahead left to right parser (LALR) generated by ruby's Racc compiler. While this type of parser has a large advantage of speed and other techincal details (such as being able to correctly parser Arithmetic and Boolean expressions) it lacks some of the flexibility of other types of parsers, such as recursive descent parsers which can be more flexible.

Even though Cadenza is implemented as this type of parser you aren't required at all to implement this as your own parser. A recursive descent parser or something else entirely should work just as well as Cadenza's LALR parser.

Cadenza requires that any custom parser have the same constructor parameters as Cadenza::Parser and also requires it define the #parse method.

class CustomParser
   def initialize(options={})
      # create a new parser object.  The options has may contain 
      # :lexer which you should use to retrieve tokens, or create 
      # a new lexer if it is not present.
   end

   def parse(source)
      # assign the source to your lexer and parse all tokens from 
      # it.  Return an AST from this method.
   end
end

Custom Renderers

Renderers take an AST and a context object and are expected to write the output template to it's given output stream. The renderer must implement the same constructor and also implement the #render method.

To make things a bit easier you may want to subclass Cadenza::BaseRenderer which will take care of some of the setup for you and make the subclass a bit cleaner. See Cadenza's YARD Documentation for details.

class CustomRenderer
   def initialize(io_object)
      # assign the IO object given, when rendering the template
      # should write directly to this IO object.
   end

   def render(document_node, context, blocks=[])
      # navigate the AST (document_node) and render the output 
      # to the IO object, the context can be used to look up 
      # values for variables, filters, etc.

      # When doing template inheritance blocks will be passed 
      # as the third parameter which are expected to be rendered 
      # in place of blocks matching the same name
   end
end

Custom Loaders

Loaders are very simple, they must implement two methods: #load_source or #load_template which are expected to return a string or an AST respectively. If they cannot load the given template name they should return nil so Cadenza knows to move onto the next available loader.

Keep in mind, just because most examples show template loading passing in paths to files you can use whatever string you want to identify your template, for example if you wrote a database loader you could specify the template's name like so: "template_id:135".

class CustomLoader
   def load_source(template_name)
      # return the unparsed source of the given template name or nil 
      # if this loader can't match a file for the name
   end

   def load_template(template_name)
      # return the parsed AST for the given template name or nil if 
      # this loader can't match a file for the name
   end
end