sy substitute --help

sy-substitute 
Substitutes templates using structured data. The idea is to build a tree of data that is used to substitute in various
templates, using multiple inputs and outputs.That way, secrets (like credentials) can be extracted from the vault just
once and used wherever needed without them touching disk.Liquid is used as template engine, and it's possible to refer
to and inherit from other templates by their file-stem. Read more on their website at https://shopify.github.io/liquid .

USAGE:
    sy substitute [FLAGS] [OPTIONS] [--] [template-spec]...

FLAGS:
    -h, --help        
            Prints help information

        --no-stdin    
            If set, we will not try to read structured data from standard input. This may be required in some situations
            where we are blockingly reading from a standard input which is attached to a pseudo-terminal.
    -v, --validate    
            If set, the instantiated template will be parsed as YAML or JSON. If both of them are invalid, the command
            will fail.

OPTIONS:
    -d, --data=<data>
            Structured data in YAML or JSON format to use when instantiating/substituting the template. If set,
            everything from standard input is interpreted as template.
    -e, --engine=<name>
            The choice of engine used for the substitution. Valid values are 'handlebars' and 'liquid'. 'liquid', the
            default, is coming with batteries included and very good at handling
                                   one template at a time.
                                   'handlebars' supports referencing other templates using partials, which is useful for
            sharing of common functionality. [default: liquid]  [possible values: handlebars, liquid]
        --partial=<template>...
            A file to be read as partial template, whose name will be the its file stem. It can then be included from
            another template, and thus act as a function call.
        --replace=<find-this:replace-with-that>...
            A simple find & replace for values for the string data to be placed into the template. The word to find is
            the first specified argument, the second one is the word to replace it with, e.g. -r=foo:bar.
    -s, --separator=<separator>
            The string to use to separate multiple documents that are written to the same stream. This can be useful to
            output a multi-document YAML file from multiple input templates to stdout if the separator is '---'. The
            separator is also used when writing multiple templates into the same file, like in 'a:out b:out'. [default: 
            ]

ARGS:
    <template-spec>...    
            Identifies the how to map template files to output. The syntax is '<src>:<dst>'. <src> and <dst> are a
            relative or absolute paths to the source templates or destination files respectively. If <src> is
            unspecified, the template will be read from stdin, e.g. ':output'. Only one spec can read from stdin. If
            <dst> is unspecified, the substituted template will be output to stdout, e.g 'input.hbs:' or 'input.hbs'.
            Multiple templates are separated by the '--separator' accordingly. This is particularly useful for YAML
            files,where the separator should be `$'---\n'`

You can also use this alias: sub.

template-specs are the bread and butter of this substitution engine. They allow to not only specify the input templates, like ./some-file.tpl, but also set the output location.

By default, this is standard ouptut, but can easily be some-file.yml, as in ./some-file.tpl:out/some-file.yml.

You can have any amount of template specs, which allows them to use the same, possibly expensive, data-model.

At first sight, it might not be so useful to output multiple templates to standard output. Some formats are built just for that usecase, provided you separate the documents correctly.

If there are multiple YAML files for instance, you can separate them like this:

echo 'value: 42' \
| sy substitute --separator=$'---\n' <(echo 'first: {{value}}') <(echo 'second: {{value}}')

first: 42
---
second: 42

Also note the explicit newline in the separator, which might call for special syntax depending on which shell you use.

In the example above, how great would it be to protect ourselves from accidentially creating invalid YAML or JSON documents?

Fortunately, sheesy has got you covered with the --validate flag.

echo 'value: 42' \
| sy substitute --validate <(echo '{"first":"{{value}}}') 

error: Validation of template output at 'stream' failed. It's neither valid YAML, nor JSON
Caused by: 
 1: while scanning a quoted scalar, found unexpected end of stream at line 1 column 10

When generating structured data files, like YAML or JSON, even with a valid template you are very vulnerable to the values contained in the data-model. Some passwords for instance may contain characters which break your output. Even though --validate can tell you right away, how can you make this work without pre-processing your data?

--replace to the rescure. The following example fails to validate as the password was now changed to contain a special character in the JSON context:

echo 'password: xyz"abc' \
| sy substitute --validate <(echo '{"pw":"{{password}}"}') 

error: Validation of template output at 'stream' failed. It's neither valid YAML, nor JSON
Caused by: 
 1: while parsing a flow mapping, did not find expected ',' or '}' at line 1 column 12

Here is how it looks like without validation:

echo 'password: xyz"abc' \
| sy substitute <(echo '{"pw":"{{password}}"}') 

{"pw":"xyz"abc"}

You can fix it by replacing all violating characters with the respective escaped version:

echo 'password: xyz"abc' \
| sy substitute --replace='":\"' --validate <(echo '{"pw":"{{password}}"}') 

{"pw":"xyz\"abc"}

You have probably seen this coming from a mile away, but this is a great opportunity for a shameless plug to advertise sy merge.

sy merge allows to merge multiple files together to become one, and even some additional processing to it. That way you can use the combined data as model during template substitution.

sy merge --at=team team.yml --at=project project.yml --at=env --environment \
| sy substitute kubernetes-manifest.yaml.tpl

apiVersion: v1
data:
  game.properties: |
    enemies=aliens
    lives=3
    enemies.cheat=true
  ui.properties: |
    color.good=purple
    color.bad=yellow
kind: ConfigMap
metadata:
  name: game-config
  namespace: default
  labels:
    team: awesomenessies
    department: finance
    project: fantasti-project
    kind: AI-research

By default, we read the data model from stdin and expect all templates to be provided by template-spec. However, sometimes doing exactly the opposite might be what you need.

In this case, just use the -d flag to feed the data model, which automatically turns standard input into expecting the template.

echo '{{greeting | capitalize}} {{name}}' | sy substitute -d <(echo '{"greeting":"hello", "name":"Hans"}')

Hello Hans

The substitution can be performed by various engines, each with their distinct advantages and disadvantages.

This section sums up their highlights.

The Liquid template engine was originally created for web-shops and is both easy to use as well as fully-featured.

It’s main benefit is its various filters, which can be used to put something into uppercase ({{ “something” | uppercase }}), or to encode text into base64 ({{ “text” | base64 }}).

There are a few filters which have been added for convenience:

• base64
  • Converts anything into its base64 representation.
  • No arguments are supported.

The first optional template engine is handlebars. Compared to Liquid, it’s rather bare-bone and does not support any filters. The filtering syntax also makes chained filters more cumbersome.

However, it allows you to use partials, which are good to model something like multiple sites, which share a header and a footer. The shared portions are filled with data that contextually originates in the page that uses them.

For example, in an invocation like this you can declare headers and footers without rendering them, and then output multiple pages that use it.

Here is the content of the files used:

cat data.json

{
  "title" : "Main Heading",
  "parent" : "base0"
}

cat base0.hbs

<html>
  <head>{{title}}</head>
  <body>
    <div><h1>Derived from base0.hbs</h1></div>
    {{~> page}}
  </body>
</html>

cat template.hbs

{{#*inline "page"}}
  <p>Rendered in partial, parent is {{parent}}</p>
{{/inline}}
{{~> (parent)~}}

When using these in substitution, this is the output:

sy substitute --engine=handlebars -d data.json base0.hbs:/dev/null template.hbs

<html>
  <head>Main Heading</head>
  <body>
    <div><h1>Derived from base0.hbs</h1></div>
  <p>Rendered in partial, parent is base0</p>

  </body>
</html>

The perceived disadvantage of having close to zero available filters would have to be compensated using a processing program which takes the data, and adds all the variations that you would need in your templates:

./data-processor < data.json | sy substitute template.tpl

The normal title: Main Heading
The capitalized title: main heading

The data-processor in the example just adds transformed values for all fields it sees:

./data-processor < data.json

{
  "title" : "Main Heading",
  "title_lowercase" : "main heading",
}