sy process --help

sy-process 
Merge JSON or YAML files from standard input from specified files. Multi-document YAML files are supported. Merging a
single file is explicitly valid and can be used to check for syntax errors.

USAGE:
    sy process [FLAGS] [OPTIONS] [--] [path-or-value]...

FLAGS:
    -h, --help            Prints help information
        --no-overwrite    If set, values in the merged document may not overwrite values already present. This is
                          enabled by default,and can be explicitly turned off with --overwrite.
        --no-stdin        If set, we will not try to read structured data from standard input. This may be required in
                          some situations where we are blockingly reading from a standard input which is attached to a
                          pseudo-terminal.
        --overwrite       If set, values in the merged document can overwrite values already present. This is disabled
                          by default,and can be explicitly turned off with --no-overwrite.

OPTIONS:
    -a, --at=<pointer>...            Use a JSON pointer to specify an existing mapping at which the next merged value
                                     should be placed. This affects only the next following --environment or <path>.
                                     Valid specifications are for example '0/a/b/4' or 'a.b.0'. If it is specified last,
                                     without a following merged value, the entire aggregated value so far is moved.
    -e, --environment=<filter>...    Import all environment variables matching the given filter. If no filter is set,
                                     all variables are imported. Otherwise it is applied as a glob, e.g. 'FOO*' includes
                                     'FOO_BAR', but not 'BAZ_BAR'.Other valid meta characters are '?' to find any
                                     character, e.g. 'FO?' matches 'FOO'. [default: *]
    -o, --output=<mode>              Specifies how the merged result should be serialized. [default: json]  [possible
                                     values: json, yaml]
    -s, --select=<pointer>...        Use a JSON pointer to specify which sub-value to use. This affects only the next
                                     following --environment or <path>. Valid specifications are for example '0/a/b/4'
                                     or 'a.b.0', and they must point to a valid value. If it is specified last, without
                                     a following merged value, a sub-value is selected from the aggregated value.

ARGS:
    <path-or-value>...    The path to the file to include, or '-' to read from standard input. It must be in a
                          format that can be output using the --output flag. Alternatively it can be a value
                          assignment like 'a=42' or a.b.c=value.

You can also use these aliases:

• merge
• show

It helps to use this powerful command by understanding its mindset a little.

• it wants to produce a single document (JSON or YAML) from multiple input documents (JSON or YAML)
• by default, it will refuse to overwrite existing values
• multi-document YAML files are fully supported
• standard input is a valid source for documents
• the order of arguments matter, as this program is implemented as state-machine
• by default it produces JSON output

This program helps to quickly manipulate various inputs and produce a new output, which can then more easily be used in programs like sy substitute, or to generate configuration files.

Now let's look typical use-cases.

This case often arises by the mere necessity of keeping things separate. Even though keeping all data in a single structured data file would work just fine, in practice not all information is under your direct control and thus pulled in separately from other locations.

For substitution, multiple files are not viable, which is why a single file should be produced instead:

sy merge --at=project project.yml --at=team team.yml

{
  "project": {
    "kind": "ios-game",
    "name": "super-punch"
  },
  "team": {
    "name": "dept. nine",
    "product-owner": "Max Owner"
  }
}

As you can see, we use the --at flag to put the contents of both files into their own namespaces. Without that, we would have a clashing name field which makes the program fail.

sy merge project.yml team.yml

error: The merge failed due to conflicts
Caused by: 
 1: Refusing to merge due to the following clashing keys:
name

Sometimes during testing, it's useful to change a single value, in the configuration for instance. You can easily do this using the key=value specification.

sy merge game-config.yml player.lives=99

error: The merge failed due to conflicts
Caused by: 
 1: Refusing to merge due to the following clashing keys:
player.lives

However, the above fails as we won't ever overwrite existing values. Let's try to argue with the program to make it work nonetheless:

sy merge game-config.yml player.lives=99 --overwrite

error: The merge failed due to conflicts
Caused by: 
 1: Refusing to merge due to the following clashing keys:
player.lives

This might appear unexpected, even though it is not when you have understood that the order matters. In the example above, --overwrite simply applies too late for overwriting the value. If we swap their positions, it will work.

sy merge game-config.yml  --overwrite player.lives=99 player.invincible=true

{
  "enemies": {
    "cheat": true,
    "type": "aliens"
  },
  "player": {
    "invincible": true,
    "lives": 99
  }
}

Please note that --overwrite acts as a toggle and affects all following arguments. You can toggle it back off with the --no-overwrite flag

sy merge game-config.yml  --overwrite player.lives=99 \
        --no-overwrite --at=project project.yml --at=team team.yml \
        -o yaml

---
enemies:
  cheat: true
  type: aliens
player:
  lives: 99
project:
  kind: ios-game
  name: super-punch
team:
  name: dept. nine
  product-owner: Max Owner

As a side-effect, you can easily convert YAML to JSON, like so...

sy process < game-config.yml

{
  "enemies": {
    "cheat": true,
    "type": "aliens"
  },
  "player": {
    "lives": 3
  }
}

...or the other way around:

sy process < game-config.yml | sy process -o yaml

---
enemies:
  cheat: true
  type: aliens
player:
  lives: 3

More often than not, the environment contains information you will want to make use of in our configuration. It's easy to bring it into your data model, and filter them by their name.

sy process --environment=HO*

{
  "HOME": "/root",
  "HOSTNAME": "52170f35ed69"
}

Of course this can be combined with all other flags:

sy process --at env --environment=HO* env.NEW=value

{
  "env": {
    "HOME": "/root",
    "HOSTNAME": "52170f35ed69",
    "NEW": "value"
  }
}

It's most common to have different sets of configuration for different environments. For example, most deploy to at least two stages: pre-production and production.

When using sy process for generating configuration to be used by tooling, it's not practical to force the tooling to know the stage.

Imagine the following configuration file:

cat multi-stage-config.yml

pre-production:
  replicas: 1
  max-cpu: 200mi
  max-memory: 64M
production:
  replicas: 3
  max-cpu: 2000mi
  max-memory: 1024M

Tools either get to know which stage configuration to use, or you 'pull it up' for them:

sy process --select=production multi-stage-config.yml -o yaml

---
max-cpu: 2000mi
max-memory: 1024M
replicas: 3

A feature that is still rare in the wild, probably due to lacking tool support, is multi-document YAML files.

We fully support them, but will merge them into a single document before processing it any further.

A file like this...

cat multi-document.yml

---
pre-production:
  replicas: 1
  max-memory: 64M
---
production:
  replicas: 3
  max-memory: 1024M

...looks like this when processing. Clashing keys will clash unless --overwrite is set.

sy process multi-document.yml -o yaml

---
pre-production:
  max-memory: 64M
  replicas: 1
production:
  max-memory: 1024M
  replicas: 3

We will read JSON or YAML from standard input if possible. To make it more flexible, any non-path flags are applied to standard input. This may lead to unexpected output if more than one document source is specified.

Let's start with a simple case:

cat team.yml | sy process --at=team-from-stdin

{
  "team-from-stdin": {
    "name": "dept. nine",
    "product-owner": "Max Owner"
  }
}

In the moment another file is added for processing, it's a bit more difficult to control which argument applies where:

cat team.yml | sy process --at=team-from-stdin --at=project project.yml

{
  "kind": "ios-game",
  "name": "super-punch",
  "project": {
    "name": "dept. nine",
    "product-owner": "Max Owner"
  }
}

As you can see, the project key is used for standard input, and the team-from-stdin is seemingly ignored.

To fix this, be explicit to make obvious what you expect:

cat team.yml | sy process --at=team-from-stdin - --at=project project.yml

{
  "project": {
    "kind": "ios-game",
    "name": "super-punch"
  },
  "team-from-stdin": {
    "name": "dept. nine",
    "product-owner": "Max Owner"
  }
}

Note the single dash (-), which indicates when to read from standard input. As standard input is always consumed entirely, it can only be specified once.