YAML Entity Store Utilities

The project is community supported and not supported by Axway Support. If you encounter any issues, please raise an GitHub issue for this project.

The project is in an early phase and not used in production yet. Please use the project for testing only.

tl;dr

YAML-ES Utilities can be used for AMPLIFY API Gateway projects - which are based on the YAML Entity Store - to verify coding rules, and to configure projects for target environments using various configuration sources.

1. Introduction

1.1. About YAML-ES Utilities

YAML-ES Utilities provide the features to verify a YAML based policy project according to defined rules (linting), and to configure the policy project for target environments.

The utilities of this project can be used to build a CI/CD pipeline for automatic configuration and deployment of YAML Entity Store based projects (see YAML Entity Store for details).

YAML-ES Utilities concept

1.2. Use Cases

The uses cases, described in this section, are just examples to illustrate the opportunities of the YAML-ES Utilities. They only give some ideas to build your own deployment pipeline.

1.2.1. Classic Topology Deployment

Classic topology with separated concerns

deployment classic topology.drw

The picture above visualize an example CI/CD pipeline. The YAML-ES Utilities tool provided by this project is used for linting and for the configuration of the stage specific project. To verify the configured project and to deploy the configured project to a gateway, the standard tools of the AMPLIFY API Gateway 7.7 distribution are used.

Configurations with shared responsibility or staged dependent values are environmentalized. A linter checks the project against coding rules (defined by the Ops team) to ensure that all required fields are environmentalized and all required entities exists.

With stage specific configurations (including certificates), a stage specific project is created.

Configuration parameter are sourced from multiple locations. Configurations are retrieved from plain text files stored in the Git repository or from encrypted data sources. Each team has its own set of configuration parameter (depending on their responsibility). Hierarchies of configurations ensures that values configured the Ops team supersede values defined by the Dev team.

The generated, stage specific project is verified for not configured fields or broken references.

After successful verification the project will be deployed to the target environment. During to protect the configurations on the server, a passphrase is added during the deployment.

1.2.2. Container Deployment (Policy Image)

deployment emt policy image.drw

1.2.3. Container Deployment (Init Container)

deployment emt init container.drw

1.3. YAML Entity Store

In newer versions of the API Gateway, policies can be stored as a YAML Entity Store (YAML-ES) instead of the classic XML based entity store (XML-ES). With the YAML-ES, the entities are stored in YAML files in a directory structure similar to the structure of the Policy Studio project.

YAML Enity Store Directory (simplified)

├── APIs
...
├── Environment\ Configuration
├── External\ Connections
...
├── Policies
├── Resources
├── Server\ Settings
│   ├── ...
│   ├── Cassandra\ Settings.yaml
...
├── System
└── values.yaml

Within the folders, entities are stored in files within the following schema (see also YAML Schema):

Entity Schema

type: string (1)
field: (2)
  fieldName: value1
  fieldNameArray:
    - item
    - item
children: (3)
  - type: string (4)
    ...

1	Type name of the entity
2	Fields of entity as name/value pair(s)
3	Optional list of children entities.
4	Type of the children entity

Below is an example of the Cassandra settings entity in the YAML format.

/Server Settings/Cassandra Settings.yaml

type: CassandraSettings (1)
fields: (2)
  name: Cassandra Settings
  replication: '{{Server_Settings.Cassandra_Settings.replication}}' (3)
  replicationStrategy: '{{Server_Settings.Cassandra_Settings.replicationStrategy}}'
  username: '{{Server_Settings.Cassandra_Settings.username}}'
  password: '{{Server_Settings.Cassandra_Settings.password}}'
children: (4)
  - type: CassandraServer
    fields:
      name: Node 1
      host: ${env.cassandra.node1}

1	Type `CassandraSettings` indicates the configuration of the Cassandra database.
2	Fields to configure the Cassandra database and keyspace.
3	Environmentalized field.
4	Children entities for each configured Cassandra node.

Within the Cassandra settings some fields are environmentalized. The double brackets {{ }} indicate an environmentalized field, where the value within the brackets reference the corresponding value within the values.yaml file of the project. The reference is a JSON Path expression, where the parent and child nodes are separated by a single dot.

values.yaml

Server_Settings:
  Cassandra_Settings:
    replicationStrategy: org.apache.cassandra.locator.SimpleStrategy
    replication: 1
    throttlingReplicationStrategy: org.apache.cassandra.locator.SimpleStrategy
    throttlingReplication: 1
    throttlingReadConsistencyLevel: ONE
    throttlingWriteConsistencyLevel: ONE
    username: ""
    password: "" (1)
    useSSL: "false"
...

1	Cassandra password is referenced by `{{Server_Settings.Cassandra_Settings.password}}` within the Cassandra settings.

2. Usage

2.1. General Usage

YAML-ES Utilities provides a built-in help for all its commands. The help message can be displayed with the help command or with the option --help or -h.

Show help message

yamlesutils.sh help
Usage: yamlesutils [-hqVv] [-a=FILE] [COMMAND]
YAML Entity Store Utilities
  -a, --audit=FILE   Audit file.
  -h, --help         Show this help message and exit.
  -q, --quiet        Disable log message to the console.
  -v, --verbose      Increase logging verbosity.
  -V, --version      Print version information and exit.
Commands:
  help    Display help information about the specified command.
  merge   Merge configuration from various sources.
  lint    Lint YAML entity store.
  config  Configure YAML-ES. Combines the 'lint', 'merge config' and 'merge
            certs' commands.
  eval    Evalulates a template expression.

There are some global options, which are available for all commands. These options have to be specified before the command.

Example: Global Options

yamlesutils.sh -q \ (1)
  merge config -c fragment.yaml -o values.yaml (2)

1	Global options are placed before the command.
2	Command to be executed.

Table 1. Options for Merge Configuration
Option	Description
`-a FILE` `--audit=FILE`	Path to a file to audit the various sources and functions used to merge the configuration fragments. If not specified, the audit messages are written to `stderr`, only.
`-q` `--quiet`	Suppress audit messages to be logged to the console. Only errors are logged to the console.
`-v` `--verbose`	Increase verbosity of standard messages. Specify multiple -v options to increase verbosity.

2.2. Linting

The tool can be used to check the project against coding guidelines. The coding guidelines are represented by rules defined in a set of YAML files.

2.2.1. Command

The lint command of yamlesutils is used to verify the project.

yamlesutils lint --project=DIR -r=FILE [-r=FILE]...

The command requires the path to the project directory and a list of files containing rules.

Example

$ yamlesutils.sh lint --project=src/apim --rules=rules/db.yaml --rules=rules/cassandra.yaml

2.2.2. Rules

The rules are defined in YAML files, having the following structure:

Structure of a rules definition file (YAML)

rules: (1)
  "rule_id": (2)
    # Description of the rule
    name: "Rule Name" (3)
    description: "Additional description (optional)"

    # Files to be checked by the rules
    fileType: ...type of the project file...
    filePatterns:
      - '...array or search patterns for files...'

    # Assertions applied to the file
    assertions:
      - '...array of assertions...'

1	Container for rules.
2	Each rule has its unique ID within the rules container.
3	Mandatory name of the rule.

An assertion checks a property in t

Table 2. Properties of Assertions
Property	Description	Mandatory
path	JSON path to the property of the YAML file to be checked	yes
type	Type of the assertion (see table below).	yes
param	Parameters to be passed to the	no
message	Message to be displayed in case of the failed assertion.	no

Table 3. Assertions
Assertion	Description and Example
environmentalized	Check if the field is environmentalized `- path: '$.fields.username' type: environmentalized message: Cassandra user name must be environmentalized`
exists	Check the existence of a property in the YAML file. `- path: '$.children[0].fields.host' type: exists message: One Cassandra node must be configured`
not_exists	Ensure that property doesn’t exist in the YAML file. `- path: '$.children[1].fields.host' type: not_exists message: Only one Cassandra node must be configured`
regex	Check the content of property against a regular expression. `- path: '$.children[0].fields.host' type: regex param: '^\$\{env\..+\}' message: Cassandra node must be configured via envSettings.props`

Example

Cassandra Nodes and KPS Tables

The following example checks the configuration of the Cassandra database and the KPS tables using Cassandra as data source.

In this example it is assumed that exactly three Cassandra nodes must be configured. To enable database center specific Cassandra nodes, the node must be retrieved from the envSettings.props of the gateway instance.

To support stage specific configurations, the following settings must be environmentalized:

Credentials for Cassandra connection
TLS security switch
Replication strategy and replication factor (KPS tables and throttling)
Read/write consistency level (KPS tables and throttling)

Example rules file for Cassandra

rules:
  #
  # Cassandra Settings
  #
  "cassandra_settings": (1)
    name: Cassandra Nodes
    description: Check Cassandra nodes

    fileType: CassandraSettings (2)
    filePatterns:
      - '/Server Settings/Cassandra Settings.yaml'

    assertions:
      # Three nodes should be configured to ensure quorum.
      - path: '$.children[0].fields.host'
        type: exists
        message: Three Cassandra nodes must be configured to ensure quorum
      - path: '$.children[1].fields.host'
        type: exists
        message: Three Cassandra nodes must be configured to ensure quorum
      - path: '$.children[2].fields.host'
        type: exists
        message: Three Cassandra nodes must be configured to ensure quorum
      - path: '$.children[3].fields.host'
        type: not_exists
        message: Only three Cassandra nodes must be configured to ensure quorum

      # Cassandra nodes should be configured by envSettings.props to support
      # datacenter aware node connections.
      #
      - path: '$.children[0].fields.host'
        type: regex
        param: '^\$\{env\..+\}'
        message: Cassandra nodes must be configured via envSettings.props

      - path: '$.children[1].fields.host'
        type: regex
        param: '^\$\{env\..+\}'
        message: Cassandra nodes must be configured via envSettings.props

      - path: '$.children[2].fields.host'
        type: regex
        param: '^\$\{env\..+\}'
        message: Cassandra nodes must be configured via envSettings.props

      # Credentials must be environmentalized
      - path: '$.fields.username'
        type: environmentalized

      - path: '$.fields.password'
        type: environmentalized

      # SSL security must be environmentalized
      - path: '$.fields.useSSL'
        type: environmentalized

      # Replication strategy and factors must be environmentalized
      - path: '$.fields.replication'
        type: environmentalized

      - path: '$.fields.replicationStrategy'
        type: environmentalized

      - path: '$.fields.throttlingReplication'
        type: environmentalized

      - path: '$.fields.throttlingReplication'
        type: environmentalized

      - path: '$.fields.throttlingReplicationStrategy'
        type: environmentalized

      # Consistency level for throttling must be environmentalized
      - path: '$.fields.throttlingReadConsistencyLevel'
        type: environmentalized

      - path: '$.fields.throttlingWriteConsistencyLevel'
        type: environmentalized

  # Cassandra for KPS Data Sources
  "kps_data_source": (3)
    name: KPS Data Source
    description: Check data source for KPS tables

    fileType: KPSCassandraDataSource (4)
    filePatterns:
      - '/Environment Configuration/Key Property Stores/.*/.+\.yaml'

    # Consistency levels must be environmentalized
    assertions:
      - path: '$.fields.readConsistencyLevel'
        type: environmentalized

      - path: '$.fields.writeConsistencyLevel'
        type: environmentalized

1	ID of the rules for Cassandra settings.
2	Cassandra settings are stored in the entity `/Server Settings/Cassandra Settings.yaml` of type `CassandraSettings`.
3	ID of the rules for KPS data sources.
4	KPS settings for Cassandra data sources are stored in entities matching the file path `/Environment Configuration/Key Property Stores/.*/.+\.yaml` and which are of type `KPSCassandraDataSource`

Database Connections

The following example checks the configuration of external database connections.

To enable stage specific configuration, the following fields of the database connections must be environmentalized:

Connection URL
User name
Password

Example rules file for database connections

rules:
  # Database connections must be environmentalized
  "db_connections": (1)
    name: DB Connection Environmentalization
    description: Connection to databases are usually stage specific and Field must be environmentalized

    fileType: DbConnection (2)
    filePatterns:
      - '/External Connections/Database Connections/.*'

    assertions:
      - path: '$.fields.username'
        type: environmentalized
        message: User for DB must be environmentalized

      - path: '$.fields.password'
        type: environmentalized
        message: Password for DB must be environmentalized

      - path: '$.fields.url'
        type: environmentalized
        message: DB connection URL must be environmentalized

1	ID of the database connection rules.
2	Database connections are stored in entities under the `/External Connections/Database Connections` folder which are of type `DbConnection`.

2.3. Evaluate Expression

For testing purpose or to generate values from various sources, an expression can be evaluated and the result is printed to stdout.

2.3.1. Command

The eval command of yamlesutils is used to evaluate an expression.

yamlesutils eval [-l FILE]... <expression>

Example

$ export NAME=World
$ yamlesutils.sh eval "Hello {{ _env('NAME') }}!"
Hello World!

2.4. Merge Configuration Fragments

For the YAML entity store, environmentalized fields are configured in the values.yaml file of the project directory. To configure the project for other stages, the whole values.yaml file has to be replaced by a file containing the new values. Also, only static values are allowed.

If you want to split the configuration into multiple files, e.g. due to separate responsibility, or, if you want to retrieve values dynamically from other sources, the values.yaml file has to be dynamically generated.

The YAML-ES utilities enables to merge a values.yaml file from multiple fragments. The order of the fragments is considered. If a field is provided by multiple fragments, the value of the latest fragment is used.

After all fragments are merged, the values of each field is evaluated as a Mustache template. If the value contains expressions delimited by double curly brackets (e.g., {{ expression }}), the block will be replaced by the result of the expression. For example, expressions can be used to encode values as Base64 (required for encrypted fields), or to lookup values from external sources, etc.

Each value of the YAML properties is handled as a separate template. So it’s not possible to use templates outside or across properties.

Example: Merge three fragments into a combined values.yaml file

Merge Configuration Fragments

In the above example, the env lookup function is used to retrieve the value from an environment variable. This function is provided as a built-in function by _YAML-ES Utilities. Custom lookup functions can be defined to retrieve values from other lookup providers (e.g. KeePass DB or AWS Secret Manager).

See Templates for more details about the evaluation of Mustache templates and the usage of lookup functions.

To merge configuration fragments, use the merge config command of YAML-ES Utilities.

yamlesutils merge [-m MODE] config
    [-f=FILE]... [-d=DIR]...
    [--lookup-functions=FILE]...
    (-o=FILE | [--project=DIR | --ignore-missing-values])

Table 4. Options for Merge Configuration
Option	Description
`-m MODE` `--mode=MODE`	Execution mode `CONFIG`: Build and write configuration (default). `DRY_RUN`: Build configuration, but don’t write to disk. `SYNTAX_CHECK`: Check syntax of configuration files only. Lookups are evaluated to empty strings; no connections to external systems (e.g. AWS Secrets Manager) are established.
`-f FILE` `--fragment=FILE` `-c FILE` `--config=FILE`	Configuration fragment to be merged to the `values.yaml` file. The fragments are files, containing a single YAML document. The parameter can be specified multiple times to merge multiple fragments. The fragments are processed in the order as specified on the command line. If a configuration parameter is configured by in multiple files, the last one wins.
`-d DIR` `--dir=DIR`	Directory to scan for YAML configuration fragments. Each file within the directory having the extension `.yml` or `.yaml` are treated as configuration fragments. The files are added to merged in alphabetical ascending order.
`-o FILE` `--output=FILE`	Target file to write the merged YAML configuration. If `-` is specified, the merged configuration is written to `stdout`.
`--project=DIR`	Path to the directory containing the YAML-ES project. If the project path is specified the `values.yaml` file within this directory will be overwritten.
`--ignore-missing-values`	If a project directory is specified, the current `values.yaml` file of this directory is loaded and all required fields are detected. Before writing the merged configuration, the merge command checks if all required fields are also provided by the merged configuration. On missing fields, the merge command stop with an error message. With this option, missing fields are ignored and the merge command continue processing. Missing or unused fields are just reported.
`-l FILE` `--lookup-functions=FILE`	YAML file to configure lookup functions. Multiple parameters can be specified to use multiple lookup configurations.

2.4.1. Basic Examples

The following files, used for this example are also located in the examples/basic/config folder.

There are two configuration fragments:

fragments/global.yaml

# Define some global defaults
global:
  greeting:
    message: "Hello World!"
    enabled: true

fragments/local.yaml

# Overwrite some global defaults
global:
  greeting:
    message: "Bonjour le monde"

# Define local configurations
local:
  country: "fr"
  application:
    name: "Application Extraordinaire"

To merge the fragments use the following command:

Simple merge of multiple fragments

yamlesutils.sh merge config \ (1)
  --fragment=fragments/global.yaml \ (2)
  --fragment=fragment/local.yaml \ (3)
  -o - (4)

1	Command to merge configuration fragments
2	The `global.yaml` fragment
3	The `local.yaml` fragment, which overwrites values from the `global.yaml`.
4	Print the result to `stdout`

This will generate the following YAML document:

Result

global:
  greeting:
    message: "Bonjour le monde" (1)
    enabled: true (2)
local: (3)
  country: "fr"
  application:
    name: "Application Extraordinaire"

1	Overwritten by `local.yaml`.
2	Provided by `global.yaml`.
3	Completely provided by `local.yaml`.

Now we want to add an additional fragment:

fragments/users.yaml

local:
  accounts:
    admin:
      upn: "{{ _users('/users/admin/id') }}" (1)
      name: "{{ _users('/users/admin/name') }}" (1)
      pwd: "{{ _users('/users/admin/password') | base64encode }}" (2)

1	Fields use Mustache templates to lookup values using the function `_users()`. The parameter specifies the JSON Path to the property within the external JSON file (see below).
2	The lookup value will be additionally encoded as Base64.

The fields contain Mustache templates, using a custom lookup function _users() to retrieve values from an external source. To use the custom function, a provider for this function must be defined. In this case we want to retrieve the values from a JSON file users.json.

lookups/users.json

{
  "users": {
    "admin": {
      "id": "admin",
      "name": "Administrator",
      "password": "changeme"
    }
  }
}

The provider is defined in a separate file lookup-func.yaml.

lookup-func.yaml

lookups:
  users: (1)
    provider: json (2)
    config:
      file: lookups/users.json (3)

1	Defines the alias `users` for the lookup function configuration. The corresponding lookup function is prefixed by an underscore `_users()`.
2	Use the JSON lookup provider to retrieve values from a JSON file.
3	Path to the JSON file. If the path is not absolute, the path is relative to the directory containing the lookup function configuration file (directory of `lookup-func.yaml`).

To merge these fragments, an additional parameter --lookup-functions must be specified to configure the required lookup function.

yamlesutils.sh merge config \ (1)
  --lookup-functions=lookup-func.yaml \ (2)
  -f fragments/global.yaml -f fragments/local.yaml -f fragments/users.yaml \ (3)
  -o - (4)

1	Command to merge configuration fragments.
2	Path to the lookup function configuration file.
3	The configuration fragments to be merged.
4	Print the result to `stdout`

Result

global:
  greeting:
    message: "Bonjour le monde"
    enabled: true
local:
  country: "fr"
  application:
    name: "Application Extraordinaire"
  accounts: (1)
    admin:
      upn: "admin" (2)
      name: "Administrator" (2)
      pwd: "Y2hhbmdlbWU=" (3)

1	The YAML document is similar to the previous result. Except the addition `accounts` object, provided by the `users.yaml` fragment.
2	The templates specified in the fragment are resolved and the values are looked up from the external JSON file.
3	The value is additionally Base64 encoded.

2.5. Merge Certificates

For the YAML entity store, certificates and private keys are configured in the Environment Configuration/Certificate Store directory of the project. For each alias configured in the Policy Studio project a corresponding YAML file exists within this folder.

Without YAML-ES Utilities, the certificates have to managed manually via Policy Studio or by editing the files directly. This may be an error prone task, makes it difficult to keep the project independent from the target environment, and may cause to store credentials (e.g., private keys), in the Git repository by accident.

YAML-ES Utilities helps to retrieve certificates from various sources and to configure the certificates during deployment time of the API Gateway.

The configuration of certificates is organized by, so called, Certificate Configuration files. Each configuration defines certificate aliases and the according source to retrieve the certificate or private key.

Certificates are provided by, so called, Certificate Providers. For each alias a certificate provider must be defined and configured accordingly.

Certificate Configuration

certificates: (1)
  <alias-1>: (2)
    provider: <provider> (3)
    config: (4)
      <provider configuration> (5)

1	Indicator for certificate configuration.
2	Certificate alias. Certificates are referenced by the alias within the policy project.
3	Type of the source of the certificate (provider of the certificate).
4	Section to configure the provider.
5	Provider specific configuration to retrieve the certificate

To configure the certificates of a project, the target project directory and a list of certificate configuration files must be specified (see help message below). Some parameters support Mustache Templates to lookup or transform values. If non built-in lookup functions are used, the lookup function have to be configured by a Lookup Function Configuration file which is specified by the --lookup-functions parameter.

yamlesutils merge [-m MODE] certs --project=DIR
    -c=FILE [-c=FILE]... [--lookup-functions=FILE]...
    [--expiration-warning=DAYS] [----expiration-error=DAYS]
    [--expiration-fail=DAYS]

Option Description

-m MODE
--mode=MODE

Execution mode

CONFIG: Build and write configuration (default).
DRY_RUN: Build configuration, but don’t write to disk.
SYNTAX_CHECK: Check syntax of configuration files only. Lookups are evaluated to empty strings; no connections to external systems (e.g. AWS Secrets Manager) are established.

-c FILE
--certs=FILE
--config=FILE

Certificates configuration file.

The parameter can be specified multiple times to merge multiple certificate configurations.

The configurations are processed in the order as specified on the command line. If an alias is defined by multiple files, the last one wins.

--project=DIR

Path to the directory containing the YAML-ES project.

To avoid any commits of confidential certificates into the repository, projects should be copied to a temporary folder before the certificate configuration is applied to the temporary folder.

--expiration-warning=DAYS

If the certificate expires within the defined number of days, a warning message will be logged into the audit log.

Set to 0 to disable warnings.

--expiration-error=DAYS

If the certificate expires within the defined number of days, an error message will be logged into the audit log.

Set to 0 to disable errors.

--expiration-fail=DAYS

If the certificate expires within the defined number of days, the command fails.

Set to -1 to disable failure.

-l FILE
--lookup-functions=FILE

YAML file to configure lookup functions.

Multiple parameters can be specified to use multiple lookup configurations.

2.5.1. Basic Example

In this example, a new public certificate has to be added to the project.

Certificate Configuration

---
certificates:
  new-pub-cert: (1)
    provider: simple (2)
    config: (3)
      cert: "MIIDdDCCAlygAwIBAgIIGNcwhDcWwucwDQYJKoZIhvcNAQELBQAwQDELMAkGA1UEBhMCREUxDzANBgNVBAcTBkJlcmxpbjEOMAwGA1UEChMFQXh3YXkxEDAOBgNVBAMTB3Jvb3QtY2EwHhcNMjIwNjA5MTUzNzAwWhcNMzIwNjA5MTUzNzAwWjBAMQswCQYDVQQGEwJERTEPMA0GA1UEBxMGQmVybGluMQ4wDAYDVQQKEwVBeHdheTEQMA4GA1UEAxMHcm9vdC1jYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM8nhhsJhlIHOJ/jTJJymP9bIB/rTx/WW/3d6imn4nJkD3oatytC2++wPkzMl0f+ROWEJ5odHBZNexHDDrdST7gB6RBu6OP7O4uqBPNf9nkYdT5n4ZS/i7sjW4YawOHqaSHq320JCkBDPtfpaXj/yFgzVYfPaHFSPwsNiz5XsfnSLmW8aIL0mit0R9OIWofExjRv1avQERbu36g0Z3rp4oE77Wk6XpcfNCrw4w3DgTIA6GBIQot2mjpfoUS7+z8xbZMJp2qlXqmdqUS3ngu/cvBEaUppzBVQA9oWsR2AWTts7/HZedZfQGj3olOVyNBVTWnS0YSm6Qgagph9f/Nk0jECAwEAAaNyMHAwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU7wJgQdJ3yFFF40yn89xrHHCZmhUwCwYDVR0PBAQDAgEGMBEGCWCGSAGG+EIBAQQEAwIABzAeBglghkgBhvhCAQ0EERYPeGNhIGNlcnRpZmljYXRlMA0GCSqGSIb3DQEBCwUAA4IBAQAUdhmR1jiRLBuf05OX58KuqyRyZ5ZrHdPG9gpZkA8gdzwYGeNgv3biBXpOmEuK1/0CpU8lNyyok+UWkfTl2Ei0JVeDqi+8cFQUQBZZ4qXG0bFjKEAXBommeyxo37MQBJOb3Fp/WKlfcR03KhUhkPmH1PirKwX1rPDiup6dIQ+OjcwnqsLBk3dcBZFDdPq+BzLJkF7hqShlmWjUJ/ffLzOPc/H2FeiADS/9hZLnHaLy+JDWDr3GsQA8lRlhYeanlw+s1V5j4UXjFq6MMabZAawZ9rzGIDs74CXeThLyAyAh8i1iW/FJYX20v+twzsJ4a1rVqAbBv0h06vW7XsnTgIuB"

1	Alias of the new certificate within the policy project.
2	Certificate provider
3	Configuration of the certificate provider. Here, a simple PEM encoded public certificate, specified directly in the configuration file.

The certificate can be add to the project, by using the following YAML-ES Utilities command.

# Copy project to temporary folder
PRJ_SRC=../../../apim
TMP_PRJ="/tmp/prj"

mkdir -p ${TMP_PRJ}
cp -rt "${TMP_PRJ}" "${PRJ_SRC}/."

# Configure certificates
yamlesutils.sh merge certs \ (1)
  --project=${TMP_PRJ} \ (2)
  --config=certificates.yaml (3)

1	Merge certificates.
2	Path to policy project.
3	Certificate configuration file.

2.6. Merge Files

The lookup functions enable YAML-ES Utilities to retrieve values from various secret stores. With the merge files command this capability can be used to generate files for which the content is retrieved from the secret stores (e.g. for licence files).

The file generator is configured by a configuration file specified by the --files=FILE parameter. With a preceeding optional --files-base-dir=DIR parameter, the base directory for target files with a relativ path is specified. Also the base directory for template files can be specified by the --files-base-dir-src parameter.

The tripple of --files-base-dir, --files-base-dir-src and --files parameter can be specified multiple times.

yamlesutils.sh merge files [-hV] [-l=FILE]... ( [--files-base-dir=DIR][--files-base-dir-src=DIR] --files=FILE)...
Generate files.
      --files=FILE              Files generator configuration.
      --files-base-dir=DIR      Base directory for generated files.
      --files-base-dir-src=DIR  Base directory for source files.
  -h, --help                    Show this help message and exit.
  -l, --lookup-functions=FILE   Configure lookup functions.
  -V, --version                 Print version information and exit.

Example

yamlesutils.sh merge files \
  --files-base-dir=/opt/Axway/apigateway \
  --files=anm-files.yaml

File Generator Configuration (anm-files.yaml)

files:
  - path: system/conf/advisorybanner.json (1)
    encoding: UTF-8 (2)
    content: | (3)
      {
        "bannerEnabled" : true,
        "bannerText" : "{{ _env('STAGE') | escape(strategy='js') }}"
      }

  - path: /opt/Axway/README.txt (4)
    encoding: ISO-8859-1
    content: Hello World (5)

  - path: /opt/Axway/test.bin
    encoding: binary (6)
    content: "{{ 'Hello World' | base64encode }}" (7)

  - path: conf/adminUsers.json
    encoding: UTF-8
    template: /merge/config/adminUsers.json.tpl (8)

  - path: /opt/Axway/apigateway/ext/custom/jvm.xml (9)
    encoding: UTF-8
    content: /merge/config/jvm.xml.tpl
    createDirs: true (10)

  - path: conf/README.txt
    encoding: UTF-8
    template: templates/README.txt.tpl (11)

1	File to be generated. Path is relative to the directory containing the files configuration file, or the directory specified by the `--files-base-dir` parameter.
2	Encoding used to write the file content.
3	Inline template.
4	File to be generated. Absolute file path.
5	Inline file content.
6	Binary file.
7	For binary files, content must be a Base64 encoded value.
8	File containing a template (Pebble Engine - see example below).
9	Target file is located in a non-existing directory.
10	Automatically create non-existing directories.
11	File containing a template. Path is relative to the directory containing the files configuration file, or the directory specified by the `--files-base-dir-src` parameter.

adminUsers.json.tpl

{
  ...

  "adminUserCredentials" : {
    "user-1" : {
      "passwordHistory" : [ "{{ _gen_anm_pwd_hash(_env('ANM_ADMIN_PASSWORD')) }}" ],
      "passwordTimestamp" : 1709227593453
    }
  },

  ...
}

2.7. Configure YAML Entity Store

The config command combines the features of the lint, merge certs and merge config commands into one single command.

yamlesutils config [-m MODE] --project=DIR
  [-r RULE]...
  [-f FILE]...
  [--lookup-functions=FILE]...
    (-o=FILE | [--project=DIR | --ignore-missing-values])

Option Description

--project=DIR

Path to the directory containing the YAML-ES project.

-m MODE
--mode=MODE

Execution mode

CONFIG: Build and write configuration (default).
DRY_RUN: Build configuration, but don’t write to disk.
SYNTAX_CHECK: Check syntax of configuration files only. Lookups are evaluated to empty strings; no connections to external systems (e.g. AWS Secrets Manager) are established.

-l FILE
--lookup-functions=FILE

YAML file to configure lookup functions.

Multiple parameters can be specified to use multiple lookup configurations.

-r FILE
--rules=FILE

Path to file containing linting rules.

Multiple files are allowed. If missing, the linting feature is not executed.

-f FILE
--fragment=FILE
--config=FILE

Configuration fragment to be merged to the values.yaml file. The fragments are files, containing a single YAML document.

The parameter can be specified multiple times to merge multiple fragments.

The fragments are processed in the order as specified on the command line. If a configuration parameter is configured in multiple files, the last one wins.

If the parameter is missing, the values.yaml file will not be configured.

--ignore-missing-values

If a project directory is specified, the current values.yaml file of this directory is loaded and all required fields are detected.

Before writing the merged configuration, the merge command checks if all required fields are also provided by the merged configuration. On missing fields, the merge command stop with an error message.

With this option, missing fields are ignored and the merge command continue processing. Missing or unused fields are just reported.

--certs=FILE

Certificates configuration file.

The parameter can be specified multiple times to merge multiple certificate configurations. If missing, no certificates will be configured.

The configurations are processed in the order as specified on the command line. If an alias is defined by in multiple files, the last one wins.

To avoid any commits of confidential certificates into the repository, projects should be copied to a temporary folder before the certificate configuration is applied to the temporary folder.

--expiration-warning=DAYS

If the certificate expires within the defined number of days, a warning message will be logged into the audit log.

Set to -1 to disable warnings.

--expiration-error=DAYS

If the certificate expires within the defined number of days, an error message will be logged into the audit log.

Set to -1 to disable errors.

--expiration-fail=DAYS

If the certificate expires within the defined number of days, the command fails.

Set to -1 to disable failure.

2.8. Built-In Documentation

YAML-ES Utilities provides a built-in documentation of the certificate providers, lookup providers, and the available lookup functions.

yamlesutils.sh merge describe help
Usage: yamlesutils merge describe [-hV] [COMMAND]
Describe features of the YAML-ES Utilities used for merging.
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
Commands:
  lookup-providers  Describe the available lookup providers.
  cert-providers    Describe the available certificate providers.
  functions         Describe available functions.
  help              Display help information about the specified command.

2.8.1. Lookup & Certificate Providers

To get an overview of the certificate and lookup providers, invoke one of the following commands:

yamlesutils.sh merge describe cert-providers
yamlesutils.sh merge describe lookup-providers

To show the full description, use the --full option.

yamlesutils.sh merge describe cert-providers --full
yamlesutils.sh merge describe lookup-providers --full

2.8.2. Lookup Functions

To view the available built-in lookup functions, use the following command:

yamlesutils.sh merge describe functions
Available Lookup Functions

_env [env+]         Lookup values from environment variables.

                    Lookup Function Arguments:
                    key: name of environment variable

_sys [sys+]         Lookup values from system properties.

                    Lookup Function Arguments:
                    key: name of system property

+    built-in lookup providers

To show all functions which are available for the merge commands, specify the same --lookup-functions options as for the merge commands.

yamlesutils.sh merge describe functions --lookup-functions=lookup-func.yaml

3. Templates

3.1. Mustache Templates

The values of configuration fragments are treated as Mustache templates. After all configuration fragments are merged, the values will be evaluated by a template engine. Values containing {{ … }} statements are handled as templates, and are processed by the built-in template engine, before the resulting values.yaml file is written.

The name Mustache is derived from the used { characters, which resemble a sideways moustache.

YAML-ES Utilities use Pebble Templates as the internal template engine. All functions provided by this engine can also be used within the Mustache templates. Additionally to the functions provided by the template engine, custom lookup functions can be defined to lookup values from external sources.

Please consult the Pebble documentation for the syntax of templates and the capabilities provided by Pebble.

Each value of the configuration fragments or configuration parameters, are handled as a separate template. So it’s not possible to use templates outside or across properties.

Invalid Configuration Fragment

db:
  user: "{{ _env('USER_NAME') }}" (1)
  {{ 'password: "changeme"' }} (2)

1	Templates within properties are allowed
2	Templates outside of properties are not supported

For the Pebble Templates engine a CVE-2022-37767 exists. According to the CVE any arbitrary Java method can be invoked by the templates.

The Pebble Templates engine within YAML-ES Utilities is configured to prevent the call of Java method. No Java method can be called within the templates. So, the example below will throw an exception.

{% set value=\"Hello\".toString() %}{{ value }}

Therefore, YAML-ES Utilities is not vulnerable by CVE-2022-37767.

As reported CVEs may be an issue in the context of an enterprise, even if the tool is not affected, the Pebble Templates engine may be replaced in the future.

3.2. Lookup Functions

Lookup functions can be used inside Mustache Templates, and are used to retrieve values from external sources. To avoid any conflict with Pebble functions, all lookup function names start with an underscore.

Arguments are passed as positional or named arguments. For positional arguments, the arguments must match the documented/expected order of the lookup function arguments. For named arguments, the order of the arguments doesn’t matter.

Named Arguments

{{ _function(key="test", option="opt") }}

Positional Arguments

{{ _function("test", "opt") }}

Lookup functions are implemented by, so called, Lookup Providers.

Built-in lookup functions are available without any additional configuration. The name of the built-in lookup function is fixed and defined by the Lookup Provider implementing the function.

For example, the Core plugin provides the following built-in lookup functions:

_env("env_name"): Lookup values from an environment variable.
_sys("prop_name"): Lookup values from system properties.

Other lookup functions can be dynamically configured via lookup function configuration files. Within these files, aliases for the lookup sources are defined. The lookup sources are configured via a parameterized lookup provider (see Plugins for a list of lookup providers implemented by the according plugin). For each alias a lookup function will be registered. The name of the function is derived from the alias name by adding an underscore.

Aliases must be unique over all function configurations. As the env and sys alias is already used by the built-in functions, these aliases are not allowed for custom lookup functions.

Lookup Functions Configuration

lookups: (1)
  dev_kdb: (2)
    provider: keepass (3)
    config: (4)
      kdb: "../all/lookups/devs-secrets.kdbx" (5)
      passphrase: "{{ _env('ENV_KDB_PASSPHRASE') }}" (6)

1	Root object for lookup definitions.
2	Alias of the lookup function. The name of lookup function is prefixed with a `_` (e.g. `_dev_kdb`). Aliases must be unique within all applied lookup provider configurations.
3	Lookup provider used by the function.
4	Configuration parameters for the lookup provider.
5	Fixed configuration parameter.
6	Configuration parameters retrieved via lookup function (only built-in functions supported).

The YAML-ES Utilities provide list of available lookup functions with the following command. To list all functions as used for the merge command, the lookup function configuration files can be specified in the same way as for the merge command. If the missing, only the built-in functions are listed.

$ yamlesutils merge [--lookup-functions=FILE]... describe functions

4. Plugins

Plugins are used to extend the features of the YAML-ES Utilities by providing Lookup- or Certificate Providers.

A plugin is a single JAR file containing the classes plugin implementation, and all of its dependencies. To prevent conflicts with different versions of dependent JARs, each plugin has its own class loader.

Are located in the plugins folder and are automatically loaded on startup of the YAML-ES Utilities.

To save the disk space, unused plugins can be removed from the folder.

Also, custom developed plugins can be copied to the folder and are loaded automatically.

The following sections document the available plugins and their provided Lookup- and Certificate Providers.

4.1. Core

4.1.1. Lookup Providers

Lookup values are provided by, so called, Lookup Providers. An instance of a Lookup Provider is configured for specific sources to provide a lookup function.

To get a list of all available Lookup Providers and their documentation, use the following describe command:

$ yamlesutils merge describe lookup-providers --full

Some lookup provider configurations require a path to a file. If the path represents a relative path, it is relative to the location of the lookup function definition file.

Environment Variables (built-in)

Name

env

Built-In

yes

Synopsis

Retrieves value from an environment variable.

Configuration Parameters

not required

Lookup Function Arguments

key

Name of environment variable.

This built-in lookup provider retrieves values from environment variables. It is used by the built-in lookup function _env. Configuration of this lookup provider is not required.

The lookup function requires a single key parameter which represents the name of the environment variable containing the value.

Example

{{ _env("MY_SECRET") }} (1)

1	Retrieves the value from the environment variable `MY_SECRET`.

System Properties (built-in)

Name

sys

Built-In

yes

Synopsis

Retrieves value from a system property.

Configuration Parameters

not required

Lookup Function Arguments

key

Name of system property.

This built-in lookup provider retrieves values from system properties passed to the YAML-ES Utilities. It is used by the built-in lookup function _sys. Configuration of this lookup provider is not required.

The lookup function requires a single key parameter which represents the name of the system property containing the value.

Example

{{ _sys("my.secret") }} (1)

1	Retrieves the value from the system property `my.secret`.

Admin Node Manager Password Hash (built-in)

Name

gen_anm_pwd_hash

Built-In

yes

Synopsis

Generates a hashed password for the Admin Node Manager.

Configuration Parameters

not required

Lookup Function Arguments

pwd

Clear text password.

This built-in lookup provider generates a hashed password, which can be used for the Admin Node Manager inside the adminUsers.json configuration file.

This provider is used by the built-in lookup function _gen_anm_pwd_hash. Configuration of this lookup provider is not required.

The lookup function requires a single key parameter which represents the clear text password to be hashed. The password can also be retrived by calling a nested lookup function.

Example

{{ _gen_anm_pwd_hash("change") }} (1)
{{ _gen_anm_pwd_hash(_env("ADMIN_PASSWORD")) }} (2)

1	Generates a hased password from a clear text password.
2	Generates a hased password from the value of the `ADMIN_PASSWORD` environment variable.

JSON from Environment Variables

Name

envjson

Built-In

Synopsis

Retrieves values from a JSON document, stored in an environment variable.

Configuration Parameters

env

Name of the environment variable storing the JSON document.

Lookup Function Arguments

key

JSON Pointer to key node (see RFC 6901).

Retrieves values from a JSON document, which is stored within an environment variable. The name of the environment variable containing the JSON document is configured by the alias in the lookup functions configuration.

The according lookup function requires a key parameter which points to the value node within the JSON document.

This feature can be used to pass the JSON output of a tool as a lookup source to the YAML-ES Utilities.

Example

This example uses a script to generate lookup values, and use them within a configuration template.

Generate Lookup Values gen-values.sh

#!/bin/sh
cat <<EOF
{
  "accounts": {
    "db": {
      "user": "admin",
      "password": "changeme"
    }
  }
}
EOF

Lookup Function Definition lookup-func.yaml

lookups:
  accounts: (1)
    provider: envjson (2)
    config:
      env: ACCOUNTS (3)

1	Define alias for the configured lookup functions. The name of the functions will be `_accounts` (alias prefixed with underscore).
2	Use this lookup provider.
3	Retrieve the JSON document from the environment variable `ACCOUNTS`.

Configuration Fragment config.yaml

db:
  user: "{{ _accounts('/accounts/db/user') }}" (1)
  pwd: "{{ _accounts('/accounts/db/password') }}" (2)

1	Lookup user name from JSON document by JSON Pointer.
2	Lookup password from JSON document by JSON Pointer.

Create configuration

ACCOUNTS=$(./gen-values.sh) (1)
yamlesutils.sh merge config \ (2)
  --lookup-functions=lookup-func.yaml \ (3)
  --config=config.yaml \ (4)
  --output=- (5)

1	Generate JSON document with lookup values and store it in the environment variable `ACCOUNTS`.
2	Create a configuration.
3	Lookup function definitions.
4	Configuration fragment.
5	Print result to `stdout`.

Result

db:
  user: admin
  password: changeme

YAML/JSON Files

Name

json or yaml

Built-In

Synopsis

Retrieves values from a JSON/YAML file.

Configuration Parameters

file

Path to the JSON/YAML file containing lookup values.

Lookup Function Arguments

key

JSON Pointer to key node (see RFC 6901).

Example

In this example there is a JSON and a YAML file containing lookup values.

JSON Lookup Values accounts.json

{
  "accounts": {
    "db": {
      "user": "admin",
      "password": "changeme"
    }
  }
}

YAML Lookup Values accounts.yaml

accounts:
  smtp:
    email: "info@axway.com"
    password: "changeme"

Lookup functions are configured to retrieve values from these files.

Lookup Function Definitions lookup-func.yaml

lookups:
  accounts_json:
    provider: json
    config:
      file: accounts.json
  accounts_yaml:
    provider: yaml
    config:
      file: accounts.yaml

A configuration fragment uses the lookup functions to retrieve the configuration values from the according files.

Configuration Fragment config.yaml

db:
  user: "{{ _accounts_json('/accounts/db/user') }}"
  pwd: "{{ _accounts_json('/accounts/db/password') }}"
mail:
  user: "{{ _accounts_yaml('/accounts/smtpd/user') }}"
  pwd: "{{ _accounts_yaml('/accounts/smtpd/password') }}"

YAML-ES Utilities is used to create a final configuration.

Create configuration

yamlesutils.sh merge config \
  --lookup-functions=lookup-func.yaml \
  --config=config.yaml \
  --output=-

Result

db:
  user: admin
  pwd: changeme
mail:
  user: info@axway.com
  pwd: changeme

File Content

Name

file

Built-In

Synopsis

Retrieves the content of a file as a string.

For binary content (encoding: binary), the content is returned as a Base64 encoded string.

Configuration Parameters

base

Base directory for relative files. If missing, the directory containing the lookup definition file is used.

Lookup Function Arguments

key

File path.

encoding

Character encoding of the content. If missing, UTF-8 is assumed.

For binary content, use binary.

Example

In this example there is a file containing a greeting message.

/opt/text/greeting.txt

Hello World!

Lookup function is configured to retrieve values from file content.

Lookup Function Definitions lookup-func.yaml

lookups:
  contents:
    provider: file
    config:
      base: "/opt/text"

A configuration fragment uses the lookup function to retrieve the content of the greeting.txt file.

Configuration Fragment config.yaml

greeting: "{{ _contents('greeting.txt') }}" (1)

1	As the file path is not absolute, the path is relative to the specified base path. So the final path is `/opt/text/greeting.txt`.

4.1.2. Certificate Providers

The purpose of a certificate provider is to retrieve certificates or private keys from external sources. Various providers exists to support various certificate sources.

To get a list of all available Certificate Providers and their documentation, use the following describe command:

$ yamlesutils merge describe cert-providers --full

Some lookup provider configurations require a path to a file. If the path represents a relative path, it is relative to the location of the certificate configuration file.

Simple

Name

simple

Synopsis

Provides certificates directly from configuration file.

Configuration Parameters

cert

PEM encoded certificate (single line).

This parameter supports a Mustache template, to lookup the certificates via lookup functions.

key

Optional PEM encoded private key (single line).

This parameter supports a Mustache template, to lookup the certificates via lookup functions.

Example

Certificate Configuration

certificates:
  cassandra-ca:
    provider: simple
    config:
      cert: "MIID...uB" (1)

  apim-server:
    provider: simple
      cert: "MIIDt...xdI=" (2)
      key: "MIIEv...CL+X"

1	Public certificate
2	Server certificate including private key.

In combination with a lookup function, the simple provider can also be used to retrieve certificates from a KeePass DB or AWS Secrets Manager.

Certificate File

Name

file

Synopsis

Provides a certificate from a DER or PEM encoded certificate file.

Configuration Parameters

path

Path to certificate file.

If a relative path is specified, the path is relative to the location of the Certificate Configuration file.

Example

Certificate Configuration

certificates:
  root-ca:
    provider: file
    config:
      path: "certs/root-ca.crt"

Keystore

Name

keystore

Synopsis

Provides certificates from a keystore file (PKCS#12 or JKS).

Configuration Parameters

path

Path to the keystore file.

If a relative path is specified, the path is relative to the location of the Certificate Configuration file.

data

Base64 encoded keystore.

This parameter supports a Mustache template, to retrieve the encoded keystore data via lookup functions.

path and data parameter are mutually exclusive

pass

Optional passphrase to access the keystore.

This parameter supports a Mustache template, to lookup the certificates via lookup functions.

alias

Optional regular expression, to select certificates within the keystore by their alias. If not specified, the target alias of the entity store certificate is used.

To select all certificates from the keystore, the regular expression .* can be used.

nokey

If true, the private key is not added to the project. For certificates having no private key, this parameter has no effect.

type

Type of the keystore. If not specified, `PKCS12`is assumed.

JKS: Java Key Store
PKCS12: PKCS#12

chain

Set to true to add the full certificate chain of the certificate to policy project.

Examples

Certificate Configuration - Single Certificate

certificates:
  example-server:
    provider: keystore
    config:
      path: keystore.p12 (1)
      pass: "{{ _kdb('/Test/Sever Certificate', 'password') }}" (2)
      alias: server (3)
      chain: true (4)

1	Path to the keystore. As the extension is `.p12` the keystore is assumed to be in the PKCS#12 format.
2	The passphrase for the keystore is retrieved from a lookup function.
3	Alias of the certificate within the keystore.
4	Add certificates of associated certificate authority. The alias for chain certificates is based on the target alias, attached by the postfix `_chain_<index>` where `<index>` is an increasing number. The root certificate of the chain has the index 0 (e.g. `example-server_chain_0`).

Certificate Configuration - All Certificates

certificates:
  trust: (1)
    provider: keystore
    config:
      path: keystore.p12
      pass: "{{ _kdb('/Test/Sever Certificate', 'password') }}"
      alias: ".*" (2)
      nokey: true (3)

1	Use `trust` prefix for alias in YAML-ES. Alias is generated by the prefix and an attached index (e.g. `trust_0`, `trust_1`, …)
2	Get all certificates from the keystore.
3	Don’t get private keys; certificates only.

Remover

Name

remover

Synopsis

Removes a certificates from the policy project.

Configuration Parameters

not required

The is not really a certificate provider. Instead it removes certificates from the policy project. The certificates are specified by their alias.

Example

Certificate Configuration

certificates:
  acme: (1)
    provider: remover

1	Alias of the certificate within the policy project.

4.2. KeePass Database

4.2.1. Lookup Providers

KeePass DB

KeePass is a popular password manager using a single encrypted file as a database. It can be used to securely store credentials, configurations and files.

Name

keepass

Built-In

Synopsis

Retrieves values from a KeePass database.

Configuration Parameters

kdb

Path to the KeePass DB file.

passphrase

Passphrase for KeePass DB.

This parameter supports a Mustache template, using built-in lookup functions.

key

Optional path to the master key file.

Lookup Function Arguments

key

Path to the entry within the DB.

what

Field to be retreived form the entry. Following values are allowed:

user: user name assigned to the entry
password: password assigned to the entry
url: URL assigned to the entry
prop: string field (property) assigned to the entry (use pname argument to specify the name of the property).
binUTF8: attachment encoded as UTF-8 (use pname argument to specify the name of the attachment).
binISO8859: attachment encoded as ISO 8859 (use pname argument to specify the name of the attachment).
binB64: attachment encoded as Base64 (use pname argument to specify the name of the attachment).

pname

Optional name of property (for fields supporting properties).

Example

Keepass Entry

To use the Keepass DB, an according lookup function has to be defined.

Lookup Function Defintion lookup-func.yaml

lookups:
  kdb:
    provider: keepass
    config:
      kdb: keepass.kdbx (1)
      passphrase: "{{ _env('PASSPHRASE') }}" (2)

1	Keepass DB file is located relative to the `lookup-func.yaml` file.
2	The `passphrase` parameter supports built-in lookup functions.

Use the lookup function _kdb to retrieve the values from the entry.

Configuration Fragment

config:
  user: "{{ _kdb('/General/generic-user-general', 'user') }}" (1)
  pwd: "{{ _kdb('/General/generic-user-general', 'password') }}"
  url: "{{ _kdb('/General/generic-user-general', 'url') }}"
  user: "{{ _kdb('/General/generic-user-general', 'user') }}"
  field: "{{ _kdb('/General/generic-user-general', 'prop', 'field') }}"
  content: "{{ _kdb('/General/generic-user-general', 'binB64', 'text.txt') }}" (2)

1	Retrieves the user name from the entry `generic-user-general` within the General group.
2	Retrieves the content of the `text.txt` attachment as an Base64 encoded string.

4.2.2. Certificate Providers

Currently this plugin doesn’t provide any certificate providers.

Certificates stored in a KeePass database can be retrieved by one of the certificate provides of the Core plugin and the lookup functions provided by this KeePass plugin.

4.3. Amazon Web Services (AWS)

This plugin retrieves lookup values and certificates from AWS Secrets Manager and AWS Certificate Manager.

It uses the AWS SDK to access the services. Therefore the default credentials chain used by the SDK are also used by the providers of this plugin.

On systems directly running on EC2 instances, IAM roles can be used to authorize the service access, without specifying any credentials. For systems running on AWS EKS, a K8s service account can be bound to an IAM role to enable the access to the services without using additional credentials.

Using environment variables (AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID), system properties (aws.accessKeyId, aws.secretAccessKey), or configuration file, are also supported.

4.3.1. Lookup Providers

This lookup providers of this plugin, retrieve the values form the AWS Secrets Manager.

The functions support structured secrets (e.g. key/values) or plain secrets (text/binary).

JSON Document Secrets

For secrets containing a JSON document, e.g. key/value pair, this lookup provider supports to read specific keys from the secret.

Name

aws_sm_json

Built-In

Synopsis

Retrieves values from JSON secrets on AWS Secrets Manager.

Configuration Parameters

secret_name

Name of the secret.

region

Optional region where the secret is located. If not specified, the default region is used.

Lookup Function Arguments

key

Secret key as JSON Pointer (e.g., /user).

Example

In this example the secret sandbox/anm is stored in AWS Secrets Manager. The secret is stored using key/value pairs.

AWS Secrets

The secret has two secret keys (user and password).

AWS Secrets

In plaintext the secret is represented as a JSON document, storing the values with the secret key as property name.

AWS Secrets

Lookup Function Configuration lookup-func.yaml

lookups:
  anm:
    provider: aws_sm_json
    config:
      secret_name: sandbox/anm

Configuration Fragment config.yaml

anm:
  user: "{{ _anm('/user') }}"
  pwd: "{{ _anm('/password') }}"

The secret is not limited to key/value pair. Any kind of JSON document is supported by this lookup provider.

Plaintext/Binary Secrets

Name

aws_sm_plain

Built-In

Synopsis

Retrieves value from plain text or binary secret on AWS Secrets Manager. The values of binary secrets are represented as Bas64 encoded strings.

Configuration Parameters

prefix

Prefix to be be added before every secret name (see key).

region

Optional region where the secret is located. If not specified, the default region is used.

Lookup Function Arguments

key

Secret name.

The resulting secret name is build by the concatenation of prefix and key.

Example

In this example the secret sandbox/certs/root-ca is stored in AWS Secrets Manager. The secret is supposed to contain a public root CA certificate in PEM format.

AWS Secrets

The PEM certificate is stored as plaintext.

AWS Secrets

A lookup function is defined to lookup certificates from secrets prefixed with sandbox/certs.

Lookup Function Configuration lookup-func.yaml

lookups:
  certs:
    provider: aws_sm_plain
    config:
      region: "us-west-1"
      prefix: "sandbox/certs"

The lookup function can be used in simple certificate providers.

Usage in certificate configuration

certificates:
  root-ca:
    provider: simple
    config:
      cert: "{{ _certs('/root-ca') }}"

4.3.2. Certificate Providers

AWS Certificate Manager

Name

aws_cm

Synopsis

Provides certificates from the AWS Certificate Manager.

Configuration Parameters

arn

ARN of the certificate stored in the AWS Certificate Manager.

chain

Set to true to add the full certificate chain to policy project.

Example

In this example, the public certificate of CA, issued the Cassandra cluster certificate, is stored in the policy project under the alias cassandra-ca. For deployment, the certificate must be replaced by the public certificate stored in AWS Certificate Manager. The full certificate chain, has to be imported also.

Certificate Configuration

certificates:
  cassandra-ca:
    provider: aws_cm
    config:
      arn: "arn:aws:acm:us-west-1:000000000000:certificate/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      chain: true

4.4. Hashicorp Vault

4.4.1. Lookup Providers

Key/Value Store

Name

vault

Built-In

Synopsis

Retrieves values from a Hashicorp Vault key/value store.

Configuration Parameters

token

Token to authorize access to Vault. It is required if the token_file parameter is not specified.

This parameter supports a Mustache template, using built-in lookup functions.

token_file

Path to token file. It is required if the token parameter is not specified.

addr

Address of the Vault server. If not specified, the default https://localhost:8200 is used.

kv_base

Path to the KV secret engine (within Vault).

Lookup Function Arguments

key

Path to KV (within the secret engine).

field

Field within the KV data.

4.4.2. Certificate Providers

Currently this plugin doesn’t provide any certificate providers.

5. FAQ

5.1. Security

5.1.1. There is a CVE-2022-37767 for Pebble Templates, is YAML-ES Utilities affected?

According to CVE-2022-37767 any Java method could be executed by a Pebble template.

In the example below, the toString() method of the String object can be invoked within a Pebble template.

{% set value=\"Hello\".toString() %}{{ value }}

The Pebble Templates engine within the YAML-ES Utilities is configured to reject the invocation of a any Java method. Therefore, the example above will throw an exception.

So, YAML-ES Utilities is not affected by CVE-2022-37767.

5.2. Usage

5.2.1. How do I add values with YAML style configuration to `values.yaml`?

Within the values.yaml file of the YAML entity store, several options exists to configure the API Gateway at runtime.

The YAML style options also uses the {{ … }} syntax, which is the same syntax as for the Mustache templates used by YAML-ES Utilities.

If values in the configuration fragments, uses values in the YAML style, they are treated as Mustache templates and will be lead to a syntax error or will be replaced.

To force the output of the YAML style, the values must be specified as Mustache template having a string literal.

Configuration Fragment

account:
  password: '{{ "{{env ACCOUNT_PASSWORD }}" }}'

Generated values.yaml

account:
  password: {{env ACCOUNT_PASSWORD }}

5.3. Contribution

5.3.1. I found a bug, what shall I do?

If you find a bug, don’t hesitate to open an issue on the GitHub project page. I’ll try to help you as soon as possible.

Please remember, it is a community project. The plugin isn’t officially supported by Axway. So, please don’t open a case in the Axway support portal.

5.3.2. I have a great idea to improve the plugin, what shall I do?

Feedback is always appreciated. So open an issue on the GitHub project and describe your ideas.

5.4. Documentation

5.4.1. What about the `.adoc` extension?

The documentation is written in Asciidoc. So we use the extension .adoc for documentation files.

5.4.2. How can I read the documentation of the cloned repository in a nice format?

There is an Asciidoctor.js extension for Chrome to view Asciidoc files in a beautiful format.

5.4.3. How do I get the documentation in PDF format?

Asciidoctor provides some tools to render Asciidoc sources into various output formats. Asciidoctor PDF can be used to convert the documentation into a well layouted PDF document. Follow the instructions on the the home page to install the tool.

To generate the PDF execute the following command in the root project folder:

$ asciidoctor-pdf -a pdf-themesdir=docs/pdf-themes -a pdf-theme=axway docs/asciidoc/user-guide.adoc -d book -o user-guide.pdf

A: Appendix

A.1. Plugins Development Guide

A.2. Example CI/CD Pipeline

A.2.1. Description

The examples folder contains an example script to simulate a CI/CD pipeline (except the final deployment).

CI/CD Pipeline

deployment classic topology.drw

The sample project is an API Manager stored in the YAML entity store format (examples/apim). The API Manager project uses environmentalized fields to enable stage specific configurations.

In this scenario there a two teams. The developers are responsible to develop the API Manager and to deploy the project for testing purpose in their own topology. The operator team is responsible to deploy the project to their topology (for test and prod). Both teams use the same tooling and an equivalent pipeline setup.

The difference between the two teams are the different responsibilities for configurations. For the developer topology, the developer team is fully responsible for all configurations as the infrastructure is completely managed by the developer team.

For the topology managed by the operator team, the development team is responsible for configurations regarding the backend infrastructure. The operator team is responsible for the configurations regarding the infrastructure managed by the operator team. This includes server certificates, Cassandra settings and connections to the metrics database.

The configuration can be retrieved from various sources, depending on the target environment.

To ensure that coding guidelines are fulfilled, the project is linted with rules provided by the operator team.

The example project is also used to demonstrate the usage of the different data sources, the full variation of the data sources may not makes sense for real projects. Following data sources are covered by the demo:

Merge values.yaml structure from multiple fragment values.yaml
Certificates
- Java Keystore (JKS)
- PKCS#12 file
- simple configuration
Providers for value lookups
- YAML file
- JSON file
- KeePass DB
- plain text from environment variable
- JSON document from environment variable

A.2.2. Prerequisites

Axway API Gateway 7.7 May'22 release or later
- Package & Deployment Tools
- PolicyStudio (recommended)
- API Gateway/Manager topology (optional)

A.2.3. Execute Demo

Clone repository

$ git clone https://github.com/Axway-API-Management-Plus/yamles-utils.git

Build project

$ cd yamles-utils
$ ./mvnw clean package

Show command line parameters

$ examples/cicd/bin/build-archive.sh
build-archive.sh  -  build and configure YAMLES archive

Usage:
  build-archive.sh -e ENV [--debug]

Options:
  -e ENV
      Target environment (local, test or prod).

  --debug
      Enable debug messages

  Configure API Manager for target environment and build
  deployment archive.

Execute demo script

$ export AXWAY_HOME="...path to Axway installation (e.g., /opt/Axway-7.7)..."
$ examples/cicd/bin/build-archive.sh -e test

On Windows, the script can be executed using Git Bash (MinGW).

A.3. Changelog

A.3.1. Version 0.7.3

ID Type Description

#29

Enhancement

File Geneator - create non-existing target directories

#31

Enhancement

File Geneator - support base directory for source (templates) files

#32

Security Fix

Parameter of _gen_anm_pwd_hash() lookup function is now masked in logs.

A.3.2. Version 0.7.2

ID Type Description

#28

Fix

Fix config command.

A.3.3. Version 0.7.1

Type

Description

#26

Fix

Fix build pipeline.

#dependabot-4

Fix

Fix json-path Out-of-bounds Write vulnerability.

A.3.4. Version 0.7.0

ID Type Description

#21

Enhancement

Generate Files.

#22

Enhancement

Evaluate Expression.

eval command to evaluate expression and print result to stdout.

Example:

$ export NAME="World"
$ yamlesutils.sh -q eval "Hello {{ _env('NAME') }}!"
Hello World!

#24

Enhancement

Lookup function to generate hashed ANM password.

Example:

$ yamlesutils.sh -q eval "{{ _gen_anm_pwd_hash('changeme') }}"
$AAGQAAAAAQAC$oALW5N6CWi0PszIfVK3w5w==$uERZwFPmZ2bEknEfZiyrK7QfqcODTyjFJNrbpEQfEDI=

A.3.5. Version 0.6.0

ID Type Description

#13

Enhancement, Refactoring

Lookup- and certificate providers as external plugins.

External plugins are dynamically loaded if exists in the plugins folder of the distribution. To disable a plugin, it can be just deleted from the folder.

Each plugin includes all dependent classes and is loaded by a separate classloaders to avoid class version conflicts.

#17

Enhancement

Support dry run and syntax check.

New parameter -m MODE or --mode=MODE added to specific the execution mode:

CONFIG: Build and write configuration (default).
DRY_RUN: Build configuration, but don’t write to disk.
SYNTAX_CHECK: Check syntax of configuration files only. Lookups are evaluated to empty strings; no connections to external systems (e.g. AWS Secrets Manager) are established.

#18

Enhancement

New parameter -f and --fragment.

The parameter --config=FILE may be misleading. The file is a fragment of the resulting values.yaml file.

New parameters -f FILE and --fragment=FILE are added as an alias of the --config parameter of the merge config and config command.

The --config and -c parameters are deprecated and may be removed in future releases.

A.3.6. Version 0.5.0

ID Type Description

Enhancement

Add multiple certificates from keystore.

#12

Enhancement

Lookup provider for AWS Secrets Manager (aws_sm_plain) supports binary secrets.

The values of binary secrets are returned as a Base64 encoded string.

#14

Enhancement

Combine feature of lint, merge certs and merge config command into one single config command.

#15

Enhancement

Check certificate expiration.

Refactoring

Various refactoring to improve code quality and to prepare for future plugin support.

A.3.7. Version 0.4.3

Type

Description

Fix

Due to a bug private keys are not retrieved from a keystore. This is fixed now.

A.3.8. Version 0.4.2

Type

Description

Fix

Add STS component for AWS to enable service account authentication in EKS.

A.3.9. Version 0.4.1

Type

Description

Fix

Fix line endings for Maven wrapper.

A.3.10. Version 0.4.0

Type

Description

Enhancement

Lookup values from file content.

Enhancement

Provide Docker image to be used as an init container for K8s deployments.

A.3.11. Version 0.3.0

Type

Description

Enhancement

Add public certificates from files in DER (.crt) or PEM (.pem) format.

A.3.12. Version 0.2.0

ID Type Description

Enhancement

Add support for plaintext secrets on AWS Secrets Manager (lookup provider: aws_sm_plain)

Breaking Change

Lookup provider for JSON secrets on AWS Secrets Manager is renamed from aws_sm to aws_sm_json. In case of usage, rename the provider in the according lookup configuration.

Enhancement

Keystore certificate provider enhanced:

Base64 encoded keystore data (config parameter data). In combination with the new plaintext support for AWS Secrets Manager, keystores can be stored in AWS SM as a Base64 encoded string.
optional add certificate chain
optional suppress private key

Refactoring

Refactor lookup providers. Now, lookup providers build specific lookup functions instead of handling lookups by them self.

Enhancement

Support usage of built-in and non-built-in lookup functions in lookup provider configuration. Lookup function must be defined before usage.

A.3.13. Version 0.1.0

Type

Description

Enhancement

After a complete redesign, this version provides a stable configuration format.

YAML Entity Store Utilities

1. Introduction

1.1. About YAML-ES Utilities

1.2. Use Cases

1.2.1. Classic Topology Deployment

1.2.2. Container Deployment (Policy Image)

1.2.3. Container Deployment (Init Container)

1.3. YAML Entity Store

2. Usage

2.1. General Usage

2.2. Linting

2.2.1. Command

2.2.2. Rules

Example

Cassandra Nodes and KPS Tables

Database Connections

2.3. Evaluate Expression

2.3.1. Command

2.4. Merge Configuration Fragments

2.4.1. Basic Examples

2.5. Merge Certificates

2.5.1. Basic Example

2.6. Merge Files

2.7. Configure YAML Entity Store

2.8. Built-In Documentation

2.8.1. Lookup & Certificate Providers

2.8.2. Lookup Functions

3. Templates

3.1. Mustache Templates

3.2. Lookup Functions

4. Plugins

4.1. Core

4.1.1. Lookup Providers

Environment Variables (built-in)

System Properties (built-in)

Admin Node Manager Password Hash (built-in)

JSON from Environment Variables

YAML/JSON Files

File Content

4.1.2. Certificate Providers

Simple

Certificate File

Keystore

Remover

4.2. KeePass Database

4.2.1. Lookup Providers

KeePass DB

4.2.2. Certificate Providers

4.3. Amazon Web Services (AWS)

4.3.1. Lookup Providers

JSON Document Secrets

Plaintext/Binary Secrets

4.3.2. Certificate Providers

AWS Certificate Manager

4.4. Hashicorp Vault

4.4.1. Lookup Providers

Key/Value Store

4.4.2. Certificate Providers

5. FAQ

5.1. Security

5.1.1. There is a CVE-2022-37767 for Pebble Templates, is YAML-ES Utilities affected?

5.2. Usage

5.2.1. How do I add values with YAML style configuration to values.yaml?

5.3. Contribution

5.3.1. I found a bug, what shall I do?

5.3.2. I have a great idea to improve the plugin, what shall I do?

5.4. Documentation

5.4.1. What about the .adoc extension?

5.4.2. How can I read the documentation of the cloned repository in a nice format?

5.4.3. How do I get the documentation in PDF format?

A: Appendix

A.1. Plugins Development Guide

A.2. Example CI/CD Pipeline

A.2.1. Description

A.2.2. Prerequisites

A.2.3. Execute Demo

A.3. Changelog

A.3.1. Version 0.7.3

A.3.2. Version 0.7.2

A.3.3. Version 0.7.1

5.2.1. How do I add values with YAML style configuration to `values.yaml`?

5.4.1. What about the `.adoc` extension?