Configuration for Terraform

In several of my projects, a variables.tf file with a set of basic variables is not adequate for configuration. An example is where a stack consists of multiple independent building blocks that each have their own configuration parameters. In that case, each module needs its own configuration file. Moreover, it can be useful to use a tree structure to represent configuration elements related to the same concepts. And finally, many of these have default values: simple default values, like if a stack does not specify a bucket name, the module tf code will generate one; and more complex types, eg the module may allow one element to be a map of arbitrary key names, but the values all have the same structure, in which case each map value has its own set of defaults.

Ideally a configuration schema would have default values or indicate that a property is required, would have type info, support some value constraints (like via a regexp), and support defaults for basic data structures like strings, numbers, lists and maps. It might look something like this:

server: 
  size: string = "t2.nano"
  ssh_key_name: string = "server_key"
  
buckets:
  _map_key_bucket_id: 
    name: string = required
    policy_filename: string = null (auto-generated)
    retention_days: number = 7
    
cidr_blocks: 
  _list_:
    ip: regexp (\d{1,3}\.){3}\d{1,3} = 0.0.0.0
    bits: number = 24
    description: text = null

This is not currently available so I use a scheme where each re-usable terraform module has json/yaml defaults, and these can be overridden in a separate "overrides" file when the module is imported into a stack. The merged config is output by terraform.

This works really well. EXCEPT that

1. The IDE has no knowledge of this configuraiton system so while editing module code that references config tree, my IDE does not give me intellisense on the configuration tree. This becomes more of an issue as the system grows.

2. I can't easily inject dynamically generated defaults into the defaults. Eg it would be nice if I could show that when config parameter "instance_name" is null, then this means "use the name computed by the module's tf code". But terraform is not designed to generate object trees like that, so this type of processing is painful in HCL.

3. Merging is too basic in terraform, and the excellent third-party module I use is not parsed properly by infracost.

A custom provider could solve the #2 and 3, but not #1. So in my next iteration I will create a wrapper that generates the necessary tf files containing variable definitions with defaults, specific to the set of modules used for a stack. I might base this on one of the open source ones that exist like terragrunt, terraspace, terramate, cdktf, or atmos, I have to investigate.