I encoutered an issue where tokens were being created without TTLs and
thus clogging up the storage of the system. I haven't found a smoking
gun pointing to this being the cause, but I do suspect that it's
_something_ to do with the Vault/Crossplane integration, since a) that's
really my only use-case for Vault, and b) there's the string
`vault-provider` in the display_name below:
```
$ vault token lookup -accessor zcRF0YAUQtP7vrbZHTW5y322
Key Value
--- -----
accessor zcRF0YAUQtP7vrbZHTW5y322
creation_time 1715766311
creation_ttl 0s
display_name token-vault-provider-token
entity_id n/a
expire_time <nil>
explicit_max_ttl 0s
id n/a
issue_time 2024-05-15T09:45:11.720412011Z
meta <nil>
num_uses 0
orphan false
path auth/token/create
policies [root]
renewable false
ttl 0s
type service
```
Due to currently-unknown fault, my Vault storage got full up (I
_suspect_ it's due to not setting a default TTL on Tokens, and so they
all hung around. Surprised they were created at such a rate, but w/e). I
wasn't able to directly expand the volume - and, anyway, it's on
Longhorn which is a Storage Provisioner that I'm moving away from - so
the solution was to:
* Create a temporary PV (on FreeNas, though that doesn't actually
matter) and copy data onto it (by mounting both it and the existing
Volume onto a debug pod, using a variant of [this
script](https://blog.scubbo.org/posts/pvc-debug-pod/))
* Delete the existing PVC and PV
* Make this update, and sync
* A new _empty_ PV will be created (and probably populated with some
stuff)
* Scale-down the StatefulSet, do the double-mount-to-debug-pod trick
again, and copy data from the temporary PV onto this one
* Delete Debug Pod, re-scale-up StatefulSet...and hope that there's
nothing stateful in the data which means that copying it from one
volume to another makes it invalid (e.g. if encrypted with an
encryption key which would change on a new spin-up of the pod - which
_seems_ unlikely, but 🤷)
Need to remove `Chart.yaml` so that Argo doesn't try to treat
`app-of-apps/` as a Helm application (because that would stop it from
using Jsonnet parsing).
