Taking the general idea from https://github.com/Puzer/stylegan and trying to estimate directions for some danbooru tags:

https://github.com/halcy/stylegan/blob/master/Stylegan-Generate-Encode.ipynb

Pre-calculated directions for some tags (and the data they were created with) included. As you can see, they're not super orthogonal - maybe this could be solved by balancing or by trying to find directions that are orthogonal to begin with. It still does work fairly well for hair and eye colours, much else, I haven't tested.

edit: And with a lot less artifacting (truncation with psi=0.7): https://i.imgur.com/Vzqhpah.jpg

double edit: mandatory Cool Video: https://icosahedron.website/@halcy/101654023952557254

triple edit: Interactive notebook https://github.com/halcy/stylegan/blob/master/Stylegan-Interactive-Modification.ipynb (video: https://www.youtube.com/watch?v=Z1-3JKDh0nI ) and a limited version to try on google colab: https://colab.research.google.com/drive/1LiWxqJJMR5dg4BxwUgighaWp2U_enaFd#offline=true&sandboxMode=true