In case you are using C# as backend with the default template, make sure to actually use the production configuration from your angular.json:
...
<Exec WorkingDirectory="$(SpaRoot)" Command="npm run build-prod" />
...
Since I switched to a newer angular version, --prod was no longer supported and I incorrectly assumed that ng build would default to production...
Furthermore I had to adapt the production configuration as described by @edyrkaj
You could change the package.json as well by adding all those options to the build command, but I prefer the angular.json approach since it's a lot nicer to read and maintain in my opinion