Can I use a VM for FNE or FNX license server setup

Yes, this is possible. The hardware / software prerequisites and the installation setup is the same for VM as for bare metal servers. 
( see TDCLHC chapter of https://cs.advantest.com/V93000/help/index.jsp#q=344041FlexNet Embedded license server setup - Prerequisites for the prerequisites and subchapters / links for the installation set up).

Only if you have a local license server serving only the local SmarTest client, then the hard disk space can be less.

Z4G5 is only allow one instance for local FNE/FNX license server: either on the HV instance or on the bare metal instance; parallel installation will corrupt database.

Note: if you have a failover FNE setup or a triad FNX server setup it is risky to use the VMs available on the same server. If this server goes down, all VMs go down.