Do I need to install Edge Software separately for BACnet and Modbus?

No, only 1 Edge software is needed to be installed and it can connect to both BACnet and Modbus. This assumes that there are no IP / subnet collisions. Multiple Edge installations are only needed in the case of multiple subnets.