You need to program the 2540USB dongle with the \Accessories\HexFiles\CC2540_USBdongle_HostTestRelease_All.hex image, the same that is used with BTool.
You need a CC debugger to do this. Alternatively you can use a SmartRF05 board programmed with CC2540_SmartRF_HostTestRelease_All.hex or 2541 of course. This will make the CC2540/41 on the EM do the same as the USB dongle, and can be used in Device Monitor or BTool via a com-port.
The SmartRF board includes a programmer available via USB that can program EM boards sitting on the SmartRF05 board. For other targets you'd need a CC debugger. Documentation on programming is provided in the kit User Guide and Quick Start guide.
Best regards,
Aslak