It does not sound to me like this is a power issue, but just to make sure, make sure the signalling levels from your custom debugger is the same as the CC2540 supply voltage.
- As stated in AN118, the only thing you should need prior to a CHIP_ERASE() command is to reset the target (CC2540) to debug mode.
- Bit 2 in the CHIP_ERASE() command is don't care, so 0x10 and 0x14 are in fact the same (ref. swra124 page 5, swru191 page 56).
- The SmartRF Flash Programmer uses the GET_CHIP_ID() command to check whether the device is in debug mode. I would recommend this as a sanity check.
Have you tried the code example that comes with AN118? Are you able to use it to program your CC2540?
Br,
ABO