Delete eq2ui_mainhud_choice.xml from your custom UI. The default UI does not use ENTER and ESC for the popup choice dialogs.
Same applies to eq2ui_inventory_loot.xml
If your set uses a custom skin for the windows in question and you don't want to loose the skin, you'll need to edit each window with Notepad and remove the following code bits...
Code:
isdefault="true"
iscancel="true"
Most UI sets added this functionality by popular demand. Yes it can result in accidentally confirming or canceling a dialog when chatting, but the problem is mostly avoided if you get into the habit of using slash(/) to initiate chat rather than ENTER.