Currently to search a reference field a dropdown is available in the custom query pop-up UI, but there is a problem with the dropdown, especially when reference data set is huge (more than 50K records). In such cases performance is getting impacted very badly. We also need to increase config values for CONTENTS__MAXRESULTS, CONTENTS__DEFAULTPAGESIZE, UI__REFERENCESDROPDOWNITEMCOUNT to populate all the data in the dropdown, not a desired way.
As a suggestion we can use list view instead of using dropdown in the custom query UI. In the list view, it’s easy to search a record without loading all the data.