I know this is not a “must have” feature, but it would be a really nice improvements in both UI and API when working with fields that will always have only one item in them.
My proposal is to start with field validation - add a checkbox “Allow multiple items”. And only if multiple items are allowed, it should be possible to allow duplicate items and set the min-max limit.
I made a mock-up for a reference field as the single/multiple problem is the same there as it is for assets.
The second part is the form itself - now it looks like this:
I think it would be sufficient to hide the “Click here to link content items” or “Drop files or click” if the field is set to contain a single item only and an item has been already selected: