Can I use the territory feature to give access to users and keep the owner field empty?

Yes, we can leave the List of users for auto-assigning leads in round robin empty and give in the users name in the List of users who have access so that these users will have access to the records in the territory and no owner will be assigned.