Rights do exist.
If you and I were living on an island, and you kill me, you've missed out on the opportunity to cooperate and trade with me. Consequently, your lifestyle will be worse than it would be had you not killed me.
On the other hand, if we were living on an island, and I was stealing from you, and you killed me, your lifestyle would be better than it would have been had you not killed me.
In both cases, you would face the natural consequences of your actions. These two hypothetical scenarios give us a wealth of insight into the true nature of morality.
To tear apart your model: If the two of us were competing for resources on the island, and I kill you, I have gained the use of all of you resources. So even without you having been infringing on my rights, I am in a better position with you being dead.
The whole "better position" fallacy was put to bed with Nozick, IIRC
Rights are "imaginary": they are a construct of the rational mind, as applied to the real world.
That doesn't mean that they are arbitrary or subjective, only that they do not exist separate from the rational mind.
Rights are an extension of the axiomatic underpinnings from Descartes and Kant, through Mises to Rothbard, that (1) "I" exist, and have rationality (whether I use it or not), (2) people have bodies that they can control through the use of their reason, (3) other people exist with what must be assumed to be equal rights to gain possession of unowned material (property) and use their own bodies how they see fit.
From those 3 undeniable propositions, we see the framework of "rights": To use your body (or property) in any way you want, limited only by the rights of others to use their body (or property).
To extend those "rights" to things such as "free speech", "Secure households", "Bearing Arms", and a "right against government taking" really is just an application of personal and property rights that arises naturally from reason alone. As codified in the Constitution, they are (intended to be) limits on the powers of the state, recognizing that personal rights are more just than the whim of whoever claims that power has been bestowed to them.
So are rights imaginary? Yes. They are. There is no existence separate from the internal workings of our minds, but they cannot be rationally denied by the mind, only skirted by those who elevate themselves above the humanness of those whose rights they violate ("3/5ths of a person", "terrorist", "illegals"). The only way to rationally violate the rights of another human is to dehumanize them, and that just another task that the state is all too familiar with and comfortable using.