Oh, please. You are all the time "at the whims" of various people, namely: everybody. You are at the whim of your wife to not kill you. She could decide to at any moment, you know? Any time you two are together and you are defenseless, your continued life is merely the whim of your wife. At any time she might decide to pass a decree or get rid of a decree and that means you're not free. Right?
Way to miss the point. Of course you are influenced by the actions of others. But my life isn't ruled by the whims of my wife. She cannot order me to do anything. And while she can certainly try to kill me, there is no assurance she would. Even if successful she would not rule me.
On the other hand, if someone else can determine your liberty or freedom, can command you to obey and compel you do so, then they rule you. Whether they choose to use that power in a way you like or not is irrelevant to the power they exercise over you. And that is what is being discussed, the extent to which the commands of others can dictate your liberty and obedience without your choice in the matter.
Your employer might decide at any moment to kidnap you -- lock the doors and not let you out. Your ferry operator may at any time choose to sink the boat. Your waiter may choose at any time to poison your food.
You must not be free.
You're absolutely right that if someone kidnaps you and holds you hostage that you then aren't free. Not only is your argument stupid here, it is entirely pointless.
For the rest of us not in La-La Land, whether we are free or not rests more upon whether our freedoms are actually being violated, not on whether they theoretically could be.
What a slave mentality. Enjoy the velvet shackles. "As long as the Massa dun't whoop me, he's a good Massa!" Never mind that the ability of the Master to do what they want to you is what constitutes slavery and servility in the first place, not whether you like it or not.