I have a fiduciary duty to my clients. Yet, usually I don't talk to them on a daily basis, (sometimes with some of them, not even on a weekly or monthly basis), but somehow still my first duty is to my clients.
Fiduciary duties are some of the strongest duties out there. I think you are minimizing what a fiduciary duty is.