Left-click skill somehow changing

This only seems to happen with my fire warlock, and I cannot quite pinpoint why. I use left click for

Sigil Lethargy (weird I know), and sometimes I will load up a fresh run and the game has changed it to something else, such as

Apocalypse or or one of my other hotkey skills. I don't even have the Sigil bound to any hotkey or anything.

Now, I can sometimes force this to happen if I spam my hotkey skills while clicking. It seems that maybe if 2 skills are used on the same frame it changes which skill is on the left-click, or something like that. But as I said, this is happening when loading into a game as well, where I know I left the previous game with the correct configuration.

This is a small annoyance, I know, but it does sometimes lead to some sketchy moments when I'm trying to lay down a Sigil on a pack of baddies and instead it summons a

Tainted or something. I tend to not notice the skill has changed until I go and try to use it, but by then it's too late. And since it's an unbound (and left rather than right click for that matter) skill, I can't quickly mouse wheel to change it back, I gotta click on it and select it!

Just wondering if anyone knows why this happens or if I can prevent it somehow. I tried changing around where the hotkeys were, whether they appear on left or right click, and that didn't seem to matter.