Apple Watch heart rate reading jumps around erratically.

I monitor my watch during weight lifting and track BPM to pace my sets. The BPM will jump around from 140 down to 80 and then back up to 130 within a one minute time period. Does it have to do with the position on my wrist - I have very small wrists and wear gloves for lifting - I'm wondering if that impacts the positioning of the watch and therefore the reading? Or is it perspiration between the watch and my skin? Appreciate any guidance.