Mate, if you went in the gym you'd struggle to lift an olympic bar. Lol
It's all muscle memory, if you've had a life on the shovel, like me, you will be a machine on that shovel, go out of the comfort zone and you will be all at sea.
I've seen plenty of lads with your body type, we have had a lad called Mad Murdoch who was hyper as feck, couldn't sit still, could graft, but put him in the gym or try and lift really heavy gear he just didn't have the pure strength.