Your workplace has to be safe and free from unnecessary hazards.
If the workmen are causing your work place to be unsafe, or creating unnecessary hazards, then you could make them stop.
For instance, if the noise created by them banging their tools (Y,J) was sufficient to cause damage to your hearing, your employer might choose to issue you with ear defenders. Obviously they would then also need to make necessary adjustments to the fire alarm so that it provides suitable visual cues (big, spinny, flashy red lights would be sufficient) so that in the event of a fire, perhaps caused by a workman accidentally drilling through a water pipe and causing it to leak on a damaged power cable, you don't just sit at your desk until you're burnt to a crisp.
As long as they are not adversely affecting your health, they can pretty much do what they want.
▪
┌────┐ ┌────┐
│ │ │ │ ▪
│ └────┘ │
│ ──┐ ┌── │ ▪
┌──────┤ ▪ ▪ │
┌┘ │ │ ▪
┌─┤ └──┐ │ │ ┌──┘
│ │ │ ││ ││ │ ┌─┐
│ │ └─┼┤ └┴─┴───┘ │
│ │ ─┘│ │
│ │ ┌──────┐ └┬──────────┘
│ │ │ │
│ │ │ │
└───┘ └───┘ |