You probably didn't break the machine when adjusting the grind setting when the grinder wasn't in use.
The reason for this recommendation is that there might be a bean which is stuck between the grind disks. If you try to adjust the grind setting with a bean stuck in between then you might need a lot of force which the adjusting mechanism isn't built for.
If you try to adjust it and it and everything moves without too much resistance then there is probably no bean stuck in between and adjusting shouldn't harm the machine. If it feels like the knob is stuck when you try to adjust it then it makes sense to turn on the grinder so that the beans blocking the disks will be pulverized as you adjust the setting.
Furthermore, adjusting the grind setting with partially ground beans between the burrs can transfer pressure to the adjusting mechanism. Even if that doesn't break the system, it can wear some parts out faster. This reasoning is given by Baratza who make burr grinders:
We also recommend, when using a Baratza grinder, that you grind coffee through the machine while adjusting your grind.
Let’s go back inside that machine again to better understand why we do this.
Grind size is regulated by the distance between each burr. If you’ve got an empty machine, these burrs can move closer or farther apart unobstructed. When coffee is in the system (especially partially ground coffee retained between the burrs), moving the burrs closer together will press down on that ground coffee. This creates resistance against the burrs which transfers to the adjustment system. The result of this kind of use is a shorter lifespan of the adjustment assembly, leading to more frequent repairs.