To make it more clear, here is the behavior of a switch press and unpress after that:
When you press a switch (button) the corresponding LED toggles. Let us say that the switch goes to a certain state, A, after pressing. When you unpress the button, the LED remains in state A.
So pressing causes the toggle, unpressing does nothing.