Just as our spirits impact our bodies, so our physical bodies affect our spirits. We cannot commit any sin with our body that doesn't damage our soul. The spiritual dimension is not immune to the physical dimension.
This is true of individuals. But it is also true of communal bodies, such as the church. When parts of the body of the church commit sin, the spirit of the entire church suffers.