When someone is grabbing and moving you, you feel the force. The attacker uses his power to do it with full force. If you cannot absorb the attacker's force, you will find it difficult to handle it.
You do parry and block against different strikes and kicks. If the timing is not right, you will miss it and that will hit the target. You have to parry at the time of striking you. Distance is another important factor that gives you an edge in a fight. When you parry to defend against the strike, you are also dealing with the attacker's force.
If you can handle it, that's great. But if you cannot handle it, you will find it hard to be stable and stand in your position. You might move away from where you stand.
Sparring is so important that you cannot imagine someone who learns martial art and self-defense but never do sparring. You can learn different things from sparring. You will learn how to handle your opponent's force and absorb that.
When you defend any strike and absorb the attacker's force, do not just keep blocking. The attacker will keep striking and doing his best to hit you. You need to go for the counter-attack immediately.
Whether you like it or not, attacking is the best defense in some situations. You have to observe the whole situation and decide what to do and how you should make your move to protect yourself.