When things are normal, you do not need to worry. But when you see the situation change, you need to change accordingly. If the attacker is aggressive and he is capable of attacking you, you need to be careful about that.
Let's watch this video. The attacker's body language gives you an indication. And when the attacker pushes you, he has already moved close to you, he can punch you as well.
As a defense, you can make enough distance so that the attacker cannot get close to you. You can talk and if that works to calm him down, that's good. Otherwise, you have to make your move to protect yourself.
You can defend a punch by outside parry or palm slap parry. While the attacker is moving towards you, put your hands forward as if you are trying to stop him or you can keep your hands up. You might not get the time to parry when he punches.
You will get the advantage when your hands are up or at least forward. You can parry and defend the punch quickly. You can also grab and throw the attacker on the ground. Here I would not recommend getting on the ground with the attacker as you watched in the video.
In case the attacker is stronger than you are, of course, he will use his strength. A fight might not happen in a nice field. It can happen anywhere and the ground can be hard. So while you are getting on the ground with the attacker, you might hurt yourself as well. Or, the attacker can use his strength to grab you.
As a result, it would not end up the way you think. You can definitely throw the attacker on the ground. But it is better not to get on the ground with the attacker in a fight for self-defense.