Birthday party

So my son goes to nursery 2 days a week and my mum takes him as I work. He has had a party invite from a child in his room but since I had my son in lock down I have started suffering from anxiety and more social anxiety recently, now I did go to the first party that he got invited to and after I had the worst head ache and felt sick and couldn’t really do much afterwards because of this, I didn’t speak to anyone but say hello to the parents who’s child’s birthday it was and sat there very awkwardly for an hour then left. Being 3 he doesn’t really have any friends yet and I don’t know any of the parents because I don’t do pick ups and that nursery is defiantly very clicky not saying there are horrible but the whole thing makes me feel uncomfortable. Is it just me and is my child missing out of course I feel awful about not wanting him to go to anymore until he’s slightly older but my family keep making me feel selfish and saying hes missing out ? Help and advice or comments would be great ? Thankyou aibu