I think the important thing is that they have the opportunity to have plenty of contact with other kids. You don't have to force it on them.
And the other most important thing :-) is learning to listen to what you're children are telling you through their actions and words. Schools teach us to see what the experts tell us to see and to ignore or doubt and not take seriously anything that doesn't match what we're taught.
Anyone saying kids need lots of friends or contact with people is wrong. But saying kids don't need lots of friends is equally wrong. Some kids need lots of kids. Some kids don't. Some kids need bursts of it. Some kids would prefer spending every day on their own.
So what's important is learning to listen to what you kids are telling you. Forget the experts. Your kids are the experts :-)
I think he would really miss playing with friends during the day.
Is the play he gets at school the same quality of play he'd get at home with friends? Unless the charter school lets kids interact freely, what you call "play" may be "contact with other kids." The interaction kids get at school is severely restricted. From how you described him, he undoubtedly needs other kids so what he gets at school is better than nothing. Basically he's making the best of what he's given.
It's sort of like craving chocolate and being given cheap waxy imitation chocolate. Cheap chocolate would be better than nothing if you were on Mars and the next shipment of good stuff didn't arrive for 6 months. But no one would choose it if they could have Godiva chocolate.