you used someone else's question which makes responding difficult because only the above title shows up in the e-mail we get and not the question you are asking which differs in detail.
many years ago would seem to indicate that this occurred prior to 2018 in which case it's too late to file amended returns.
however to answer your question. the children seem to be qualifying children of both you and their grandfather. the tax laws say in cases where multiple people can claim a person as a qualifying child it's the parent that gets to claim them.
who supported the children is irrelevant
you mention nothing about their mother because if she lived with you and her adjusted gross income was higher than yours, then it would e her that could properly claim the children if no joint return was filed.
