Congress said it is not an "atheist party" that it would oppose a temple or religious ritual, but had declined the invite as it is a "political" event