they asked me to go with them.
they asked me if I would like to come with them.
they invited me to go with them.
they told me to come with them.
The correct indirect speech would be 'They invited me to go with them'. Here, 'us' changes to 'them' and verb 'come' changes to 'go'. Also said changes to 'invited' as it is a verbal invitation.