Hi
1. I set from admin that user must login in order to post ad. And  When I am logged out and click post add, it takes me to login page.  Which is good.

2. I installed osclass pay plugin and set group that user must be a member of a group in order to post listing. 
Now when I am logged out, I click on post ad, it does not take me to login page, instead it showing me message that I must be a member of a group. 

Let me login first then I will become member.

It is jist informing you that limit was hot.
Increase limit for logged-out users might solve issue (as logged out cannot publish listing)