select max(id) from yourtable where sts=0 and ref_id is null;
That assumes that ref_id is null. It would help to see your table structure and the query you tried that doesn't work. If ref_id is actually a character string then you might need ref_id='' or coalesce(ref_id,'')='' if it can be null or empty string.
select max(id) from yourtable where sts=0 and id not in (select ref_id from yourtable);
Isn't it?
The OP will need to explain further as we are all guessing. As I mentioned in my earlier (accidental top - curses GMail) post, table structures and the query or queries that don't work would be useful. So would a description of the problem that is being solved since there could be better approaches.