Just my 2 cents worth. Objectively, the nature of work in Ministries and SBs can be quite diff. Ms focuses a lot on policies, studies and governance while SBs are more focused in the execution of policies/regulations in their respective domains.

There are significantly more scholars in the Ms simply because the govt thinks that's where the thinking caps should go. And to justify their investments in the scholarships, they are obliged to give the scholars more opportunities to shine. Competing with these scholars require you to be on par with them in terms of eloquence, confidence and insight. These aren't exactly quantifiable traits, hence it can seem unfair during perfomance evaluation.

On the other hand, work at SBs often has more defined KPIs. If you are a do-er more than a thinker, you have more opportunities to shine there. Scholars here will still have advantages, but end results are more visible. I have seen farmers rise faster than scholars because they delivered higher KPIs. But course, there isn't the norm.