Don't compare with the elite Administrative Service. There are only about 400 Administrative Officers and Management Associates (387 according to Peter Ho's speech in 2007).
400 against a labour force of about million (or more), is insignificant. They were the black swans.
These lucky folks got in because they did well in their studies in JCs and managed to impress the interview panel during their scholarship interviews. They were reasonably smart students and more importantly they can speak and present themselves well. That said, there are also a lot of "farmer" scholars who couldn't become AOs (not lucky enough or chose the wrong scholarships).
If you want to compare, you might as well compare with the thousands of rich kids who inherited millions from their rich parents.