I think you are doing just fine.
Let's assume you work till 67, that's another 20 years.
By then, you would have cleared your mortgage and your condo would have gone up in value to say, $1.6m. So you have $1.6m in equity in your condo.
Since you save $40k pm, over 20 years, your additional savings will be $800k.
Your CPF, which is now $420k, would have grown since you will earn 2.5% pa on your OA and 4% on your SA and MA. Let's say it will grow to $600k.

So, at 67, your net worth will be $(1.6 + 0.8 + 0.6 + 0.1) = $3.1m.
This is sufficient for you and your wife.
You can sell your condo and downgrade to buy a 4 room HDB flat, costing say $500k, and you will have cash balance of $1.1m and plus your savings, you have $2m to invest in high dividend yield stocks, giving you 5% pa or $100k pa dividends. You will also get a CPF retirement account payout of $24k pa in total. So, you will have $124k pa or $10k pm. For an old couple, $10k pm is a lot of money to retire on, the most you would spend is $5k pm if you don't drive nor have a maid. You don't even need to depend on your children to give you both any allowances, if they do, that will be a bonus.

So, you are on your way to financial freedom at 67. Hopefully, you will still be alive to enjoy your retirement. All the best.