Its has to do with the "expected value" or EV of the action.
It is difficult to "see" in blackjack because there are millions of permutations as to how a single hand can play out when 8 decks are in play, but the sum of them can be figured, and it has been done in the past.
To make the concept easier to understand, i will first use the example i always use being that of a bizzare roullete wheel.
You can win by standing, and you can win by hitting, but it all depends on what happens after you make your action, somethign that you cannot predict, but rather only speculate on the probibilities governing it. Once your money is in the betting circle and you are dealt cards, the only think you can do is give the money the best chance it can of winning, even if it means its goign to lose most fo the time.
Compare it to a bizzare roullette wheel, where once you have your money on the felt, you must move it to either red or black, only...on this roullette wheel, there are
30 red spaces
40 black spaces
and 30 green spaces but unfortunately, you cant bet on green.
If you bet on red, you will win only 30 % of the time, and lose a frusterating 70% of the time
However, if you bet on black, you will stll lose a frusterating 60% of the time, but now you will win 40% of the time.
Obviously, the best action you could take, if you were forced to bet on this wheel, would be to simply always bet on black.
This is just as it is in blackjack, sincce once, you money is on the table and you are dealt your cards, you cannot retreive it, and once you know your cards, there is a plying action, that in the sum of all possible events(just like the total of all the spaces on the roullette wheel) that will cause you to win more often than **any OTHER action you can make****
That is how and why basic strategy is what it is, for infor on the EV of every action, for every hand you coudl possibly be dealt under a variety of playing conditions, check out this link....
http://wizardofodds.com/blackjack/appendix9-6ds17r4.html
---------------------------------------------------
About soft doubling, it is a little more difficult to explain because when doubling down becomes the most favorable option, it is because of the EV, not so much weather you win or lose, (Ev is the measurment used to formulate ALL playin decisions, howveer, winning and losing is easiest to illustrate by example.)
EV is the hypothetical percentage of your exhisting bet, that you are expecting to win/lose on average in the sum of every possible way the random events governing your money in the game can play out. This includes, times when you win double you bet, and lose double your bet, and is what
When you double down, more money goes into action, so weather you win or lose has less of an impact for comparable events where you dont double down. Often the likly hood of a win in conbination to putting more money into action will boost the ev substancially for double down, and plit opotunities to make them the most valuable action.