Quantcast
Channel: SCN: Message List
Viewing all articles
Browse latest Browse all 10473

Re: Age Old Floating Point Problem

$
0
0

Yeah, after re-reading your description, I realized that Truncate(...) wouldn't work either.

 

I've also run into weird rounding problems with currency calculations. The workaround was to store every product in an intermediate variable, and round each value at each step (I used 4 decimal places).

 

Does this give a different result?

 

decimal ld_temp

 

ld_temp = round(ldRegTime * ldRate, 4)

ld_temp = round(ld_temp * ldMultiplier, 4)


Viewing all articles
Browse latest Browse all 10473

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>