by Doug Coulter » Mon Mar 17, 2014 5:41 pm
Here's the code for the crucial math in my 4d plot program - changes a/d input to millibars:
$c = 10**((1.667*$a2*$calfactor*2.013)-11.33); # pressure
$a2 is the a/d input - it's that 10 to the (rest of math)....that makes things look bad.
The 2.013 is because to get the output into a 5v range, I needed a divide by two in hardware, and the 1% resistors didn't quite match. Calfactor is derived by sampling a reference diode to compensate a/d full scale changes due to 5v changing (the data aq box is USB powered). It's 12 bits, and still not that great, frankly. Gets really "notchy" like the Pfeiffer controller does, which is also 12 bits. Cramming 12 (1e3 mbar to 1e-9) decades into a little less than 10v range...via an analog log = issues.
But yeah, for a mass spec...not a problem, just that it's only good to about 2 digit accuracy, almost (notchy, not all the numbers in two digits exist) and the right exponent is good enough for that.
It also reads right around factor of two high on things like deuterium. It's in the manual.
Posting as just me, not as the forum owner. Everything I say is "in my opinion" and YMMV -- which should go for everyone without saying.