Data Logger
12th June 2008
New website at magics-notebook.com
Introduction.
I have brain damage which always makes doing a project a race against time, before I wake up one day with the project completely wiped from my memory. This is one of the reasons I maintain this web site, so if I get a memory wipe at least I can see what I was supposed to be doing Hi!. This has already happen to me once on the data logger project. The only reason I know I have done a data logger is because I saw a photo of it on the PIC DVD page !.
I need a data logger to hand out to the kids for the "Owned" project. I see the one I did used flash eprom for storage and for some reason I think it would be better to use an SD card for storage. Unfortunately everything I learned about SD storage went out of the window too. So I need to somehow try and do it again ... but this time make more comprehensive notes of what I am doing ... ready for the next memory failure !. Isn't life great Hi!.
Options
1. Home brew design solves copyright problems. Cheap(er). Software in Proton.
2. Elektor design. see below.
3. Little Doctor board off the shelf (not SD). USB.
4. Jianping Electronics module. Very promising. Serial Basic control, needs porting over to Proton. Would work with 18F452 card or STAMP. Ex USA shipping $12, 5 days delivery. Total cost $61.95. An alternative would be to homebrew PCB and use the Proton SD file system library, lack of instructions problem, more comprehensive than JP but how complicated do we need to get ?. Get an example !!!!
Notes..
1. I have an 18F452 on the development board so I might as well use that. Also on the development board is a DS1307 real time clock, which strangely enough I do know how to use.
2. I use the Crownhill Basic Proton compiler for my PIC Projects and I see that on the Proton user site there is a "SD file system library" which means nothing to me.
3. I have found a Futurlec SD card PCB and that contains a 512 mB SD card that I must have successfully formatted with FAT16. The card does not appear to have ever been connected up so I must have done it in the PC card reader. I also found a Futurlec DS1307 RTC PCB so I must have intended to use the two boards for some reason. This is a bit odd because I usually find it better to layout a PCB from scratch. I am wondering if I bought this PCB to get the SD socket ?.
4. I think I need 4 - 6 channels on the logger with perhaps another channel to monitor the battery voltage. It would make sense to port out all 8 analogue inputs.
5. I need over voltage protection on all analogue inputs, maybe a simple 5.1 volt Zenner and limiting resistor.
6. I could hang a 16 x 2 or 20 x 4 LCD on port A for software development and setting up channels etc.. Do I need on board analysis ?.
7. I seem to remember that Fat16 spit out 512 bytes, so either need a buffer or write off some storage space, which I can easily do using a 1 gB SD card (I think!). Must map out the maximum amount of space I need for n channels using n samples.
8. I could use the 1 Hz clock from the DS1307 for timing samples but that would limit the sample rate to 1 second, to go down to the next lower clock frequency might be too fast. Must check. I am wondering if the kids might want to look at transients and means a very short sample period. Thinking perhaps could change the clock output on the fly ?. I think it was a register poke on the RTC.
9. Cannot see any need to use USB, other than ICSP with laptops because no Com1 and Com2. Can remove card from logger and read on any PC card reader. SD cards are down to a couple of pounds at this time. Thinking I don't want to go down to surface mount because it would complicate things for the assemblers and test equipment.
10 Functions ... select sample rate, select run time ?. Select channels (thinking 0 - 5 v input or LM35 sensor link selected ?). Or via menu ?. Logic says stick to 0-5 volts for all inputs and use project MicroBoards from project for all sensors). Maybe knock up extra temp sensor board for the extended range, yes do that. Use diode solution to sort out negative supply requirement}. Use 3 pin R/A polarised headers. Look at finger spacing.
11. Find out what quiesant current SD card takes in between samples. Use 2500mA rechargeable AA's to power. Check to see what LM3940 5/ 3.3 volt regulator drop out voltage is. Thinking 4.8 volts battery pack or use 6 volt pack and low drop out 5 volt regulator. Need charging socket for an intelligent charger.
12. Housing ?. or use waterproof bag concept ?. Still need battery pack unless fit battery holders under main PCB ?.
13. Use dividers or a 3.3 volt PIC ?. Problem is a reduced voltage available for MicroBoards which were tested at 5 volts. Best to stick to 5 volt PIC and use dividers to CS, SDI and CLK.
14. Discovered Elektor SD data logger design.
Plus Protected inputs, PCF8583 RTC. Multiple files. On board time date selection switches. On board analysis.
Minus. Four channel. 12 volt supply. Max232. Needs MikroC compiler !. Kit cost's £71.75, PCB £16.30, no case.
---------------------------------------------------------------------------------------------------------------------------
13th June 2008
I found the help files for the Proton SD Filing system .... and very good they are too !. A very comprehensive list of commands with some excellent example in Proton Basic. I now have a lot of reading to do !. You can find the files at .....
Macros for accessing Secure Digital (SD) or Multimedia Memory Card (MMC) cards. The commands include ......Thank you very much Mr. Steven Wright !.
-------------------------------------------------------------------------------------------------------------------------
17th June 2008
1. Urban legend has it that the cheapest and easiest way of getting an SD socket is to buy a card reader and canabalise it . So I bid for a card reader on EBAY, won it for 10p with free postage and am now wondering what the catch is !.
2. This is the Futurlec SD socket card ....
It is far from ideal, but good value and since I have it, I decided to use it to make an SD plug in module for the PIC DVD project. That means I can develop the SD logger on the PIC DVD development board. To make the Futurlec card useful it really needs a piggyback board containing a 5 volt to 3.3v PSU and three signal voltage dividers plus a pull up. The Futurlec board just plugs into the new PCB below. to allow this seven way SIL pin strip is soldered to the central row of via's.. There will be a six way ribbon cable that plugs into the Dev board. I have laid the board out so that it could actually plug straight into port C and take it's 5 volts from one of the unused MicroBoard plockets. The other alternative is to use the 6 way Microboard socket which does have 5V and ground on it as well as 4 signal lines. That board will look like this.
--------------------------------------------------------------------------------------------------------
23 June 2008
1. Modified Futurlec SD PCB by soldering a 7 way header plug into the via's on the bottom of the PCB.
This header will allow all connections to the adaptor PCB.
2. Left. The adaptor board, populated. At the top is the 5V's to 3.3 volt, low dropout regulator. At center are the three voltage dividers to drop the signal inputs from the the PIC to the correct level. At center is a seven way header socket that mates with the header on the SD card.
Right, the module assembled. A six way ribbon cable feeds back to the six way module socket on the development board.
Pin connections on the white header ............. +5 volts, SDO, SDI, CLK, CS and GND. They are all one needs to use the module as mass storage filing system. To the left of the SD card is a green LED which indicates 'Card present'. To the right of the card is a red LED indicating power on. The onboard pull up resistors are not used. All I have to do now .... is make it work Hi!. I have already formatted the card with FAT16 and it tests out OK in a card reader, but will also require initialising before use in software.
-----------------------------------------------------------------------------------------------------------
25th June 2008
Above is a shot of SD_Init_FS running. What this example file does is set up the connections to the PIC. It then checks if a card is present and prompts for one.
Next it opens a file called "TESTFILE" and write a string to it. The file then is closed .... hopefully leaving the file on the SD card.
This is shot taken on the second run. Again the program tries to save a file called "TESTFILE" but it already exists and tells you so.
If one then takes the card and put it into a card reader the file can be seen (and opened if required) .
So far it means I can open a file on an SD card,, write to it and then close it. So far .... so good. The important thing to note, is not only does the software work, but also the SD card module.
-----------------------------------------------------------------------------------------------------
26th June 2008
Now that I have the SD card and RTC software working, I wanted to layout a data logger PCB. I had ordered both an MMC socket and MMC adaptor from Crownhill ... which arrived this morning. Unfortunately both Verbatium and Kodak brand SD cards will not fit into these sockets !. I think the problem is that SD cards must be a bit thicker than MMC cards and the metal bar stops a SD card being inserted into this particular type of MMC socket. Fortunately, The Futurlec PCB SD socket works with both SD and MMC cards.
Just as a reminder to me .... both the RTC and the SD card use the SPI line on the PIC .... I am wondering if that will cause any conflicts. Must also consider whether I want to make a 'Universal master time injector module' for accurately setting the time etc on any design using an RTC. Base it on the Futurlec DS1307 RTC PCB, with a piggy back PCB containing a 18F1330 'parallel to serial circuit'. It could plug into and be powered from the host ICSP plocket.
Brown announced the Low Carbon initiative today. Check to see if solar water heater panels need to be drained in winter !. Why are they not using concentrators and trackers ?. Flue regulations for wood burners ?.
---------------------------------------------------------------------------------------------------
27th June 2008
File formats. I can find no standard format for fields in sample records. It is not really a problem at this stage because I can always alter the sequence later. Now need to work out a method of getting sensor data onto the SD card.
1. AN1 etc give me a variable, correct to one decimal place in the range 0 - 5 volts. V = Vraw * 5/1024. ie. 3.7v. For temperature want 101.3 degrees C . Special degree character, no because we have º . Six elements. Maybe we only need symbol if we use on screen analysis ?.
2. Can I extract date and insert as a file name ?.
3. Date 30-11-08. Time 12:57:46. Six elements.
4. 4 - 8 channels ?
5. Variable to string ?. String = string1 + " , " + string2 + " , " etc. Write string to SD in one line, CR, LF ?. How many characters max in that line ? (255 ?)
6. Need 4 - 8 Anal inputs ... each with V/º switch. Also over volts protection (5.1 volts) and series limiting R. Need to try thicker ribbon in Header contacts ?. Screened cable ?.
What do we want on LCD ?
a. SD announcements. SD free space ?. Write LED ?
b. Auto bat check. Auto run ?.
c. Channel configuration menu.
d. Sample number
e. Do we need to program run stop time ?
7. Battery recharge socket. On off switch. LCD backlight switch (powered from charger ?).
--------------------------------------------------------------------------------------------------
29th June 2008
For some reason or other, my bootloader now works with 16F's but not 18F's. I can only surmise that both of the 18F452 chips I have tried both have the same fault ..... or the 18F452 4 mHz bootloader file is corrupt. I have emailed Crownhill to see if they will sell me a couple of 18F452 chips with a preloaded bootloader. I also ordered two 18F4420's from EBAY .... just incase !. Usually I get this kind of problem because I forget to change Xtal frequencies. Ever so slightly annoying ! .
It turned out to be two faulty PIC's both with the same symptoms !. New ones with bootloader work OK.
------------------------------
Note. Special format for HOBO reporting to data@globe.gov. Primary and secondary schools.
//AA
YYYYMMDD , HHmm (,ss), Ch 1, Ch 2, Ch 3, ..... Ch 7.
//ZZ
-------------------------------
2nd August 2008.
Had another memory wipe !. Kid's school holidays re-kindled interest. Now looking for an SD breakout PCB suitable to put into a final design. Sparkfun have them around $17 !. Have emailed Futurlec with photo's.
Need a 'set up' screen to show all transducers are functioning. Then screen last sample.
EXCEL spreadsheet is 256 columns x 16,384 rows !. For 24 hour sample period ..... 24 x 60 = 1440 minutes. 1440/256 = 5.6 samples per minute ... roughly one sample every 10.7 seconds. Need to look for alternative software ... maybe !.
Had a look at stripping SD sockets from card readers ... too labour intensive !.
Think 0.5 volts for digital inputs to avoid overwriting analogue traces.
------------------------
9th August.
Have done the PCB layout, for the main board. Decided to put the input components on a separate board and use a ribbon cable with polarised connectors. Cannot locate an SD socket at the moment, have emailed Futurlec to see if they stock the SD socket on their SD mini board. So I decided to leave the socket off the main board to give a choice where to locate it one the final layout. Have also designed an SD socket breakout board for the Futurlec socket. Do I want to get into surface mount ?.
Have made the PCB for the main board and have included a battery volts pot, plus an LED for development work. The picture is about 30% oversize.
The DS1307 RTC is bottom left, with the backup battery mounted above it. I decided to put the SD socket attenuators on the main board (right). The socket at bottom right is for a ribbon cable to the remote switch 'panel'. At the top left to right DC input socket .... +5volt regulator .... programming socket .... +3volt regulator .... link to disable the LED when installed in it's box. I can use the programming socket to also set the time with an 'injector' I was running out of pins so I decided to do channel set-up in software. Have ordered 20 x 4 line blue LCD with backlight, for Port A from HK.
I think I will use 6 x 1.2v 2500mA/H rechargeable AA cells and a 9 volt regulated charger (series R) with a voltage detector to end charge. Need low dropout voltage regulator !. It can be left connected and that will also solve the self discharge problem. Use a Walmart plug in PSU (get some more at the boot sale !). Could use the spare pins on port A for the degrees / volts function. D0, RC1, RC2 & RA.4 also free.
Need sensor sockets to layout input board. DIN 4 ?.
-------------------------
August 10th 2008
I have laid out the input board ... but then realised it would be an ideal application to use a double-decker board arrangement, so I may redo that later ... depends upon the thickness of the main board, when assembled. There is 10.7mm clearance using the headers for inter-board connection and that still leaves plenty of headroom in the box. What I have done will be fine for development of the software. Still no news on the SD socket.
Since the data logger will either be sensing temperature in degree's C, or 0 - 5 volts, I have arranged it so that the processor automatically knows how to configure a channel, , depending upon what kind of sensor is plugged into that channel.
-----------------------
August 11th 2008
Re-laid the input board to piggy back it on top of the main PCB, it now leaves enough space if I decide to add auto trigger etc. It plugs into port A and derives it's power for the sensors from Port A header. Will have to have a look at the height of the regulators on the main board. Also changed port B header, SD card header and programming port header to right angle ... also power in header.
Each channel now has over - voltage protection, hence the components on the left. Just thinking maybe I can use the spare space for the charging circuit !.
--------------------------------
August 12th 2008
I was thinking that it does not make much sense to use cables to connect to sensors. At least one temperature sensor needs to be wireless. I don't want to change the design of the main board again, so maybe I can knock one up and use a plug in Rx. Have to look at the scaling for the AN. If I move to SM for the new sensors, then I could include an op-amp at source either x2 or x4 and that would also get the signal out of the noise as well.
Think remote. LM35 ..... PWM ..... 433 TX.
433 Rx ..... integrator ..... DC out to data logger ?
Calibrate using look-up table to specific sensor ie (1).
August 20th 2008
Now that I have the kids project presentation wrapped up on the new laptop, I thought that I would have a go at the data logger mother board. After I have checked it out, I will insert the chips and start testing it. I have to admit that I am not looking forward to writing the software for this one Hi!. Here it is, minus chips ......
As per usual, I have hidden most of the messy bit under the 40 way socket. The plockets are all right angle because the input board plugs into the top of this one. The +5v and +3.3vlt PSU components are at the top of the board. Real time clock and clock backup battery are on the left. On the right are the DS card attenuators and SD output socket. Bottom right is the connector for the switch panel, this is used for the multi-level menu. The input board plugs straight into the left 20 way header socket and the 4 x 20 LCD plugs into port A on the right hand 20 way header socket.
The xtal and oscillator caps are all plug-in, because I am not sure whether I can get the SD side of things to run at 20 mHz. Top center is my standard bootloader connector that goes to the RS232 programming adaptor. I had purchased two USB to RS232 cables from Ebay some time ago and both seemed not to work. However I had to have a USB to RRS232 adaptor for the laptop and managed to get one of them to work after buying a new USB cable for it. The original one was intermittently open circuit. Providing the logger works out OK I intend to use it with the laptop. The laptop does not have a conventional card reader, but I had bought three from EBAY for about 4 pence each (including postage!) .... intending to canabilise them for the SD sockets .... but they all work as card readers, so that is another problem fixed.
The small 10K pot at the top is for tapping off a percentage of the battery voltage for battery monitoring purposes ...... and also as a sensing point for the smart charger circuit. In the top left hand corner is a small LED and jumper. As writing the software will be a lengthy process, I like to have an on board 'power on' indicator. When it is eventually fitted into it's working case ... this link will be removed.
The kid's project has so far consumed nine months of my life. Working in complete isolation can be difficult at times, especially when one run's into problems. For that reason I would like to make a special 'thank you' to Jack Lim, of Singapore who took the trouble to send me an email of encouragement. Thanks Jack !.
14th September 2008
Life goes on. Components ordered and sometime components received and those little interventions like a new 3 x 3 metre shed for the solar energy project. In the last month I did very little except, mix hundreds of barrow loads of concrete !. I finished laying the base yesterday .... Hooray !. I am just wondering how long it is going to take me to recover !.
I did locate some SD card sockets at Toby Electronics' in UK and that allowed me to have a look at a break away SD socket PCB. he sockets of course are surface mount !. I actually assembled one this morning ... only to discover that I had put the negative in the light box .... upside down. So shed deliveries permitting, I will make a couple of new PCB's tomorrow. I must admit I got into a right muddle laying out the PCB .... because of all the interconnection's. I sometimes think it miraculous that I ever get anything right !. I also have another component delivery on it's way, with the front panel switches and that will be the last PCB to do for the logger. I also managed to blow up the blue 20 x 4 LCD I had got for the logger. I accidentally fitted a 15 volt regulator to the mother board instead of a 5 volt one !. It also wiped out an 18F452 as well .... Ho Hum. The truth is that I have been so tired after every day of manual labour ..... I was working half asleep !. So a new LCD is also on it's way from China.
15th September 2008
By the time I got around to the PCB's I was dead on my feet again. After six tries I eventually got a usable one. I have been having a lot of problems with board from one particular supplier .... I always seem to end up with the edges of the larger board and dodgy coating !.
Same board front and back. The two 3mm countersunk screws are for mounting purpose. My cock-up today was after I had finished soldering the board I gave it a brush over with a steel brush ... back and front and damaged two SD socket contacts. One I managed to bend back into place and the other which was beyond ANY repair .... was pin 9, which I am not using anyway ... so I removed it completely .... and I hope now that all is well.
7th November 2008
Having finished off three sets of hardware for the kids project and also completed the first panel for the solar energy project I am finally back to have another go at the data logger. I had run out of header cable crimps and the cable I was using was far too long at 9". Crimps arrived today so I can now make a much shorter cable. As a confidence measure I did tidy up the Futurlec board and make a flying lead cable to plug into it the PIC board. The end result is that I can again open, check for duplicated file names, save and close files ..... so I know that the motherboard and basic program is OK.
I also had a look at the idea of using the date as the file name. The SD routines are limited to 8 characters, but they can be all numbers .... so one possibility is MMHHDDMM. I am not yet sure if I actually need this yet. Below are a couple of example files MM HH DD MM (43 12 09 12, just to prove it does work. Just realised I used the wrong month !
The real time clock (DS1307) is also working OK on the PCB. My next job is to use the 1 second square wave output on the DS1307 to time data sampling etc..
20th December 2008
Another memory wipe !. Fortunately I did make a note to order a couple of new 18F452 PIC's from Mechanique with preloaded 20 MHz bootloaders. They have now arrived ......
This will give a very noticeable increase in the speed of saving data to the SD card. I know because I started off using a 20 MHz xtal .... I just cannot remember why I ended up using a 4 MHz xtal .... unless I blew up the PIC with the 20 MHz bootloader .... when I was testing the SD break out PCB.
Copyright John Kent 2008