SparkFun OpenLog solderpad
sparkfun-openlog / OpenLog_v2 / Changes.txt
12-3-09
Copyright SparkFun Electronics� 2010
Nathan Seidle
spark at sparkfun.com
OpenLog is a simple serial logger based on the ATmega328 running at 16MHz. The ATmega328
should be able to talk to high capacity (larger than 2GB) SD cards. The whole purpose of this
logger was to create a logger that just powered up and worked. OpenLog ships with standard
57600bps serial bootloader running at 16MHz so you can load new firmware with a simple serial
connection. This makes it compatible with Arduino if needed.
OpenLog runs at 9600bps by default. This is configurable to 2400, 9600, 57600, and 115200bps. We recommend
you attach a serial connection to reconfigure the unit to work at a different serial speed, but you
should be able to do it in software.
Type '?' to get a list of supported commands.
During power up, you will see '12<'. '1' indicates the serial connection is established. '2' indicates
the SD card has been successfully initialized. '<' indicates OpenLog is ready to receive serial characters.
Recording constant 115200bps datastreams are supported. Throw it everything you've got!
'cd ..' is a bit weird. Normally it's 'cd..' but to change to a lower dir, use 'cd ..'
Currently, the firmware supports creating a new file or directory up to 16 characters including the
'.' and extension. "123456789012.txt" is the longest name. Any longer and the module will re-initialize
as if there was a variable sizeof error.
Capital letters, white space, and other characters are supported ("Hi there#$_.txt").
Type 'set' to enter baud rate configuration menu. Select the baud rate and press enter. You will then
see a message 'Going to 9600bps...' or some such message. You will need to power down OpenLog, change
your system UART settings to match the new OpenLog baud rate and then power OpenLog back up.
If you get OpenLog stuck into an unknown baudrate, there is a safety mechanism built-in. Tie the RX pin
to ground and power up OpenLog. You should see the LEDs blink back and forth for 2 seconds, then blink
in unison. Now power down OpenLog and remove the RX/GND jumper. OpenLog is now reset to 9600bps.
Please note: The preloaded STK500 serial bootloader is 2k, and begins at 0x7800 (30,720). If the code is
larger than 30,719 bytes, you will get verification errors during serial bootloading.
SD vs HCSD configuration is found in sd_raw_config.h - Currently only 512MB, 1GB, 2GB, and some
4GB cards work (not yet compatible with HCSD cards).
STAT1 LED is sitting on PD5 (Arduino D5) - toggles when character is received
STAT2 LED is sitting on PB5 (Arduino D13) - toggles when SPI writes happen
LED Flashing errors @ 2Hz:
No SD card - 3 blinks
Baud rate change (requires power cycle) - 4 blinks
During an append, OpenLog will buffer 512 characters at a time. That means that if the system loses power
while reading in characters, you may loose up to, but no more than, 511 characters. This is important for low
power systems where you may not know when the battery or power will die. OpenLog should record each buffer as
it receives each 512 byte chunk. The only way to exit an append is with Ctrl+z (ASCII 26).
8mA idle
18mA actively writing
Input voltage on VCC can be 3.3 to 12V. Input voltage on RX-I pin must not exceed 6V. Output voltage on
TX-O pin will not be greater than 3.3V. This may cause problems with some systems - for example if your
attached microcontroller requires 4V minimum for serial communication (this is rare).
v1.1
Adding better defines for EEPROM settings
Adding new log and sequential log functions.
Code is acting very weird with what looks to be stack crashes. I can get around this by turning the optimizer off ('0').
Found an error : EEPROM functions fail when optimizer is set to '0' or '1'.
sd-reader_config.h contains flag for USE_DYNAMIC_MEMORY
Looks like tweaking the optimization higher causes the asm("nop"); to fail inside the append_file routine. Changing this to
delay_us(1); works.
I have a sneaking suspicion that I was having buffer overrun problems when defining the input_buffer at 1024 bytes. The
ATmega328 should have enough RAM (2K) but weird reset errors were occuring. With the buffer at 512bytes, append_file runs
just fine and is able to log at 115200 at a constant data rate.
Added file called 'lots-o-text.txt' to version control. This text contains easy to scan text to be used for full data
rate checking and testing.
v1.2
ringp added:
Adding support for splitting command line parameters into arguments
Adding support for reading files sequencially
read <filename> <start> <length>
New log now for sequencial log functions supports 0-65535 files
Adding support for wildcard listing or deletion of files
ls <wildcard search>
rm <wildcard delete>
Really great additions. Thanks ringp!
Nate added error testing within newlog()
Checks to see if we have 65534 logs. If so, error out to command prompt with "!Too many logs:1"
v1.3
Added sd_raw_sync() inside append_file. I believe this was why tz1's addition of the timeout buffer update feature
was not working. Auto buffer update now working. So if you don't send anything to OpenLog for 5 seconds,
the buffer will automatically record/update.
Need to create 'Testing' page to outline the battery of tests we need to throw at any OpenLog after a firmware
submission and update is complete.
Testing
create 40,000 logs
Record at full speed:
Run at full 115200, load lotsoftext.txt and verify no characters are dropped.
Detect too many logs:
Create new log at 65533 (do this by editing 'zero' function to set EEPROM to 0xFF and oxFD)
and power cycle. Verify unit starts new log. Power cycle and verify unit errors out and drops to command prompt.
Record buffer after timeout:
Create new log. Type 20 characters and wait 5 seconds. Unit should auto-record buffer. Power down unit.
Power up unit and verify LOG has 20 characters recorded.
v1.4
Added exit options to the two menus (set and baud)
Also added display of current settin to two menus (Ex: "Baud currently: 57600bps")
Added '!' infront of 'error opening'. This pops up if there is an error while trying to append
to a freshly created new log (ex: LOG00548.txt is created, then errors out because it cannot append).
'!' was added so that user can parse against it.
Replicated logging errors at 57600 using 5V Arduino
Unit would systematically glitch during logging of 111054 bytes
Increasing buffer to 1000 characters caused URU error.
URU: Unit Resets Unexpectedly
To recreate URU error. Type "append ". Include the space. If you get "!error opening", then things are
fine. If you get "!error opening#" where # is a weird character, then type 'ls' and the unit will
unexpectedly reset (URU error). I believe this is attributed to a memory overrun somewhere in the
FAT system.
Changed buffer size to 900 and declared the character buffer as volatile
#define BUFF_LEN 900
volatile char input_buffer[BUFF_LEN];
This increase to the buffer allows for clean logging of 444055 bytes with no URU errors.
Experimenting with Scott's SD cards (customer gave cards on loan for recreating logging errors):
Card with single ~740mb file produces errors when trying to open/append to new log.
Card with less stuff on it logs full 444055 bytes correctly.
v1.5
Added 4800bps and 19200bps support
Added power saving features. Current consumption at 5V is now:
In default append mode:
6.6/5.5mA while receiving characters (LED On/Off)
2.1mA during idle
In command mode: 3.2/2.1mA (LED On/Off)
So if you're actively throwing characters at the logger, it will be ~6mA. If you send the logger
characters then delay 5-10 seconds, current will be ~2.5mA. (Unit records the characters in the buffer
and goes into idle more if no characters are received after 5 seconds)
These power savings required some significant changes to uart.c / uart_getc()
v1.51 check_emergency_reset, default break character is ctrl+z 3 times, example Arduino sketch
Added function from mungewell - check_emergency_reset. This has improved testing of the RX pin.
There was potential to get a false baud reset. There is still a chance but it's now much less likely.
If OpenLog is attached to a Arduino, during bootloading of the Arduino, ctrl+z will most likely be sent
to the Arduino from the computer. This character will cause OpenLog to drop to command mode - probably
not what we want. So I added user selectable character (ctrl+x or '$' for example) and I added
user selectable number of escape characters to look for in a row (example is 1 or 2 or 3, '$$$' is a
common escape sequence). The default is now ctrl+z sent 3 times in a row.
Added an example Arduino sketch (from ScottH) to GitHub so that people can easily see how to send characters to
OpenLog. Not much to it, but it does allow us to test large amounts of text thrown at OpenLog
at 57600bps.
v1.6 Adding config file.
What happens if I reset the system by pulling RX low, but the config file has corrupt values in it?
If config file has corrupt values in it, system will default to known values 9600/ctrl+z/3/newlog
If config file is empty, system resets to known values
After some massive testing, and lots of code to check for illegal states, it looks to be pretty stable.
The only problem is that we're running out of RAM. The buffer had to be decreased from 900 bytes
to 700 bytes to facilitate all the config file checking. Testing at 57600bps, unit runs very well
over 40kb test file on straight RS232 connection. That's pretty good. Testing at 115200 on straight
connection, unit will drop a buffer every once and a while. Not great, but not much we can do if the
SD card times out for ~150ms while it's writing.
8 bits to the byte plus a start/stop bit = 10 bits per byte
@ 9600bps = 960 bytes per second. Buffer will last for 729ms
@ 57600bps = 5760 bytes per second. Buffer will last for 121ms
@ 115200bps = 11520 bytes per second. Buffer will last for 60.7ms
So if the SD card pauses for more than 60ms, 115200 will have lost data, sometimes. All other baud rates
should be covered for the most part.
SD cards with larges amounts of data will have increased pause rates. Always use a clean card where possible.
v1.61 Small PCB change. Fixed version # in help menu.
Fixed the firmware version in the help menu to v1.61.
Updated Eagle files to Eagle v5.9. Fixed weird airwire. Changed D1 LED from Green to Blue.
Will only affect new production after 4-28-10.
Closed some tickets and wrote some more example Arduino code:
http://forum.sparkfun.com/viewtopic.php?t=21438
v2.0 - 25986 bytes out of 30720
Welcome to version 2! We've moved from Roland Riegel's FAT library to Bill Greiman's sdfatlib. OpenLog now works with SD cards
up to 16GB (that is the current largest microSD card we can get our hands on). OpenLog automatically detects and works with FAT16/FAT32
file systems. It also automatically works with normal SD cards as well as SDHC cards.
Almost all the previous commands have been ported from v1 to v2. The current commands that do not work:
cd.. - does not work. You can change to an upper directory, but you cannot navigate back down the tree.
cat - this command was depricated. HEX printing is now done with the 'read' command. We've added a 5th argument to select between ASCII and HEX printing.
Wild cards do not yet work. So rm and ls do not have wild cards enabled - yet. Help us out!
Porting OpenLog to work directly under Arduino to work with sdfatlib (http://code.google.com/p/sdfatlib/) by Bill Greiman.
sdfatlib intrinsically supports FAT16, FAT32 as well as SD and HCSD cards. In a word, it's amazing.
Needs to be done:
Done - Get config file reading/loading working
Document config file in wiki: if no config file is found, current system settings are used. If config is found, system switches to settings found in file. If system settings are changed, then config file is changed and system uses new settings immediately.
Done - We don't want to constantly record a new config file on each power on. Only record when there is a change.
Get cd.. working
Seperate OpenLog_v2 into multiple files
Re-test too many log files created in the newlog setting - 65535. Potentially create thousands of files and see how sdfatlib handles it.
Done - Test sequential logging.
Get wild card support working for ls and rm
Get backspace working
Test rm with directories, and directory wildcards? Maybe not.
Get power save working
Test compile on a computer that doesn't have WinAVR
Test commands:
new - works, also in sub dirs
append - works, also in sub dirs
rm - works, but does not yet support wild cards.
md - works, also in sub dirs
cd - you can change up to a sub-directory, but you cannot navigate back down the tree. The current work around is to type 'init'. This will re-init the card and set the directory back to root.
ls - works pretty well but does not yet support wild cards. Shows directories, files, and file sizes. Would be cool if it listed dirs first.
read - works well. Tested 0, 1, 2, 3, 4, and 5 arguments (included and left off). Fails gracefully. Now prints in HEX as well!
size - works well
disk - works well, prints more information than ever before!
init - works well
sync - works, but not really tested
cat - I've decided to drop this command. You can now print in hex using the read command and using a 5th argument as '1' for ASCII (default) and '2' for HEX.
set - works well
baud - works well
v2.1 - Power save not working. Fixed issue 35. Dropping characters at 57600bps.
26058 bytes out of 30720
Fixed a bug found by Salient (Issue 35). User settings where declared at chars which allowed them to be signed. If a user went from old firmware, to v2,
the safety checks would fail because the settings would be read at -1 instead of 255. Declaring user settings as byte fixed issue.
Added "a) Clear user settings" to set menu. This allows us to completely wipe an OpenLog (user settings and config file) to see how it will respond
to future firmware changes.
Improved the file 'size' command.
Sequential logging is tested and works.
Receive testing: Using the Test_Sketch found on Github, I am testing the receive reliability at different UART speeds.
We need to test a lot of received data. At 57600, 115200, and both from an Arduino (lots of time in between characters becuase of software overhead)
and from a raw serial port (almost no time in between characters). I am hoping to make sdfatlib hiccup at 115200, full speed, across a 1MB file. If
I can make it fail, then we can start to increase the buffer size and look at how much RAM sdfatlib has left open for the buffer.
9600bps from Arduino works fine
57600bps from Arduino drops characters
115200 from Arduino drops characters
It seems that sdfatlib takes longer to write to the SD card than the original file system from Robert Reigel. I'm thinking perhaps
we should create a version of OpenLog firmware that is just sequantial logging, no fancy system menus... It might open up some RAM.
If only we could get control of the UART from Arduino's clutches, we could probably handle the ring buffer much better. Not sure how to handle UART
interrupts without tearing out HardwareSerial.cpp.
Added README to the Test sketch. Added 115200bps to test sketch.
v2.11 Tested with 16GB microSD. Fixed some general bugs. Lowered power consumption.
26136 bytes out of 30720
Fixed issue 30. I added printing a period ('.') for non-visible ASCII characters during a 'read' command. This cleans up the output a bit. HEX
printing is still available.
Fixed issue 34. When issuing a long command such as "read log00056.txt 100 200 2" (read from spot 100 to spot 200 and print in HEX), the
command shell would die at 24 spots. I increased both the general_buffer and 'buffer' in the command shell from 24 to 30. The limit is now
30 characters, so "read log00056.txt 100 20000 2" is allowed.
Works with a 16GB microSD card! High volume test: loaded 16GB card with 5GB of data. Basic serial tests work. When running at 57600, there
is an odd delay. I think it has to do with the file system doing an initial scan for an available cluster. Takes 2-3 seconds before OpenLog
returns to normal. This can cause significant data loss.
Fixing power management in v2. Power down after no characters for 3 seconds now works. Unit drops to 2.35mA in sleep. 7.88mA in sitting
RX mode (awake but not receiving anything). There is still a weird bug where the unit comes on at 30mA. After sleep, it comes back at the
correct 7-8mA. Is something not getting shut off?
v2.2 Modified append_file() to use a single buffer. Increased HardwareSerial.cpp buffer to 512 bytes.
More testing at 57600. Record times look to be 2, 5, and zero milliseconds when doing a record. This means that the split buffer doesn't
really make a difference? There are some records that take 150ms, 14ms, etc. At 57600bps, that's 7200 bytes/s, 138us per byte. With a 150ms
pause, that's 1,086 bytes that need to be buffered while we wait... Grrr. Too many.
I re-wrote the append_file function to use only one buffer. This allows us to more quickly pull data from the hardware serial buffer. Hardware
serial buffer has to be increased manually to 512. This file (hardwareserial.cpp) is stored in the Arduino directory. With testing,
it seems like recording is working more solidly at 57600bps. But now I'm seeing glitches at 19200bps so more testing is required before we
make this the official OpenLog release.
Moved input_buffer into within the append function. Attempting to shave bytes of RAM.
v2.21 ringp fork brought in. Wildcard remove and list commands now work. Remove directory now works! Change directory up/down the tree works again.
28440 bytes used of 30720.
ringp brought back many great commands! Thanks ringp!
rm LOG*.* works
ls *.TXT works
cd .. works again
ls now correctly shows directories first and files following the directories.
To remove a directory, you have to navigate into that directory. For example:
>cd TEMP (you are now in TEMP directory)
>rm -f TEMP (you are now removing TEMP, and you will be pushed back down one level of the tree)
>ls (shows files and directories where TEMP directory used to be, but TEMP directory should be gone)
ringp added new commands:
efcount: gives the number of files in the current directory. Example: "efcount" returns "count|3". There are 3 files in the current directory.
efinfo <spot>: gives info about the file in <spot>. Example: "efinfo 2" reports "LOG00588.TXT|45". File number 2 is named LOG00588.TXT and is 45 bytes in size.
verbose <"on"|"off">: sets whether command errors are verbose (long winded) or just the "!" character. Example: "verbose off" sets verbose off. Then if a
command like "blah" is received, then only "!>" is seen from the command line interface. This makes it easier for embedded systems to recognize there
was an error. This setting is not recorded to EEPROM.