If you’re anything like me, and learning how to use micro controllers, you have probably learned that there is value in being able to save data, or settings between power cycles of your device.
For a long time, there was a significant learning curve, and no good explanation of how that would work. However, during a recent project, the incentives to learn became significant enough, that I took it upon myself to do research, and piece together what tidbits I could find, until things started making sense.
Now that I have a working knowledge of the process, I though I might attempt to share that information, so that the learning process might be easier for others.
First, flash memory structure.
It is imporant to understand that memory ultimately deals with bits and bytes. When you look at your ESP8266, and it says that it has 4MB of flash or 32Mb, what that means is, there are actually 4,194,304 bytes, or 33,554,432 bits.
Hypothetically, the smallest amount of data that you can work with at one time is 1 byte, meaning that any data value/setting will occupy, at minimum, 8 bits.
Imagine all of this as if it were a book, with enough room for 4,194,304 letters (each letter is equal to 1 byte).
As a rough estimate: If each word is 6.1 characters, that would be enough room for ~686,919 words. In comparison, Harry Potter and the Order of the Phoenix is 257,045 words.
Now, in addition to the memory being divided into bytes, the memory is divided into sectors of of 4096 bytes. These sectors are a way of managing the memory. When you want to save data, you have to use a whole sector, even if your data is only a few bytes. However, if your data is more than 4096 bytes, you can split that data into multiple sectors.
Thinking of a sector as a page, If you want to write anything at all, it has to go on one of these pages. If you write a single letter, it takes a page, if you want to write a sentence, or paragraph, it takes a page, even if it doesn’t fill it up. If you write a chapter, it may take up several whole pages, plus a little bit of another page.
Second, reading the flash memory.
When you want to start reading from Flash, you have to know two things: the address to start at, and the number of bytes to read (length).
In shool, you may have received instructions from a teacher, “Read chapter 3, on page 148”. You would then, without thinking about it, start on page 148, and read until you reached the end of chapter 3. However, when dealing with computers, you have to be very literal. The computer doesn’t know what the end of chapter 3 looks like, it doesn’t know the end of chapter 3 from the beginning of chapter 4. A better way of giving instructions would be, “Start reading at the beginning of page 148, and read for 2737 characters”.
Additionally, when reading from flash memory, you need to know how the data you’re reading is structured. If you previously wrote data with a structure, that consisted of:
- int (1 byte)
- long (4 bytes)
- long (4 bytes)
it was written as a total of 9 bytes, one after the other, into flash memory.
In the context of a book, it would be akin to writting a sentence without spaces. You would take some letters that make sense to you, “i love cats”, (1 character, 4 characters, 4 characters) and write them on the page in a little less organized, but more compact, way “ilovecats” (9 characters).
Later, when you come back to read that data out of flash memory, you need to make sure that you read it into the same structure that you wrote it from. (1 byte, 4 bytes, 4 bytes)
If you don’t do that properly, you’re going to be data out, but it won’t be accurate and wont’ make sense.
So, when reading the letters we wrote earlier, you need to know how it was organized before, if you read letter back from the page incorrectly (4 characters, 4 characters, 1 character) instead of 1, 4, and 4 characters, you get incomprehensible words: “ilov ecat s”.
In C, the language that was used for programming the ESP8266, this is done be creating a data structure, the same that was used to store the data before it was written, and then effectively, read the data from flash, into that data structure, that way, ints, longs, and strings end up where they are supposed to.
Third: Writing to flash memory.
When writing to memory, you have to erase the entire sector first, you can’t just write to the first 9 characters of the sector, instead, you have to erase all 4096 bytes, and then write 9 bytes of data, and effectively, 4087 bytes of nothing.
Like, when writing in a book, you have to start with the clean page, you can’t erase the first 9 letters of the page with existing content, and then add your own stuff. If you want to add something, you need to erase the whole page, and then add your content.
When you see documentation referring to the memory addresses, these are the address for the start of a sector. Each of these sectors are 4096 bytes long. The first sector starts at 0x00000, the second sector starts at 0x01000, etc.
### Flash size 32Mbit-C1: 1024KB+1024KB
esp_init_data_default.bin 0x3fc000 (optional)
blank.bin 0xfe000 & 0x3fe000
In the scenario above, these different files are going to be written to flash memory, starting at the specified addresses, and continuing on for as many sectors as necessary, until the entire file is written.
To illustrate this idea further, take a look at the Google Sheet showing the flash layout for my project.
The boot.bin files is written to sector 0x00000, and takes up all 4096 bytes.
The user1.4096.new.4.bin file (my application) is written to flash starting at address 0x1000, and, because it 302080 bytes in size, it will fill up all the sectors until 0x4C000.
Hypothetically, I could save my data to any sector that isn’t being used by some other file. I decided to use a sector near the “end” so that it is far away from my other files, and isn’t likely to be accidentally overwritten if I work on my program, and it grows in size. The address that I used was 0x7B000.
Because my settings persist between power cycles, if I want those settings to be erased, I can write blank.bin to that same address (0X7B000), if I don’t write blank.bin to that address, the program will be able to load those settings once it powers back up.