[Server-devel] ext2 vs ext4 vs exFAT for XO content SD cards?

[Adam Holt]

What filesystem would people recommend for ~128GB SD cards inserted into XO
laptops or XSCE servers heading far afield, to insert very large
(evolving?) digital libraries just like Internet-in-a-Box?

Multiyear reliability for all this content would be important, but some say
ext4 (slow journaled filesystems?) have their own problems on SD?

Is wear-leveling of modern SD cards (Flash memory) fully taken care of by
the largest manufacturers/drivers already, or should we seek out particular
filesystems/drivers? And unmount / power off carefully etc?

In-country copying and eventually in-country remixing of SD cards
(containing local-language, local-vid cultural jewels) would be Absolutely
Wonderful -- if we can achieve that by using a filesystem that works
instantly (exFAT? NTFS?) across all Windows and Mac computers ideally? For
all static content anyway, on 1 partition of several? Accelerating dup'ing
of SD cards as a bonus?!

Many paths possible for sure, but most important: what
interop/longevity/other issues are we overlooking??

[James Cameron]

Thanks, interesting questions.

No, ext4 is not a slow journaled filesystem, and no, there are no
obvious problems on SD when using ext4 given your use case. But it
isn't operating system portable, and as your content is static no need
for a journal. Other features of ext4 make mounting or filesystem
check faster.

Yes, wear-leveling is taken care of by the firmware in the card, put
there by the manufacturers. Wear-levelling also critical during
reading, since a flash page can't be read repeatedly without
disturbance eventually requiring it to be written to a freshly erased
page. This is all handled by the firmware. Happens way more
frequently than it does on a hard drive.

Duplication time of SD cards won't be affected by your filesystem or
partition decision.

One partition is sufficient. MBR partition type best, for
compatibility across the operating systems.

For filesystem, use FAT32, mounted read-only. FAT32 works across most
Windows and Mac computers, at media sizes up to 2 TB, for file sizes
up to 4 GB.

Where content cannot live on FAT32 due to file name character set or
metadata, it can be placed in disk image bundles of ISO-9660,
squashfs, or ext4 and loop mounted. The content curation process for
the end user might be easier if bundles can be added and removed as
needed.

What you might be overlooking; I/O bandwidth of the connection to the
media, endurance impact of reading data from the card slowing
performance one year on, backups, content bundle tamper checks, risk
of filesystem format incompatibilities introduced by new versions of
operating systems after your product is in the field, risk of cross
system malware infections, electrostatic discharge damage to the card,
and how modern cards can change performance behaviour as a result of a
production state awareness flag stored by card firmware.

On the other hand the alternatives have their own problems.

[Sameer Verma]

James,
Would it help to mark the content partition(s) as read only?

Sameer

Post by James Cameron
Thanks, interesting questions.
No, ext4 is not a slow journaled filesystem, and no, there are no
obvious problems on SD when using ext4 given your use case. But it
isn't operating system portable, and as your content is static no need
for a journal. Other features of ext4 make mounting or filesystem
check faster.
Yes, wear-leveling is taken care of by the firmware in the card, put
there by the manufacturers. Wear-levelling also critical during
reading, since a flash page can't be read repeatedly without
disturbance eventually requiring it to be written to a freshly erased
page. This is all handled by the firmware. Happens way more
frequently than it does on a hard drive.
Duplication time of SD cards won't be affected by your filesystem or
partition decision.
One partition is sufficient. MBR partition type best, for
compatibility across the operating systems.
For filesystem, use FAT32, mounted read-only. FAT32 works across most
Windows and Mac computers, at media sizes up to 2 TB, for file sizes
up to 4 GB.
Where content cannot live on FAT32 due to file name character set or
metadata, it can be placed in disk image bundles of ISO-9660,
squashfs, or ext4 and loop mounted. The content curation process for
the end user might be easier if bundles can be added and removed as
needed.
What you might be overlooking; I/O bandwidth of the connection to the
media, endurance impact of reading data from the card slowing
performance one year on, backups, content bundle tamper checks, risk
of filesystem format incompatibilities introduced by new versions of
operating systems after your product is in the field, risk of cross
system malware infections, electrostatic discharge damage to the card,
and how modern cards can change performance behaviour as a result of a
production state awareness flag stored by card firmware.
On the other hand the alternatives have their own problems.
--
James Cameron
http://quozl.linux.org.au/
_______________________________________________
Server-devel mailing list
http://lists.laptop.org/listinfo/server-devel

[James Cameron]

No, it won't help.

1. reading data an SD card (or eMMC, or USB flash drives) does cause
writes internal to the card, and does reduce life,

2. there's no such flag to set in an MBR partition table,

Perhaps you mean a write protect switch on the card? This is a
plastic slide, sensed by a switch in the socket, and provides _advice_
to the system. Some systems don't receive the advice. The system can
ignore that advice. Even if acted upon, it only stops change to the
data, it doesn't stop writes internal to the firmware of the card.

Post by Sameer Verma
James,
Would it help to mark the content partition(s) as read only?
Sameer
Thanks, interesting questions.
No, ext4 is not a slow journaled filesystem, and no, there are no
obvious problems on SD when using ext4 given your use case.  But it
isn't operating system portable, and as your content is static no need
for a journal.  Other features of ext4 make mounting or filesystem
check faster.
Yes, wear-leveling is taken care of by the firmware in the card, put
there by the manufacturers.  Wear-levelling also critical during
reading, since a flash page can't be read repeatedly without
disturbance eventually requiring it to be written to a freshly erased
page.  This is all handled by the firmware.  Happens way more
frequently than it does on a hard drive.
Duplication time of SD cards won't be affected by your filesystem or
partition decision.
One partition is sufficient.  MBR partition type best, for
compatibility across the operating systems.
For filesystem, use FAT32, mounted read-only.  FAT32 works across most
Windows and Mac computers, at media sizes up to 2 TB, for file sizes
up to 4 GB.
Where content cannot live on FAT32 due to file name character set or
metadata, it can be placed in disk image bundles of ISO-9660,
squashfs, or ext4 and loop mounted.  The content curation process for
the end user might be easier if bundles can be added and removed as
needed.
What you might be overlooking; I/O bandwidth of the connection to the
media, endurance impact of reading data from the card slowing
performance one year on, backups, content bundle tamper checks, risk
of filesystem format incompatibilities introduced by new versions of
operating systems after your product is in the field, risk of cross
system malware infections, electrostatic discharge damage to the card,
and how modern cards can change performance behaviour as a result of a
production state awareness flag stored by card firmware.
On the other hand the alternatives have their own problems.
--
James Cameron
[2]http://quozl.linux.org.au/
_______________________________________________
Server-devel mailing list
[4]http://lists.laptop.org/listinfo/server-devel
[2] http://quozl.linux.org.au/
[4] http://lists.laptop.org/listinfo/server-devel
_______________________________________________
Devel mailing list
http://lists.laptop.org/listinfo/devel

[Adam Holt]

A "Custom_Content" folder is eventually demanded by almost every IIAB-like
deployment.

The reason is that every local school / librarian quite naturally wants a
Non-Bureaucratic process, to add their own language/videos/curriculum,
copying their own file-tree onto the SD card, using their own Windows/Mac
computer hassle-free.

Towards this quite universal demand, an exFAT partition seems much better
than FAT32, as exFAT works with most all recent Windows and Mac machines,
without filename limitations. (Not unrelated to exFAT being the modern SD
Card industry standard.) Reason being that teachers/principals/pedagogy
folk just don't have hours and hours to squash every filename down to
FAT32's 8+3 character limit.

Likewise for the same reason NTFS is another strong possibility, with
better file ownership/permissions possibilities than exFAT, however NTFS
does not work out-of-the-box on Mac. Which doesn't much matter across the
preponderance of lower-class schools planetwide, where Windows 7 and
similar are the norm. However deep-pocketed NGO educators bring
Mercedes-class MacBooks to the table for sure; yes rich voluntourists can
be annoying but when they invest for the long-term let's absolutely make a
place for them too =)

Post by James Cameron
No, it won't help.
1. reading data an SD card (or eMMC, or USB flash drives) does cause
writes internal to the card, and does reduce life,
2. there's no such flag to set in an MBR partition table,
Perhaps you mean a write protect switch on the card? This is a
plastic slide, sensed by a switch in the socket, and provides _advice_
to the system. Some systems don't receive the advice. The system can
ignore that advice. Even if acted upon, it only stops change to the
data, it doesn't stop writes internal to the firmware of the card.

Post by Sameer Verma
James,
Would it help to mark the content partition(s) as read only?
Sameer
Thanks, interesting questions.
No, ext4 is not a slow journaled filesystem, and no, there are no
obvious problems on SD when using ext4 given your use case. But it
isn't operating system portable, and as your content is static no

need

Post by Sameer Verma
for a journal. Other features of ext4 make mounting or filesystem
check faster.
Yes, wear-leveling is taken care of by the firmware in the card, put
there by the manufacturers. Wear-levelling also critical during
reading, since a flash page can't be read repeatedly without
disturbance eventually requiring it to be written to a freshly erased
page. This is all handled by the firmware. Happens way more
frequently than it does on a hard drive.
Duplication time of SD cards won't be affected by your filesystem or
partition decision.
One partition is sufficient. MBR partition type best, for
compatibility across the operating systems.
For filesystem, use FAT32, mounted read-only. FAT32 works across

most

Post by Sameer Verma
Windows and Mac computers, at media sizes up to 2 TB, for file sizes
up to 4 GB.
Where content cannot live on FAT32 due to file name character set or
metadata, it can be placed in disk image bundles of ISO-9660,
squashfs, or ext4 and loop mounted. The content curation process for
the end user might be easier if bundles can be added and removed as
needed.
What you might be overlooking; I/O bandwidth of the connection to the
media, endurance impact of reading data from the card slowing
performance one year on, backups, content bundle tamper checks, risk
of filesystem format incompatibilities introduced by new versions of
operating systems after your product is in the field, risk of cross
system malware infections, electrostatic discharge damage to the

card,

Post by Sameer Verma
and how modern cards can change performance behaviour as a result of

a

Post by Sameer Verma
production state awareness flag stored by card firmware.
On the other hand the alternatives have their own problems.
--
James Cameron
[2]http://quozl.linux.org.au/
_______________________________________________
Server-devel mailing list
[4]http://lists.laptop.org/listinfo/server-devel
[2] http://quozl.linux.org.au/
[4] http://lists.laptop.org/listinfo/server-devel
_______________________________________________
Devel mailing list
http://lists.laptop.org/listinfo/devel

--
James Cameron
http://quozl.linux.org.au/
--
<http://quozl.linux.org.au/>
<http://quozl.linux.org.au/>
http://unleashkids.org !

[James Cameron]

Post by Adam Holt
Towards this quite universal demand, an exFAT partition seems much
better than FAT32, as exFAT works with most all recent Windows and
Mac machines, without filename limitations.  (Not unrelated to exFAT
being the modern SD Card industry standard.)  Reason being that
teachers/principals/pedagogy folk just don't have hours and hours to
squash every filename down to FAT32's 8+3 character limit.

You are quite wrong here.

LFN (long filename) support is present in all the operating systems
you've mentioned, and works fine with FAT32.

In the OLPC ecosystem, only Open Firmware has an 8.3 limit, and there's
nothing in your use case which suggests that Open Firmware is to be
used for content delivery!

[Adam Holt]

Post by James Cameron
LFN (long filename) support is present in all the operating systems
you've mentioned, and works fine with FAT32.

Thanks for the correction. FAT32 is indeed about as universal/tolerable of
a standard as possible in 2015. But might not be wise on large SD cards
going forward, given 32GB is the traditional Windows limit for FAT32
partitions. And of course FAT32's a bit slower than modern filesystems,
and worse 4GB content zip/repos (FAT32's upper limit) will appear before we
know it?

On balance, SD Card industry standard exFAT seems (to me) more future-proof
for a hassle-free "grassroots content" partition over coming years, for 1%
of the SD card of whatev. Or perhaps NTFS ~if~ file owners/permissions are
more important than Mac support.

In the OLPC ecosystem, only Open Firmware has an 8.3 limit, and there's

Post by James Cameron
nothing in your use case which suggests that Open Firmware is to be
used for content delivery!

:)

[James Cameron]

Post by Adam Holt
On balance, SD Card industry standard exFAT seems (to me) more
future-proof for a hassle-free "grassroots content" partition over
coming years,
[...]

If you're able to control the desktops and laptops that will be used
to add or remove content, sure, but exFAT is quite recent as far as
remote villages are concerned; what will you do when you get teachers
who can't even open it? Any systems with unpatched XP or Vista will
be affected. Any Mac earlier than 10.6.5 too.

[Emmanuel Engelhart]

Just to chime in, as has been already noted the max file size on a FAT32
system is 4GB. Some of the files we deal with are much larger than that.
Ex. the Zim files for TED talks etc. are 8GB+ in size. Now we could
always break them into smaller chunks, but that is another step.

ZIM files can be splitted in chunks (like in ou "portable versions").
Concerning Kiwix, the only point which stops us to say FAT32 is (still)
the best filesystem is that in case of Wikipedia in English, the
fullsearch text index (a directory) contains files which are over the
4GB limit (and there is not easy way to split them).

As a consequence, on the Kiwix side, our conclusion was that ext4 is the
best solution for our kiwix-plugs servers (we control the OS) and exfat
otherwise.

Emmanuel

[Adam Holt]

Great points all around.

To give a genuine voice to innovative but non-technical grassroots
teachers, I'd very much advocate for a 1GB exFAT/NTFS/FAT32 partition (or
whatever, let's say 1% of the 64GB or 128GB or 256GB SD card) to give
streets-is-talkin local educators authentic voice, alongside neocolonial
megapublishers birthing new 21st century learning cultures. (We ARE the
99%, so at least give us 1% of the disk :-)

Much as Daniel Drake's 2009-era http://wiki.laptop.org/go/OS_Builder is a
far better tool for national deployments, as compared to Michael Stone's
2008-era http://wiki.laptop.org/go/Customization_stick which was so
effective for community deployments during its heyday, much was also lost
(beyond just Michael Stone's paycheck ;) As Mass Customization is never as
easy as the buzzword implies, and supporting both approaches became too
expensive for OLPC.

But both styles have their essential place (mass-production and
in-field-artisanry) so going forward I strongly hope we make a place for
both: megapublishers mass-cloned content a-la-IIAB, and equally important:
in-field customization by amateur/local true-as-can-be educators &
librarians, rise-of-the-rest folks who will rarely speak English nor
understand our documentation, let's be honest!

(Even if God Forbid teachers occasionally bootleg motivating content, using
this 1% partition of static local content, just like every interesting
teacher has done over the past half century, at the mimeograph/xerox
duplication machine...)

I've some questions and statements.

Hi Adam,
This is an excellent initiative and just what we are looking for in the
Pacific. We are thinking of 3 versions of the SD Card,

Hi Ian,
I'm wondering about your creation method that you plan on using, are you
planning to create the initial SDcard then adding your content with the
intent on cloning the configured SDcard later?

one each for
primary, secondary and TVET. We are collecting a good range of content
with
some focussed on the Pacific, but we are also looking at people creating
their own or adapting them to suit local conditions and adding it to the
pool
BTY, have you seen the TVET equivalent of Khan?

Please keep us in the loop
Ian

Now to clear some misconceptions and classify some limitations.

Post by Adam Holt
What filesystem would people recommend for ~128GB SD cards inserted
into
XO laptops or XSCE servers heading far afield, to insert very large
(evolving?) digital libraries just like Internet-in-a-Box?
Multiyear reliability for all this content would be important, but some
say ext4 (slow journaled filesystems?) have their own problems on SD?
Is wear-leveling of modern SD cards (Flash memory) fully taken care of by
the largest manufacturers/drivers already, or should we seek out particular
filesystems/drivers? And unmount / power off carefully etc?

/library contains databases and squid's cache it is going to be linux only
period. Which type of linux filesystem is open for debate.

Post by Adam Holt
In-country copying and eventually in-country remixing of SD cards
(containing local-language, local-vid cultural jewels) would be Absolutely
Wonderful -- if we can achieve that by using a filesystem that works
instantly (exFAT?

That needs added support to be useful.

Post by Adam Holt
NTFS

That should be included out of the box but speak up if you find that
lacking in what you are using.
?) across all Windows and Mac computers ideally? For

Post by Adam Holt
all static content anyway, on 1 partition of several? Accelerating dup'ing
of SD cards as a bonus?!

I support the reading of all filesystem types for content importing
purposes for use within the XSCE but we can't really make use of non-linux
filesystems within the running system without jumping through hoops(with
lookback mounts sort of like a swapfile). We can set aside some space but
it would be a trade-off against what is available for the system. I think
the entire configured SDcard should be linux for the system, with content
that has not be pre-installed be available on different physical media,
like with activities and usbmount.
Once the content is added/imported you can clone the SDcard, now there is a
project create a cloning playbook.
Now *maybe* in the future new playbooks to export data, think directory
tree here for lets say portal for use/editing elsewhere in zip format. The
export could go to a non-linux file system if that support is installed.
Now the import could unzip the tree back into place, ensure the permissions
are sane and restart the service. Or whatever data/zims/pubs/ you're
messing with the pull request would be welcome. We have a pretty good
handle on consuming content, we need to work through sharing the data with
others and are open to ideas.
Jerry
--
---
You received this message because you are subscribed to the Google Groups
"Unleash Kids" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
--

[Tim Moody]

I agree that http://www.tvetacademy.org/ looks interesting, provided you can meet their staff requirements. In your case probably you can use a lot of their Eng materials so no need to shoot videos, etc. It would be great if you would contact them and see what develops.



From: ***@googlegroups.com [mailto:***@googlegroups.com] On Behalf Of Ian Thomson



BTY, have you seen the TVET equivalent of Khan?

http://youtu.be/PrqcaxsyK0E

[Adam Holt]

Thanks You Arnd, we have many months work ahead of us and this will be a
big help!

Post by Adam Holt
What filesystem would people recommend for ~128GB SD cards inserted into

XO

Post by Adam Holt
laptops or XSCE servers heading far afield, to insert very large
(evolving?) digital libraries just like Internet-in-a-Box?
Multiyear reliability for all this content would be important, but some

say

Post by Adam Holt
ext4 (slow journaled filesystems?) have their own problems on SD?

ext4 is better than most file systems in this regard (much better than
ext3,
also better than btrfs).

Post by Adam Holt
Is wear-leveling of modern SD cards (Flash memory) fully taken care of by
the largest manufacturers/drivers already, or should we seek out

particular

Post by Adam Holt
filesystems/drivers? And unmount / power off carefully etc?

Some cards handle it better than others. Traditionally, Kingston cards
were completely unusable after writing data on the order of the card size,
while Samsung and Sandisk did much better (I have not checked in a few
years
though, so YMMV). Most other manufacturers have multiple sources for the
embedded controllers on the cards, so it's a bit of hit and miss. The
controllers have all evolved a lot in the past few years, the simple
controllers that used to be on 8GB cards can no longer cope with the
increased block sizes on today's cards (32GB and larger).

Post by Adam Holt
In-country copying and eventually in-country remixing of SD cards
(containing local-language, local-vid cultural jewels) would be

Absolutely

Post by Adam Holt
Wonderful -- if we can achieve that by using a filesystem that works
instantly (exFAT? NTFS?) across all Windows and Mac computers ideally?

For

Post by Adam Holt
all static content anyway, on 1 partition of several? Accelerating

dup'ing

Post by Adam Holt
of SD cards as a bonus?!
Many paths possible for sure, but most important: what
interop/longevity/other issues are we overlooking??

SDXC cards are mandated to be using exFAT (just like SDHC cards have
to use VFAT, and indeed this is the only difference between the two)
by the SD card standard. If you don't use this, you are strictly speaking
in violation of the standard and the cards might not behave as designed.
In particular, the card is allowed to only do efficient garbage collection
for the access patterns that you get with a single exFAT partition that
spans the entire card and has all its metadata aligned exactly in the
way that the spec defines.
In practice, things tend to work mostly ok with other file systems, but
if you use NTFS or ext3 (rather than ext4), you are usually asking for
trouble.
The best longevity would be provided by f2fs, which is designed to work
fine on most SD cards. The downside is that it only works on relatively
modern Linux kernels (3.x or higher).
I would expect that cards today use only dynamic wear leveling, not
static wear leveling as real SSDs do. This means that content on a
read-only partition will decay with the normal life of the card
(several years, but depending on the quality of the card and the
environmental conditions, e.g. not too hot), independent of the
presence of partitions you write to.
Dynamic wear leveling works best if you have a lot of unused blocks,
so a good strategy for long life would be to leave a whole partition
(e.g. 20% of the size of the writable partition, the more you have,
the longer the card will survive) that never gets written after
manufacturing, or at least gets erased using the fitrim ioctl
command after the initial imaging.
For a 128 GB card with 115GB of actual space, you could then use something
* 80GB zisofs/cramfs/squashfs for static data
* 30GB f2fs/ext4 for writable data
* 5GB unused space for wear leveling
Arnd