Binghua Duan's Linux & Windows CE Blog

Best Practice to Make Linux/Android Filesystem Robust

2011-03-04T10:03:00.000+08:00

Introduction ¶

There is a common concern in Linux-based design: the buffering of filesystem writes is very good for performance, but may cause the buffered data to be lost and so filesystem corruption when the system go down unexpectedly.
This documents presents some solutions you can take to make your Linux-based system perform more robustly.

1. Use a Read-Only Partition for Root Filesystem ¶

The simplest solution to avoiding power failures during writes is to avoid all writes - files that don't to be modified should be kept on a partition that is mounted read-only. In most cases (not include NAND flash device) it means that there will be no modifications to the entire partition, and as a result there is no risk of corruption. If local data writing is required, an extra read-write partition can be established for that purpose only. With this setup, Even in worst-case, we can still ensure system will boot correctly.
In Android platform, there're several options to use read-only partitions:

Root filesystem as ramdisk and Android system partition as read-only partition.

Root filesystem and Android system files are in a same read-only partition.

Root filesystem and Android system files are in separate read-only partition.

Which option to use depends on the image size of the root filesystem. Ramdisk can only hold small rootfs, and a too large partition is not convenient to do upgrading.

2. Use the Right Filesystem for Read-write partitions ¶

Linux offers many many filesystems, such as jffs2, yaffs2 and ubifs for flash devices, ext2, ext3, ext4 and btrfs for disk-like devices.
If the Linux-based system (i.e. Android platform) is booting from a SD/MMC card or managed NAND, the ext3 or ext4 is a good choice.
The ext3 or third extended filesystem (wiki: http://en.wikipedia.org/wiki/Ext3) is a journaled file system that is commonly used by Linux kernel. Its main advantage over ext2 is journaling which improves reliability and eliminates the need to check the file system after an unclean shutdown. The ext4 is its successor which has many extensions and performance improvements but also brings more data loss probabilities to be paid more attention.
Starting from Android-2.3 (Gingerbread), many Android devices are going to be moving from YAFFS to the ext4 filesystem according to the post on Google official Android developer blog. ( http://android-developers.blogspot.com/2010/12/saving-data-safely.html)
With the ext3/4 filesystem, system crashes or power loss are far less likely to make system become corrupted because of the journaling design.

One of the ext3 features to be noted is that by default, ext3 will commit changes to its journal every 5 seconds. So, in general, 5 seconds worth of writes might be lost as the result of a system crash or power loss. The value can be tuned.

One feature to be noted in ext4 is the delayed allocation which means that the filesystem tries to delay the allocation of physical disk blocks for written data as long as possible. This policy brings some important performance benefits. Many files are short-lived; delayed allocation can keep the system from writing fleeting temporary files to disk at all. And, for longer-lived files, delayed allocation allows the kernel to accumulate more data and to allocate the blocks for data contiguously, speeding up both the write and any subsequent reads of that data. However, this also brings more data loss probabilities.

There are some sysctl variables to shorten the system’s writeback time and so as to reduce the data loss possibility:

/proc/sys/vm/dirty_expire_centiseconds
/proc/sys/vm/dirty_writeback_centiseconds

The first variable controls how long written data can sit in the page cache before it’s considered “expired” and queued to be written to disk; it defaults to 30 seconds.

The second variable controls how ofter the pdflush process wakes up to actually flush expired data to disk; it defaults to 5 seconds.

Lowering these values will cause the system to flush data to disk more aggressively, with a cost in the form of reduced performance.

3. Use Right Solution When Developing Applications ¶

The final solution to this problem is to fix the applications which are expecting the filesystem to provide more guarantees than it really is.

Application developer should keep in mind that data doesn’t actually consistently reach the storage media when write() or even close() is called.

Applications which want to be sure that their critical data have been committed to the media can use the fsync() system call; that can be slow, so be careful not to call it carelessly.

In Android application, if you just use SharedPreferences? or SQLite, you don't need to worry about the data loss, because Android has already used fsync() to do the right thing about buffering.

Ubuntu - Install Redmine + MySQL in Ubuntu 10.04

2010-05-17T18:51:00.007+08:00

Introduciton

Redmine is an open source, web-based project management and bug-tracking tool. It includes calendar and gantt charts to aid visual representation of projects and their deadlines. It supports multiple projects. Redmine is free and open source software which provides integrated project management features, issue tracking, and support for multiple version control options.

The design of Redmine is significantly influenced by Trac, a software package with some similar features.

Redmine is written using the Ruby on Rails framework. It is cross-platform and cross-database.

Installation Steps
The following are my steps to install it on the new Ubuntu 10.04 system:
1) install mysql server & phpmyadmin tool

$ sudo apt-get install mysql-server phpmyadmin

You can just set all initial passwords to 'admin'. When finished, you can use http://localhost/phpmyadmin to manage your databases.
2) change mysql database directory (optional, I just want to store it in the /srv directory)

$ mysqladmin -u root -p shutdown
$ sudo mv /var/lib/mysql /srv/
$ sudo vi /etc/mysql/my.cnf (change datadir value from "/var/lib/mysql" to "/srv/mysql")
$ sudo vi /etc/apparmor.d/usr.sbin.mysqld (replace all "/var/lib/mysql" to "/srv/mysql")
$ sudo /etc/init.d/mysql restart (FIXME: doesn't work, I just reboot the system instead)

3) install redmine & redmine-mysql

$ sudo apt-get install redmine redmine-mysql

4) install rails...

$ sudo apt-get install mongrel ruby1.8-dev

5) change redmine files default directory (optional, I prefer to store them in the /srv directory)

$ sudo mv /var/lib/redmine /srv/
$ sudo vi /usr/share/redmine/config/environment.rb (replace all "/var/lib" to "/srv")

6) test redmine

$ cd /usr/share/redmine
$ sudo mkdir /usr/share/redmine/log
$ sudo ruby /usr/share/redmine/scripts/server -e production

(now in PC web-browser, you can use http://localhost:3000 to do the test)
7) configure apache to run redmine

$ sudo apt-get install libapache2-mod-passenger (seems it has been already installed)
$ sudo ln -s /usr/share/redmine/public /var/www/redmine
$ sudo a2enmod passanger (seems already been enabled too)
$ sudo vi /etc/apache2/site-available/default
+ RailsEnv production
+ RailsBaseURI /redmine
$ sudo chown www-data /usr/share/redmine/config/environment.rb
$ sudo /etc/init.d/apache2 reload

That's all. In your PC, you can use: http://localhost/redmine to access Redmine now.

Ubuntu - Install privoxy to visit blocked web

2008-04-22T19:46:00.001+08:00

Privoxy is worked with Tor. And it can forward http requests to socks requests used by Tor.

1) To install privoxy:

sudo apt-get install privoxy

2) To config privoxy

sudo vi /etc/privoxy/config

listen-address :8118
forward-socks4a / localhost:9050 .
3) To restart privoxy

sudo /etc/init.d/privoxy restart

4) To configure Firefox
Edit->Preferences->Advanced->Network->Settings->Manual proxy Configuration:
http prosy: localhost Port: 8118

Ubuntu - Install Tor to use the blocked MSN

2008-01-21T20:38:00.000+08:00

The following step shows how to setup Tor to use the blocked MSN:
1) Install Tor

sudo apt-get install tor
sudo /etc/init.d/tor start

2) Config MSN for localhost
If your MSN and Tor are running at the same computer, the configuration is, Pidgin for example: Accounts --> Add/Edit --> (account) --> Modify Account --> Advanced --> Proxy Type --> Socks 5 --> Host: localhost, Port: 9050
3) Config for LAN
Open the config file:

sudo vi /etc/tor/torrc

Find the following line:

SocksListenAddress 127.0.0.1 # accept connections only from localhost

Add a new line:

SocksListenAddress XXX.XXX.XXX.XXX:PORT # accept connections from local network

XXX.XXX.XXX.XXX is the IP address of the pc running Tor, Port is any number not used.
For example:

172.21.18.247:9100

And then restart Tor:

sudo /etc/init.d/tor restart

Pidgin settings: ... --> Host: 172.21.18.247, Port: 9100
That's all.

Ubuntu - Play RM format movies

2007-12-19T13:26:00.000+08:00

When I tried to play the rmvb format movies the first, the totem player said that the "Realvideo 4.0 decode can not be found", so there was only audio but no video. After I searched the internet about that, I found this site: medibuntu.org which has what I need.
The following shows the step to make my Ubuntu 7.10 pc play the rm movie:
1) Install the mplayer (Add/Remove --> Sound & Video --> check the MPlayer Movie Player)
2) Install w32codecs & libdvdcss2

a) sudo wget http://www.medibuntu.org/sources.list.d/gutsy.list -O /etc/apt/sources.list.d/medibuntu.list
b) wget -q http://packages.medibuntu.org/medibuntu-key.gpg -O- | sudo apt-key add - && sudo apt-get update
c) sudo apt-get install w32codecs libdvdcss2

3) Now you can remove the default totem movie player if you like:
Open the Synaptic Package Manager, uncheck the totem-gstreamer (totem, totem-mozilla) and press Apply button
That's all!

Indent - Beautify your C source code

2007-11-08T04:00:00.000+08:00

Introduction

GNU Indent is a GNU tool used to make code easier to read. It can also convert from one style of writing C to another.

Default Parameters file

indent will use the ~/.indent.pro file as the default parameter file. My content of the .indent.pro is:

-bad -bap -bbb -bbo -nbc -br -bli0 -bls -c33 -cd33 -ncdb -ncdw -ce -cli0 -cp33 -cs -d0 -nbfda -di2 -nfc1 -nfca -hnl -ip5 -l75 -lp -npcs -nprs -npsl -saf -sai -saw -nsc -nsob -nss -i4 -ts4 -ut

Reference:
http://linux.die.net/man/1/indent

ARM GCC inline assembly

2007-11-05T02:33:00.000+08:00

Introduction

Just now I want to print the ARM CP15 registers' value in a C function, I know I need the inline assembly technology. After read some codes in linux kernel and searched the internet, I have got some simple idea.

Examples
Read ARM ID code:

u32 val;
__asm__ __volatile__ ("mrc p15, 0, %0, c0, c0, 0" : "=r" (val));
printk("ARM ID code: 0x%x\n", val);

Write TTB:

u32 val
/*... */
__asm__ __volatile__ ("mcr p15, 0, %0, c2, c0, 0" : : "r" (val));

For more information, see:
http://www.ibiblio.org/gferg/ldp/GCC-Inline-Assembly-HOWTO.html

ARM coprocessor CP15 Summary

2007-11-05T00:39:00.000+08:00

Introduction
* CP15 is a coprocessor of ARM chip, it used to configure and control the ARM core modules - The caches, Tightly-Coupled Memories (TCMs), Memory Management Unit (MMU), and most other system options.

Instructions
* You can only access CP15 registers with MRC and MCR instructions in a privileged mode.

mcr{cond} p15, <Opcode_1>, <Rd>, <CRn>, <CRm>, <Opcode_2>
mrc{cond} p15, <Opcode_1>, <Rd>, <CRn>, <CRm>, <Opcode_2>

- The Opcode_1 and Opcode_2 field Should Be Zero except when the values specified are used to select the desired operations.
- The CRm field should be c0 except when the the register specified are used to select the desired behavior.

Registers
* c0~c15
- c0: used to read ID code/Cache type/TCM status
> read ID code:

mrc p15, 0, <Rd>, c0, c0, 0

> read Cache type:

mrc p15, 0, <Rd>, c0, c0, 1

> read TCM status:

mrc p15, 0, <Rd>, c0, c0, 2

- c1: used to enable/disable caches and MMU
> read control register:

mrc p15, 0, <Rd>, c1, c0, 0

> write control register:

mcr p15, 0, <Rd>, c1, c0, 0

- c2: r/w Translation Table Base (TTB)

mrc p15, 0, <Rd>, c2, c0, 0; read TTBR
mcr p15, 0, <Rd>, c2, c0, 0; write TTBR

- c3: Domain access control

mrc p15, 0, <Rd>, c3, c0, 0; read domain access permissions
mcr p15, 0, <Rd>, c3, c0, 0; write domain access permissions

- c4: Reserved in ARM926EJ-S
- c5: Data fault status/Instruction fault status
- c6: Fault address
- c7: Cache operations

mrc p15, 0, <Rd>, c7, c7, 0; Invalidate ICache and DCache
mcr p15, 0, <Rd>, c7, c5, 0; Invalidate ICache
mcr p15, 0, <Rd>, c7, c5, 1; Invalidate ICache single entry (MVA)
mcr p15, 0, <Rd>, c7, c5, 2; Invalidate ICache single entry (Set/Way)
mcr p15, 0, <Rd>, c7, c13, 1; Prefetch ICache line (MVA)
mcr p15, 0, <Rd>, c7, c6, 0; Invalidate DCache
mcr p15, 0, <Rd>, c7, c6, 1; Invalidate DCache single entry (MVA)
mcr p15, 0, <Rd>, c7, c6, 2; Invalidate DCache single entry (Set/Way)
mcr p15, 0, <Rd>, c7, c10, 1; Clean DCache single entry (MVA)
mcr p15, 0, <Rd>, c7, c10, 2; Clean DCache single entry (Set/Way)
mrc p15, 0, <Rd>, c7, c10, 3; Test and clean DCache
mcr p15, 0, <Rd>, c7, c14, 1; Clean and invalidate DCache entry (MVA)
mcr p15, 0, <Rd>, c7, c14, 2; Clean and invaliate DCache entry (Set/Way)
mrc p15, 0, <Rd>, c7, c14, 3; Test, clean, and invalidate DCache
mcr p15, 0, <Rd>, c7, c10, 4; Drain write buffer
mcr p15, 0, <Rd>, c7, c0, 4; Wait for interrupt

> Clean the entire DCache:

tc_loop:
mrc p15, 0, r15, c7, c10, 3; test and clean, pc is not changed by this instrunction
bne tc_loop

> Clean and invalidate the entire DCache:

tci_loop:
mrc p15, 0, r15, c7, c14, 3; test clean and invalidate
bne tci_loop

- c8: (W) TLB operations
- c9: Cache lockdown/TCM region

mrc p15, 0, <Rd>, c9, c0, 0; read DCache lockdown register
mcr p15, 0, <Rd>, c9, c0, 0; write DCache lockdown register
mrc p15, 0, <Rd>, c9, c0, 1; read ICache lockdown register
mcr p15, 0, <Rd>, c9, c0, 1; write ICache lockdown register

- c10: TLB lockdown
- c11/c12: Reserved in ARM926EJ-S
- c13: FCSE PID/Context ID
- c14: Reserved in ARM926EJ-S
- c15: Test configuration

Ubuntu - Install subversion over apache2 & trac

2007-10-22T21:26:00.000+08:00

This article covers installation subversion over apache2 and trac so that it can be easily accessed from other systems on a public network. Multiple projects are supported in this installation.

1. install related packages
You can use the Synaptic Package Manager or the sudo apt-get install command to download and install packages. The following steps only show the sudo apt-get install method.
1) install subversion

sudo apt-get install subversion

2) install apache2
Install apache2 (+apache2-mpm-worker, apache2-utils, apache2.2-common, apache2-doc & libapache2-svn)

sudo apt-get install apache2 libapache2-svn

3) install trac
Install trac (+python-clearsilver, python-pysqlite2, python-subversion, libapache2-mod-python & libapache2-mod-python-doc)

sudo apt-get install trac libapache2-mod-python libapache2-mod-python-doc

2. create svn repository
There are several typical places to put a subversion repository; most common places are: /srv/svn, /usr/local/svn & /home/svn. The following comand uses the /srv/svn:

sudo mkdir -p /srv/svn

The /srv/svn is a root directory for multiple projects. To create a project use the command like this:

sudo svnadmin create /srv/svn/project1

The project1 is the project name, you can change to your project name. And then change the owner of the folder:

sudo chown -R www-data:www-data /srv/svn/project

Modify the configuration file:

sudo vi /etc/apache2/mods-available/dav_svn.conf

<Location /svn>
DAV svn
SVNParentPath /srv/svn
AuthType Basic
AuthName "Subversion Repository"
AuthUserFile /etc/apache2/dav_svn.passwd
&LimitExcept ...> </LimitExcept>
</Location>

After the changes have been made, restart the apache2:

sudo /etc/init.d/apache2 restart

Create user account and set password (please change bhduan to your own user account):

sudo htpasswd -cm /etc/apache2/dav_svn.passwd bhduan

3. Create trac Environments Directory

sudo mkdir /srv/trac

This is the root directory for multiple projects. If you put your environment somewhere else, make sure to note that and use that location in the appropriate places in the next steps. To enable the trac site by the apache2 server:

cd /etc/apache2/sites-available
sudo cp default trac
sudo vi trac
<Location>
SetHandler mod_python
PythonHandler trac.web.modpython_frontend
PythonOption TracEnvParentDir /srv/trac
PythonOption TracUrlRoot /trac
</Location>
sudo a2dissite default
sudo a2ensite trac
sudo /etc/init.d/apache2 reload

To create and initialize the trac environment:

sudo trac-admin /srv/trac/project1 initenv

Please note that you must fill the correct svn path (/srv/svn), change the owner:

sudo chown -R www-data:www-data /srv/trac

Set authenfication:

sudo vi /etc/apache2/sites-available/trac

Add the following lines before </Location>

<Location /trac>
# ...
AuthType Basic
AuthName "Trac Projects"
AuthUserFile /etc/apache2/dav_svn.passwd
Require valid-user
</Location>

4. Create new project example
The following example shows the steps to create a project named as linux:
1) Open a Terminal and change the current directory to /srv/svn

cd /srv/svn

2) Create a new project named as linux

sudo svnadmin create linux

3) Change the owner of the folder and its sub directories and files:

sudo chown -R www-data:www-data linux

4) Change current directory to trac

cd /srv/trac

5) Create and initialize the linux trac environment, set Project Name [My Project] > as Linux and set Path to repository [/path/to/repos]> to /srv/svn/linux, and use default value for other options:

sudo trac-admin /srv/trac/linux initenv

6) Also change its owner

sudo chown -R www-data:www-data linux

7) Change current directory to your project's working folder, the following just shows the empty project in a tmp foler:

mkdir ~/tmp
cd ~/tmp
mkdir branches tags trunk

8) Import the project to the svn

svn import -m "original version for testing" ./ http://localhost/svn/linux

Here I use the localhost as the svn server address, you can change it to your real svn server ip address.

Ubuntu - Setup Samba

2007-10-22T16:43:00.000+08:00

If you want to share files between your Ubuntu and Windows computers, your best option is to use Samba file sharing. The following steps describe howto setup Samba:
1. Install Samba
If Samba is not installed in your system, you must first install it. You can use the Synaptic Package Manager or the following command:

sudo apt-get install samba

2. Configure smb
We've got samba installed, but now we'll need to configure it to make it accessible. Run following command to open the configuration file, substituting your editor of choice:

sudo vim /etc/samba/smb.conf

Uncomment some lines to make the following valid:

[homes]
security = user
comment = Home Directories
browseable = no
valid users = %S
writable = yes
create mask = 0600
directory mask = 0700

3. Create a Samba User

sudo smbpasswd -a [username]

4. Restart Samba

sudo /etc/init.d/samba restart

Now your home directory is shared, I tried this in the Ubuntu 7.04 & 7.10.

WinCE - Share a data segment in a DLL

2006-11-23T20:36:00.000+08:00

Introduction

A DLL can be loaded by more than one process. For every process the DLL's code segment is shared but each process gets its own data segment by default. Therefore if one process changes the value of the DLL's global variable, the other process can not get the modified value. For sharing some global variables of the DLL among several processes, we can use the shared data segment.

Example Code

// Global and static member variables that are not shared defined here.
...
// Begin the shared data segment
#pragma data_seg("SHARED")
// Define simple variables, the variable must be initialized here, otherwise it will not be shared.
int gSharedTest = 0;
// Do not define classes that require 'deep' copy constructors.
#pragma data_seg()
// End the shared data segment and default back to the normal data segment behavior.
// Tells the linker to generate the shared data segment.
#pragma comment(linker, "/section:SHARED,RWS")

Remarks

1) The upper "SHARED" is the name of the segment, you can use another name. It should be appeared both in the data_seg and comment part.
2) The address of the shared variable must lie in this data segment, so variables allocated from heap can not be shared.
3) The variable must be initialized otherwise it won't be shared.

WinCE - Using Protected Server Libraries (PSL)

2006-11-22T15:45:00.000+08:00

Introduction

As we know, Windows CE threads can migrate between processes while executing API calls, it borrows the caller's thread and places it in the server process. Such servers are called "Protected Server Libraries" (PSLs) and are processes that export services like DLLs.

Protected Server Libraries (PSL) are used by Windows CE to create API sets that in their own process address space and can server multiple clients concurrently. Windows CE implements all of its system APIs using this technique including all WIN32 APIs, User interface, device driver, and many of its DirectX interfaces. There is a one to one mapping between an API call and its corresponding PSL entry.

Windows CE implements PSL support through a clever technique generating special processor fault conditions that cause the execution of PSL entry points. The system maintains a table of 32 PSL entries. All entries between 0 and 24 are preassigned and may not be used for custom PSLs. PSLs register their APIs, associating their API with one particular entry in the PSL table. Part of the PSL registration process is providing the system with a function table of all the API entry points for a particular PSL. There are also a set of macros defined in psyscall.h which take the PSL id and an index into the registered function table to indicate which particular function for the specified PSL is to be called. This macro generates a 'function pointer' that really is an invalid memory address. When this function pointer gets used it causes the processor to fault. The fault handling code then inspects the faulty address and sees that it is one of the special PSL addresses and extracts the PSL id and function table index. It then turns around and calls the correct entry point in the PSL, just as if the caller had directly jumped to it – with one significant difference in that process address spaces have been jumped across as well. When the PSL function finishes execution and returns, the OS resumes the application that called the PSL to begin with by returning from the fault condition with the return value from the PSL function. The application resumes execution and it looks just as if a function call had occurred. Because PSLs are the core way APIs are implemented in the system the code path for executing this process is very streamlined.

Components of PSL

PSLs have four main components:

Functions implementing the various entry points of the PSL.
Function table (a Vtable) that is an array of function pointers containing all the entry points of the PSL. This is an array of pointers to the functions making up the API set (PSL entries). The first two entries are reserved for the kernel. The first entry is the notification procedure that the kernel will call any time a thread or a process is stopped or started, low memory occurs, or the system is booted. The second entry is reserved by the kernel and should be zero. All remaining entries are available to the PSL implementer.
Signature table describing all the argument types for each function in the function table. Made up of an array of 2 or more entry points with a one to one correspondence to the API function table. You can use these macros: FNSIG0-12, DW - DWORD, PTR - Pointer, I64 - 64 bit integer. PSL functions can have one of three argument types (DW, PTR, I64) or no arguments at. Those types are a DWORD (unsigned long), a pointer (void *), or a 64 bit integer (long long). There return values are the same. They can be used in any combination within a function or across function. There can be no other parameter types but the current parameter types can be used to contain other types (like a WORD or WCHAR in a DWORD, a BSTR in a void *). PSL functions can have from zero to 12 arguments. As can be seen above, each PSL function signiture is defined by using an FNSIG macro that is followed by the number of arguments the function has. The FNSIG macro is then filled in with an argument type specification for each argument it has.
The data in the signature table is used by the kernel to marshal arguments for the PSL function calls across process address spaces so it is imperative that these be correct or arguments will not get passed correctly from a client calling your PSL into the actual function in your PSL that implements the API call.
A .lib that clients of the PSL link to which contains the code that generates the special invalid memory address that causes the processor to fault and execute the PSL. The code also can be inline functions defined in the header file.

Example

The following is the code fragment of using the PSL technique to add APIs to our customizing Power Manager. The example shows the steps to add the SetSystemPowerRequirement API.

// 1. Functions
// 1.1 Notify function
VOID PmAPINotifyProc(DWORD dwEvent, DWORD dwProcessId, DWORD dwThreadId) { switch (dwEvent) { case DLL_PROCESS_ATTACH: PMLOGMSG(ZONE_INIT,(TEXT("DLL_PROCESS_ATTACH ProcessId %X, ThreadId %X\r\n"), dwProcessId, dwThreadId)); break; case DLL_THREAD_ATTACH: PMLOGMSG(ZONE_INIT,(TEXT("DLL_THREAD_ATTACH ProcessId %X, ThreadId %X\r\n"), dwProcessId, dwThreadId)); break; case DLL_THREAD_DETACH: PMLOGMSG(ZONE_INIT,(TEXT("DLL_THREAD_DETACH ProcessId %X, ThreadId %X\r\n"), dwProcessId, dwThreadId)); break; case DLL_PROCESS_DETACH: PMLOGMSG(ZONE_INIT,(TEXT("DLL_PROCESS_DETACH ProcessId %X, ThreadId %X\r\n"), dwProcessId, dwThreadId)); break; case DLL_PROCESS_EXITING: PMLOGMSG(ZONE_INIT,(TEXT("DLL_PROCESS_EXITING ProcessId %X, ThreadId %X\r\n"), dwProcessId, dwThreadId)); break; case DLL_SYSTEM_STARTED: PMLOGMSG(ZONE_INIT,(TEXT("DLL_SYSTEM_STARTED ProcessId %X, ThreadId %X\r\n"), dwProcessId, dwThreadId)); break; default: break; } }
// 1.2 Real API
EXTERN_C HANDLE WINAPI PmSetSystemPowerRequirement(LPCTSTR pszName) { HANDLE hRequirement = NULL; //... return hRequirement; }
// 2. API function table
static VOID PmAPINotifyProc(DWORD dwEvent, DWORD dwProcessId, DWORD dwThreadId); const PFNVOID VTable[] = { (PFNVOID)PmAPINotifyProc, (PFNVOID)NULL, (PFNVOID)PmSetSystemPowerRequirement, };
// 3. Signatures table
#define NUM_APIS (sizeof(VTable) / sizeof(VTable[0])) const DWORD SigTable[NUM_APIS] = { FNSIG3(DW, DW, DW), // PmAPINotifyProc FNSIG0(), // Kernel reserved entry FNSIG1(PTR), // Signature of PmSetSystemPowerRequirement };
// 4. Client APIs (defined in header file)
#define PM_APISET_ID 25 // APISet ID #define PM_APISET_NAME "CPM" // APISet Name #define PmSetSystemPowerRequirement_xxx \ PRIV_IMPLICIT_DECL(HANDLE, PM_APISET_ID, 3, (LPCTSTR psz)) __inline HANDLE SetSystemPowerRequirement(LPCTSTR pszState) { ASSERT(QueryAPISetID(PM_APISET_NAME) == PM_APISET_ID); return IsAPIReady(PM_APISET_ID) ? PmSetSystemPowerRequirement_xxx(pszState) : NULL; }

// 5. Create and register APISet
// PSL Initialization ghPmAPISet = CreateAPISet(PM_APISET_NAME, NUM_APIS, VTable, SigTable); if (ghPmAPISet == NULL) { PMLOGMSG(ZONE_ERROR, (_T("%s: CreateAPISet() failed: %d\r\n"), pszFname, GetLastError())); return FALSE; } if (!RegisterAPISet(ghPmAPISet, PM_APISET_ID)) { PMLOGMSG(ZONE_ERROR, (_T("%s: RegisterAPISet() failed: %d\r\n"), pszFname, GetLastError())); CloseHandle(ghPmAPISet); return FALSE; }

FlashDisk Bad Block Management

2006-09-09T13:43:00.000+08:00

Introduction

Bad blocks are blocks that contain one or more invalid bits whose reliability is not guaranteed. Bad blocks may be present when the device is shipped, or may develop during the lifetime of the device.
Devices with bad blocks have the same quality level and the same AC and DC characteristics as devices where all the blocks are valid. A bad block does not affect the performance of valid blocks because it is isolated from the bit line and common source line by a select transistor.

As the failure of a page program operation does not affect the data in other pages in the same block, the block can be replaced by re-programming the current data and copying the rest of the replaced block to an available valid block.

FlashDisk BBM

FlashDisk implements the Bad Block Management in the Flash Translation Layer (FTL), not in the FMD layer. In FlashDisk, some good blocks are reserved for garbage collection and for replacing the runtime bad blocks.

FlashDisk Garbage Collection

2006-09-08T23:55:00.000+08:00

Introduction

In FlashDisk when the file system write a sector through FlashDisk interface, the data is written to a new free sector position and the old sector with the same logical number is marked as invalid state. After a certain number of sector writes, the free sectors will become low, so a garbage collection is used to free the invalid flash memory space to allow further write operations.

Garbage Collection

Garbage collection is performed when the number of free sectors in the whole Flash is lower than a specified threshold value. It copies the valid sectors into new (free) sectors and erases the original invalid block.
The basic operations involved in Garbage Collection are the following:
1) The block with invalid sectors are selected for garbage collection. The selection follows an algorithm considered the performance and wear leveling.
2) The valid sectors are copied into a free sector in another block.
3) The block without valid sectors is erased and put to free block list.

Background Thread

There is a background thread in the FlashDisk which can be activated to optimize the overall performance. With this thread, the long erase times of Flash devices are completely transparent as the Garbage Collection is activated during the system's idle time. In this way, the FlashDisk erases sectors to free flash memory space automatically, and not only when the number of data sectors to be written exceeds the number of free sectors.

FlashDisk Wear Leveling Algorithm

2006-09-08T23:06:00.000+08:00

Introduction

As we know that in SLC (Single Level Cell) NAND Flash memories each physical block can be programmed or erased reliably over 100,000 to 1,000,000 times on current technology. To increase the lifetime of the NAND Flash, a Wear Leveling Algorithm is implemented in FlashDisk to monitor and spread the number of write cycles per block.

First-Level Wear Leveling

In the first level Wear Leveling new data is programmed to the free blocks that have had the fewest write cycles. In FlashDisk, all the free physical blocks are gathered to a free blocks chain. Each time a virtual block needs to be allocated, a physical block is selected (by the Wear Leveling algorithm) from the chain.

Second-Level Wear Leveling

In the 2nd level wear leveling, long-lived data is copied to another block so that the original block can be used for more frequently-changed data.
The 2nd level wear leveling is triggered when the difference between the maximum and the minmum number of write cycles per block reaches a specific threshold. With this particular technique, the mean age of physical NAND blocks is maintained constant.

Installable ISR

2006-09-06T21:59:00.000+08:00

Origin of writing this article is that when I am working to support the Microsoft Windows Automotive for ARM platform project, there is a requirement to support the installable isr for LCD interrupt. After I finished it, I summarize the process and write this article.

Introduction

An installable interrupt service routine (ISR) is one that can be installed and allowed to hook an interrupt after the kernel is built. Usually, if an interrupt had to be hooked to service some event, the code had to be built into the kernel when it was originally built. If a new device was inserted into the board, the code to handle the IRQ would need to already be in the kernel; otherwise, the IRQ could not be hooked.

Interrupt Handling
The main technique for handling an interrupt is to associate an event to a specified ISR. Windows CE schedules your IST when the event is triggered.

1) Typical normal interrupt handling sequence:
a) Hardware raises an interrupt.
b) Kernel looks up IRQ of the interrupt, calls the registered ISR, and disables all lower-priority interrupts.
c) ISR performs necessary handling, and then returns an interrupt identifier (SYSINTR_xxx).
d) If the interrupt identifier value returned by the ISR is SYSINTR_NOP, kernel completes processing without setting an event. all interrupts enabled. Otherwise, kernel sets the event. When the ISR returns, the kernel enables all other interrupts except the one that is being processed.
e) The kernel schedules the IST indicated by the SYSINTR to run.
f) IST call InterruptDone, InterruptDone calls OEMInterruptDone to perform any hardware actions necessary to enable the next interrupt of the target device.

2) Function call sequence in the interrupt handling
InterruptInitialize --> OEMInterruptEnable --> OALIntrEnableIrqs
InterruptDisable --> OEMInterruptDisable --> OALIntrDisableIrqs
InterruptDone --> OEMInterruptDone --> OALIntrDoneIrqs
Hardware IRQ --> OEMInterruptHandler --> static ISRs or installable ISRs --> IST

For an installable ISR, driver's IST is the same with the normal ISR, both of them will call InterruptInitialize, WaitForSingleObject, InterruptDone, InterruptDisable functions. Installable ISR will be called by the OEMInterruptHandler, If OEMInterruptHandler doesn't NKCallIntChain, the installable ISR will not be called. Installable ISR does things just like the static ISR does in the OEMInterruptHandler called functions, so the enable/disable interrupt still need be implemented in the OAL. We can only use the installable ISR for those IRQs that had been handled in the OALIntrEnableIrqs, OALIntrDisableIrqs & OALIntrDoneIrqs function.

Related Functions in Driver

1) LoadIntChainHandler
> This function is called by a driver to install an ISR to handle a particular interrupt.
> Parameters: DLL name, ISR function name, IRQ value.
> Return value: the handle to the installed handler, NULL if failed.
> Default ISR for all interrupt is the OEMInterruptHandler
> Sample code:
HANDLE hIISR = LoadIntChainHandler(_T("lcd_isr.dll"), _T("LcdISR"), IRQ_LCD);

2) FreeIntChainHandler
> This function unloads an existing interrupt handler.
> Parameters: handle of the installed ISR.
> Return value: TRUE/FALSE.
> Code of the ISR will not be freed from memory until reset.
> Sample code: FreeIntChainHandler(hIISR);

3) KernelIoControl
> This function provides the kernel with a generic I/O control for carrying out I/O operations.
> Will use this function to call IOCTL_HAL_REQUEST_SYSINTR to map IRQ-to-SYSINTR.
> Will call IOCTL_HAL_RELEASE_SYSINTR also.
> Parameters: IOCTL code, Out & In buffer.
> Return value: TRUE/FALSE.
> Sample code:
DWORD dwSYSINTR, dwIRQ = IRQ_LCD;
KernelIoControl(IOCTL_HAL_REQUEST_SYSINTR, &dwIRQ, sizeof(dwIRQ), &dwSYSINTR, sizeof(dwSYSINTR), NULL);

4) KernelLibIoControl
> This function is called by a driver to communicate with an interrupt handler.
> Parameters: ISR handle, OEM or ISV specified IOCTL, ...
> Return value: TRUE/FALSE (ERROR: Use GetLastError() to retrieve)
> This function will call IOControl function implemented in the interrupt handler.
> Sample code:
GIISR_INFO iisr_info;
memset(&iisr_info, 0, sizeof(GIISR_INFO));
iisr_info.SysIntr = dwSYSINTR;
if (!KernelLibIoControl(hIISR, IOCTL_GIISR_INFO, &iisr_info, sizeof(iisr_info), NULL, 0, NULL))
{
RETAILMSG(1, (TEXT("KernelLibIoControl failed!\r\n")));
//...
}

5) BusTransBusAddrToStatic
> This function translate a bus address to a system address. Then, it creates a static, process independent, virtual address mapping for that address.
> Parameters:
- hBusAccess: handle obtained from CreateBusAccessHandle
- InterfaceType: Bus type
- BusNumber: bus number where the device resides.
- BusAddress: bus-relative address of registers and ports on the device.
- Length: number of bytes to map on the device.
- AddressSpace: flag
- MappedAddress: virtual address mapped.
> Return value: TRUE/FALSE

Related Functions in OAL
1) NKCallIntChain
> This function is called by OEM from the OAL ISR routine (such as OEMInterruptHandler).
> This function determines which chained or shared interrupt device triggered an IRQ event.
> Parameter: IRQ value.
> Return value: return SYSINTR_CHAIN if no ISR has handled the IRQ event, otherwise return valid SYSINTR value to the OEM, and OEM passes back this value to kernel to trigger IST.

> Sample code: DWORD dwSysIntr = NKCallIntChain(nIrq);

2) OALIntrTranslateSysIntr
> This function returns the list of IRQs for a specified SYSINTR. It will be called by OEMInterruptEnable, OEMInterruptDisable & OEMInterruptDone function.
> Parameters: SYSINTR value, buffer for the IRQs list
> Return value: TRUE/FALSE

3) OALIntrEnableIrqs
> This function enables the list of interrupts identified by an IRQ.
> Parameters: number of IRQs in the list, Pointer to the list of IRQs.
> Return value: TRUE/FALSE

4) OALIntrDisableIrqs
> This function disables the list of interrupts identified by an IRQ.

5) OALIntrDoneIrqs
> This function scans the list of identified by an IRQ, signaling the end of interrupt processing.

6) OEMInterruptHandler
> This function is called by the kernel when an interrupt occurs. This function is only used by the ARM kernel, and provides all ISR functionality for ARM-based platforms. You do not need to call HookInterrupt in OEMInit for ARM-based platforms.
> Parameters: instruction counter when the interrupt occurred.
> Return value: a SYSINTR_* value.

Related Functions in an Installable ISR DLL
Installable ISR DLL must implement the following functions:
1) DllEntry
> DllEntry is a placeholder for the library-defined function name. You must specify the name you use when you build your DLL.

2) CreateInstance
> This function is implemented by an ISR to return a value that references an instance of the ISR. It's the DLL's first function called when the LoadIntChainHandler function is called to install the ISR handler, after the handler is loaded.
> Parameters: none

> Return value: a reference value that identifies an instance of the ISR handler, -1 identifies an error in the ISR.

3) DestroyInstance
> This function is called when an installable ISR is unloaded using the FreeIntChainHandler function.
> Parameters: instance of the ISR being unloaded.
> Return value: TRUE/FALSE.

4) ISRHandler
> This function prototype is used by an OEM/IHV to create and export an installable interrupt handler.
> Parameters: Instance of the ISR being registered.
> Return value: the SYSINTR value that corresponds to the IST that should be schedule to run. This can also include returning SYSINTR_CHAIN if the IRQ is not handled by your handler.

4) IOControl
> This function, exported by the installable ISR DLL, enables a communication path from the interrupt service thread (IST) to the ISR.

Notes about the Installable ISR DLL
1) The DLL is loaded into the kernel process space.
2) The DLL must be in the FILES section with no fixup variable or in the MODULES section with the kernel flag, K, set for a kernel-style fixup variable. A fixup variable is functionality of ROMIMAGE that allows you to initialize a global variable in the NK.exe at MAKEIMG time.
3) The DLL can contain more than one ISR handler and can be used in several call to LoadIntChainHandler. All ISR handlers have a full stack, based on the CPU.
4) The DLL cannot have dependent DLLs.
5) The DLL cannot implicitly to any other DLL, and all code must be contained in the DLL. You can set NOMIPS16CODE=1 in the sources file to prevent importing external functions.
6) Must be compiled to eliminate the C Run-Time Library from being included by default.

Samples of the Installable ISR
There is a sample of the installable ISR -GIISR in Windows CE under the public\common\oak\drivers\giisr\ folder.

FlashDisk - Flash Translation Layer (FTL)

2006-09-03T00:10:00.000+08:00

The FlashDisk is designed to fully replace Microsoft FAL for better stability and better performance. The original purpose of designing FlashDisk attributes to that the Microsoft FAL has some unconquerable shortcomings which are the critical features in some hardware dependent system.

Introduction

The FlashDisk is a Flash Translation Layer (FTL) which Microsoft named as Flash Abstraction Layer (FAL) used to link with Flash Media Driver (FMD) to make a Flash Block Driver. It is designed to fullly replace Microsoft FAL for better stability and better performace.

Microsoft FAL Known Disadvantages

1) Boot time is very long if the large size flash is full filled.
2) Disk may be crashed if power failure suddenly during data write process. And this will cause system can not boot up any longer if the crashed part lies in the hive-based registry area. We often have to face this issue reported by our QA engineer.
3) Read/Write performance is not well-pleasing.

FlashDisk Advantages

1) Quick Boot - Boot up very quickly even large size disk is full filled.
2) Read/Write performance is better than MS FAL driver.
3) Power failure is safe.
4) Better and more controllable wear-leveling policy.
5) Better bad block management arithmetic.
6) Automatic Garbage Collection.
7) Memory usage is acceptable.
8) Can replace MS FAL fully.

Performance Comparison

The following is a simple comparison between FlashDisk and the Microsoft FAL performance. The test platform is: 200MHz ARM920 based processor; WinCE4.2 OS; K9F1208+TC58NVG (every 2048 page is divided to four 512 pages) NAND Flash; BINFS Partition Size: 22MB, Data Partition Size: 162MB; Read/Write Performance Test App: Use 1MB buffer to R/W 5MB file 10 times.
1) Boot time
FlashDisk: 10s (empty), 10s (full)
MS FAL: 10s (empty), 35s(full)
2) Read/Write speed:
FlashDisk: R: 3563KB/s, W: 816KB/s (empty), R: 2966KB/s, W: 658KB/s (full)
MS FAL: R: 2138KB/s, W: 598KB/s (empty), R: 1854KB/s, W: 459KB/s (full)
3) Stability:
4) Memory:
5) CPU loading:
6) Bad block management:

Usage

1. Copy the FlashDisk.lib, FlashDisk.pdb to your nandflash driver path.
2. Modify the sources file in which linked with microsoft FAL.lib:
a) Remove: $(_COMMONOAKROOT)\lib\$(_CPUINDPATH)\FAL.LIB \
b) Add: \$(_CPUINDPATH)\FlashDisk.lib \
c) Modify line: DLLENTRY=DllEntry to DLLENTRY=DllMain

Remarks

To use FlashDisk, the following constraints must be followed (also these constraints are compatible with Microsoft FAL):
1) All blocks used by OEM must have the OEM_BLOCK_RESERVED flag (recommend to set OEM_BLOCK_READONLY and bad block flag, but they are optional). These blocks begin from the head of the NAND flash and include NBOOT, TOC, EBOOT, ...
2) If you use monolithic IMAGE, the blocks used by IMAGE also should have at least set the OEM_BLOCK_RESERVED flag.
3) If your system have BINFS partition, every block of the BINFS partition should have the OEM_BLOCK_READONLY flag.
4) If you want to try FlashDisk, please comment here or send mail to me.

Scatter Loading Mechanism in ADS

2006-09-02T09:56:00.000+08:00

Origin of this article: we have falled arcoss one requirement that some functions of the booter loader project need to build to another bin file and execute in the known fixed address. This is something like the DLL in the windows system but it is compiled by ADS. After I study the ADS linker guider I know that just using more than one load region can resolve this problem.

Introduction

Scatter loading is a mechanism provided by the ARM linker, which enables you to partition an executable image into regions that can be positioned independently in memory.

In a simple embedded computer system, memory is divided into ROM and RAM. The image produced by the linker is divided into the "Read-Only" segment, which contains the code and read-only data, and the "Read-Write" segment, which contains the initialized and non-initialized or zero-initialized (ZI) data. Usually, the "Read-Only" segment is placed in ROM and the "Read-Write" segment is copied from ROM to RAM before execution begins.

Embedded systems often use a more complex memory map, which can consist of ROM, SRAM, DRAM, FLASH and so on. The scatter loading mechanism lets you place various parts of the image in these distinct memory areas.

Scatter loading enables you to partition your program image into several regions of code and data which can be placed separately in the memory map. Each region is placed in a contiguous chunk of memory space. The location of a region can differ between load time and execute time, which the application copying code and data from its load address to its execution address.

The placement information is contained in a description file, the name of which is passed as a command line parameter to the linker.

Load Regions and Execution Regions
A program image consists of regions which may occupy different locations at load time and execution time.

This means that just before an image is executed, there are some regions which need to be moved from the locations at which they were initially loaded in memory. For example, initialized read-write data may reside in ROM, but it must be copied into RAM when the program starts executing.
There are two mechanisms available to describle where image regions should be placed in memory at execution time:
1) Using -RO and -RW command line options.
2) Scatter loading

Scatter Loading Definitions
1) Load region
The memory which is occupied by a program before it starts executing, but after it has been loaded into memory, can be split into a set of disjoint load regions, each of which is contiguous chunk of bytes.
2) Execution region
The memory used by a program while it is executing can also be split into a set of disjoint execution regions.

Placing Regions with Scatter Loading
1) Use the command line options "-scatter descritption-file", this option will cause the linker to ignore the "-RO -RW" options.
2) Scatter load images can be output in the following formats:
BIN
Generates one file for each load region, in the directory given as the output filename. These can then be blown into ROM, Flash and so on as appropriate. The output name is treated as a directory name. Each load region is placed in a separate file in that directory, with the same name as the load region. Load region names must therefore not contain characters, or be of a length, unacceptable to the host file system.

ELF
Generates a single executable ELF file suitable for loading into the debugger. A single output file, containing one section per load region, is produced. The name of the file is given by the -output option.

Linker pre-defined symbols
Using the region names given in the scatter loading description file, the linker generates the symbols required to allow each region to be copied from its load address to its execution address.
Neither the linker nor the C library provide the code required to copy an execution region from its load address or create a zero-initialized region; you must do this, as the application code writer.

The linker generates symbols which allow your routine to initialize all the execution regions that have different load and execution address. These symbols give the length, load address and execution address of each region.

For RO and RW segments:
Load$$region_name$$Base is the load address of the region
Image$$region_name$$Base is the execution address of the region
Image$$region_name$$Length is the execution region length in bytes
For zero-initiailized segments:
Image$$region_name$$ZI$$Base is the execution address of the region
Image$$region_name$$ZI$$Length is the execution region length in bytes.
These symbols can be imported and used by assembly language programs, or referred to as extern address from C (using the -fc compiler option which allow $ in identifiers).

Note
These symbols are generated for every region named in the scatter load description.
A scatter load image is not padded with zero, and requires the ZI data area to be created dynamically. This is similar to the case with a normal -bin file when the -nozeropad option is used. There is therefore no need for a load address symbol for ZI data.
The linker sorts AREAs within execution regions according to their attributes. For example, all initialized data area are grouped together. Therefore, you can assume that all initialized data that needs coping is contiguous.

Area ordering
The linker orders AREAs within each execution region by attributes. The ordering is:
Read-only code
Read-only based data
Read-only data
Read-write code
Based data
Other initialized data
Zero-initialized data
The pseudo-attributes FIRST and LAST can be used in the description file to mark the first and last AREAs in an execution region if the placement order is important (for example, if the ENTRY must be first and a checksum last).

Scatter loading and long distance branching
The ARM instruction set has branch instructions that allow a branch forwards or backwards up to 32Mb. The BL (Branch with Link) instruction also preserves the return address in register 14 (Link Register, LR).

The Thumb instruction set has much shorter branch ranges. From 256 bytes in the case of conditional branches to 2048 bytes for unconditional branches. The BL instruction has a range of 4Mb.

The linker has to ensure that no branch or subroutine call violates these range restrictions. If you place your execution regions in such a way as to require inter-region branches beyond the range, the linker generates an error stating Relocated value too big for instruction sequence.

The description file
A Scatter Load Description is a text file describing how the AREAs in a linked image are assigned to separate regions of memory.
In a scatter load description
. You list the separate regions of memory in which your image will execute, and specify an execution base address of each region.
. You describe how execution regions are packed into regions of physical memory (called load regions). The linker generates a separately loadable chunk of image for each load region. In some image formats (for example BIN), each load region is written to its own output file; In others (for example ELF, AIF), each load region has its own section within a single output file. You can think of each load region corresponding to a separate persistent memory such as ROM, EPROM, FLASH, and so on.
. Using simple patterns and attributes you describe how armlink should assign the constituent AREAs of your image to execution regions.

Notes:
. If you want to execute a code region directly then you must ensure its execution address is the same as its load address. Consequently, you must assign the AREA containing your image's ENTRY point to such a region.
. Your scatter load description does not describe the objects which make up your image. You describe that in the same way for all image on its command line. The patterns you write in a scatter load description describe how to assign the AREAs which armlink has already selected to the execution regions you defined.
. You can describe execution regions which overlap, provided that you give each region the OVERLAY attribute. If you do this, the linker generates support code and data to allow overlapping regions to be swapped dynamically at execution time and adds a reference from the support code to an overlay manager. You must have included ARM's standard overlay manager, or one compatible with it, in your list of object files.
. ARM-Thumb interworking veneers are built an AREA called IWV$$Code, You can assign this AREA to an execution region just like any other area using the AREA selector:
*(IWV$$Code)
Although there is no associated module, * still matches. Because there is only one IWV$$Code AREA, this section is unambiguous.

Example of the scatter loading file: