Samuel Jacob's Weblog

Just another technical blog

USB image creation using dd command

without comments

Wrote this post while waiting for a “dd” command to finish USB image creation. So this post is mainly for dd tips.

How to find where my usb drive is attached?

Attach USB and then issue the following command and then look for ‘/dev/sd??’
dmesg | less

How to create a USB drive clone?

sudo dd if=/dev/xxx of=clone.img
The output image will be in clone.img.
If you want to restore to the original disk or another USB drive then use
sudo dd of=/dev/xxx if=clone.img

How long will it take for a X GB drive?

Depends on your usb drive transfer rate. But also depends on the buffer size used by dd, I use
sudo dd of=/dev/xxx if=clone.img bs=10M

iostat will give you the transfer rate of your devices

Dont know whether dd is dead or not?

If you send USR1 signal to dd it will print the status.
sudo kill -USR1 `pidof dd`
If you want auto update every 10 sec, try
watch -n 10 sudo kill -USR1 `pidof dd`

Written by samueldotj

February 9th, 2011 at 3:22 am

Posted in C,gcc,Tools

PS3 File Conversion(mkv to vob)

with one comment

PlayStation 3 supports lot of file formats. However PS3 doesnt play most famous HD files container mkv. Another limitation of PS3 is it doesn’t support any file system except FAT32. And FAT32 has limitation of 4GB file size. And most of the HD files will easily cross 4GB limit.
I used to watch HD movies on my PS3 since it is connected to TV and HT; and other movies(which doesnt require special video/audio effects) on my desktop computer.
So to watch movie in mkv format on PS3, it should be converted to PS3 compatible format(VOB) and the output file(s) size should be less than 4GB. For this I have been using mkv2vob, this tool transcodes DTS to AC3(if necessary) otherwise just copies video/audio streams to output file which in vob format. It also does good job by splitting the files so that a single file is always <>

Recently this tool was not able to convert a mkv file, so I have to try different tools. The first tool I tried was media coder; after few trial and errors I was able to convert mkv to vob. However it doesnt have provision for splitting the file. So I went through few forums to find GOTSent. It did the job but took lot of time(~15-20min) than mkv2vob.
After all this, I started watching that movie and realized that particular doesn’t have any special effects(audio/video) but needs subtitles; instead of transcoding it again – I just watched it in my computer. Either I should lay cable between my router and PS3 so that PS3 Media Server or I should develop a wireless router which can communicate at 1G speed. I feel the later more interesting 🙂

Written by samueldotj

December 21st, 2010 at 8:20 am

Posted in Tools

Eva

without comments

Eva is an expression evaluator developed as a cross platform utility. It is written using ANTLR(targetting C), gcc. Eva supports all C expressions including bitwise operations. It is created to aid developers to convert between hexa to binary operations easily. The main goal was very simple code base ~1000 lines.

The UI part was initially coded using wxWidgets and now contains – simple command line interface, wxWidgets and Windows specific GUI interface. Here is a video demonstrating different operations possible with Eva.

Here is the source code – https://bitbucket.org/samueldotj/eva

Written by samueldotj

July 31st, 2010 at 2:55 am

Posted in C,Compiler,Tools

Kernel(FreeBSD) Concurrency

without comments

Today, I had to revisit how FreeBSD system call path was synchronized. And here it is:
Kernel data structures are tradionally protected by the following synchronization methods.
  • Interrupt disabling
  • Spinlock
  • System priority level
  • Mutex
Disabling interrupts is the easiest method because only one code path can be executed with in a interrupt disbale/enable boundary. However exceptions are there which dont honour interrupts disabled attribute.
Eg:
  1. Exceptions
  2. Faults
  3. NMI
Disabling interrupts wont work on SMP because interrupts can be disabled/enabled only on current CPU. Disabling interrupts is problematic if the code path takes longer time to complete. Because the kernel would have lost interrupts from devices.
Spinlocks is a datastructure residing on memory and acceissible from all the processors on the system. Before entering a critical code path spinlock is taken and all other access to that code path is prohibted and the other cpus will be in a loop trying to get the lock and enter. There are certain rules while creating a spinlock to avoid dead locks and gain performance from SMP.
  1. Locking order heierarchy should be maintained to avoid dead locks.
  2. Fine grained locks instead of single big lock to gain performance on SMP.
Spinlock alone would cause a dead lock in a preemptive kernel. Avoiding this in easy way is disabling interrupt because scheduler wont run. However it again would introduce a problem of losing interrupts.
system priority level or SPL is the prefered method to use with spinlock to avoid lock issues and also get interrupts from devices. SPL rules:
  • And all spinlocks are at greater SPL than scheduler.
  • All spinlocks should have a associated SPL and at that or greater SPL level only the spinlock should be taken.
Note:
  1. In simpler term spl is equivalent to masking interrupts.
  2. SPL is per processor, so the spinlock should have locking order hierarchy, otherwise deadlock for sure.
Mutexes used because even with fine grained spinlock there is performance penalty when other CPUs are waiting(looping) for a spinlock. Mutex solves this problem by scheduling out the looping threads in the other CPUs and selecting other threads to run.
Note:
  • Mutex cant be used in certain situations eg: a interrupt handler which runs without context cant take mutex because it might be put into sleep.
  • Combining mutex and spinlock requires only one condition – after a spinlock is taken no mutex acquire should be attempted.
FreeBSD 6.x uses mutex and spinlock – MUTEX_DEF and MUTEX_SPIN.
MUTEX_DEF – Is a mutex which result in going to sleep.
MUTEX_SPIN – Is a spinlock which never puts the caller into sleep but spins and also disables interrupts.
When FreeBSD was optimized for SMP, spinlocks were removed and mutex was introduced.
It also introduced context full interrupt handlers; now interrupt handlers can sleep, so can use mutex. Since most of the system calls was not ported to SMP, they were protected using a big mutex called Giant.
Giant – Gaint is a MUTEX_DEF which protects the code path of most system calls.

Written by samueldotj

February 10th, 2010 at 4:17 am

Posted in Uncategorized

Size of all data structures in a C program

without comments

There are some cases when you want to find size of all data structures in your program/project. For a given program it is easy because we can manually calculate or put a printf sizeof() to calculate a few structures. But for a project with few hundred files it is difficult. There is noway AFAIK in gcc to dump all the structure sizes while it compiles.

Here is a way todo it using DWARF debugging info if you are using gnu compiler tools. Compile your project output file with -g option, this will generate debugging information in your program. Then run objdump to dump the debug info. This debugging information contains all the information about your program, filter the output for “DW_TAG_structure_type” and you will get only structure information.

Here is how I did it:

objdump -W kernel.debug.reloc > debug.info
grep -e "DW_TAG_structure_type" debug.info --after-context=5 --mmap > structs.txt

Written by samueldotj

November 19th, 2009 at 3:58 am

Posted in C,Debugger,gcc

Call Trace without modifying the source

without comments

While investigating about gcc flag “-fprofile-arcs”, I came to know about a new(to me) gcc flag and this blog entry is about it. For any large C project it is hard to learn/find call graph through code walk. From C prospective unless otherwise you put a printf in each function entry/exit it is hard to find the call trace.

GCC and ICC has a wonderful option “-finstrument-functions” to solve this. This option instructs the compiler to emit instructions to call a external function on each function’s entry/exit. Defining these two functions like the following and adding the above option -finstrument-function to your makefile will do the magic.

void noinstrument __cyg_profile_func_enter(void *this_fn, void *call_site)
{
  printf("%p called from %p\n", this_fn, call_site);
}

void noinstrument __cyg_profile_func_exit(void *this_fn, void *call_site)
{
  printf("%p returns\n", this_fn);
}

Of course, you can do anything in these functions, for simplicity sake I just defined them as printfs.

Written by samueldotj

October 3rd, 2009 at 1:04 pm

GCC debugging switches

without comments

While compiling a FreeBSD kernel I encountered the following error message from GNU assembler

Error: suffix or operands invalid for 'mov'

I was not sure whether the error is because of my changes are not. But still I have to debug the problem and solve it, so I decided to find what is wrong with the mov instruction. But since gcc created a temporary file and invoked as to assemble it, I couldn’t find the temp file anymore.

Googled and find the following gcc options to find what gcc does.

gcc –v is verbose mode, which prints all the action/scripts gcc is doing.
gcc -### similar to –v but wont execute any commands.
gcc –save-temps saves all the temporary files generated by the gcc. Useful if you want to see the assembly output passed to gnu assembler.
gcc –E Just preprocess the c file.
gcc –S generate assembly file.

With gcc –v –save-temps, I got the temporary assembler file used by assembler. Since reading machine generated assembly file is difficult, I tried to find the C source code corresponding to the error. From nearest .file assembler directive I found the C file name and from the .loc directive I found the line number. And the following was the inline assembly in the C file.
__asm __volatile("<b>movl </b>%%ss,%0" : "=rm" (sel));

Was surprised what is wrong with this and why my assembler giving error. Googled again and found this link – http://kerneltrap.org/node/5785. It seems newer gnu assembler(>=2.16) wont support long read/write operations(mov) on segment registers(ds,ss…). So changed the above source to movw %%ss, %o and it compiled perfectly. 🙂

Written by samueldotj

August 28th, 2009 at 2:23 am

Unreferenced symbols – EXPORT_SYMBOL

without comments

I wrote ACPI driver for Ace and try to load it into kernel. It was failing with the error message symbol not found.

The log file content says AcpiGetName was not found.

$ cat log.txt
Driver for id acpi : acpi.sys
../src/kernel/pm/elf.c:343:FindElfSymbolByName(): Symbol not found AcpiGetName(c01a2148:1869) string table c01aa619
[shell]

After exploring the Acpi-CA source code and Ace kernel code, I couldn’t figure out why the symbol is missing. Then I tried to dump the symbol table for Acpi library and kernel.sys.
[shell]
$ nm build/default/src/kernel/libacpi.a | grep GetName
U AcpiExGetNameString
000001e3 T AcpiExGetNameString
000001b7 T AcpiPsGetName
000000f4 T AcpiGetName

$ nm build/default/src/kernel/kernel.sys | grep GetName
c0135163 T AcpiExGetNameString
c0137bfb T AcpiPsGetName

It is obvious during kernel link process some how the linker is losing AcpiGetName. Within few seconds I realized that the linker is doing optimization on the output to remove unreferenced sections. So I googled for how to avoid this and found --no-gc-sections is the ld option to do it. However even after giving this option to linker to was removing the unreferenced symbols; I kept on trying to find a solution for making the linker to include unreferenced symbols in the output… I didn’t find the solution.
Then I started thinking how linux kernel is avoiding this problem and I it flashed seeing EXPORT_SYMBOL kind of macro in some source. So I googled for it and got the answer – use the symbol in storage, so that it will be referenced, so that linker will include it in the output.

struct kernel_symbol
{
   unsigned long symbol;
   char * name;
};

#ifndef MODULE_SYMBOL_PREFIX
#define MODULE_SYMBOL_PREFIX “”
#endif

#define __EXPORT_SYMBOL(sym, sec) \
static const char __kstrtab_##sym[] \
__attribute__((section("__ksymtab_strings"))) \
= MODULE_SYMBOL_PREFIX#sym; \
static const struct kernel_symbol __ksymtab_##sym \
__attribute__((section("__ksymtab" sec), unused)) \
= { (unsigned long)&sym, __kstrtab_##sym }

#define EXPORT_SYMBOL(sym) \
__EXPORT_SYMBOL(sym, "")

Written by samueldotj

April 9th, 2009 at 4:22 am

Posted in Ace,C,gcc,Programming,Tools

Better build system

with one comment

I was using “make”(http://en.wikipedia.org/wiki/GNU_build_system) to build and clean my projects. The makefiles grew bigger enough to not maintainable state. Recently I came to know about waf(http://code.google.com/p/waf/) from a forum, so decided to give a try. I went through the waf documentation and it took 2 hours to understand how it works and it took 2 days to convert all my makefiles to waf wscripts and wscript_build files.

The basic problem with make was I have write rules to how to build object files from source and

then to clean I have to write how to delete the object files. With waf the problem is simplified, you have to tell what all source files and it will take care of how to compile and link. It will also take care of how to clean the project and the dependency between files are also taken care, so no need for gcc –M and dependency files etc. The feature I liked in waf is it has colored output with progress bar 🙂

The other problem solved in Ace project compilation was build configuration. Waf will configure the project for the first time, so it solved the different cross compiler required for Ace OS.

Waf(http://code.google.com/p/waf/) is really good alternative/replacement for make tools.

Written by samueldotj

March 17th, 2009 at 3:54 am

Posted in Uncategorized

Porting to “Intel C Compiler”

without comments

In this post, I will explain how to easy it is to compile/port C programs and makefiles designed for GCC(GNU compiler collection) can be used with ICC(Intel C Compiler)

The necessity for using ICC happened to me because gcc was generating too big object files. Ace is using ACPI-CA(http://acpica.org/) as it is ACPI driver. After compilation the ACPI library is around 7M in debug version and around 500K in non-debug version. But from http://acpica.org/download/changes.txt it seems Microsoft C compiler produce smaller code.

  Non-Debug Version:  81.2K Code, 17.0K Data,  98.2K Total
  Debug Version:     155.8K Code, 49.1K Data, 204.9K Total

So I decided to try “Intel Compiler” as replacement for GCC. Things were easy, since icc has options which is similar to gcc.
You can get the details of portability options from icc documentation – http://www.intel.com/software/products/compilers/docs/clin/main_cls/index.htm.

I just changed my make.conf file to check the compiler in use and use appropriate options. I removed -fno-leading-underscore and –Wall. I removed –Wall because it was giving lot of warnings, we have to fix our code sometime soon. I removed -fno-leading-underscore because there is no equivalent option in icc, however icc does not adds underscores to symbols anyway.

I added –O1 option because the kernel was panicked during boot with invalid opcode exception, I believe it was because of some MMX register usage by icc. So I settled with –O1 now.

ifeq ($(CC),gcc)
    DEBUG_FLAGS= -gdwarf-2 -g3
    CFLAGS+= -Wall -Wno-multichar $(DEFINES) -nostartfiles -ffreestanding -funsigned-char -fno-leading-underscore -c -fno-stack-protector $(DEBUG_FLAGS) $(CUSTOM_FLAG)
else
    DEBUG_FLAGS= -gdwarf-2 -g
    CFLAGS+= -O1 -Wno-multichar $(DEFINES) -nostartfiles -ffreestanding -funsigned-char -c -fno-stack-protector $(DEBUG_FLAGS) $(CUSTOM_FLAG)
endif

I also made some static variables in kernel/i386/gdb_stub.c to non-static because icc appends .0 to static variables but it forgets the same name when referenced in a inline assembly code in the same file. After removing the static attribute it is working fine.

Compilation time:
gcc compiler took the following time to compile Ace source code.

real    0m51.125s
user    0m36.975s
sys     0m11.185s

icc compiler took the following time to compile Ace source code.

real    1m44.366s
user    0m59.107s
sys     0m20.946s

Size:
Gcc produced double the size of kernel.sys produced by icc. (I used GNU linker(ld) and ar).
gcc output size:

$ ls -lh /home/samuel/Projects/Ace3/obj/
-rw-rw-r– 1 samuel samuel 372K 2008-11-22 18:17 acpi.a
-rw-rw-r– 1 samuel samuel 545K 2008-11-22 18:17 arch.a
-rwxrwxr-x 1 samuel samuel 1.3M 2008-11-22 18:17 kernel.sys
-rw-rw-r– 1 samuel samuel  62K 2008-11-22 18:17 libds.a
-rw-rw-r– 1 samuel samuel  40K 2008-11-22 18:17 libheap.a
-rw-rw-r– 1 samuel samuel  39K 2008-11-22 18:17 libstring.a
-rw-rw-r– 1 samuel samuel 9.5K 2008-11-22 18:17 libsync.a
-rw-rw-r– 1 samuel samuel  91K 2008-11-22 18:17 pic.a
-rw-rw-r– 1 samuel samuel  23K 2008-11-22 18:17 pit.a
-rw-rw-r– 1 samuel samuel  21K 2008-11-22 18:17 rtc.a

icc output:
$ ls -lh /home/samuel/Projects/Ace3/obj/
-rw-rw-r– 1 samuel samuel 352K 2008-11-22 16:25 acpi.a
-rw-rw-r– 1 samuel samuel 230K 2008-11-22 16:24 arch.a
-rwxrwxr-x 1 samuel samuel 528K 2008-11-22 16:25 kernel.sys
-rw-rw-r– 1 samuel samuel  38K 2008-11-22 16:23 libds.a
-rw-rw-r– 1 samuel samuel  22K 2008-11-22 16:23 libheap.a
-rw-rw-r– 1 samuel samuel  21K 2008-11-22 16:23 libstring.a
-rw-rw-r– 1 samuel samuel  11K 2008-11-22 16:23 libsync.a
-rw-rw-r– 1 samuel samuel  19K 2008-11-22 16:24 pic.a
-rw-rw-r– 1 samuel samuel 8.4K 2008-11-22 16:24 pit.a
-rw-rw-r– 1 samuel samuel  13K 2008-11-22 16:24 rtc.a

Conclusion: So it is easy to switch to icc instead of gcc.

Written by samueldotj

November 22nd, 2008 at 5:36 am

Posted in Ace,C,Compiler,gcc