{"id":231,"date":"2015-09-11T21:51:24","date_gmt":"2015-09-12T04:51:24","guid":{"rendered":"http:\/\/samueldotj.com\/blog\/?p=231"},"modified":"2015-09-11T21:51:24","modified_gmt":"2015-09-12T04:51:24","slug":"swig-and-complex-c-structures","status":"publish","type":"post","link":"http:\/\/samueldotj.com\/blog\/swig-and-complex-c-structures\/","title":{"rendered":"SWIG and Complex C structures"},"content":{"rendered":"<p>I had to use SWIG to access a kernel module&#8217;s chardev interface through python and found SWIG examples are not enough, so adding my own.<\/p>\n<p>Lets take the following example header file.<br \/>\nI will explain how to access all the members in <em>complex_struct_t<\/em> from python.<br \/>\nAlso extend these structures so that python code would look little better.<br \/>\n[codegroup]<br \/>\n[c tab=&#8221;header&#8221;]<\/p>\n<p>\/* test.h *\/<\/p>\n<p>#ifndef _TEST_STRUCT_H<br \/>\n#define _TEST_STRUCT_H<\/p>\n<p>\/* a simple structure &#8211; no problem with SWIG *\/<br \/>\ntypedef struct simple_struct {<br \/>\n   int int_var;<br \/>\n   long long_var;<br \/>\n   float float_var;<br \/>\n} simple_struct_t;<\/p>\n<p>typedef struct tlv_base {<br \/>\n   int type;<br \/>\n   int length;<br \/>\n   unsigned char value[];<br \/>\n} tlv_base_t;<\/p>\n<p>typedef struct tlv_type1 {<br \/>\n   tlv_base_t base;<br \/>\n   int stat;<br \/>\n   int info;<br \/>\n   long something;<br \/>\n} tlv_type1_t;<\/p>\n<p>\/* relatively complex C structure. *\/<br \/>\ntypedef struct complex_struct {<br \/>\n   char string[10];           \/\/SWIG considers this as null terminated string<br \/>\n   unsigned char bytes[10];   \/\/SWIG wont considers this as string<\/p>\n<p>   simple_struct_t embedded;<\/p>\n<p>   int pointer_array_count;<br \/>\n   simple_struct_t *pointer_array; \/\/SWIG supports only accessing first element.<\/p>\n<p>   tlv_base_t tlv;   \/\/How do cast this to derived_struct_1 ?<br \/>\n} complex_struct_t;<\/p>\n<p>complex_struct_t * alloc_complex_struct(int array_count);<br \/>\nvoid free_complex_struct(complex_struct_t *cs);<br \/>\nvoid print_complex_struct(complex_struct_t *cs);<\/p>\n<p>#endif<br \/>\n[\/c]<\/p>\n<p>[c tab=&#8217;source&#8217;]<\/p>\n<p>\/* test.c *\/<\/p>\n<p>#include <stdlib.h><br \/>\n#include <stdio.h><\/p>\n<p>#include &#8220;test.h&#8221;<\/p>\n<p>complex_struct_t *<br \/>\nalloc_complex_struct(int array_count)<br \/>\n{<br \/>\n   complex_struct_t *result;<br \/>\n   size_t size;<\/p>\n<p>   result = (complex_struct_t *)malloc(sizeof(complex_struct_t) + sizeof(tlv_type1_t));<br \/>\n   if (result == NULL){<br \/>\n      return NULL;<br \/>\n   }<\/p>\n<p>   result->tlv.type = 1;<br \/>\n   result->tlv.length = sizeof(tlv_type1_t);<\/p>\n<p>   size = sizeof(simple_struct_t) * array_count;<br \/>\n   result->pointer_array = (simple_struct_t *)malloc(size);<br \/>\n   if (result->pointer_array == NULL) {<br \/>\n      free(result);<br \/>\n      return NULL;<br \/>\n   }<br \/>\n   memset(result->pointer_array, 0, size);<br \/>\n   result->pointer_array_count = array_count;<\/p>\n<p>   return result;<br \/>\n}<\/p>\n<p>void<br \/>\nfree_complex_struct(complex_struct_t *cs)<br \/>\n{<br \/>\n   free(cs->pointer_array);<br \/>\n   free(cs);<br \/>\n}<\/p>\n<p>static inline void<br \/>\nprint_simple_struct(simple_struct_t *ss)<br \/>\n{<br \/>\n   printf(&#8220;int %d long %ld float %f\\n&#8221;, ss->int_var, ss->long_var, ss->float_var);<br \/>\n}<\/p>\n<p>void<br \/>\nprint_complex_struct(complex_struct_t *cs)<br \/>\n{<br \/>\n   int i;<\/p>\n<p>   printf(&#8220;String = %s\\n&#8221;, cs->string);<br \/>\n   printf(&#8220;Embedded : &#8220;);<br \/>\n   print_simple_struct(&#038;cs->embedded);<br \/>\n   printf(&#8220;External : \\n&#8221;);<br \/>\n   for (i=0; i<cs->pointer_array_count; i++) {<br \/>\n      printf(&#8220;%d) &#8220;, i + 1);<br \/>\n      print_simple_struct(&#038;cs->pointer_array[i]);<br \/>\n   }<br \/>\n}<\/p>\n<p>[\/c]<\/p>\n<p>[raw tab=&#8217;interface&#8217;]<br \/>\n\/\/ test_swig.i<br \/>\n%module test_struct<br \/>\n%{<br \/>\n   #include &#8220;..\/test.h&#8221;<br \/>\n%}<\/p>\n<p>%include &#8220;test.h&#8221;<br \/>\n[\/raw]<\/p>\n<p>[shell tab=&#8217;commands&#8217;]<br \/>\n# Commands to make the shared library<br \/>\n$ mkdir -p _build<br \/>\n$ gcc -I \/usr\/include\/python2.7\/ -fPIC -c -o _build\/test.o test.c<br \/>\n$ swig -python -outdir _build -o _build\/test_swig_wrap.c test_swig.i<br \/>\n$ gcc -I \/usr\/include\/python2.7\/ -fPIC -c -o _build\/test_swig_wrap.o _build\/test_swig_wrap.c<br \/>\n$ ld -shared  _build\/test.o  _build\/test_swig_wrap.o -o _build\/_test_struct.so<br \/>\n$ rm _build\/test_swig_wrap.c<br \/>\n[\/shell]<\/p>\n<p>[raw tab=&#8217;makefile&#8217;]<br \/>\n#Makefile<br \/>\nBLD_DIRECTORY = _build<\/p>\n<p>SWIG_SRCS = test_swig.i<br \/>\nC_SRCS = test.c<br \/>\nCFLAGS = -I \/usr\/include\/python2.7\/ -fPIC -c -Wall<\/p>\n<p>OBJS = $(patsubst %.c, $(BLD_DIRECTORY)\/%.o, $(C_SRCS)) \\<br \/>\n\t\t $(patsubst %.i, $(BLD_DIRECTORY)\/%_wrap.o, $(SWIG_SRCS))<\/p>\n<p>$(BLD_DIRECTORY)\/%.o: %.c %.h<br \/>\n\tgcc $(CFLAGS)  -o $@ $&lt;<\/p>\n<p>$(BLD_DIRECTORY)\/%.o: $(BLD_DIRECTORY)\/%.c<br \/>\n\tgcc $(CFLAGS) -o $@ $&lt;<\/p>\n<p>$(BLD_DIRECTORY)\/%_wrap.c: %.i<br \/>\n\tswig -python -outdir $(BLD_DIRECTORY) -o $@ $&lt;<br \/>\n\tcp $@ $@.bak<\/p>\n<p>$(BLD_DIRECTORY):<br \/>\n\tmkdir -p $(BLD_DIRECTORY)<\/p>\n<p>clean:<br \/>\n\trm -rf $(BLD_DIRECTORY)<\/p>\n<p>all: $(BLD_DIRECTORY) $(OBJS) $(C_SRCS) $(SWIG_SRCS)<br \/>\n\tld -shared $(OBJS) -o $(BLD_DIRECTORY)\/_test_struct.so<\/p>\n<p>.PHONY: all clean<\/p>\n<p>.DEFAULT_GOAL := all<br \/>\n[\/raw]<br \/>\n[\/codegroup]<\/p>\n<p>With this simple interface file, SWIG would be able to create a _test_struct.so and test_struct.py which is perfect for most cases.<br \/>\n[raw tab=&#8217;interface&#8217;]<br \/>\n%module test_struct<br \/>\n%{<br \/>\n   #include &#8220;..\/test.h&#8221;<br \/>\n%}<\/p>\n<p>%include &#8220;test.h&#8221;<br \/>\n[\/raw]<\/p>\n<p>[python]<br \/>\nimport test_struct as ts<br \/>\ncs = ts.alloc_complex_struct(1)<br \/>\ncs.string = &#8220;Hello&#8221;<br \/>\ncs.embedded.int_var = 9<br \/>\ncs.embedded.long_var = 10<br \/>\ncs.embedded.float_var = 11.23<br \/>\nts.print_complex_struct(cs)<br \/>\nts.free_complex_struct(cs)<br \/>\n[\/python]<\/p>\n<p>This shows SWIG&#8217;s ability to convert C string to python string and vice versa.<br \/>\nSimilarly accessing primitive structure members is very easy.<br \/>\nHere is the output of the program when ran from _build directory. <\/p>\n<p>[raw]<br \/>\nString = Hello<br \/>\nEmbedded : int 9 long 10 float 11.230000<br \/>\nPointer Array :<br \/>\n1) int 0 long 0 float 0.000000<br \/>\n[\/raw]<\/p>\n<p>It also shows how to call function call into C library. If you have noticed this program looks more like a C program rather than a python program &#8211; mainly because it manages the memory allocation\/free. Python can informed that [c] alloc_complex_struct() [\/c] returns a new object and it is the responsibility of the caller to free it by using the SWIG typemap [python] newobject [\/python]. Now python garbage collector will free the object when there is no reference. But python does not know how to free the object(complex_struct_t) &#8211; this can be done by using [python]newfree[\/python] typemap.<\/p>\n<p>By adding the following to the test_swig.i, we can avoid calling free_complex_struct() in python program.<br \/>\n[c]<br \/>\n%typemap(newobject) alloc_complex_struct;<br \/>\n%typemap(newfree) complex_struct_t * {<br \/>\n  free_complex_struct($1);<br \/>\n}<br \/>\n[\/c]<\/p>\n<p>Lets modify the program a little bit and access the pointer_array elements.<\/p>\n<p>[python]<br \/>\nimport test_struct as ts<br \/>\ncs = ts.alloc_complex_struct(5)<br \/>\nprint &#8216;Pointer array count &#8216;, cs.pointer_array_count<br \/>\nprint cs.pointer_array[0]<br \/>\nts.free_complex_struct(cs)<br \/>\n[\/python]<\/p>\n<p>This will fail with the following error:<br \/>\n[raw]<br \/>\nPointer array count  5<br \/>\nTraceback (most recent call last):<br \/>\n  File &#8220;.\/test.py&#8221;, line 4, in <module><br \/>\n    print cs.pointer_array[0]<br \/>\nTypeError: &#8216;simple_struct_t&#8217; object does not support indexing<br \/>\n[\/raw]<\/p>\n<p>The reason is SWIG does not really know [c]simple_struct_t *pointer_array;[\/c] actually points to an array of [c]simple_struct_t[\/c]. In other words SWIG safely assumes it points to a single entry. If pointer_array was &#8220;array of simple_struct_t pointers&#8221; then <a href=\"http:\/\/www.swig.org\/Doc3.0\/SWIGDocumentation.html#Library_carrays\" target=\"_blank\">carrays.i<\/a> macro would have been helped. But pointer_array is actually &#8220;pointer to array of simple_struct_t&#8221; so carrays.i won&#8217;t help.<\/p>\n<p>The easiest way is extending complex_struct_t and add a new member(kind of) function to it.<br \/>\n[c]<br \/>\n%extend complex_struct_t{<br \/>\n    simple_struct_t *get_array_element(int i) {<br \/>\n        return &#038;$self->pointer_array[i];<br \/>\n    }<br \/>\n}<br \/>\n[\/c]<\/p>\n<p>This way cs.get_array_element(4) will return 4th element in the array.<br \/>\nSimilarly tlv elements can be accessed also &#8211; but this time I decided to override indexing operator([]).<\/p>\n<p>[c]<br \/>\n%extend complex_struct_t{<br \/>\n    unsigned char __getitem__(int i) {<br \/>\n        return $self->tlv[i];<br \/>\n    }<br \/>\n}<br \/>\n[\/c]<\/p>\n<p>However this is not useful since python cant cast from [c](struct tlv_base *)[\/c] to [c]struct tlv_type1 *[\/c]. To cast, a C function can be coded or SWIG&#8217;s <em>cpointer.i<\/em> can be used.<\/p>\n<p>Here is the full test_swig.i<br \/>\n[c]<br \/>\n%module test_struct<br \/>\n%{<br \/>\n   #include &#8220;..\/test.h&#8221;<br \/>\n%}                                                                                                                                                                       <\/p>\n<p>%include &#8220;test.h&#8221;                                                                                                                                                        <\/p>\n<p>%extend complex_struct{<br \/>\n    simple_struct_t *get_array_element(int i) {<br \/>\n        return &#038;$self->pointer_array[i];<br \/>\n    }<br \/>\n}                                                                                                                                                                        <\/p>\n<p>%typemap(newobject) alloc_complex_struct;<br \/>\n%typemap(newfree) complex_struct_t * {<br \/>\n      free_complex_struct($1);<br \/>\n}                                                                                                                                                                        <\/p>\n<p>%include <cpointer.i><br \/>\n%pointer_cast(tlv_base_t *,  tlv_type1_t *, cast_to_tlv_type1);<br \/>\n[\/c]<\/p>\n<p>And test code:<br \/>\n[python]<br \/>\nimport test_struct as ts<br \/>\ncs = ts.alloc_complex_struct(5)<br \/>\ncs.string = &#8216;Hello&#8217;<br \/>\nprint &#8216;Pointer array count &#8216;, cs.pointer_array_count<br \/>\nfor i in range(cs.pointer_array_count):<br \/>\n   simple_struct = cs.get_array_element(i)<br \/>\n   simple_struct.int_var = i * 10<br \/>\n   simple_struct.long_var = i * 20<br \/>\n   simple_struct.float_var = i * 3.3<br \/>\nts.print_complex_struct(cs)                                                                                                                                              <\/p>\n<p>tlv = ts.cast_to_tlv_type1(cs.tlv)<br \/>\nprint tlv.stat, tlv.info, tlv.something<br \/>\n[\/python]<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I had to use SWIG to access a kernel module&#8217;s chardev interface through python and found SWIG examples are not enough, so adding my own. Lets take the following example header file. I will explain how to access all the members in complex_struct_t from python. Also extend these structures so that python code would look [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,5],"tags":[],"class_list":["post-231","post","type-post","status-publish","format-standard","hentry","category-c","category-programming"],"_links":{"self":[{"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/posts\/231","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/comments?post=231"}],"version-history":[{"count":10,"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/posts\/231\/revisions"}],"predecessor-version":[{"id":268,"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/posts\/231\/revisions\/268"}],"wp:attachment":[{"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/media?parent=231"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/categories?post=231"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/samueldotj.com\/blog\/wp-json\/wp\/v2\/tags?post=231"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}