After doing some reading about x86_64 asm, I ‘ported’ the example kernel to 64bit, and gave it a try. And of course, it crashed. While I was responsible for the first couple of crashes (for which btw, I can write at least 2-3 additional blog posts ), I got stuck with this error:

traps.c:470:d100 Unhandled bkpt fault/trap [#3] on VCPU 0 [ec=0000]
RIP:    e033:<0000000000002271>

when trying to boot the example kernel as a domU (under xen-unstable).

0×2000 is the address where XEN maps the hypercall page inside the domU’s address space. The guest crashed when trying to issue any hypercall (HYPERCALL_console_io in this case). At first, I thought I had screwed up with the x86_64 extended inline asm, used to perform the hypercall, so I checked how the hypercall macros were implemented both in the Linux kernel (wow btw, it’s pretty scary), and in the mini-os kernel. But, I got the same crash with both of them.

After some more debugging, I made it work. In my Makefile, I used gcc to link all of the object files into the guest kernel. When I switched to ld, it worked. Apparently, when using gcc to link object files, it calls the linker with a lot of options you might not want. Invoking gcc using the -v option will reveal that gcc calls collect2 (a wrapper around the linker), which then calls ld with various options (certainly not only the ones I was passing to my ‘linker’). One of them was –build-id, which generates a .note.gnu.build-id” ELF note section in the output file, which contains some hash to identify the linked file.

Apparently, this note changes the layout of the resulting ELF file, and ‘shifts’ the .text section to 0×30 from 0×0, and hypercall_page ends up at 0×2030 instead of 0×2000. Thus, when I ‘called’ into the hypercall page, I ended up at some arbitrary location instead of the start of the specific hypercall handler I was going for. But it took me quite some time of debugging before I did an objdump -dS [kernel] (and objdump -x [kernel]), and found out what was going on.

The code from bootstrap.x86_64.S looks like this (notice the .org 0×2000 before the hypercall_page global symbol):

        .text
        .code64
	.globl	_start, shared_info, hypercall_page
_start:
	cld
	movq stack_start(%rip),%rsp
	movq %rsi,%rdi
	call start_kernel

stack_start:
	.quad stack + 8192
	
	.org 0x1000
shared_info:
	.org 0x2000

hypercall_page:
	.org 0x3000	

One solution, mentioned earlier, is to switch to ld (which probalby makes more sense), instead of using gcc. The other solution is to tweak the ELF file layout, through the linker script (actually this is pretty much what the Linux kernel does, to work around this):

OUTPUT_FORMAT("elf64-x86-64", "elf64-x86-64", "elf64-x86-64")
OUTPUT_ARCH(i386:x86-64)
ENTRY(_start)

PHDRS {
	text PT_LOAD FLAGS(5);		/* R_E */
	data PT_LOAD FLAGS(7);		/* RWE */
	note PT_NOTE FLAGS(0);		/* ___ */
}

SECTIONS
{
	. = 0x0;			/* Start of the output file */
	_text = .;			/* Text and ro data */
	.text : {
		*(.text)
	} :text = 0x9090 

	_etext = .;			/* End ot text section */

	.rodata : {			/* ro data section */
		*(.rodata)
		*(.rodata.*)
	} :text

	.note : { 
		*(.note.*)
	} :note

	_data = .;
	.data : {			/* Data */
		*(.data)
	} :data

	_edata = .;			/* End of data section */	
}

And now that my kernel boots, I can go back to copy-pasting code from the book … erm hacking.

Disclaimer: I’m not very familiar with lds scripts or x86_64 asm, so don’t trust this post too much.

Long time, no post.

For about a year now, I’ve been working at GRNET on its (OpenStack API compliant) open source IaaS cloud platform Synnefo, which powers the ~okeanos service.

Since ~okeanos is mainly aimed towards the Greek academic community (and thus has restrictions on who can use the service), we set up a ‘playground’ ‘bleeding-edge’ installation (okeanos.io) of Synnefo, where anyone can get a free trial account, experiment with the the Web UI, and have fun scripting with the kamaki API client. So, you get to try the latest features of Synnefo, while we get valuable feedback. Sounds like a fair deal.

Unfortunately, being the only one in our team that actually uses Gentoo Linux, up until recently Gentoo VMs were not available. So, a couple of days ago I decided it was about time to get a serious distro running on ~okeanos (the load of our servers had been ridiculously low after all ). For future reference, and in case anyone wants to upload their own image on okeanos.io or ~okeanos, I’ll briefly describe the steps I followed.

1) Launch a Debian-base (who needs a GUI?) VM on okeanos.io

Everything from here on is done inside our Debian-base VM.

2) Use fallocate or dd seek= to create an (empty) file large enough to hold our image (5GB)

fallocate -o $((5 * 1024 * 1024 *1024)) -l 1 gentoo.img


3) Losetup the image, partition and mount it

losetup -f gentoo.img
parted /dev/loop0 mklabel msdos
parted /dev/loop0 mkpart primary ext4 2048s 5G
kpartx -a /dev/loop0
mkfs.ext4 /dev/mapper/loop0p1
losetup /dev/loop1 /dev/mapper/loop0p1 (trick needed for grub2 installation later on)
mount /dev/loop1 /mnt/gentoo -t ext4 -o noatime


4) Chroot and install Gentoo in /mnt/gentoo. Just follow the handbook. At a minimum you’ll need to extract the base system and portage, and set up some basic configs, like networking. It’s up to you how much you want to customize the image. For the Linux Kernel, I just copied directly the Debian /boot/[vmlinuz|initrd|System.map] and /lib/modules/ of the VM (and it worked! ).

5) Install sys-boot/grub-2.00 (I had some *minor* issues with grub-0.97 ).

6) Install grub2 in /dev/loop0 (this should help). Make sure your device.map inside the Gentoo chroot looks like this:

(hd0) /dev/loop0
(hd1) /dev/loop1

and make sure you have a sane grub.cfg (I’d suggest replacing all references to UUIDs in grub.cfg and /etc/fstab to /dev/vda[1]).
Now, outside the chroot, run:

grub-install --root-directory=/mnt --grub-mkdevicemap=/mnt/boot/grub/device.map /dev/loop0


Cleanup everything (umount, losetup -d, kpartx -d etc), and we’re ready to upload the image, with snf-image-creator.

snf-image-creator takes a diskdump as input, launches a helper VM, cleans up the diskdump / image (cleanup of sensitive data etc), and optionally uploads and registers our image with ~okeanos.

For more information on how snf-image-creator and Synnefo image registry works, visit the relevant docs [1][2][3].

0) Since snf-image-creator will use qemu/kvm to spawn a helper VM, and we’re inside a VM, let’s make sure that nested virtualization (OSDI ’10 Best Paper award btw ) works.

First, we need to make sure that kvm_[amd|intel] is modprobe’d on the host machine / hypervisor with the nested = 1 parameter, and that the vcpu, that qemu/kvm creates, thinks that it has ‘virtual’ virtualization extensions (that’s actually our responsibility, and it’s enabled on the okeanos.io servers).

Inside our Debian VM, let’s verify that everything is ok.

grep [vmx | svm] /proc/cpuinfo
modprobe -v kvm kvm_intel


1) Clone snf-image-creator repo

git clone https://code.grnet.gr/git/snf-image-creator


2) Install snf-image-creator using setuptools (./setup.py install) and optionally virtualenv. You’ll need to install (pip install / aptitude install etc) setuptools, (python-)libguestfs and python-dialog manually. setuptools will take care of the rest of the deps.

3) Use snf-image-creator to prepare and upload / register the image:

snf-image-creator -u gentoo.diskdump -r "Gentoo Linux" -a [okeanos.io username] -t [okeanos.io user token] gentoo.img -o gentoo.img --force


If everything goes as planned, after snf-image-creator terminates, you should be able to see your newly uploaded image in https://pithos.okeanos.io, inside the Images container. You should also be able to choose your image to create a new VM (either via the Web UI, or using the kamaki client).

And, let’s install kamaki to spawn some Gentoo VMs:

git clone https://code.grnet.gr/git/kamaki

and install it using setuptools (just like snf-image-creator). Alternatively, you could use our Debian repo (you can find the GPG key here).

Modify .kamakirc to match your credentials:

[astakos]
enable = on
url = https://astakos.okeanos.io
[compute]
cyclades_extensions = on
enable = on
url = https://cyclades.okeanos.io/api/v1.1
[global]
colors = on
token = [token]
[image]
enable = on
url = https://cyclades.okeanos.io/plankton
[storage]
account = [username]
container = pithos
enable = on
pithos_extensions = on
url = https://pithos.okeanos.io/v1


Now, let’s create our first Gentoo VM:

kamaki server create LarryTheCow 37 `kamaki image list | grep Gentoo | cut -f -d ' '` --personality /root/.ssh/authorized_keys


That’s all for now. Hopefully, I’ll return soon with another more detailed post on scripting with kamaki (vkoukis has a nice script using kamaki python lib to create from scratch a small MPI cluster on ~okeanos ).

Cheers!

Now with this change, the number of arguments passed to the kmap_atomic() function changed too, and thus you end up with a huge patch covering all the tree, which fixed the issue (changing kmap_atomic(p, KM_TYPE) to kmap_atomic(p)).

But there was another way to go. Some C preprocessor magic.

#define kmap_atomic(page, args...) __kmap_atomic(page)

Yes, the C preprocessor supports va_args.
(which I found out when going through the reptyr code, but I’ll talk about it in an other post.)

Today, I saw a thread at the lkml, which actually did the cleanup I described. Andrew Morton responded:

I’m OK with cleaning all these up, but I suggest that we leave the back-compatibility macros in place for a while, to make sure that various stragglers get converted. Extra marks will be awarded for working out how to make unconverted code generate a compile warning

And Nick Bowler responded with a very clever way to do this (which involved abusing heavily the C preprocessor ):

  #include <stdio.h>

  int foo(int x)
  {
     return x;
  }

  /* Deprecated; call foo instead. */
  static inline int __attribute__((deprecated)) foo_unconverted(int x, int unused)
  {
     return foo(x);
  }

  #define PASTE(a, b) a ## b
  #define PASTE2(a, b) PASTE(a, b)
  
  #define NARG_(_9, _8, _7, _6, _5, _4, _3, _2, _1, n, ...) n
  #define NARG(...) NARG_(__VA_ARGS__, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0)

  #define foo1(...) foo(__VA_ARGS__)
  #define foo2(...) foo_unconverted(__VA_ARGS__)
  #define foo(...) PASTE2(foo, NARG(__VA_ARGS__)(__VA_ARGS__))

  int main(void)
  {
    printf("%d\n", foo(42));
    printf("%d\n", foo(54, 42));
    return 0;
  }

The actual warning is printed due to the deprecated attribute of the foo_unconverted() function.

The fun part, however, is how we get to use the foo ‘identifier’/name to call either foo() or foo_uncoverted() depending on the number of arguments given.

The trick is to use the __VA_ARGS__ to ‘shift’ the numbers 9-0 in the NARG macro, so that when calling the NARG_ macro, _9 will match with the first __VA_ARGS__ argument, _8 with the second etc, and so n will match with actual number of arguments (I’m not sure I described it very well, but if you try doing it by hand, you’ll understand how it’s working).

Now that we have the number of arguments given to foo, we use the PASTE macro to ‘concatenate’ the number of the arguments with the function name, and the actual arguments given, and call the appropriate wrapper macro (foo1, foo2 etc).

Another interesting thing, which I didn’t know, is about argument expansion in macros. For macros that concatenate (##) or stringify (#) the arguments are not expanded beforehand. That’s why we have to use PASTE2 as a wrapper, to get the NARG() argument/macro fully expanded before concatenating.

Ok, C code can get at times a bit obfuscated, and yes you don’t have type safety etc etc, but, man, you can be really creative with the C language (and the C preprocessor)!
And the Linux kernel development(/-ers) prove just that.

Yesterday, I ended up looking the xargs Wikipedia article, and there I found an interesting note:

Under the Linux kernel before version 2.6.23, arbitrarily long lists of parameters could not be passed to a command,[1] so xargs breaks the list of arguments into sublists small enough to be acceptable.

Along with a link to the GNU coreutils FAQ.

And from there a link to the Linux Kernel mainline git repository.

After a bit of googling, I found a very nice article describing in great detail the ARG_MAX variable, which defines the maximum length of the arguments passed to execve.

Traditionally Linux used a hardcoded:

#define MAX_ARG_PAGES 32

to limit the total size of the arguments passed to the execve() (including the size of the ‘environment’). That limited the maxlen of the arguments passed to about 128KB (minus the size of the ‘environment’).

(Note: actually, very early Linux kernels did not have support for ARG_MAX and didn’t use MAX_ARG_PAGES, but back then I was probably 2-3 years old, so it’s ancient history for me )

With Linux-2.6.33, this hardcoded limit was removed. Actually it was replaced by a more ‘flexible’ limit. The maximum length of the arguments can now be as big as the 1/4th of the user-space stack. For example, in my desktop, using ulimit -s I get a stack size of 8192KB, which means 2097152 maxlength for the arguments passed. The same value you can obtain by using getconf. Now, if I increase the soft limit on the stack size, the maxlength allowed will also increase, although with a 8192KB soft limit, the ‘ARGS_MAX’ is already big enough. Two new limits where also introduced, one on the maxlength of each argument (equal to PAGE_SIZE * 32), and the total number of arguments, equal to 0x7FFFFFFF, or as big as a signed integer can be.

Linux headers however use the MAX_ARG_STRLEN, I think, as the ARG_MAX limit, which forces libc to #undef it in its own header files. I’m not sure, since I haven’t looked into code yet, but at least for Linux, ARG_MAX is not statically defined anymore by libc (ie in a header file), but libc computes its value from the userspace stack size.
(edit: that’s indeed how it works for >=linux-2.6.33 — code in sysdeps/unix/sysv/linux/sysconf.c:

    case _SC_ARG_MAX:
  #if __LINUX_KERNEL_VERSION < 0x020617
        /* Determine whether this is a kernel 2.6.23 or later.  Only
           then do we have an argument limit determined by the stack
           size.  */
        if (GLRO(dl_discover_osversion) () >= 0x020617)
  #endif
          {
            /* Use getrlimit to get the stack limit.  */
            struct rlimit rlimit;
            if (__getrlimit (RLIMIT_STACK, &rlimit) == 0)
              return MAX (legacy_ARG_MAX, rlimit.rlim_cur / 4);
          }
  
        return legacy_ARG_MAX;

).

And the kernel code that enforces that limit:

               struct rlimit *rlim = current->signal->rlim;
               unsigned long size = bprm->vma->vm_end - bprm->vma->vm_start;

               /*
                * Limit to 1/4-th the stack size for the argv+env strings.
                * This ensures that:
                *  - the remaining binfmt code will not run out of stack space,
                *  - the program will have a reasonable amount of stack left
                *    to work from.
                */
               if (size > rlim[RLIMIT_STACK].rlim_cur / 4) {
                       put_page(page);
                       return NULL;
               }

The whole kernel patch is a bit complicated for me to understand, since I don’t have digged much into kernel mm code, but from what I understand, instead of copying the arguments into pages, and then mapping those pages into the new process address space, it setups a new mm_struct, and populates it with a stack VMA. It then copies the arguments into this VMA (expanding it as needed), and then takes care to ‘position’ it correctly into the new process. But since I’m not very familiar with the Linux Kernel mm API, it’s very likely that what I said is totally wrong (I really have to read the mm chapters from “Understanding the Linux Kernel” ).

ketchup is an awesome utility/script, written by Matt Mackall in Python, which makes it very easy to manage kernel sources. You can very easily upgrade to a newer kernel version, downgrade to older releases, and even switch between different patchsets. The ketchup ebuild I found in Portage (and in every Linux distro I know about) was fetching the original and out-of-date version of ketchup. Steven Rostedt had pulled the original ketchup code (v0.9) into his git repo @ kernel.org. However, there were no commits/updates to ketchup for 1-2 years, I think.

So, I decided to cleanup some of the old trees that ketchup supported, but were no longer maintained, and add support for new trees (or some updated ‘versions’ of the old trees). I sent the patches to Steven Rostedt, and he proposed that I take over and maintain ketchup.

I cloned the ketchup git repo to Github, applied the patches I’d written, plus quite a lot of patches that the Debian ketchup package provided.

Now, with the Linux-3.0 release approaching, I tried to add (at least) partial support for the new 2 digit version numbers, but there are still some issues, which will hopefully get resolved once Linux-3.0 gets released, and the new versioning scheme gets standarized (for example the EXTRAVERSION Makefile variable will probably not get removed from 3.0, as it breaks some userspace utils, like uptime etc from procps utils, some depmod issues etc).

The new code for 3.x kernels is currently in the linux-3 branch, from which I took a snapshot and pushed it to Portage as dev-util/ketchup-1.1_beta. I will hopefully merge it back with master, after the first -stable release comes out (Linux-3.0.1), just to make sure that everything works.

Feel free to give it a try, and report any bugs/issues.

I’ll be helping the Gentoo Kernel Project, with bug fixing at first, and help with the maintenance of some of the kernel sources in the tree.

Many thanks to mpagano for mentoring me, tampakrap for his help with the quizzes, and of course hwoarang, who had no problem to do all of the review sessions during his vacations.

It took me *3 days* of ‘debugging’, until I was able to find why that happened!

I tried various changes to the iptables and iproute2 configuration, giving more hints to both utilities in order to use the correct routing table, mark the packets correctly etc, but it still wouldn’t work.

After a lot of time tweaking the configuration, without results, I saw that, although ping -Ieth0 ${VPN_SERVER}, didn’t ‘work’ (with openvpn running, and tap0 configured with the correct address/netmask), I could see with tcpdump the ‘ECHO REPLY’ packets sent by the VPN server, with correct source and destination addresses.

After stracing the ping command, I saw that when ping issued a recvmsg syscall, recvmsg returned with -EAGAIN. So, now I know that the packets do arrive to the interface with correct addresses, but they couldn’t ‘reach’ the upper network stacks of the kernel.

The problem was that both machines were running vanilla kernels, so I couldn’t blame any Debian or Gentoo specific patches. But since I knew that the problem was in the kernel, I tried to see if any kernel .config options, regarding NETFILTER, and multiple routing tables didn’t match between the two configs. But I couldn’t find anything that could cause that ‘bug’.

So since the kernel sources are the same, and I can’t find anything in the .configs that could cause the problem, I try tweaking some /proc/sys/net ‘files’, although I couldn’t see why these would differ between the two machines. And then I saw some /proc/sys/net/ipv4/ files in Gentoo, that didn’t show up in Debian (/proc/sys/net/ipv4/cipso*).

I googled to find what cipso is, and I finally found out that it was part of the NetLabel project. CIPSO (Common IP Security Option) is an IETF draft (it’s quite old actually) and is implemented like a ‘security module’ in the Linux Kernel, and it was what it caused the problem, probably because it tried to do some verification on the inbound packets, which failed, and therefore the packets were ‘silently’ dropped. LWN has an article with more infromation about packet labeling and CIPSO, and there’s also related Documentation in the Linux Kernel.

make defconfig enbales Netlabel, but Debian’s default configuration had it disabled, and that’s why Openvpn/iproute2/iptables configuration worked with Debian, but failed on Gentoo.

Instead of compiling a new kernel, one can just do

echo 0 > /proc/sys/net/ipv4/cipso_rbm_strict_valid

and disable CIPSO verification on inbound packets, so that multiple routing tables and packet marking work as expected.

I prepared a presentation about DNS tunneling and how to bypass Captive Portals at Wifi Hotspots, which require authentication.
(We want to do another presentation, to test ICMP/ping tunnel too ).

I had blogged on that topic some time ago.
It was about time for a test-drive.

I set up iodine, a DNS tunneling server(and client), and I was ready to test it, since I would be travelling with Minoan Lines the next day.

I first did some tests from my home 24Mbps ADSL connection, and the results weren’t very encouraging. Although the tunnel did work, and I could route all of my traffic through the DNS tunnel, and over a second OpenVPN secure tunnel, bandwidth dropped to ~30Kbps, when using the NTUA FTP Server, through the DNS tunnel.
(The tunnel also worked with the NTUA Wifi Captive Portal, although at first we had some ‘technical issues’, ie I hadn’t set up NAT on the server to masquarade and forward the traffic coming from the tunnel ).

The problem is that the bandwidth of the Minoan Lines(actually Forthnet ‘runs’ it afaik) Wifi(not inside the ‘local’ network of course) was ~30Kbps(terrible, I know), without using DNS tunneling. So, I wasn’t very optimistic. (I think they have some satelite connection, or something like that from the Wifi to the Internet).

When I was on the ship, I tried to test it. At first, I encountered another technical issue(the local DNS had an IP inside the Wifi local network, and due to NAT the IP our server was ‘seeing’, was different than the IP of the DNS packets, so we had to run iodined with the -c flag). Luckily, FOSS NTUA members(who had root access on the computer running iodined) are 1337 and fixed that in no time.

And at last, I had a ‘working’ DNS tunnel, but with extremely high ping times(2sec RTT) to the other end of the tunnel, and when I tried to route all traffic through the tunnel I had a ridiculous 22sec RTT to ntua.gr. Of course even browsing the Web was impossible, since all the HTTP requests timed out before an answer could reach my laptop.

However, because I am a Forthnet customer(for my ADSL connection), I was able to use my Username/Password of my home ADSL connection, and have free access to the Internet, from their hotspot(with the amaing bandwidth of ~30Kbps ). At least they do the authentication over SSL.

Although DNS tunneling didn’t really work in this case(the tunnel itself worked, but due to the bandwidth being so low, I didn’t have a ‘usable’ connection to the Internet), I think that in other hotspots, which provide better bandwidth/connection, it can be a very effective way to bypass the authentication and use them for free.

Probably, there’ll be a Part 3, with results from bandwidth benchmarks inside the NTUA Wifi, and maybe some ICMP tunneling stuff.

Cheers!

As the APM 1.2 specification states:

The APM BIOS 32-bit protected mode interface requires 3 consecutive
selector/segment descriptors for use as 32-bit code, 16-bit code, and data segments,
respectively. Both 32-bit and 16-bit code segment descriptors are necessary so the
APM BIOS 32-bit interface can call other BIOS routines in a 16-bit code segment if
necessary. The caller must initialize these descriptors using the segment base and
length information returned from this call to the APM BIOS. These selectors may
either be in the GDT or LDT, but must be valid when the APM BIOS is called in
protected mode.

So, at boot time, Linux will query the BIOS/APM for information about the base and length of the segments that APM code uses(query_apm_bios()):

        /* 32-bit connect */
	ireg.al = 0x03;
	intcall(0x15, &ireg, &oreg);

	boot_params.apm_bios_info.cseg        = oreg.ax;
	boot_params.apm_bios_info.offset      = oreg.ebx;
	boot_params.apm_bios_info.cseg_16     = oreg.cx;
	boot_params.apm_bios_info.dseg        = oreg.dx;
	boot_params.apm_bios_info.cseg_len    = oreg.si;
	boot_params.apm_bios_info.cseg_16_len = oreg.hsi;
	boot_params.apm_bios_info.dseg_len    = oreg.di;

These are the values that Linux will use to set-up the appropriate segments in the Global Descriptor Table(GDT).

	/*
	 * Set up the long jump entry point to the APM BIOS, which is called
	 * from inline assembly.
	 */
	apm_bios_entry.offset = apm_info.bios.offset;
	apm_bios_entry.segment = APM_CS;
	/*
	 * The APM 1.1 BIOS is supposed to provide limit information that it
	 * recognizes.  Many machines do this correctly, but many others do
	 * not restrict themselves to their claimed limit.  When this happens,
	 * they will cause a segmentation violation in the kernel at boot time.
	 * Most BIOS's, however, will respect a 64k limit, so we use that.
	 *
	 * Note we only set APM segments on CPU zero, since we pin the APM
	 * code to that CPU.
	 */
	gdt = get_cpu_gdt_table(0);
	set_desc_base(&gdt[APM_CS >> 3],
		 (unsigned long)__va((unsigned long)apm_info.bios.cseg << 4));
	set_desc_base(&gdt[APM_CS_16 >> 3],
		 (unsigned long)__va((unsigned long)apm_info.bios.cseg_16 << 4));
	set_desc_base(&gdt[APM_DS >> 3],
		 (unsigned long)__va((unsigned long)apm_info.bios.dseg << 4));

So, now the APM segments(their descriptors) are ready to use.

The code used to make an APM BIOS call looks like this:

static inline void apm_bios_call_asm(u32 func, u32 ebx_in, u32 ecx_in,
					u32 *eax, u32 *ebx, u32 *ecx,
					u32 *edx, u32 *esi)
{
	/*
	 * N.B. We do NOT need a cld after the BIOS call
	 * because we always save and restore the flags.
	 */
	__asm__ __volatile__(APM_DO_ZERO_SEGS
		"pushl %%edi\n\t"
		"pushl %%ebp\n\t"
		"lcall *%%cs:apm_bios_entry\n\t"
		"setc %%al\n\t"
		"popl %%ebp\n\t"
		"popl %%edi\n\t"
		APM_DO_POP_SEGS
		: "=a" (*eax), "=b" (*ebx), "=c" (*ecx), "=d" (*edx),
		  "=S" (*esi)
		: "a" (func), "b" (ebx_in), "c" (ecx_in)
		: "memory", "cc");
}

It took me a while to figure out the long jump.

lcall *%%cs:apm_bios_entry

because apm_bios_entry is defined as:

static struct {
        unsigned long   offset;
        unsigned short  segment;
} apm_bios_entry;

At first I though that the struct should be defined the other way around(first the segment and then the offset).
I experimented a bit with inline asm, and after lots of segmentation faults, and some time going over Intel x86 manuals about the ljmp instruction, I figured it out.

Well, I think it took much much longer than it should to understand was going on. :S

The ljmp expects a mem16:mem32 operand, where mem16 is the segment, and mem32 the offset.
And that’s exactly how the struct apm_bios_entry is stored in memory.
However, as I ‘read’ mem16:mem32, I thought that mem16 should be stored before mem32. :S

And thus I lost several hours writing and experimenting with segfaulting C code.
For something pretty obvious…

Well, the patch itself isn’t a big deal, since I didn’t write any code. It was a cleanup of asm-offsets.

Afaik, Linux has quite a lot of assembly code, which needs the offsets of various struct members. Of course, assembly code(or even toplevel inline assembly) cannot use the offsetof marco. That’s also the case for some C constants, eg:

#define PAGE_SIZE (1UL << PAGE_SHIFT)) 

Thus, these offsets and constants are ‘calcualted’ at build time as ‘absolute values’ so that gas will be ok.
Using the macros defined in incldue/linux/kbuild.h

#define DEFINE(sym, val) \
        asm volatile("\n->" #sym " %0 " #val : : "i" (val))

#define BLANK() asm volatile("\n->" : : )

#define OFFSET(sym, str, mem) \
	DEFINE(sym, offsetof(struct str, mem))

#define COMMENT(x) \
	asm volatile("\n->#" x)

the asm-offsets.c is used to create a ‘special’ asm fle, which is then parsed by the kernel build system, in order to create the include/generated/asm-offsets.h file.

However, a patch introduced two new macros in include/linux/const.h

#ifdef __ASSEMBLY__
#define _AC(X,Y)	X
#define _AT(T,X)	X
#else
#define __AC(X,Y)	(X##Y)
#define _AC(X,Y)	__AC(X,Y)
#define _AT(T,X)	((T)(X))
#endif

so that some constants defined in C work with gas too.

And now, the PAGE_SIZE is defined as

#define PAGE_SIZE       (_AC(1,UL) << PAGE_SHIFT

and thus, the PAGE_SIZE_asm defined in x86/asm-offsets.c, and used in some places in x86 code was no longer needed(that’s also the case with PAGE_SHIFT and THREAD_SIZE).

So, I deleted PAGE_SIZE_asm/PAGE_SHIFT_asm/THREAD_SIZE_asm from x86/kernel/asm-offsets.c, and replaced them with their non-asm counterparts, in the code that used them.

I posted it to lkml(after some hours experimenting with git format-patch and git send-email ), and it got accepted.